Raw measurement archival and UQ characterization.

All climate data products are based on original raw measurements but often involve subtle processing and transformations of the raw observations. Sometimes it is the direct observations, with quality checking and homogenization, that constitute the data product, although at other times the raw measurements are used to derive the product through a forward model. Questions were raised regarding the availability of original records and accurate quantification of the associated measurement errors. It was acknowledged that U.S. government agencies are successfully archiving and stewarding most remotely sensed raw measurements in addition to the derived products. However, the records of ground-based measurements still face issues with digitization, archival, and retention. A significant and sustained effort is required to save these data, which constitute the long-term (greater than 50 years) record. Figure 2 illustrates the idealization of climate data record stewardship.

View Full Size

Climate observation stewardship. This conceptual framework idealizes the stewardship of climate observations from the original raw measurements, through homogenization, and derivation of products. Along the way, all processing should be documented and archived in a way that is easily accessible to the user community.

Citation: Bulletin of the American Meteorological Society 94, 3; 10.1175/BAMS-D-12-00042.1

• Download Figure
• Download figure as PowerPoint slide

The characterization of UQ in remotely sensed data products can be done through two main ways: 1) propagation of uncertainties from instrument measurement errors through the retrievals to the final product, or 2) comparison to in situ measurements. More often, scientists choose to use in situ measurements to validate remotely sensed data products rather than the more complex alternative of propagation of uncertainties. Although there are some notable successes in characterizing UQ for the raw measurements, there remain large geographic regions, vertical domains, and measurement types where associated validation data are not available.

Quantification of the measurement errors traceable to absolute International System of Units (SI) standards in the ground-based measurement networks would be prohibitively expensive across the network as a whole. However, there is a growing realization of the potential value of reference networks such as the Global Climate Observing System (GCOS) Reference Upper-Air Network (GRUAN) and the U.S. Climate Reference Network (USCRN), which are traceable in this manner and can form the backbone of a global system of systems approach to observations as envisioned by the Global Earth Observation System of Systems (GEOSS) and World Meteorological Organization (WMO) Integrated Global Observing System (WIGOS).

Homogenization techniques.

The workshop focused on the current procedure for removing outlier and poor quality measurements from the data record. Climate scientists sometimes find it necessary to identify data points that suffer from measurement inconsistencies or underepresentativeness and then omit them from further analysis. In doing so, artifacts and systematic bias are removed from the data record. Given the skepticism with which so-called data cleansing procedures are received by both scientific and lay audiences, it was recommended that all such procedures should be formulated into a mathematical algorithm that is fully described to promote reliability and transparency. Even after homogenization techniques are applied, improved exploration of structural uncertainty and undertaking a consistent suite of benchmarking assessment exercises to understand more fully the strengths and limitations of each method will enable the data record to be better characterized. All participants agreed that care needs to be taken with any modification to, or subsetting of, the complete dataset as it changes the distribution of measurements, which are in fact only estimates of what is considered “truth.”

Challenges faced when applying UQ methods to climate observations.

Although this workshop was intended to further collaboration between the statistical and mathematical communities and climate scientists to apply UQ methods to climate observations, it became clear that each side has different challenges. One issue of concern for statisticians is the time investment required to educate themselves about a particular application, but it was agreed that this is where the collaboration with climate scientists could be instrumental.

Climate scientists and data providers have concerns related to choosing an appropriate statistical methodology. Many UQ techniques appear equally good, and it is difficult to choose which method is appropriate. Moreover, it is often unclear as to the scope of a particular method, where it is valid, and where it should not be used. It is equivocal to the community at large how to correctly intercom-pare uncertainty estimates derived via different techniques.

The remote sensing data product community has begun to address UQ issues by including data quality flags along with data. Although this is an excellent first step in qualitatively assessing the uncertainty associated with observations, a more quantitative representation is desirable. One suggestion for a quantitative UQ characterization was the inclusion of Monte Carlo ensemble predictions for data products, rather than only reporting means as is the current practice. From the data provider perspective, there are storage concerns for archiving a suite of ensemble predictions. Further, there is concern about the ability of the general user to understand how to use an ensemble product, given the diverse range of background experience for end users. A middle ground would be supplying standard errors or quantile values to indicate the range of uncertainty, which may be a more feasible option to address the interests of all parties.

All participants agreed that there is too often a lack of documentation in both products and papers describing the uncertainty calculation methods. Many techniques rely on varying assumptions, which may not be clearly communicated to, or understood by, data product users. The end users of climate data, which can include policymakers, comprise a wide range of backgrounds. This diversity of backgrounds needs to be considered when choosing uncertainty products to associate with climate data. The uncertainty estimates need to be robust, completely explained, and intellectually accessible to all users. It was agreed that the more approximate quality statements intended for nonexpert users should be traceable back to more detailed quality measures.

Reproducibility of applied UQ methods.

Transparency of applied UQ methods is a major concern. Accomplishment of transparency includes auditability, reproducibility, and replication of results; that is, every UQ method applied should be accompanied by an explanation of what, how, and why it was done.

One obvious way to facilitate auditability and reproducibility is with provision of code that operates on the raw measurement inputs to produce data products. Even in this relatively small subset of participants there was a wide range of perspectives regarding code provision. Some are willing to provide code, although there is doubt whether code alone will explain methodologies and whether external users can simply copy code and reproduce results. Although code provision can directly explain what and how analysis was done, it may be silent on why the particular methodology and implementation was selected. Some agencies have extremely strict requirements for code release, insisting on near-commercial-grade quality, which most researchers do not have the time or resources to develop. Therefore, code release is not a feasible option under these conditions. Another subset of statisticians feel that it is better to provide inputs and outputs to interested parties. Supplying (nonpeer reviewed) code can be a dangerous practice, while leaving it to the user to develop the code and reproduce results independently encourages thought and engagement by the potential user. In this replication of results, real scientific value is realized through the analysis of structural uncertainty.

Another effort that also achieves transparency is developing detailed technical documentation that outlines all assumptions, error handling, and procedures used to create the associated data product. This documentation is a living entity, modified as reprocessing methods are developed while retaining the historical records for version control.

Collaboration possibilities.

The intent of this workshop was to facilitate collaboration between the statistical and climate science communities. As such, ways for improving and strengthening the relationship between them were discussed. Ideally, this should be a synergistic relationship. Not only can statisticians significantly assist climate scientists with data, allowing for new interpretations of data, but also careful modeling and analysis of important climate datasets can suggest new statistical methods.

Several ideas for collaboration avenues were raised:

• The need for statistical input during the development of experimental schemes.

• Creation of more flexible data products, designed with input from both climatologists and statisticians, to allow for easy data manipulation for users.

• A few mathematicians were interested in providing sensitivity analysis capabilities for measurement devices as well as forward models to connect what is being measured and what is being derived.

• Scaling issues, both spatial and temporal, were a recurrent theme where the advice of statisticians would be greatly valued by climate scientists. A specific example of this is the collaboration to devise a statistical model for surface temperatures calibrated from ground station data.

• There was common interest in working toward a communitywide definition for terminology describing trends and truth. Additionally, a joint effort is warranted to communicate clearly with the public what is meant by “uncertainty quantification methods,” and how uncertainty calculations are themselves uncertain.