WORKSHOP ON UNCERTAINTY QUANTIFICATION IN CLIMATE MODELING AND PROJECTION
What: Approximately 70 participants from 30 countries on all five continents met to discuss the need, approaches, and challenges in uncertainty quantification in climate modeling and projection. More details, including workshop presentations, can be found online (http://indico.ictp.it/event/a14268/)
When: 13–17 July 2015
Where: Trieste, Italy
The projection of future climate is one of the most complex problems undertaken by the scientific community. Although scientists have been striving to better understand the physical basis of the climate system and to improve climate models, the overall uncertainty in projections of future climate has not been significantly reduced [e.g., from the Intergovernmental Panel on Climate Change (IPCC) Fourth Assessment Report (AR4) to Fifth Assessment Report (AR5)]. With the rapid increase of complexity in Earth system models, reducing uncertainties in climate projections becomes extremely challenging. Since uncertainties always exist in climate models, interpreting the strengths and limitations of future climate projections is key to evaluating risks, and climate change information for use in vulnerability, impact, and adaptation (VIA) studies should be provided with both well-characterized and well-quantified uncertainty.
The workshop aimed at providing participants, many of them from developing countries, information on strategies to quantify the uncertainty in climate model projections and to assess the reliability of climate change information for decision-making. The program included a mixture of lectures on fundamental concepts in Bayesian inference and sampling, applications, and hands-on computer laboratory exercises employing software packages for Bayesian inference, Markov chain Monte Carlo methods, and global sensitivity analyses. The lectures covered a range of scientific issues underlying the evaluation of uncertainties in climate projections, such as the effects of uncertain initial and boundary conditions, uncertain physics, and the limitations of observational records. Progress in quantitatively estimating uncertainties in hydrologic, land surface, and atmospheric models at both regional and global scales was also reviewed.
The application of uncertainty quantification (UQ) concepts to coupled climate system models is still in its infancy. The Coupled Model Intercomparison Project (CMIP) multimodel ensemble currently represents the primary data for assessing reliability and uncertainties of climate change information. An alternative approach is to generate similar ensembles by perturbing parameters within a single-model framework. One of the workshop’s objectives was to give participants a deeper understanding of these approaches within a Bayesian statistical framework. However, there remain significant challenges still to be resolved before UQ can be applied in a convincing way to climate models and their projections.
BAYESIAN STATISTICAL FRAMEWORK.
The workshop especially emphasized the use of a Bayesian framework for synthesizing uncertainties affecting climate projections. Computation for the Bayesian approach can make use of a simple relationship in conditional probability to divide a large and seemingly intractable problem into a sequence of smaller calculations from which one can construct the desired estimates. The workshop focused on uncertainties in representing climate system physics in current models. Our best knowledge of climate modeling uncertainty can be currently derived from the CMIP framework, which includes results from a standard set of experiments by ∼30–40 state-of-the-art climate system models constructed by different teams of experts and centers throughout the world. This “ensemble of opportunity” is the best available source of information on the range of physically plausible climate evolutions over the next century. Bayesian inference provides an unproven, but potentially powerful, alternative approach to quantify climate model uncertainties from individual models. While the approach is most easily adapted to consider uncertain model parameter settings (i.e., uncertainty in model tuning), it can also be used to explore alternate parameterization schemes.
Applying Bayesian inference to climate projections is not trivial because of the challenges imposed by the computational expense of running large ensembles of climate simulations and the nonlinear interactions of climate system processes. For example, it has been found that changes of model parameters aimed at improving the simulation of Greenland’s surface mass balance can have major consequences on how the model simulates tropical water vapor. To meet these challenges, the Bayesian statistical community has developed strategies for reducing the size of the problem by developing 1) global sensitivity analyses that can weed out unimportant parameters; 2) surrogate models, such as Gaussian process emulation, that can predict the response of a climate model to arbitrary changes in model parameter settings; and 3) adaptive sampling strategies that can significantly reduce the number of model experiments needed in the construction of response functions.
ISSUES AND APPLICATIONS.
Sensitivity analysis.
One of the problems in climate projections is the extrapolation of sensitivities in the present climate to future climate conditions. Parameters or processes that are important for simulating today’s climate are not necessarily important regulators of how climate will change. Thus, one of the first challenges is to identify what are the physical processes and parameters that need to be considered. For example, analyses of model climate sensitivities within the CMIP ensemble point to the important role of uncertainties in cloud feedbacks. There have been several extensive studies of the sensitivity of climate models to different (up to 30) uncertain model parameters. These studies address two important questions: the first is whether the parameters affect feedbacks leading to substantial scatter in climate projections. The second is the extent to which each parameter needs to be considered in combination with other parameters. A parameter that does not affect climate change feedbacks and does not interact with the choice of other parameters does not require complex UQ tools. One of the limitations of current studies is that they only consider global metrics. This is not sufficient given that participant concerns are largely regional for which conclusions at the global scale may not be as relevant. Thus, existing efforts identifying process uncertainties need to be extended to regional scales.
Calibration.
Calibration in Bayesian inference refers to the identification of optimal interdependent model parameters using observed data. While the main objective of the calibration process is to reduce model biases, so far bias reduction in climate models has been relatively limited. There remain significant systematic model errors whose reduction will likely come from improved understanding and representation of system physics. On the other hand, some notable successes have been achieved in using calibration frameworks to select physically reasonable parameter settings. However, it is important to realize that model biases have the potential to skew parameter estimates through compensating errors. While the calibration of model parameters does not address questions concerning the effects of missing or inaccurate physics, it offers new opportunities to explore relationships between modeled processes and regional impacts using single-model, perturbed physics ensembles.
Climate projections.
With larger VIA communities assessing different aspects of climate change impacts, there is an increasing demand on climate scientists to provide quantitative measures of skill and uncertainty in detailed projections of a wider range of variables, often expressed in probabilistic terms. Such estimates are difficult to make with the relatively small ensemble size represented by the CMIP program and may require the use of targeted statistical approaches.
The most recent national climate projection assessment in the United Kingdom [United Kingdom Climate Projections 2009 (UKCP09)] provides an example of such an approach. UKCP09 made use of emulation techniques and multiple simulations with a single climate model and applied a Bayesian methodology to quantify a (conditional) probability for future climate changes at United Kingdom scales. Based on multiple climate model simulations, the emulators are used to estimate the wider climate response space of all possible parameter combinations. The Bayesian framework can then be used to evaluate the relative likelihood of each of these possible simulated climates relative to the observed world. There are a number of other aspects to this approach (such as the use of available multimodel ensembles to include an estimate of the intermodel structural uncertainties), but at its core the application uses a standard set of UQ tools that were presented at the workshop. Similar probabilistic frameworks are being built by other national modeling centers. The statistical framework on which these estimates are built still requires various assumptions that may be difficult to fully justify. Nevertheless, they do provide a feasible way to combine different sources of information relevant to assessments of climate change impacts.
Impacts.
With so much work still needed to quantify and possibly reduce uncertainties in climate projections, it may seem premature to consider risk assessments. One of the messages of the workshop is that this notion should be flipped around. In other words, the identification of key vulnerabilities should drive the subsequent identification of the set of questions that warrant more careful consideration. An example was presented at the workshop regarding the Sustainable Climate Risk Management (SCRiM) project, which focuses on the risks associated with sea level rise and on the regionalization of certain climate change impacts. In these cases, a key point was that some critical elements are not addressed in state-of-the-art models and therefore alternative methods need to be considered. As an example, uncertainty distributions will be impacted by the lack of well-known feedbacks such as carbon cycle or land-use change that are directly linked to human behavior. Ultimately, these must be incorporated into decision support tools. The other result from this project that was relevant to the workshop participants is that the currently generated projection ensembles (e.g., CMIP) may underrepresent the range of possible impacts in particular regions where there has not been a concerted effort to explore parameters or forcings important to that region.
CHALLENGES AND WAY FORWARD.
Several challenges in UQ were discussed during the workshop. One is the quantification of the effect of missing or incorrect physics on model projections, a problem often referred to as structural uncertainty. Accounting for such effects often requires observations of model errors. While we have plenty of observations of such errors for the present climate, we do not yet have observations of future climate. Currently, concerns about model deficiencies in projections are assessed by the level of agreement within a multimodel ensemble and metrics that summarize how a certain response is obtained. There do not seem to be clear-cut answers on how to account for the effects of missing physics.
One of the challenges for calibrating climate models with observational data is the interdependency that exists in the selection of parameters affecting different component models, especially with components of the climate system that take a long time to adjust to a change in forcing or parameters such as the ocean and ice sheets. The time scales of adjustment in the ocean make any practical testing of a range of alternate model configurations untenable.
While the challenges for placing defensible error bars on projections are daunting, progress is being made on several fronts. A 2010 IPCC Expert Meeting report reviews many of the above-mentioned issues and makes recommendations concerning the use of the CMIP multimodel ensemble in VIA assessments. For example, the report discourages the weighting of individual models outside of a statistical framework. Scientific criteria for excluding certain models may be acceptable as long as the criteria are applied to all models equally. The issue of the distillation of relevant and robust climate change information from multiple, and often conflicting, sources (e.g., global and regional climate models), is an evolving area of research that will draw increasing attention in the future.
One of the key limitations toward the application of full UQ approaches to current climate models is computational. The challenge is how to run sufficiently large numbers of simulations to estimate the uncertainties without degrading either the resolution or complexity beyond a level where the simulations would not be representative of the state of the art. One way forward could be to make use of information on shorter, weather time scales. A number of systematic model biases arise from the fast physics that can emerge on short 2–3-day time scales (such as vertical stability and cloud responses). An increasing number of modeling groups now make use of a “seamless” approach of using model evaluation on weather, seasonal, and climate time scales to inform model development. The workshop participants discussed how information from short time-scale simulations could be used in UQ approaches.
Overall, the workshop stressed the need for more critical and wide-ranging analyses of models and tools used in the production of climate projections. Uncertainty quantification provides a framework for bringing various pieces of this effort together, although the standard UQ tools will need to be better fitted to the targeted problems. More importantly, UQ crosses disciplinary boundaries and requires a level of communication and commitment across different research communities. The workshop provided a unique opportunity to gain insights on these topics from the perspective of a very diverse group of scientists from all over the world.
ACKNOWLEDGMENTS
The workshop was jointly sponsored by the Abdus Salam International Centre for Theoretical Physics (ICTP) and the International Union of Geodesy and Geophysics (IUGG). This study was supported by the U.S. Department of Energy’s Office of Science as part of the Earth System Modeling Program. The Pacific Northwest National Laboratory is operated for DOE by Battelle Memorial Institute under Contract DE-AC05–76RL01830.
