Risks of Model Weighting in Multimodel Climate Projections

Andreas P. Weigel Federal Office of Meteorology and Climatology, MeteoSwiss, Zurich, Switzerland

Search for other papers by Andreas P. Weigel in
Current site
Google Scholar
PubMed
Close
,
Reto Knutti Institute for Atmospheric and Climate Science, ETH, Zurich, Switzerland

Search for other papers by Reto Knutti in
Current site
Google Scholar
PubMed
Close
,
Mark A. Liniger Federal Office of Meteorology and Climatology, MeteoSwiss, Zurich, Switzerland

Search for other papers by Mark A. Liniger in
Current site
Google Scholar
PubMed
Close
, and
Christof Appenzeller Federal Office of Meteorology and Climatology, MeteoSwiss, Zurich, Switzerland

Search for other papers by Christof Appenzeller in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Multimodel combination is a pragmatic approach to estimating model uncertainties and to making climate projections more reliable. The simplest way of constructing a multimodel is to give one vote to each model (“equal weighting”), while more sophisticated approaches suggest applying model weights according to some measure of performance (“optimum weighting”). In this study, a simple conceptual model of climate change projections is introduced and applied to discuss the effects of model weighting in more generic terms. The results confirm that equally weighted multimodels on average outperform the single models, and that projection errors can in principle be further reduced by optimum weighting. However, this not only requires accurate knowledge of the single model skill, but the relative contributions of the joint model error and unpredictable noise also need to be known to avoid biased weights. If weights are applied that do not appropriately represent the true underlying uncertainties, weighted multimodels perform on average worse than equally weighted ones, which is a scenario that is not unlikely, given that at present there is no consensus on how skill-based weights can be obtained. Particularly when internal variability is large, more information may be lost by inappropriate weighting than could potentially be gained by optimum weighting. These results indicate that for many applications equal weighting may be the safer and more transparent way to combine models. However, also within the presented framework eliminating models from an ensemble can be justified if they are known to lack key mechanisms that are indispensable for meaningful climate projections.

Corresponding author address: Andreas Weigel, MeteoSwiss, Krähbühlstrasse 58, P.O. Box 514, CH-8044 Zürich, Switzerland. Email: andreas.weigel@meteoswiss.ch

Abstract

Multimodel combination is a pragmatic approach to estimating model uncertainties and to making climate projections more reliable. The simplest way of constructing a multimodel is to give one vote to each model (“equal weighting”), while more sophisticated approaches suggest applying model weights according to some measure of performance (“optimum weighting”). In this study, a simple conceptual model of climate change projections is introduced and applied to discuss the effects of model weighting in more generic terms. The results confirm that equally weighted multimodels on average outperform the single models, and that projection errors can in principle be further reduced by optimum weighting. However, this not only requires accurate knowledge of the single model skill, but the relative contributions of the joint model error and unpredictable noise also need to be known to avoid biased weights. If weights are applied that do not appropriately represent the true underlying uncertainties, weighted multimodels perform on average worse than equally weighted ones, which is a scenario that is not unlikely, given that at present there is no consensus on how skill-based weights can be obtained. Particularly when internal variability is large, more information may be lost by inappropriate weighting than could potentially be gained by optimum weighting. These results indicate that for many applications equal weighting may be the safer and more transparent way to combine models. However, also within the presented framework eliminating models from an ensemble can be justified if they are known to lack key mechanisms that are indispensable for meaningful climate projections.

Corresponding author address: Andreas Weigel, MeteoSwiss, Krähbühlstrasse 58, P.O. Box 514, CH-8044 Zürich, Switzerland. Email: andreas.weigel@meteoswiss.ch

Save
  • Allen, M. R., and W. J. Ingram, 2002: Constraints on future changes in climate and the hydrological cycle. Nature, 419 , 224232.

  • Boé, J., A. Hall, and X. Qu, 2009: September sea-ice cover in the Arctic Ocean projected to vanish by 2100. Nat. Geosci., 2 , 341343. doi:10.1038/NGEO467.

    • Search Google Scholar
    • Export Citation
  • Buizza, R., 1997: Potential forecast skill of ensemble prediction, and spread and skill distributions of the ECMWF Ensemble Prediction System. Mon. Wea. Rev., 125 , 99119.

    • Search Google Scholar
    • Export Citation
  • Buser, C. M., H. R. Künsch, D. Lüthi, M. Wild, and C. Schär, 2009: Bayesian multimodel projection of climate: Bias assumptions and interannual variability. Climate Dyn., 33 , 849868. doi:10.1007/s00382-009-0588-6.

    • Search Google Scholar
    • Export Citation
  • Christensen, J. H., F. Boberg, O. B. Christensen, and P. Lucas-Picher, 2008: On the need for bias correction of regional climate change projections of temperature and precipitation. Geophys. Res. Lett., 35 , L20709. doi:10.1029/2008GL035694.

    • Search Google Scholar
    • Export Citation
  • Cox, P., and D. Stephenson, 2007: A changing climate for prediction. Science, 317 , 207208.

  • Déqué, M., and Coauthors, 2007: An intercomparison of regional climate simulations for Europe: Assessing uncertainties in model projections. Climatic Change, 81 , 5370.

    • Search Google Scholar
    • Export Citation
  • Doblas-Reyes, F. J., R. Hagedorn, and T. N. Palmer, 2005: The rationale behind the success of multimodel ensembles in seasonal forecasting. Part II: Calibration and combination. Tellus, 57A , 234252.

    • Search Google Scholar
    • Export Citation
  • Epstein, E. S., 1969: A scoring system for probability forecasts of ranked categories. J. Appl. Meteor., 8 , 985987.

  • Frame, D. J., N. E. Faull, M. M. Joshi, and M. R. Allen, 2007: Probabilistic climate forecasts and inductive problems. Phil. Trans. Roy. Soc., 365A , 19711992.

    • Search Google Scholar
    • Export Citation
  • Giorgi, F., and L. O. Mearns, 2002: Calculation of average, uncertainty range, and reliability of regional climate changes from AOGCM simulations via the “reliability ensemble averaging” (REA) method. J. Climate, 15 , 11411158.

    • Search Google Scholar
    • Export Citation
  • Giorgi, F., and L. O. Mearns, 2003: Probability of regional climate change based on the reliability ensemble averaging (REA) method. Geophys. Res. Lett., 30 , 1629. doi:10.1029/2003GL017130.

    • Search Google Scholar
    • Export Citation
  • Gleckler, P. J., K. E. Taylor, and C. Doutriaux, 2008: Performance metrics for climate models. J. Geophys. Res., 113 , D06104. doi:10.1029/2007JD008972.

    • Search Google Scholar
    • Export Citation
  • Greene, A. M., L. Goddard, and U. Lall, 2006: Probabilistic multimodel regional temperature change projections. J. Climate, 19 , 43264343.

    • Search Google Scholar
    • Export Citation
  • Hagedorn, R., F. J. Doblas-Reyes, and T. N. Palmer, 2005: The rationale behind the success of multimodel ensembles in seasonal forecasting. Part I: Basic concept. Tellus, 57A , 219233.

    • Search Google Scholar
    • Export Citation
  • Hawkins, E., and R. Sutton, 2009: The potential to narrow uncertainty in regional climate predictions. Bull. Amer. Meteor. Soc., 90 , 10951107.

    • Search Google Scholar
    • Export Citation
  • Hawkins, E., and R. Sutton, 2010: The potential to narrow uncertainty of regional precipitation change. Climate Dyn., in press, doi:10.1007/s00382-010-0810-6.

    • Search Google Scholar
    • Export Citation
  • Jun, M., R. Knutti, and D. W. Nychka, 2008: Spatial analysis to quantify numerical model bias and dependence: How many climate models are there? J. Amer. Stat. Assoc., 103 , 934947.

    • Search Google Scholar
    • Export Citation
  • Kalnay, E., 2003: Atmospheric Modeling, Data Assimilation and Predictability. Cambridge University Press, 341 pp.

  • Kharin, V. V., and F. W. Zwiers, 2003: Improved seasonal probability forecasts. J. Climate, 16 , 16841701.

  • Knutti, R., 2008: Should we believe model predictions of future climate change? Philos. Trans. Roy. Soc., 366A , 46474664.

  • Knutti, R., R. Furrer, C. Tebaldi, and J. Cermak, 2010: Challenges in combining projections from multiple climate models. J. Climate, 23 , 27392758.

    • Search Google Scholar
    • Export Citation
  • Kumar, A., 2009: Finite samples and uncertainty estimates for skill measures for seasonal prediction. Mon. Wea. Rev., 137 , 26222631.

  • Liniger, M. A., H. Mathis, C. Appenzeller, and F. J. Doblas-Reyes, 2007: Realistic greenhouse gas forcing and seasonal forecasts. Geophys. Res. Lett., 34 , L04705. doi:10.1029/2006GL028335.

    • Search Google Scholar
    • Export Citation
  • Lucas-Picher, P., D. Caya, R. de Elía, and R. Laprise, 2008: Investigation of regional climate models’ internal variability with a ten-member ensemble of 10-year simulations over a large domain. Climate Dyn., 31 , 927940.

    • Search Google Scholar
    • Export Citation
  • Meehl, G. A., and Coauthors, 2009: Decadal prediction: Can it be skillful? Bull. Amer. Meteor. Soc., 90 , 14671485.

  • Murphy, J. M., D. M. H. Sexton, D. N. Barnett, G. S. Jones, M. J. Webb, M. Collins, and D. A. Stainforth, 2004: Quantification of modelling uncertainties in a large ensemble of climate change simulations. Nature, 430 , 768772.

    • Search Google Scholar
    • Export Citation
  • Nakicenovic, N., and R. Swart, Eds. 2000: Special Report on Emissions Scenarios. A Special Report of Working Group III of the Intergovernmental Panel on Climate Change. Cambridge University Press, 599 pp.

    • Search Google Scholar
    • Export Citation
  • Palmer, T. N., and Coauthors, 2004: Development of a European Multimodel Ensemble System for Seasonal-to-Interannual Prediction (DEMETER). Bull. Amer. Meteor. Soc., 85 , 853872.

    • Search Google Scholar
    • Export Citation
  • Perkins, S. E., and A. J. Pitman, 2009: Do weak AR4 model bias projections of future climate change over Australia? Climatic Change, 93 , 527558.

    • Search Google Scholar
    • Export Citation
  • Popper, K. R., 1959: The propensity interpretation of probability. Brit. J. Philos. Sci., 10 , 2542.

  • Raftery, A. E., T. Gneiting, F. Balabdaoui, and M. Polakowski, 2005: Using Bayesian model averaging to calibrate forecast ensembles. Mon. Wea. Rev., 133 , 11551174.

    • Search Google Scholar
    • Export Citation
  • Räisänen, J., 2007: How reliable are climate models? Tellus, 59A , 229.

  • Rajagopalan, B., U. Lall, and S. E. Zebiak, 2002: Categorical climate forecasts through regularization and optimal combination of multiple GCM ensembles. Mon. Wea. Rev., 130 , 17921811.

    • Search Google Scholar
    • Export Citation
  • Reifen, C., and R. Toumi, 2009: Climate projections: Past performance no guarantee of future skill? Geophys. Res. Lett., 36 , L13704. doi:10.1029/2009GL038082.

    • Search Google Scholar
    • Export Citation
  • Robertson, A. W., U. Lall, S. E. Zebiak, and L. Goddard, 2004: Improved combination of multiple atmospheric GCM ensembles for seasonal prediction. Mon. Wea. Rev., 132 , 27322744.

    • Search Google Scholar
    • Export Citation
  • Rougier, J., 2007: Probabilistic inference for future climate using an ensemble of climate model evaluations. Climatic Change, 81 , 247264.

    • Search Google Scholar
    • Export Citation
  • Roulston, M. S., and L. A. Smith, 2002: Evaluating probabilistic forecasts using information theory. Mon. Wea. Rev., 130 , 16531660.

  • Scherrer, S. C., 2010: Present-day interannual variability of surface climate in CMIP3 models and its relation to future warming. Int. J. Climatol., in press, doi:10.1002/joc.2170.

    • Search Google Scholar
    • Export Citation
  • Smith, L. A., 2002: What might we learn from climate forecasts? Proc. Natl. Acad. Sci. USA, 99 , 24872492.

  • Solomon, S., D. Qin, M. Manning, M. Marquis, K. Averyt, M. M. B. Tignor, H. L. Miller Jr., and Z. Chen, Eds. 2007: Climate Change 2007: The Physical Sciences Basis. Cambridge University Press, 996 pp.

    • Search Google Scholar
    • Export Citation
  • Stainforth, D. A., M. R. Allen, E. R. Tredger, and L. A. Smith, 2007: Confidence, uncertainty and decision-support relevance in climate predictions. Philos. Trans. Roy. Soc. London, 365A , 21452161.

    • Search Google Scholar
    • Export Citation
  • Stephenson, D. B., C. A. S. Coelho, F. J. Doblas-Reyes, and M. Balmaseda, 2005: Forecast assimilation: A unified framework for the combination of multimodel weather and climate predictions. Tellus, 57A , 253264.

    • Search Google Scholar
    • Export Citation
  • Stott, P. A., S. F. B. Tett, G. S. Jones, M. R. Allen, J. F. B. Mitchell, and G. J. Jenkins, 2000: External control of 20th century temperature by natural and anthropogenic forcings. Science, 290 , 21332137.

    • Search Google Scholar
    • Export Citation
  • Tebaldi, C., and R. Knutti, 2007: The use of the multimodel ensemble in probabilistic climate projections. Philos. Trans. Roy. Soc., 365A , 20532075.

    • Search Google Scholar
    • Export Citation
  • Tebaldi, C., R. L. Smith, D. Nychka, and L. O. Mearns, 2005: Quantifying uncertainty in projections of regional climate change: A Bayesian approach to the analysis of multimodel ensembles. J. Climate, 18 , 15241540.

    • Search Google Scholar
    • Export Citation
  • Uppala, S. M., and Coauthors, 2005: The ERA-40 Re-Analysis. Quart. J. Roy. Meteor. Soc., 131 , 29613012.

  • Van der Linden, P., and J. F. B. Mitchell, Eds. 2009: ENSEMBLES: Climate change and its impacts at seasonal, decadal and centennial timescales. Summary of research and results from the ENSEMBLES project. Met Office Hadley Centre, 160 pp. [Available from Met Office Hadley Centre, FitzRoy Road, Exeter AU3 EX1 3PB, United Kingdom].

    • Search Google Scholar
    • Export Citation
  • Walsh, J. E., W. L. Chapman, V. Romanovsky, J. H. Christensen, and M. Stendel, 2008: Global climate model performance over Alaska and Greenland. J. Climate, 21 , 61566174.

    • Search Google Scholar
    • Export Citation
  • Webster, M. D., 2003: Communicating climate change uncertainty to policy-makers and the public. Climatic Change, 61 , 18.

  • Weigel, A. P., M. A. Liniger, and C. Appenzeller, 2007: Generalization of the discrete Brier and ranked probability skill scores for weighted multimodel ensemble forecasts. Mon. Wea. Rev., 135 , 27782785.

    • Search Google Scholar
    • Export Citation
  • Weigel, A. P., D. Baggenstos, M. A. Liniger, F. Vitart, and C. Appenzeller, 2008a: Probabilistic verification of monthly temperature forecasts. Mon. Wea. Rev., 136 , 51625182.

    • Search Google Scholar
    • Export Citation
  • Weigel, A. P., M. A. Liniger, and C. Appenzeller, 2008b: Can multimodel combination really enhance the prediction skill of ensemble forecasts? Quart. J. Roy. Meteor. Soc., 134 , 241260.

    • Search Google Scholar
    • Export Citation
  • Weigel, A. P., M. A. Liniger, and C. Appenzeller, 2009: Seasonal ensemble forecasts: Are recalibrated single models better than multimodels? Mon. Wea. Rev., 137 , 14601479.

    • Search Google Scholar
    • Export Citation
  • Whetton, P., I. Macadam, J. Bathols, and J. O’Grady, 2007: Assessment of the use of current climate patterns to evaluate regional enhanced greenhouse response patterns of climate models. Geophys. Res. Lett., 34 , L14701. doi:10.1029/2007GL030025.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 2125 648 50
PDF Downloads 935 292 40