Editorial Type:
Article
Article Type:
Research Article

Seasonal Ensemble Forecasts: Are Recalibrated Single Models Better than Multimodels?

Andreas P. Weigel Federal Office of Meteorology and Climatology, MeteoSwiss, Zürich, Switzerland

Search for other papers by Andreas P. Weigel in
Current site
Google Scholar
PubMed Close
,
Mark A. Liniger Federal Office of Meteorology and Climatology, MeteoSwiss, Zürich, Switzerland

Search for other papers by Mark A. Liniger in
Current site
Google Scholar
PubMed Close
, and
Christof Appenzeller Federal Office of Meteorology and Climatology, MeteoSwiss, Zürich, Switzerland

Search for other papers by Christof Appenzeller in
Current site
Google Scholar
PubMed Close
Print Publication:
01 Apr 2009
DOI:
https://doi.org/10.1175/2008MWR2773.1
Page(s):
1460–1479
Restricted access

Abstract

Multimodel ensemble combination (MMEC) has become an accepted technique to improve probabilistic forecasts from short- to long-range time scales. MMEC techniques typically widen ensemble spread, thus improving the dispersion characteristics and the reliability of the forecasts. This raises the question as to whether the same effect could be achieved in a potentially cheaper way by rescaling single model ensemble forecasts a posteriori such that they become reliable. In this study a climate conserving recalibration (CCR) technique is derived and compared with MMEC. With a simple stochastic toy model it is shown that both CCR and MMEC successfully improve forecast reliability. The difference between these two methods is that CCR conserves resolution but inevitably dilutes the potentially predictable signal while MMEC is in the ideal case able to fully retain the predictable signal and to improve resolution. Therefore, MMEC is conceptually to be preferred, particularly since the effect of CCR depends on the length of the data record and on distributional assumptions. In reality, however, multimodels consist only of a finite number of participating single models, and the model errors are often correlated. Under such conditions, and depending on the skill metric applied, CCR-corrected single models can on average have comparable skill as multimodel ensembles, particularly when the potential model predictability is low. Using seasonal near-surface temperature and precipitation forecasts of three models of the Development of a European Multimodel Ensemble System for Seasonal-to-Interannual Prediction (DEMETER) dataset, it is shown that the conclusions drawn from the toy-model experiments hold equally in a real multimodel ensemble prediction system. All in all, it is not possible to make a general statement on whether CCR or MMEC is the better method. Rather it seems that optimum forecasts can be obtained by a combination of both methods, but only if first MMEC and then CCR is applied. The opposite order—first CCR, then MMEC—is shown to be of only little effect, at least in the context of seasonal forecasts.

Corresponding author address: Andreas Weigel, MeteoSwiss, Krähbühlstrasse 58, P.O. Box 514, CH-8044 Zürich, Switzerland. Email: andreas.weigel@meteoswiss.ch

Abstract

Multimodel ensemble combination (MMEC) has become an accepted technique to improve probabilistic forecasts from short- to long-range time scales. MMEC techniques typically widen ensemble spread, thus improving the dispersion characteristics and the reliability of the forecasts. This raises the question as to whether the same effect could be achieved in a potentially cheaper way by rescaling single model ensemble forecasts a posteriori such that they become reliable. In this study a climate conserving recalibration (CCR) technique is derived and compared with MMEC. With a simple stochastic toy model it is shown that both CCR and MMEC successfully improve forecast reliability. The difference between these two methods is that CCR conserves resolution but inevitably dilutes the potentially predictable signal while MMEC is in the ideal case able to fully retain the predictable signal and to improve resolution. Therefore, MMEC is conceptually to be preferred, particularly since the effect of CCR depends on the length of the data record and on distributional assumptions. In reality, however, multimodels consist only of a finite number of participating single models, and the model errors are often correlated. Under such conditions, and depending on the skill metric applied, CCR-corrected single models can on average have comparable skill as multimodel ensembles, particularly when the potential model predictability is low. Using seasonal near-surface temperature and precipitation forecasts of three models of the Development of a European Multimodel Ensemble System for Seasonal-to-Interannual Prediction (DEMETER) dataset, it is shown that the conclusions drawn from the toy-model experiments hold equally in a real multimodel ensemble prediction system. All in all, it is not possible to make a general statement on whether CCR or MMEC is the better method. Rather it seems that optimum forecasts can be obtained by a combination of both methods, but only if first MMEC and then CCR is applied. The opposite order—first CCR, then MMEC—is shown to be of only little effect, at least in the context of seasonal forecasts.

Corresponding author address: Andreas Weigel, MeteoSwiss, Krähbühlstrasse 58, P.O. Box 514, CH-8044 Zürich, Switzerland. Email: andreas.weigel@meteoswiss.ch

Save
Cover Monthly Weather Review
Monthly Weather Review

  • Adler, R. F., and Coauthors, 2003: The Version-2 Global Precipitation Climatology Project (GPCP) monthly precipitation analysis (1979–present). J. Hydrometeor., 4 , 1147–1167.

    • Search Google Scholar
    • Export Citation

  • Anderson, D. L. T., and Coauthors, 2003: Comparison of the ECMWF seasonal forecast systems 1 and 2, including the relative performance for the 1997/8 El Niño. ECMWF Tech. Memo. 404, 93 pp.

    • Search Google Scholar
    • Export Citation

  • Atger, F., 2003: Spatial and interannual variability of the reliability of ensemble-based probabilistic forecasts: Consequences for calibration. Mon. Wea. Rev., 131 , 1509–1523.

    • Search Google Scholar
    • Export Citation

  • Box, G. E. P., and D. R. Cox, 1964: An analysis of transformations. J. Roy. Stat. Soc. Ser. B. Methodol., 26 , 211–252.

  • Buizza, R., and T. N. Palmer, 1998: Impact of ensemble size on ensemble prediction. Mon. Wea. Rev., 126 , 2508–2518.

  • Buizza, R., M. Miller, and T. N. Palmer, 1999: Stochastic representation of model uncertainties in the ECMWF Ensemble Prediction System. Quart. J. Roy. Meteor. Soc., 125 , 2887–2908.

    • Search Google Scholar
    • Export Citation

  • Buizza, R., P. L. Houtekamer, Z. Toth, G. Pellerin, M. Wei, and Y. Zhu, 2005: A comparison of the ECMWF, MSC, and NCEP global ensemble prediction systems. Mon. Wea. Rev., 133 , 1076–1097.

    • Search Google Scholar
    • Export Citation

  • DelSole, T., 2007: A Bayesian framework for multimodel regression. J. Climate, 20 , 2810–2826.

  • Déque, M., 2001: Seasonal predictability of tropical rainfall: Probabilistic formulation and validation. Tellus, 53A , 500–512.

  • Doblas-Reyes, F. J., R. Hagedorn, and T. N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting. Part II: Calibration and combination. Tellus, 57A , 234–252.

    • Search Google Scholar
    • Export Citation

  • Epstein, E. S., 1969: A scoring system for probability forecasts of ranked categories. J. Appl. Meteor., 8 , 985–987.

  • Feddersen, H., and U. Andersen, 2005: A method of statistical downscaling of seasonal ensemble predictions. Tellus, 57A , 398–408.

  • Ferro, C. A. T., D. S. Richardson, and A. P. Weigel, 2008: On the effect of ensemble size on the discrete and continuous ranked probability scores. Meteor. Appl., 15 , 19–24.

    • Search Google Scholar
    • Export Citation

  • Goddard, L., S. J. Mason, S. E. Zebiak, C. F. Ropelewski, and R. Basher, 2001: Current approaches to seasonal to interannual climate predictions. Int. J. Climatol., 21 , 1111–1152.

    • Search Google Scholar
    • Export Citation

  • Gordon, C., C. Cooper, C. A. Senior, H. Banks, J. M. Gregory, T. C. Johns, J. F. B. Mitchell, and R. A. Wood, 2000: The simulation of SST, sea ice extents and ocean heat transports in a version of the Hadley Centre coupled model without flux adjustments. Climate Dyn., 16 , 147–168.

    • Search Google Scholar
    • Export Citation

  • Hagedorn, R., F. J. Doblas-Reyes, and T. N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting. Part I: Basic concept. Tellus, 57A , 219–233.

    • Search Google Scholar
    • Export Citation

  • Kharin, V. V., and F. W. Zwiers, 2003: Improved seasonal probability forecasts. J. Climate, 16 , 1684–1701.

  • Krishnamurti, T. N., C. M. Kishtawal, T. E. LaRow, D. R. Bachiochi, Z. Zhang, C. E. Williford, S. Gadgil, and S. Surendran, 1999: Improved weather and seasonal climate forecasts from a multimodel superensemble. Science, 285 , 1548–1550.

    • Search Google Scholar
    • Export Citation

  • Kumar, A., A. G. Barnston, and M. P. Hoerling, 2001: Seasonal predictions, probabilistic verifications, and ensemble size. J. Climate, 14 , 1671–1676.

    • Search Google Scholar
    • Export Citation

  • Liniger, M. A., H. Mathis, C. Appenzeller, and F. J. Doblas-Reyes, 2007: Realistic greenhouse gas forcing and seasonal forecasts. Geophys. Res. Lett., 34 , L04705. doi:10.1029/2006GL028335.

    • Search Google Scholar
    • Export Citation

  • Mason, S. J., 2004: On using “climatology” as a reference strategy in the Brier and ranked probability skill scores. Mon. Wea. Rev., 132 , 1891–1895.

    • Search Google Scholar
    • Export Citation

  • Mason, S. J., 2008: From dynamical model predictions to seasonal climate forecasts. Seasonal Climate Variability: Forecasting and Managing Risk, A. Troccoli et al., Eds., Springer Academic Publishers, 167–206.

    • Search Google Scholar
    • Export Citation

  • Mason, S. J., and O. Baddour, 2008: Statistical modelling. Seasonal Climate Variability: Forecasting and Managing Risk, A. Troccoli et al., Eds., Springer Academic Publishers, 167–206.

    • Search Google Scholar
    • Export Citation

  • Mason, S. J., and D. B. Stephenson, 2008: How do we know whether seasonal climate forecasts are any good? Seasonal Climate Variability: Forecasting and Managing Risk, A. Troccoli et al., Eds., Springer Academic Publishers, 167–206.

    • Search Google Scholar
    • Export Citation

  • Mason, S. J., and A. P. Weigel, 2009: A generic forecast verification framework for administrative purposes. Mon. Wea. Rev., 137 , 331–349.

    • Search Google Scholar
    • Export Citation

  • Müller, W. A., C. Appenzeller, and C. Schar, 2004: Probabilistic seasonal prediction of the winter North Atlantic Oscillation and its impact on near surface temperature. Climate Dyn., 24 , 213–226.

    • Search Google Scholar
    • Export Citation

  • Murphy, A. H., 1969: On the ranked probability skill score. J. Appl. Meteor., 8 , 988–989.

  • Murphy, A. H., 1971: A note on the ranked probability skill score. J. Appl. Meteor., 10 , 155–156.

  • Murphy, A. H., 1991: Forecast verification: Its complexity and dimensionality. Mon. Wea. Rev., 119 , 1590–1601.

  • Palmer, T. N., and Coauthors, 2004: Development of a European Multimodel Ensemble System for Seasonal-to-Interannual Prediction (DEMETER). Bull. Amer. Meteor. Soc., 85 , 853–872.

    • Search Google Scholar
    • Export Citation

  • Palmer, T. N., R. Buizza, R. Hagedorn, A. Lawrence, M. Leutbecher, and L. Smith, 2006: Ensemble prediction: A pedagogical perspective. ECMWF Newsletter, No. 106, ECMWF, Reading, United Kingdom, 10–17.

    • Search Google Scholar
    • Export Citation

  • Palmer, T. N., F. J. Doblas-Reyes, A. Weisheimer, and M. J. Rodwell, 2008: Toward seamless prediction—Calibration of climate change projections using seasonal forecasts. Bull. Amer. Meteor. Soc., 89 , 459–470.

    • Search Google Scholar
    • Export Citation

  • Pellerin, G., L. Lefaivre, P. Houtekamer, and C. Girard, 2003: Increasing the horizontal resolution of ensemble forecasts at CMC. Nonlinear Processes Geophys., 10 , 463–468.

    • Search Google Scholar
    • Export Citation

  • Peña, M., and H. van den Dool, 2008: Consolidation of multimodel forecasts by ridge regression: Application to Pacific sea surface temperature. J. Climate, 21 , 6521–6538.

    • Search Google Scholar
    • Export Citation

  • Rajagopalan, B., U. Lall, and S. E. Zebiak, 2002: Categorical climate forecasts through regularization and optimal combination of multiple GCM ensembles. Mon. Wea. Rev., 130 , 1792–1811.

    • Search Google Scholar
    • Export Citation

  • Richardson, D. S., 2001: Measures of skill and value of ensemble prediction systems, their interrelationship and the effect of ensemble size. Quart. J. Roy. Meteor. Soc., 127 , 2473–2489.

    • Search Google Scholar
    • Export Citation

  • Robertson, A. W., U. Lall, S. E. Zebiak, and L. Goddard, 2004: Improved combination of multiple atmospheric GCM ensembles for seasonal prediction. Mon. Wea. Rev., 132 , 2732–2744.

    • Search Google Scholar
    • Export Citation

  • Roulston, M. S., and L. A. Smith, 2003: Combining dynamical and statistical ensembles. Tellus, 55A , 16–30.

  • Rowell, D. P., 1998: Assessing potential predictability with an ensemble of multidecadal GCM simulations. J. Climate, 11 , 109–120.

  • Schwierz, C., C. Appenzeller, H. C. Davies, M. A. Liniger, W. Müller, T. F. Stocker, and M. Yoshimore, 2006: Challenges posed by and approaches to the study of seasonal-to-decadal climate variability. Climatic Change, 79 , 31–63.

    • Search Google Scholar
    • Export Citation

  • Sheshkin, D. J., 2007: Handbook of Parametric and Nonparametric Statistical Procedures. Chapman and Hall/CRC, 1776 pp.

  • Stephenson, D. B., C. A. S. Coelho, F. J. Doblas-Reyes, and M. Balmaseda, 2005: Forecast assimilation: A unified framework for the combination of multi-model weather and climate predictions. Tellus, 57A , 253–264.

    • Search Google Scholar
    • Export Citation

  • Tippett, M. K., A. G. Barnston, and A. W. Robertson, 2007: Estimation of seasonal precipitation tercile-based categorical probabilities from ensembles. J. Climate, 20 , 2210–2228.

    • Search Google Scholar
    • Export Citation

  • Toth, Z., O. Talagrand, G. Candille, and Y. Zhu, 2003: Probability and ensemble forecasts. Forecast Verification—A Practitioner’s Guide in Atmospheric Science, I. T. Joliffe and D. B. Stephenson, Eds., John Wiley and Sons, 137–163.

    • Search Google Scholar
    • Export Citation

  • Uppala, S. M., and Coauthors, 2005: The ERA-40 Re-Analysis. Quart. J. Roy. Meteor. Soc., 131 , 2961–3012.

  • Vitart, F., and Coauthors, 2007: Dynamically-based seasonal forecasts of Atlantic tropical storm activity issued in June by EUROSIP. Geophys. Res. Lett., 34 , L16815. doi:10.1029/2007GL030740.

    • Search Google Scholar
    • Export Citation

  • Weigel, A. P., and N. Bowler, 2009: Comment on “Can multi-model combination really enhance the prediction skill of probabilistic ensemble forecasts?”. Quart. J. Roy. Meteor. Soc., 135 , 535–539.

    • Search Google Scholar
    • Export Citation

  • Weigel, A. P., F. K. Chow, and M. W. Rotach, 2007a: The effect of mountainous topography on moisture exchange between the “surface” and the free atmosphere. Bound.-Layer Meteor., 125 , 227–244.

    • Search Google Scholar
    • Export Citation

  • Weigel, A. P., M. A. Liniger, and C. Appenzeller, 2007b: The discrete Brier and ranked probability skill scores. Mon. Wea. Rev., 135 , 118–124.

    • Search Google Scholar
    • Export Citation

  • Weigel, A. P., M. A. Liniger, and C. Appenzeller, 2007c: Generalization of the discrete Brier and ranked probability skill scores for weighted multimodel ensemble forecasts. Mon. Wea. Rev., 135 , 2778–2785.

    • Search Google Scholar
    • Export Citation

  • Weigel, A. P., D. Baggenstos, M. A. Liniger, F. Vitart, and C. Appenzeller, 2008a: Probabilistic verification of monthly temperature forecasts. Mon. Wea. Rev., 136 , 5162–5182.

    • Search Google Scholar
    • Export Citation

  • Weigel, A. P., M. A. Liniger, and C. Appenzeller, 2008b: Can multi-model combination really enhance the prediction skill of probabilistic ensemble forecasts? Quart. J. Roy. Meteor. Soc., 134 , 241–260.

    • Search Google Scholar
    • Export Citation

  • Wilks, D. S., 2002: Smoothing forecast ensembles with fitted probability distributions. Quart. J. Roy. Meteor. Soc., 128 , 2821–2836.

  • Wilks, D. S., 2006: Statistical Methods in the Atmospheric Sciences. 2nd ed. International Geophysics Series, Vol. 91, Academic Press, 627 pp.

    • Search Google Scholar
    • Export Citation

  • Yoo, J. H., and I-S. Kang, 2005: Theoretical examination of a multi-model composite for seasonal prediction. Geophys. Res. Lett., 32 , L18707. doi:10.1029/2005GL023513.

    • Search Google Scholar
    • Export Citation

  • Zwiers, F. W., 1996: Interannual variability and predictability in an ensemble of AMIP climate simulations conducted with the CCC GCM2. Climate Dyn., 12 , 825–848.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 571 164 14
PDF Downloads 350 69 8