Editorial Type:
Article
Article Type:
Research Article

Best Practices for Postprocessing Ensemble Climate Forecasts. Part I: Selecting Appropriate Recalibration Methods

Philip G. Sansom Exeter Climate Systems, University of Exeter, Exeter, United Kingdom

Search for other papers by Philip G. Sansom in
Current site
Google Scholar
PubMed Close
,
Christopher A. T. Ferro Exeter Climate Systems, University of Exeter, Exeter, United Kingdom

Search for other papers by Christopher A. T. Ferro in
Current site
Google Scholar
PubMed Close
,
David B. Stephenson Exeter Climate Systems, University of Exeter, Exeter, United Kingdom

Search for other papers by David B. Stephenson in
Current site
Google Scholar
PubMed Close
,
Lisa Goddard International Research Institute for Climate and Society, The Earth Institute, Columbia University, Palisades, New York

Search for other papers by Lisa Goddard in
Current site
Google Scholar
PubMed Close
, and
Simon J. Mason International Research Institute for Climate and Society, The Earth Institute, Columbia University, Palisades, New York

Search for other papers by Simon J. Mason in
Current site
Google Scholar
PubMed Close
Print Publication:
15 Oct 2016
DOI:
https://doi.org/10.1175/JCLI-D-15-0868.1
Page(s):
7247–7264
Restricted access

Abstract

This study describes a systematic approach to selecting optimal statistical recalibration methods and hindcast designs for producing reliable probability forecasts on seasonal-to-decadal time scales. A new recalibration method is introduced that includes adjustments for both unconditional and conditional biases in the mean and variance of the forecast distribution and linear time-dependent bias in the mean. The complexity of the recalibration can be systematically varied by restricting the parameters. Simple recalibration methods may outperform more complex ones given limited training data. A new cross-validation methodology is proposed that allows the comparison of multiple recalibration methods and varying training periods using limited data.

Part I considers the effect on forecast skill of varying the recalibration complexity and training period length. The interaction between these factors is analyzed for gridbox forecasts of annual mean near-surface temperature from the CanCM4 model. Recalibration methods that include conditional adjustment of the ensemble mean outperform simple bias correction by issuing climatological forecasts where the model has limited skill. Trend-adjusted forecasts outperform forecasts without trend adjustment at almost 75% of grid boxes. The optimal training period is around 30 yr for trend-adjusted forecasts and around 15 yr otherwise. The optimal training period is strongly related to the length of the optimal climatology. Longer training periods may increase overall performance but at the expense of very poor forecasts where skill is limited.

Corresponding author address: Philip G. Sansom, Exeter Climate Systems, University of Exeter, Laver Building, North Park Road, Exeter EX4 4QE, United Kingdom. E-mail: p.g.sansom@exeter.ac.uk

Abstract

This study describes a systematic approach to selecting optimal statistical recalibration methods and hindcast designs for producing reliable probability forecasts on seasonal-to-decadal time scales. A new recalibration method is introduced that includes adjustments for both unconditional and conditional biases in the mean and variance of the forecast distribution and linear time-dependent bias in the mean. The complexity of the recalibration can be systematically varied by restricting the parameters. Simple recalibration methods may outperform more complex ones given limited training data. A new cross-validation methodology is proposed that allows the comparison of multiple recalibration methods and varying training periods using limited data.

Part I considers the effect on forecast skill of varying the recalibration complexity and training period length. The interaction between these factors is analyzed for gridbox forecasts of annual mean near-surface temperature from the CanCM4 model. Recalibration methods that include conditional adjustment of the ensemble mean outperform simple bias correction by issuing climatological forecasts where the model has limited skill. Trend-adjusted forecasts outperform forecasts without trend adjustment at almost 75% of grid boxes. The optimal training period is around 30 yr for trend-adjusted forecasts and around 15 yr otherwise. The optimal training period is strongly related to the length of the optimal climatology. Longer training periods may increase overall performance but at the expense of very poor forecasts where skill is limited.

Corresponding author address: Philip G. Sansom, Exeter Climate Systems, University of Exeter, Laver Building, North Park Road, Exeter EX4 4QE, United Kingdom. E-mail: p.g.sansom@exeter.ac.uk
Save
Cover Journal of Climate
Journal of Climate

  • Arribas, A., and Coauthors, 2011: The GloSea4 ensemble prediction system for seasonal forecasting. Mon. Wea. Rev., 139, 1891–1910, doi:10.1175/2010MWR3615.1.

    • Search Google Scholar
    • Export Citation

  • Baran, S., and D. Nemoda, 2016: Censored and shifted gamma distribution based EMOS model for probabilistic quantitative precipitation forecasting. Environmetrics, 27, 280–292, doi:10.1002/env.2391.

    • Search Google Scholar
    • Export Citation

  • Brohan, P., J. J. Kennedy, I. Harris, S. F. B. Tett, and P. D. Jones, 2006: Uncertainty estimates in regional and global observed temperature changes: A new data set from 1850. J. Geophys. Res., 111, D12106, doi:10.1029/2005JD006548.

    • Search Google Scholar
    • Export Citation

  • Coelho, C. A. S., S. Pezzulli, M. Balmaseda, F. J. Doblas-Reyes, and D. B. Stephenson, 2004: Forecast calibration and combination: A simple Bayesian approach for ENSO. J. Climate, 17, 1504–1516, doi:10.1175/1520-0442(2004)017<1504:FCACAS>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Dee, D. P., and Coauthors, 2011: The ERA-Interim reanalysis: Configuration and performance of the data assimilation system. Quart. J. Roy. Meteor. Soc., 137, 553–597, doi:10.1002/qj.828.

    • Search Google Scholar
    • Export Citation

  • Eade, R., D. Smith, A. Scaife, E. Wallace, N. Dunstone, L. Hermanson, and N. Robinson, 2014: Do seasonal-to-decadal climate predictions underestimate the predictability of the real world? Geophys. Res. Lett., 41, 5620–5628, doi:10.1002/2014GL061146.

    • Search Google Scholar
    • Export Citation

  • Garthwaite, P., I. Jolliffe, and B. Jones, 2002: Statistical Inference. 2nd ed. Oxford University Press, 342 pp.

  • Glahn, B., M. Peroutka, J. Wiedenfeld, J. Wagner, G. Zylstra, B. Schuknecht, and B. Jackson, 2009: MOS uncertainty estimates in an ensemble framework. Mon. Wea. Rev., 137, 246–268, doi:10.1175/2008MWR2569.1.

    • Search Google Scholar
    • Export Citation

  • Glahn, H. R., and D. A. Lowry, 1972: The use of model output statistics (MOS) in objective weather forecasting. J. Appl. Meteor., 11, 1203–1211, doi:10.1175/1520-0450(1972)011<1203:TUOMOS>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Gneiting, T., A. E. Raftery, A. H. Westveld, and T. Goldman, 2005: Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation. Mon. Wea. Rev., 133, 1098–1118, doi:10.1175/MWR2904.1.

    • Search Google Scholar
    • Export Citation

  • Goddard, L., and M. Dilley, 2005: El Niño: Catastrophe or opportunity. J. Climate, 18, 651–665, doi:10.1175/JCLI-3277.1.

  • Goddard, L., and Coauthors, 2013: A verification framework for interannual-to-decadal predictions experiments. Climate Dyn., 40, 245–272, doi:10.1007/s00382-012-1481-2.

    • Search Google Scholar
    • Export Citation

  • Goswami, B. N., J. R. Kulkarni, V. R. Mujumdar, and R. Chattopadhyay, 2010: On factors responsible for recent secular trend in the onset phase of monsoon intraseasonal oscillations. Int. J. Climatol., 30, 2240–2246, doi:10.1002/joc.2041.

    • Search Google Scholar
    • Export Citation

  • Hamill, T. M., J. S. Whitaker, and X. Wei, 2004: Ensemble reforecasting: Improving medium-range forecast skill using retrospective forecasts. Mon. Wea. Rev., 132, 1434–1447, doi:10.1175/1520-0493(2004)132<1434:ERIMFS>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • ICPO, 2011: Decadal and bias correction for decadal climate predictions. International CLIVAR Project Office Tech. Rep. 150, 3 pp. [Available online at http://www.wcrp-climate.org/decadal/references/DCPP_Bias_Correction.pdf.]

  • Jolliffe, I. T., and D. B. Stephenson, Eds., 2011: Forecast Verification: A Practitioner’s Guide in Atmospheric Science. 2nd ed. Wiley, 292 pp.

    • Search Google Scholar
    • Export Citation

  • Jones, P. D., M. New, D. E. Parker, S. Martin, and I. G. Rigor, 1999: Surface air temperature and its changes over the past 150 years. Rev. Geophys., 37, 173–199, doi:10.1029/1999RG900002.

    • Search Google Scholar
    • Export Citation

  • Kharin, V. V., and F. W. Zwiers, 2003: Improved seasonal probability forecasts. J. Climate, 16, 1684–1701, doi:10.1175/1520-0442(2003)016<1684:ISPF>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Kharin, V. V., G. J. Boer, W. J. Merryfield, J. F. Scinocca, and W.-S. Lee, 2012: Statistical adjustment of decadal predictions in a changing climate. Geophys. Res. Lett., 39, L19705, doi:10.1029/2012GL052647.

    • Search Google Scholar
    • Export Citation

  • Knutti, R., R. Furrer, C. Tebaldi, J. Cermak, and G. A. Meehl, 2010: Challenges in combining projections from multiple climate models. J. Climate, 23, 2739–2758, doi:10.1175/2009JCLI3361.1.

    • Search Google Scholar
    • Export Citation

  • Krishnamurthy, V., and B. N. Goswami, 2000: Indian monsoon–ENSO relationship on interdecadal timescale. J. Climate, 13, 579–595, doi:10.1175/1520-0442(2000)013<0579:IMEROI>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Krzanowski, W. J., 1998: An Introduction to Statistical Modelling. John Wiley & Sons, 264 pp.

  • MacLachlan, C., and Coauthors, 2015: Global Seasonal Forecast System version 5 (GloSea5): A high-resolution seasonal forecast system. Quart. J. Roy. Meteor. Soc., 141, 1072–1084, doi:10.1002/qj.2396.

    • Search Google Scholar
    • Export Citation

  • Malhi, Y., and J. Wright, 2004: Spatial patterns and recent trends in the climate of tropical rainforest regions. Philos. Trans. Roy. Soc. London, 359B, 311–329, doi:10.1098/rstb.2003.1433.

    • Search Google Scholar
    • Export Citation

  • Matheson, J. E., and R. L. Winkler, 1976: Scoring rules for continuous probability distributions. Manage. Sci., 22, 1087–1096, doi:10.1287/mnsc.22.10.1087.

    • Search Google Scholar
    • Export Citation

  • Merryfield, W. J., and Coauthors, 2013: The Canadian Seasonal to Interannual Prediction System. Part I: Models and initialization. Mon. Wea. Rev., 141, 2910–2945, doi:10.1175/MWR-D-12-00216.1.

    • Search Google Scholar
    • Export Citation

  • Molteni, F., and Coauthors, 2011: The new ECMWF Seasonal Forecast System (System 4). ECMWF Tech. Memo. 656, 49 pp. [Available online at http://www.ecmwf.int/en/elibrary/11209-new-ecmwf-seasonal-forecast-system-system-4.]

  • Raftery, A. E., T. Gneiting, F. Balabdaoui, and M. Polakowski, 2005: Using Bayesian model averaging to calibrate forecast ensembles. Mon. Wea. Rev., 133, 1155–1174, doi:10.1175/MWR2906.1.

    • Search Google Scholar
    • Export Citation

  • Roulston, M. S., and L. A. Smith, 2003: Combining dynamical and statistical ensembles. Tellus, 55A, 16–30, doi:10.1034/j.1600-0870.2003.201378.x.

    • Search Google Scholar
    • Export Citation

  • Schefzik, R., T. L. Thorarinsdottir, and T. Gneiting, 2013: Uncertainty quantification in complex simulation models using ensemble copula coupling. Stat. Sci., 28, 616–640, doi:10.1214/13-STS443.

    • Search Google Scholar
    • Export Citation

  • Scheuerer, M., 2014: Probabilistic quantitative precipitation forecasting using ensemble model output statistics. Quart. J. Roy. Meteor. Soc., 140, 1086–1096, doi:10.1002/qj.2183.

    • Search Google Scholar
    • Export Citation

  • Siegert, S., P. G. Sansom, and R. M. Williams, 2016: Parameter uncertainty in forecast recalibration. Quart. J. Roy. Meteor. Soc., 142, 1213–1221, doi:10.1002/qj.2716.

    • Search Google Scholar
    • Export Citation

  • Sinha, A., G. Kathayat, H. Cheng, S. F. M. Breitenbach, M. Berkelhammer, M. Mudelsee, J. Biswas, and R. L. Edwards, 2015: Trends and oscillations in the Indian summer monsoon rainfall over the last two millennia. Nat. Commun., 6, 6309, doi:10.1038/ncomms7309.

    • Search Google Scholar
    • Export Citation

  • Smith, D. M., A. A. Scaife, and B. P. Kirtman, 2012: What is the current state of scientific knowledge with regard to seasonal and decadal forecasting? Environ. Res. Lett., 7, 015602, doi:10.1088/1748-9326/7/1/015602.

    • Search Google Scholar
    • Export Citation

  • Smith, D. M., R. Eade, and H. Pohlmann, 2013: A comparison of full-field and anomaly initialization for seasonal to decadal climate prediction. Climate Dyn., 41, 3325–3338, doi:10.1007/s00382-013-1683-2.

    • Search Google Scholar
    • Export Citation

  • Stephenson, D. B., C. A. S. Coelho, F. J. Doblas-Reyes, and M. Balmaseda, 2005: Forecast assimilation: A unified framework for the combination of multi-model weather and climate predictions. Tellus, 57A, 253–264, doi:10.1111/j.1600-0870.2005.00110.x.

    • Search Google Scholar
    • Export Citation

  • Stephenson, D. B., M. Collins, J. C. Rougier, and R. E. Chandler, 2012: Statistical problems in the probabilistic prediction of climate change. Environmetrics, 23, 364–372, doi:10.1002/env.2153.

    • Search Google Scholar
    • Export Citation

  • Stockdale, T. N., 1997: Coupled ocean–atmosphere forecasts in the presence of climate drift. Mon. Wea. Rev., 125, 809–818, doi:10.1175/1520-0493(1997)125<0809:COAFIT>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Taylor, K. E., R. J. Stouffer, and G. A. Meehl, 2012: An overview of CMIP5 and the experiment design. Bull. Amer. Meteor. Soc., 93, 485–498, doi:10.1175/BAMS-D-11-00094.1.

    • Search Google Scholar
    • Export Citation

  • Tebaldi, C., and R. Knutti, 2007: The use of the multi-model ensemble in probabilistic climate projections. Philos. Trans. Roy. Soc. London, 365A, 2053–2075, doi:10.1098/rsta.2007.2076.

    • Search Google Scholar
    • Export Citation

  • Trenberth, K. E., and Coauthors, 2007: Observations: Surface and atmospheric climate change. Climate Change 2007: The Physical Science Basis, S. Solomon et al., Eds., Cambridge University Press, 235–336.

  • Wang, M., and J. E. Overland, 2009: A sea ice free summer Arctic within 30 years? Geophys. Res. Lett., 36, L07502, doi:10.1029/2009GL037820.

    • Search Google Scholar
    • Export Citation

  • Weigel, A. P., M. A. Liniger, and C. Appenzeller, 2009: Seasonal ensemble forecasts: Are recalibrated single models better than multimodels? Mon. Wea. Rev., 137, 1460–1479, doi:10.1175/2008MWR2773.1.

    • Search Google Scholar
    • Export Citation

  • Weijs, S. V., and N. van de Giesen, 2011: Accounting for observational uncertainty in forecast verification: An information-theoretical view on forecasts, observations, and truth. Mon. Wea. Rev., 139, 2156–2162, doi:10.1175/2011MWR3573.1.

    • Search Google Scholar
    • Export Citation

  • Wilks, D. S., 2002: Smoothing forecast ensembles with fitted probability distributions. Quart. J. Roy. Meteor. Soc., 128, 2821–2836, doi:10.1256/qj.01.215.

    • Search Google Scholar
    • Export Citation

  • Wilks, D. S., 2006: Comparison of ensemble-MOS methods in the Lorenz ’96 setting. Meteor. Appl., 13, 243–256, doi:10.1017/S1350482706002192.

    • Search Google Scholar
    • Export Citation

  • Wilks, D. S., 2011: Statistical Methods in the Atmospheric Sciences. 3rd ed. Elsevier, 704 pp.

  • Williams, R. M., C. A. T. Ferro, and F. Kwasniok, 2014: A comparison of ensemble post-processing methods for extreme events. Quart. J. Roy. Meteor. Soc., 140, 1112–1120, doi:10.1002/qj.2198.

    • Search Google Scholar
    • Export Citation

  • WMO, 2007: Commission for basic systems: Extraordinary session. WMO Tech. Rep. 1017, 350 pp. [Available online at http://library.wmo.int/opac/index.php?lvl=notice_display&id=9680.]

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 2249 1573 75
PDF Downloads 627 77 12