Assessing the Ensemble Spread–Error Relationship

T. M. Hopson Research Applications Laboratory, National Center for Atmospheric Research, Boulder, Colorado

Search for other papers by T. M. Hopson in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

The potential ability of an ensemble prediction system (EPS) to represent its own varying forecast error provides strong motivation to produce an EPS over a less expensive deterministic forecast. Traditionally, this ability has been assessed by correlating the realized forecast error with the ensemble's dispersion. This paper revisits the limitations of the skill–spread correlation, but uses aspects of the correlation to introduce two metrics to assess an EPS's capacity to provide a reliable likelihood of its own error. Using a perfect EPS, skill–spread correlation is shown to be limited by its dependence on how “skill” and “spread” are defined and, perhaps most fatally, by its inability to distill the skill–spread reliability from the stability properties of the physical system being modeled. Building from this, it is argued there are two aspects of an ensemble's dispersion that should be assessed. First, is there enough variability in the dispersion to justify the expense of the EPS? The factor that controls the theoretical upper limit of the spread–error correlation can be useful in diagnosing this. Second, does the variable dispersion of an ensemble relate to a variable expectation of the forecast error? Representing the spread–error correlation in relation to its theoretical limit can provide a simple diagnostic of this attribute. A context for these concepts is provided by assessing two operational ensembles: western U.S. temperature forecasts and Brahmaputra River flow before and after postprocessing. It is shown that “skill–spread” reliability can be improved by postprocessing to that of a perfect EPS, but at the cost of the potential information content of the EPS's variable dispersion.

Corresponding author address: T. M. Hopson, NCAR/RAL, P.O. Box 3000, Boulder, CO 80307-3000. E-mail: hopson@ucar.edu

Abstract

The potential ability of an ensemble prediction system (EPS) to represent its own varying forecast error provides strong motivation to produce an EPS over a less expensive deterministic forecast. Traditionally, this ability has been assessed by correlating the realized forecast error with the ensemble's dispersion. This paper revisits the limitations of the skill–spread correlation, but uses aspects of the correlation to introduce two metrics to assess an EPS's capacity to provide a reliable likelihood of its own error. Using a perfect EPS, skill–spread correlation is shown to be limited by its dependence on how “skill” and “spread” are defined and, perhaps most fatally, by its inability to distill the skill–spread reliability from the stability properties of the physical system being modeled. Building from this, it is argued there are two aspects of an ensemble's dispersion that should be assessed. First, is there enough variability in the dispersion to justify the expense of the EPS? The factor that controls the theoretical upper limit of the spread–error correlation can be useful in diagnosing this. Second, does the variable dispersion of an ensemble relate to a variable expectation of the forecast error? Representing the spread–error correlation in relation to its theoretical limit can provide a simple diagnostic of this attribute. A context for these concepts is provided by assessing two operational ensembles: western U.S. temperature forecasts and Brahmaputra River flow before and after postprocessing. It is shown that “skill–spread” reliability can be improved by postprocessing to that of a perfect EPS, but at the cost of the potential information content of the EPS's variable dispersion.

Corresponding author address: T. M. Hopson, NCAR/RAL, P.O. Box 3000, Boulder, CO 80307-3000. E-mail: hopson@ucar.edu
Save
  • Atger, F., 1999: The skill of ensemble prediction systems. Mon. Wea. Rev., 127, 19411953.

  • Barker, T. W., 1991: The relationship between spread and forecast error in extended-range forecasts. J. Climate, 4, 733742.

  • Buizza, R., 1997: Potential forecast skill of ensemble prediction and spread and skill distributions of the ECMWF Ensemble Prediction System. Mon. Wea. Rev., 125, 99119.

    • Search Google Scholar
    • Export Citation
  • Gneiting, T., and A. E. Raftery, 2007: Strictly proper scoring rules, prediction, and estimation. J. Amer. Stat. Assoc., 102, 359378, doi:10.1198/016214506000001437.

    • Search Google Scholar
    • Export Citation
  • Grimit, E. P., and C. F. Mass, 2007: Measuring the ensemble spread–error relationship with a probabilistic approach: Stochastic ensemble results. Mon. Wea. Rev., 135, 203221.

    • Search Google Scholar
    • Export Citation
  • Hopson, T. M., and P. J. Webster, 2010: Operational flood forecasting for Bangladesh using ECMWF ensemble weather forecasts. J. Hydrometeor., 11, 618641.

    • Search Google Scholar
    • Export Citation
  • Hopson, T. M., and Coauthors, 2010: Quantile regression as a means of calibrating and verifying a mesoscale NWP ensemble. Proc. 20th Conf. on Probability and Statistics in the Atmospheric Sciences, Atlanta, GA, Amer. Meteor. Soc., 7.4. [Available online at https://ams.confex.com/ams/90annual/techprogram/paper_163208.htm.]

  • Houtekamer, P. L., 1993: Global and local skill forecasts. Mon. Wea. Rev., 121, 18341846.

  • Houtekamer, P. L., L. Lefaivre, J. Derome, H. Ritchie, and H. L. Mitchell, 1996: System simulation approach to ensemble prediction. Mon. Wea. Rev., 124, 12251242.

    • Search Google Scholar
    • Export Citation
  • Huffman, G. J., R. F. Adler, S. Curtis, D. T. Bolvin, and E. J. Nelkin, 2005: Global rainfall analyses at monthly and 3-hr time scales. Measuring Precipitation from Space: EURAINSAT and the Future, V. Levizzani, P. Bauer, and J. F. Turk, Eds., Springer, 291–306.

  • Huffman, G. J., and Coauthors, 2007: The TRMM Multisatellite Precipitation Analysis (TMPA): Quasi-global, multiyear, combined sensor precipitation estimates at fine scales. J. Hydrometeor., 8, 3855.

    • Search Google Scholar
    • Export Citation
  • Joyce, R. J., J. E. Janowiak, P. A. Arkin, and P. P. Xie, 2004: CMORPH: A method that produces global precipitation estimates from passive microwave and infrared data at high spatial and temporal resolution. J. Hydrometeor., 5, 487503.

    • Search Google Scholar
    • Export Citation
  • Koenker, R. W., and G. W. Bassett Jr., 1978: Regression quantiles. Econometrica, 46, 3350.

  • Kolczynski, W. C., D. R. Stauffer, S. E. Haupt, N. S. Altman, and A. Deng, 2011: Investigation of ensemble variance as a measure of true forecast variance. Mon. Wea. Rev., 139, 39543963.

    • Search Google Scholar
    • Export Citation
  • Kruizinga, S., and C. J. Kok, 1988: Evaluation of the ECMWF experimental skill prediction scheme and a statistical analysis of forecast errors. Proc. Workshop on Predictability in the Medium and Extended Range, Reading, United Kingdom, ECMWF, 403415.

  • Leith, C. E., 1974: Theoretical skill of Monte Carlo forecasts. Mon. Wea. Rev., 102, 409418.

  • Liu, Y., M. Xu, J. Hacker, T. Warner, and S. Swerdlin, 2007: A WRF and MM5-based 4-D mesoscale ensemble data analysis and prediction system (E-RTFDDA) developed for ATEC operational applications. Preprints, 18th Conf. on Numerical Weather Prediction, Park City, UT, Amer. Meteor. Soc., 7B.7. [Available online at https://ams.confex.com/ams/pdfpapers/124739.pdf.]

  • Marzban, C., R. Wang, F. Kong, and S. Leyton, 2011: On the effect of correlations on rank histograms: Reliability of temperature and wind speed forecasts from finescale ensemble reforecasts. Mon. Wea. Rev., 139, 295310.

    • Search Google Scholar
    • Export Citation
  • Molteni, F., R. Buizza, T. N. Palmer, and T. Petroliagis, 1996: The ECMWF Ensemble Prediction System: Methodology and validation. Quart. J. Roy. Meteor. Soc., 122, 73119.

    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1973: Hedging and skill scores for probability forecasts. J. Appl. Meteor., 12, 215223.

  • Palmer, T. N., 2002: The economic value of ensemble forecasts as a tool for risk assessment: From days to decades. Quart. J. Roy. Meteor. Soc., 128, 747774.

    • Search Google Scholar
    • Export Citation
  • Richardson, D. S., 2000: Skill and relative economic value of the ECMWF Ensemble Prediction System. Quart. J. Roy. Meteor. Soc., 126, 649667.

    • Search Google Scholar
    • Export Citation
  • Scherrer, S. C., C. Appenzeller, P. Eckert, and D. Cattani, 2004: Analysis of the spread–skill relations using the ECMWF ensemble prediction system over Europe. Wea. Forecasting, 19, 552565.

    • Search Google Scholar
    • Export Citation
  • Stigler, S. M., 1989: Francis Galton's account of the invention of correlation. Stat. Sci., 4, 7379, doi:10.1214/ss/1177012580.

  • Toth, Z., and E. Kalnay, 1997: Ensemble forecasting at NCEP and the breeding method. Mon. Wea. Rev., 125, 32973319.

  • Toth, Z., O. Talagrand, G. Candille, and Y. Zhu, 2003: Probability and ensemble forecasts. Forecast Verification: A Practitioner's Guide in Atmospheric Science, I. T. Jolliffe and D. B. Stephenson, Eds., J. Wiley and Sons, 137–164.

  • Wang, X., and C. H. Bishop, 2003: A comparison of breeding and ensemble transform Kalman filter ensemble forecast schemes. J. Atmos. Sci., 60, 11401158.

    • Search Google Scholar
    • Export Citation
  • Whitaker, J. S., and A. F. Loughe, 1998: The relationship between ensemble spread and ensemble mean skill. Mon. Wea. Rev., 126, 32923302.

    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 1995: Statistical Methods in the Atmospheric Sciences: An Introduction. Academic Press, 467 pp.

  • Wilks, D. S., 2011: On the reliability of the rank histogram. Mon. Wea. Rev., 139, 311316.

  • Xie, P. P., B. Rudolf, U. Schneider, and P. A. Arkin, 1996: Gauge-based monthly analysis of global land precipitation from 1971 to 1994. J. Geophys. Res., 101 (D14), 19 02319 034.

    • Search Google Scholar
    • Export Citation
  • Zhu, Y., Z. Toth, R. Wobus, D. Richardson, and K. Mylne, 2002: The economic value of ensemble-based weather forecasts. Bull. Amer. Meteor. Soc., 83, 7383.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 1765 596 87
PDF Downloads 1105 269 41