Editorial Type:
Article
Article Type:
Research Article

Decomposition of the Continuous Ranked Probability Score for Ensemble Prediction Systems

Hans Hersbach Koninklijk Nederlands Meteorologisch Instituut, De Bilt, Netherlands

Search for other papers by Hans Hersbach in
Current site
Google Scholar
PubMed Close
Print Publication:
01 Oct 2000
DOI:
https://doi.org/10.1175/1520-0434(2000)015<0559:DOTCRP>2.0.CO;2
Page(s):
559–570
Restricted access

Abstract

Some time ago, the continuous ranked probability score (CRPS) was proposed as a new verification tool for (probabilistic) forecast systems. Its focus is on the entire permissible range of a certain (weather) parameter. The CRPS can be seen as a ranked probability score with an infinite number of classes, each of zero width. Alternatively, it can be interpreted as the integral of the Brier score over all possible threshold values for the parameter under consideration. For a deterministic forecast system the CRPS reduces to the mean absolute error.

In this paper it is shown that for an ensemble prediction system the CRPS can be decomposed into a reliability part and a resolution/uncertainty part, in a way that is similar to the decomposition of the Brier score. The reliability part of the CRPS is closely connected to the rank histogram of the ensemble, while the resolution/uncertainty part can be related to the average spread within the ensemble and the behavior of its outliers. The usefulness of such a decomposition is illustrated for the ensemble prediction system running at the European Centre for Medium-Range Weather Forecasts. The evaluation of the CRPS and its decomposition proposed in this paper can be extended to systems issuing continuous probability forecasts, by realizing that these can be interpreted as the limit of ensemble forecasts with an infinite number of members.

Corresponding author address: Dr. Hans Hersbach, KNMI, P.O. Box 201, 3730 AE Utrecht, Netherlands.

Abstract

Some time ago, the continuous ranked probability score (CRPS) was proposed as a new verification tool for (probabilistic) forecast systems. Its focus is on the entire permissible range of a certain (weather) parameter. The CRPS can be seen as a ranked probability score with an infinite number of classes, each of zero width. Alternatively, it can be interpreted as the integral of the Brier score over all possible threshold values for the parameter under consideration. For a deterministic forecast system the CRPS reduces to the mean absolute error.

In this paper it is shown that for an ensemble prediction system the CRPS can be decomposed into a reliability part and a resolution/uncertainty part, in a way that is similar to the decomposition of the Brier score. The reliability part of the CRPS is closely connected to the rank histogram of the ensemble, while the resolution/uncertainty part can be related to the average spread within the ensemble and the behavior of its outliers. The usefulness of such a decomposition is illustrated for the ensemble prediction system running at the European Centre for Medium-Range Weather Forecasts. The evaluation of the CRPS and its decomposition proposed in this paper can be extended to systems issuing continuous probability forecasts, by realizing that these can be interpreted as the limit of ensemble forecasts with an infinite number of members.

Corresponding author address: Dr. Hans Hersbach, KNMI, P.O. Box 201, 3730 AE Utrecht, Netherlands.

Save
Cover Weather and Forecasting
Weather and Forecasting

  • Anderson, J. L., 1996: A method for producing and evaluating probabilistic forecasts from ensemble model integrations. J. Climate,9, 1518–1530.

    • Crossref
    • Export Citation

  • Bouttier, F., 1994: Sur la prévision de la qualité des prévisions météorologiques. Ph.D. thesis, Université Paul Sabatier, Toulouse, France, 240 pp. [Available from Libray, Université Paul Sabatier, route de Narbonne, Toulouse, France.].

  • Brier, G. W., 1950: Verification of forecasts expressed in terms of probability. Mon. Wea. Rev.,78, 1–3.

    • Crossref
    • Export Citation

  • Brown, T. A., 1974: Admissible scoring systems for continuous distributions. Manuscript P-5235, The Rand Corporation, Santa Monica, CA, 22 pp. [Available from The Rand Corporation, 1700 Main St., Santa Monica, CA 90407-2138.].

  • Buizza, R., and T. N. Palmer, 1998: Impact of ensemble size on ensemble prediction. Mon. Wea. Rev.,126, 2503–2518.

    • Crossref
    • Export Citation

  • ——, A. Hollingsworth, F. Lalaurette, and A. Ghelli, 1999: Probabilistic predictions of precipitation using the ECMWF Ensemble Prediction System. Wea. Forecasting,14, 168–189.

    • Crossref
    • Export Citation

  • Epstein, E. S., 1969: A scoring system for probability forecasts of ranked categories. J. Appl. Meteor.,8, 985–987.

  • Hamill, T., and S. J. Colucci, 1997: Verification of Eta–RSM short-range ensemble forecasts. Mon. Wea. Rev.,125, 1312–1327.

    • Crossref
    • Export Citation

  • Katz, R. W., and A. H. Murphy, 1997: Economic Value of Weather and Climate Forecasts. Cambridge University Press, 222 pp.

    • Crossref
    • Export Citation

  • Mason, I., 1982: A model for assessment of weather forecasts. Aust. Meteor. Mag.,30, 291–303.

  • Matheson, J. E., and R. L. Winkler, 1976: Scoring rules for continuous probability distributions. Manage. Sci.,22, 1087–1095.

    • Crossref
    • Export Citation

  • Molteni, F., R. Buizza, T. N. Palmer, and T. Petroliagis, 1996: The ECMWF Ensemble Prediction System: Methodology and validation. Quart J. Roy. Meteor. Soc.,122, 73–119.

    • Crossref
    • Export Citation

  • Murphy, A. H., 1969: On the “ranked probability score.” J. Appl. Meteor.,8, 988–989.

  • ——, 1971: A note on the ranked probability score. J. Appl. Meteor.,10, 155–156.

  • ——, 1973: A new vector partition of the probability score. J. Appl. Meteor.,12, 595–600.

    • Crossref
    • Export Citation

  • ——, and E. S. Epstein, 1989: Skill scores and correlation coefficients in model verification. Mon. Wea. Rev.,117, 572–581.

    • Crossref
    • Export Citation

  • Press, W. H., B. P. Flannery, S. A. Teukolsky, and W. T. Vetterling, 1989: Numerical Recipes: The Art of Scientific Computing. Cambridge University Press, 818 pp.

  • Richardson, D., 1998: Obtaining economic value from the EPS. ECMWF Newsletter, Vol. 80, 8–12.

  • ——, 2000: Skill and relative economic value of the ECMWF Ensemble Prediction System. Quart J. Roy. Meteor. Soc.,126, 649–668.

    • Crossref
    • Export Citation

  • Stanski, H. R., L. J. Wilson, and W. R. Burrows, 1989: Survey of common verification methods in meteorology. Atmospheric Environment Service Research Rep. 89-5, 114 pp. [Available from Forecast Research Division, 4905 Dufferin St., Downsview, ON M3H 5T4, Canada.].

  • Talagrand, O., and R. Vautard, 1997: Evaluation of probabilistic prediction systems. Proc. ECMWF Workshop on Predictability, Reading, United Kingdom, ECMWF, 1–25.

  • Unger, D. A., 1985: A method to estimate the continuous ranked probability score. Preprints, Ninth Conf. on Probability and Statistics in Atmospheric Sciences, Virginia Beach, VA, Amer. Meteor. Soc., 206–213.

  • Wilks, D. S., 1995: Statistical Methods in the Atmospheric Sciences. Academic Press, 467 pp.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 15232 3609 249
PDF Downloads 13091 3008 211