Scoring Probabilistic Forecasts: The Importance of Being Proper

Jochen Bröcker Centre for the Analysis of Time Series, London School of Economics, London, United Kingdom

Search for other papers by Jochen Bröcker in
Current site
Google Scholar
PubMed
Close
and
Leonard A. Smith Centre for the Analysis of Time Series, London School of Economics, London, and Pembroke College, Oxford University, Oxford, United Kingdom

Search for other papers by Leonard A. Smith in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Questions remain regarding how the skill of operational probabilistic forecasts is most usefully evaluated or compared, even though probability forecasts have been a long-standing aim in meteorological forecasting. This paper explains the importance of employing proper scores when selecting between the various measures of forecast skill. It is demonstrated that only proper scores provide internally consistent evaluations of probability forecasts, justifying the focus on proper scores independent of any attempt to influence the behavior of a forecaster. Another property of scores (i.e., locality) is discussed. Several scores are examined in this light. There is, effectively, only one proper, local score for probability forecasts of a continuous variable. It is also noted that operational needs of weather forecasts suggest that the current concept of a score may be too narrow; a possible generalization is motivated and discussed in the context of propriety and locality.

Corresponding author address: Jochen Bröcker, Centre for the Analysis of Time Series, London School of Economics, Houghton St., London WC2A 2AE, United Kingdom. Email: cats@lse.ac.uk

Abstract

Questions remain regarding how the skill of operational probabilistic forecasts is most usefully evaluated or compared, even though probability forecasts have been a long-standing aim in meteorological forecasting. This paper explains the importance of employing proper scores when selecting between the various measures of forecast skill. It is demonstrated that only proper scores provide internally consistent evaluations of probability forecasts, justifying the focus on proper scores independent of any attempt to influence the behavior of a forecaster. Another property of scores (i.e., locality) is discussed. Several scores are examined in this light. There is, effectively, only one proper, local score for probability forecasts of a continuous variable. It is also noted that operational needs of weather forecasts suggest that the current concept of a score may be too narrow; a possible generalization is motivated and discussed in the context of propriety and locality.

Corresponding author address: Jochen Bröcker, Centre for the Analysis of Time Series, London School of Economics, Houghton St., London WC2A 2AE, United Kingdom. Email: cats@lse.ac.uk

Save
  • Bernardo, J. M., 1979: Expected information as expected utility. Ann. Stat., 7 , 686690.

  • Candille, G., and Talagrand O. , 2005: Evaluation of probabilistic prediction systems for a scalar variable. Quart. J. Roy. Meteor. Soc., 131 , 21312150.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gneiting, T., and Raftery A. , 2004: Strictly proper scoring rules, prediction, and estimation. Tech. Rep. 436, Department of Statistics, University of Washington.

    • Crossref
    • Export Citation
  • Good, I. J., 1952: Rational decisions. J. Roy. Stat. Soc., XIV , 107114.

  • Jolliffe, I. T., and Stephenson D. B. , 2003: Forecast Verification: A Practitioner’s Guide in Atmospheric Science. Wiley, 256 pp.

  • Kleeman, R., 2002: Measuring dynamical prediction utility using relative entropy. J. Atmos. Sci., 59 , 20572072.

  • Kullback, S., and Leibler R. A. , 1951: On information and sufficiency. Ann. Math. Stat., 22 , 7986.

  • Murphy, A. H., and Winkler R. L. , 1987: A general framework for forecast verification. Mon. Wea. Rev., 115 , 13301338.

  • Palmer, T. N., 2000: Predicting uncertainty in forecasts of weather and climate. Rep. Prog. Phys., 63 , 2. 71116.

  • Palmer, T. N., Shutts G. J. , Hagedorn R. , Doblas-Reyes F. , Jung T. , and Leutbecher M. , 2005: Representing model uncertainty in weather and climate prediction. Annu. Rev. Earth Planet. Sci., 33 , 163193.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Petterssen, S., 1956: Weather Analysis and Forecasting. 2d ed. McGraw Hill, 505 pp.

  • Roulston, M. S., and Smith L. A. , 2002: Evaluating probabilistic forecasts using information theory. Mon. Wea. Rev., 130 , 16531660.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Selten, R., 1998: Axiomatic characterisation of the quadratic scoring rule. Exp. Econ., 1 , 4362.

  • Wilks, D. S., 1995: Statistical Methods in the Atmospheric Sciences: An Introduction. International Geophysics Series, Vol. 59, Academic Press, 464 pp.

    • Search Google Scholar
    • Export Citation
  • Wilson, L. J., Burrows W. R. , and Lanzinger A. , 1999: A strategy for verification of weather element forecasts from an ensemble prediction system. Mon. Wea. Rev., 127 , 956970.

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 1930 520 37
PDF Downloads 1687 488 44