• Atger, F., 2003: Spatial and interannual variability of the reliability of ensemble-based probabilistic forecasts: Consequences for calibration. Mon. Wea. Rev., 131 , 15091523.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Atger, F., 2004: Estimation of the reliability of ensemble-based probabilistic forecasts. Quart. J. Roy. Meteor. Soc., 130 , 627646.

  • Brier, G. W., 1950: Verification of forecasts expressed in terms of probabilities. Mon. Wea. Rev., 78 , 13.

  • Candille, G., and Talagrand O. , 2005: Evaluation of probabilistic prediction systems for a scalar variable. Quart. J. Roy. Meteor. Soc., 131 , 21312150.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Epstein, E. S., 1969: A scoring system for probability forecasts of ranked categories. J. Appl. Meteor., 8 , 985987.

  • Ferro, C. A. T., 2007: Comparing probabilistic forecasting systems with the Brier score. Wea. Forecasting, 22 , 10761088.

  • Jolliffe, I. T., and Stephenson D. B. , 2003: Forecast Verification: A Practitioner’s Guide in Atmospheric Science. John Wiley and Sons, 254 pp.

    • Search Google Scholar
    • Export Citation
  • Mullen, S. L., and Buizza R. , 2002: The impact of horizontal resolution and ensemble size on probabilistic forecasts of precipitation by the ECMWF Ensemble Prediction System. Wea. Forecasting, 17 , 173191.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1971: A note on the ranked probability score. J. Appl. Meteor., 10 , 155156.

  • Murphy, A. H., 1973: A new vector partition of the probability score. J. Appl. Meteor., 12 , 595600.

  • Murphy, A. H., 1986: A new decomposition of the Brier score: Formulation and interpretation. Mon. Wea. Rev., 114 , 26712673.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Palmer, T. N., and Coauthors, 2004: Development of a European Multimodel Ensemble System for Seasonal to Interannual Prediction (DEMETER). Bull. Amer. Meteor. Soc., 85 , 853872.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sanders, F., 1963: On subjective probability forecasting. J. Appl. Meteor., 2 , 191201.

  • Stephenson, D. B., Coelho C. A. S. , Balmaseda M. , and Doblas-Reyes F. J. , 2005: Forecast assimilation: A unified framework for the combination of multi-model weather and climate predictions. Tellus, 57A , 253264.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 1154 692 0
PDF Downloads 1037 627 0

Two Extra Components in the Brier Score Decomposition

View More View Less
  • 1 School of Engineering, Computing, and Mathematics, University of Exeter, Exeter, United Kingdom
  • | 2 Centro de Previsão de Tempo e Estudos Climáticos, Instituto Nacional de Pesquisas Espaciais, Cachoeira Paulista, São Paulo, Brazil
  • | 3 School of Engineering, Computing, and Mathematics, University of Exeter, Exeter, United Kingdom
Restricted access

Abstract

The Brier score is widely used for the verification of probability forecasts. It also forms the basis of other frequently used probability scores such as the rank probability score. By conditioning (stratifying) on the issued forecast probabilities, the Brier score can be decomposed into the sum of three components: uncertainty, reliability, and resolution. This Brier score decomposition can provide useful information to the forecast provider about how the forecasts can be improved.

Rather than stratify on all values of issued probability, it is common practice to calculate the Brier score components by first partitioning the issued probabilities into a small set of bins. This note shows that for such a procedure, an additional two within-bin components are needed in addition to the three traditional components of the Brier score. The two new components can be combined with the resolution component to make a generalized resolution component that is less sensitive to choice of bin width than is the traditional resolution component. The difference between the generalized resolution term and the conventional resolution term also quantifies how forecast skill is degraded when issuing categorized probabilities to users. The ideas are illustrated using an example of multimodel ensemble seasonal forecasts of equatorial sea surface temperatures.

Corresponding author address: Dr. David B. Stephenson, School of Engineering, Computing, and Mathematics, University of Exeter, North Park Rd., Exeter EX4 4QF, United Kingdom. Email: d.b.stephenson@exeter.ac.uk

Abstract

The Brier score is widely used for the verification of probability forecasts. It also forms the basis of other frequently used probability scores such as the rank probability score. By conditioning (stratifying) on the issued forecast probabilities, the Brier score can be decomposed into the sum of three components: uncertainty, reliability, and resolution. This Brier score decomposition can provide useful information to the forecast provider about how the forecasts can be improved.

Rather than stratify on all values of issued probability, it is common practice to calculate the Brier score components by first partitioning the issued probabilities into a small set of bins. This note shows that for such a procedure, an additional two within-bin components are needed in addition to the three traditional components of the Brier score. The two new components can be combined with the resolution component to make a generalized resolution component that is less sensitive to choice of bin width than is the traditional resolution component. The difference between the generalized resolution term and the conventional resolution term also quantifies how forecast skill is degraded when issuing categorized probabilities to users. The ideas are illustrated using an example of multimodel ensemble seasonal forecasts of equatorial sea surface temperatures.

Corresponding author address: Dr. David B. Stephenson, School of Engineering, Computing, and Mathematics, University of Exeter, North Park Rd., Exeter EX4 4QF, United Kingdom. Email: d.b.stephenson@exeter.ac.uk

Save