• Brier, G. W., 1950: Verification of forecasts expressed in terms of probability. Mon. Wea. Rev, 78 , 13.

  • Epstein, E. S., 1969: A scoring system for probability forecasts of ranked categories. J. Appl. Meteor, 8 , 985987.

  • Gandin, L. S., , and A. H. Murphy, 1992: Equitable scores for categorical forecasts. Mon. Wea. Rev, 120 , 361370.

  • Gerrity, J. P., 1992: A note on Gandin and Murphy's equitable skill score. Mon. Wea. Rev, 120 , 27072712.

  • Hsu, W-R., , and A. H. Murphy, 1986: The attributes diagram: A geometrical framework for assessing the quality of probability forecasts. Int. J. Forecasting, 2 , 285293.

    • Search Google Scholar
    • Export Citation
  • Jolliffe, I. T., , and D. B. Stephenson, 2003: Introduction. Forecast Verification: A Practitioner's Guide in Atmospheric Science, I. T. Jolliffe and D. B. Stephenson, Eds., Wiley, 1–12.

    • Search Google Scholar
    • Export Citation
  • Kumar, A., , A. G. Barnston, , and M. P. Hoerling, 2001: Seasonal predictions, probabilistic verifications, and ensemble size. J. Climate, 14 , 16711676.

    • Search Google Scholar
    • Export Citation
  • Livezey, R. E., 2003: Categorical events. Forecast Verification: A Practitioner's Guide in Atmospheric Science, I. T. Jolliffe and D. B. Stephenson, Eds., Wiley, 77–96.

    • Search Google Scholar
    • Export Citation
  • Mason, I. T., 2003: Binary events. Forecast Verification: A Practitioner's Guide in Atmospheric Science, I. T. Jolliffe and D. B. Stephenson, Eds., Wiley, 37–76.

    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1966: A note on the use of probabilistic predictions and the probability score in the cost-loss ratio decision situation. J. Appl. Meteor, 5 , 534537.

    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1969: On the “ranked probability score.”. J. Appl. Meteor, 8 , 988989.

  • Murphy, A. H., 1971: A note on the ranked probability score. J. Appl. Meteor, 10 , 155156.

  • Murphy, A. H., 1973: A new vector partition of the probability score. J. Appl. Meteor, 12 , 595600.

  • Murphy, A. H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8 , 281293.

    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., , and E. S. Epstein, 1967: A note on probability forecasts and “hedging.”. J. Appl. Meteor, 6 , 10021004.

  • Murphy, A. H., , and R. L. Winkler, 1992: Diagnostic verification of probability forecasts. Int. J. Forecasting, 7 , 435455.

  • Potts, J. M., 2003: Basic concepts. Forecast Verification: A Practitioner's Guide in Atmospheric Science, I. T. Jolliffe and D. B. Stephenson, Eds., Wiley, 13–36.

    • Search Google Scholar
    • Export Citation
  • Potts, J. M., , C. K. Folland, , I. T. Jolliffe, , and D. Sexton, 1996: Revised “LEPS” scores for assessing climate model simulations and long-range forecasts. J. Climate, 9 , 3453.

    • Search Google Scholar
    • Export Citation
  • Toth, Z., , O. Talagrand, , G. Candille, , and Y. Zhu, 2003: Probability and ensemble forecasts. Forecast Verification: A Practitioner's Guide in Atmospheric Science, I. T. Jolliffe and D. B. Stephenson, Eds., Wiley, 137–163.

    • Search Google Scholar
    • Export Citation
  • Ward, N. M., , and C. K. Folland, 1991: Prediction of seasonal rainfall in the north Nordeste of Brazil using eigenvectors of sea surface temperatures. Int. J. Climatol, 11 , 711743.

    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 1995: Statistical Methods in the Atmospheric Sciences. Academic Press, 467 pp.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 140 140 33
PDF Downloads 212 212 43

On Using “Climatology” as a Reference Strategy in the Brier and Ranked Probability Skill Scores

View More View Less
  • 1 International Research Institute for Climate Prediction, Columbia University, Palisades, New York
© Get Permissions
Restricted access

Abstract

The Brier and ranked probability skill scores are widely used as skill metrics of probabilistic forecasts of weather and climate. As skill scores, they compare the extent to which a forecast strategy outperforms a (usually simpler) reference forecast strategy. The most widely used reference strategy is that of “climatology,” in which the climatological probability (or probabilities in the case of the ranked probability skill score) of the forecast variable is issued perpetually. The Brier and ranked probability skill scores are often considered harsh standards. It is shown that the scores are harsh because the expected value of these skill scores is less than 0 if nonclimatological forecast probabilities are issued. As a result, negative skill scores can often hide useful information content in the forecasts. An alternative formulation of the skill scores based on a reference strategy in which the outcome is independent of the forecast is equivalent to using randomly assigned probabilities but is not strictly proper. Nevertheless, positive values of the Brier skill score with random guessing as a strategy correspond to positive-sloping reliability curves, which is intuitively appealing because of the implication that the conditional probability of the forecast event increases as the forecast probability increases.

Corresponding author address: Dr. Simon J. Mason, International Research Institute for Climate Prediction, 61 Route 9W, P.O. Box 1000, Columbia University, Palisades, NY 10964-8000. Email: simon@iri.columbia.edu

Abstract

The Brier and ranked probability skill scores are widely used as skill metrics of probabilistic forecasts of weather and climate. As skill scores, they compare the extent to which a forecast strategy outperforms a (usually simpler) reference forecast strategy. The most widely used reference strategy is that of “climatology,” in which the climatological probability (or probabilities in the case of the ranked probability skill score) of the forecast variable is issued perpetually. The Brier and ranked probability skill scores are often considered harsh standards. It is shown that the scores are harsh because the expected value of these skill scores is less than 0 if nonclimatological forecast probabilities are issued. As a result, negative skill scores can often hide useful information content in the forecasts. An alternative formulation of the skill scores based on a reference strategy in which the outcome is independent of the forecast is equivalent to using randomly assigned probabilities but is not strictly proper. Nevertheless, positive values of the Brier skill score with random guessing as a strategy correspond to positive-sloping reliability curves, which is intuitively appealing because of the implication that the conditional probability of the forecast event increases as the forecast probability increases.

Corresponding author address: Dr. Simon J. Mason, International Research Institute for Climate Prediction, 61 Route 9W, P.O. Box 1000, Columbia University, Palisades, NY 10964-8000. Email: simon@iri.columbia.edu

Save