• Brier, G. W., 1950: Verification of forecasts expressed in terms of probability. Mon. Wea. Rev., 78 , 13.

  • Dahni, R. R., and Stern H. , 1995: The development of a generalised UNIX version of the Victorian Regional Office’s operational analogue statistics model. BMRC Research Rep. 47, Melbourne, VIC, Australia, 34 pp.

  • Drosdowsky, W., and Chambers L. E. , 2001: Near-global sea surface temperature anomalies as predictors of Australian seasonal rainfall. J. Climate, 14 , 16771687.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fawcett, R. J. B., Jones D. A. , and Beard G. S. , 2005: A verification of publicly issued seasonal forecasts issued by the Australian Bureau of Meteorology: 1998–2003. Aust. Meteor. Mag., 54 , 113.

    • Search Google Scholar
    • Export Citation
  • Hartmann, H. C., Pagano T. C. , Sorooshian S. , and Bales R. , 2002: Confidence builders: Evaluating seasonal climate forecasts from user perspectives. Bull. Amer. Meteor. Soc., 83 , 683698.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jones, D. A., 1998: The prediction of Australian land surface temperatures using near global sea surface temperature patterns. BMRC Research Rep. 70, Melbourne, VIC, Australia, 44 pp.

  • Mason, I. B., 1982: A model for assessment of weather forecasts. Aust. Meteor. Mag., 30 , 291303.

  • Mason, I. B., 2003: Binary events. Forecast Verification: A Practitioner’s Guide, I. T. Jolliffe and D. B. Stephenson, Eds., Wiley, 37–76.

    • Search Google Scholar
    • Export Citation
  • Mason, S. J., and Graham N. E. , 1999: Conditional probabilities, relative operating characteristics, and relative operating levels. Wea. Forecasting, 14 , 713725.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Potts, J. M., Folland C. K. , Jolliffe I. T. , and Sexton D. , 1996: Revised “LEPS” scores for assessing climate model simulations and long-range forecasts. J. Climate, 9 , 3453.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Stern, H., 1980: A system for automated forecasting guidance. Aust. Meteor. Mag., 28 , 141154.

  • Stone, R. C., Hammer G. L. , and Marcussen T. , 1996: Prediction of global rainfall probabilities using phases of the Southern Oscillation index. Nature, 384 , 252255.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2006: Statistical Methods in the Atmospheric Sciences: An Introduction. 2nd ed. Academic Press, 467 pp.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 8 8 8
PDF Downloads 8 8 8

Verification Techniques and Simple Theoretical Forecast Models

View More View Less
  • 1 National Climate Centre, Australian Bureau of Meteorology, Melbourne, Victoria, Australia
Restricted access

Abstract

This paper investigates the performance of some skill measures [e.g., linear error in probability space, (LEPS), relative operating characteristics score (ROCS), Brier scores, and proportion correct rates], commonly used in the validation and verification of seasonal climate forecasts, within the context of some simple theoretical forecast models. The models considered include linear regression and linear discriminant analysis types, where the forecasts are presented in the form of above/below median probabilities and tercile probabilities. Above and below the median categories are also explored within the context of stratified climatology models, while tail categories are explored within the context of the linear regression type. The skill scores for the models are calculated in each case as functions of a parameter that expresses the strength of the relationship between the predictor and predictand. The skill scores investigated are found to exhibit different dependencies on the model parameter, implying that a given skill score value (0.1 say) can imply a range of strengths in the relationship between predictor and predictand, depending on which skill score is being considered. On the other hand, interrelationships between pairs of skill scores are found to be similar across the different types of models, provided model reliability is preserved. The two-category and three-category LEPS skill scores are found to be on approximately the same scale for the linear regression–type model, thereby enabling a direct comparison.

Corresponding author address: Robert Fawcett, National Climate Centre, Australian Bureau of Meteorology, GPO Box 1289, Melbourne, VIC 3001, Australia. Email: r.fawcett@bom.gov.au

Abstract

This paper investigates the performance of some skill measures [e.g., linear error in probability space, (LEPS), relative operating characteristics score (ROCS), Brier scores, and proportion correct rates], commonly used in the validation and verification of seasonal climate forecasts, within the context of some simple theoretical forecast models. The models considered include linear regression and linear discriminant analysis types, where the forecasts are presented in the form of above/below median probabilities and tercile probabilities. Above and below the median categories are also explored within the context of stratified climatology models, while tail categories are explored within the context of the linear regression type. The skill scores for the models are calculated in each case as functions of a parameter that expresses the strength of the relationship between the predictor and predictand. The skill scores investigated are found to exhibit different dependencies on the model parameter, implying that a given skill score value (0.1 say) can imply a range of strengths in the relationship between predictor and predictand, depending on which skill score is being considered. On the other hand, interrelationships between pairs of skill scores are found to be similar across the different types of models, provided model reliability is preserved. The two-category and three-category LEPS skill scores are found to be on approximately the same scale for the linear regression–type model, thereby enabling a direct comparison.

Corresponding author address: Robert Fawcett, National Climate Centre, Australian Bureau of Meteorology, GPO Box 1289, Melbourne, VIC 3001, Australia. Email: r.fawcett@bom.gov.au

Save