• Barnston, A. G., 1992: Correspondence among the correlations, RMSE, and Heidke forecast verification measures; refinement of the Heidke score. Wea. Forecasting,7, 699–709.

    • Crossref
    • Export Citation
  • Bettge, A. G., D. P. Baumhefner, and R. M. Chervin, 1981: On the verification of seasonal climate forecasts. Bull. Amer. Meteor. Soc.,62, 1654–1665.

    • Crossref
    • Export Citation
  • Casey, T. M., 1995: Optimal linear combination of seasonal forecasts. Aust. Meteor. Mag.,44, 219–224.

  • ——, 1998: Assessment of a seasonal forecast model. Aust. Meteor. Mag.,47, 103–113.

  • Doswell, C. A., III, and J. A. Flueck, 1989: Forecasting and verifying in a field research project: DOPLIGHT 87. Wea. Forecasting,4, 97–109.

    • Crossref
    • Export Citation
  • Egan, J. P., 1975: Signal Detection Theory and ROC Analysis. Academic Press, 277 pp.

  • Fraedrich, K., and L. M. Leslie, 1987: Combining predictive schemes in short-term forecasting. Mon. Wea. Rev.,115, 1640–1644.

    • Crossref
    • Export Citation
  • Gandin, L. S., and A. H. Murphy, 1992: Equitable skill scores for categorical forecasts. Mon. Wea. Rev.,120, 361–370.

    • Crossref
    • Export Citation
  • Green, D. M., and J. A. Swets, 1966: Signal Detection Theory and Psychophysics. Wiley, 455 pp. (Reprinted by Robert E. Krieger Publishing Co., 1974.).

  • Hammer, G. L., D. P. Holzworth, and R. Stone, 1996: The value of skill in seasonal climate forecasting to wheat crop management in a region with high climatic variability. Aust. J. Agric. Res.,47, 717–737.

    • Crossref
    • Export Citation
  • Hanssen, A. W., and W. J. A. Kuipers, 1965: On the relationship between the frequency of rain and various meteorological parameters. Meded. Verh.,81, 3–15.

  • Heidke, P., 1926: Berechnung des erfolges und der gute der windstarkevorhersagen im sturmwarnungsdientst. Geogr. Ann.,8, 301–349.

  • Huntrieser, H., H. H. Schiesser, W. Schmid, and A. Waldvogel, 1997:Comparison of traditional and new developed thunderstorm indices for Switzerland. Wea. Forecasting,12, 108–125.

    • Crossref
    • Export Citation
  • Lee, R. R., and J. E. Passner, 1993: The development and verification of TIPS: An expert system to forecast thunderstorm occurrence. Wea. Forecasting,8, 271–280.

  • Mason, I., 1979: On reducing probability forecasts to yes/no forecasts. Mon. Wea. Rev.,107, 207–211.

    • Crossref
    • Export Citation
  • ——, 1982: A model for assessment of weather forecasts. Aust. Meteor. Mag.,30, 291–303.

  • Murphy, A. H., 1977: The value of climatological categorical and probabilistic forecasts in the cost–loss ratio situation. Mon. Wea. Rev.,105, 803–816.

    • Crossref
    • Export Citation
  • ——, 1988: Skill scores based on the mean square error and their relationships to the correlation coefficient. Mon. Wea. Rev.,116, 2417–2424.

    • Crossref
    • Export Citation
  • ——, 1991: Forecast verification: Its complexity and dimensionality. Mon. Wea. Rev.,119, 1590–1601.

    • Crossref
    • Export Citation
  • Potts, J. M., C. K. Folland, I. T. Jolliffe, and D. Sexton, 1996: Revised“LEPS” scores for assessing climate model simulations and long-range forecasts. J. Climate,9, 34–53.

    • Crossref
    • Export Citation
  • Schaefer, J. T., 1990: The critical success index as an indicator of warning skill. Wea. Forecasting,5, 570–575.

    • Crossref
    • Export Citation
  • Stanski, H. R., L. J. Wilson, and W. R. Burrows, 1989: Survey of common verification methods in meteorology. World Weather Watch Tech. Rep. 8, WMO/TD No.-358, 114 pp.

  • Swets, J. A., 1973: The relative operating characteristic in psychology. Science,182, 990–1000.

    • Crossref
    • Export Citation
  • Ward, M. N., and C. K. Folland, 1991: Prediction of seasonal rainfall in the north Nordeste of Brazil using eigenvectors of sea-surface temperature. Int. J. Climatol.,11, 711–743.

    • Crossref
    • Export Citation
  • Wilks, D. S., 1995: Statistical Methods in the Atmospheric Sciences:An Introduction. Academic Press, 467 pp.

  • Woodcock, F., 1976: The evaluation of yes/no forecasts for scientific and administrative purpose. Mon. Wea. Rev.,104, 1209–1214.

    • Crossref
    • Export Citation
  • Zhang, X.-G., and T. M. Casey, 1992: Long-term variation in the Southern Oscillation and relationships with Australian rainfall. Aust. Meteor. Mag.,40, 211–225.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 363 105 6
PDF Downloads 185 96 6

Verification of Categorical Probability Forecasts

View More View Less
  • 1 Bureau of Meteorology Research Centre, Melbourne, Victoria, Australia
Restricted access

Abstract

This paper compares a number of probabilistic weather forecasting verification approaches. Forecasting skill scores from linear error in probability space and relative operating characteristics are compared with results from an alternative approach that first transforms probabilistic forecasts to yes/no form and then assesses the model forecasting skill. This approach requires a certain departure between the categorical probability from forecast models and its random expectation. The classical contingency table is revised to reflect the “nonapplicable” forecasts in the skill assessment.

The authors present a verification of an Australian seasonal rainfall forecast model hindcasts for the winter and summer seasons over the period from 1900 to 1995. Overall skill scores from different approaches demonstrate similar features. However there are advantages and disadvantages in each of those approaches. Using more than one skill assessment scheme is necessary and is also of practical value in the evaluation of the model forecasts and their applications.

Corresponding author address: Dr. H. Zhang, Bureau of Meteorology Research Centre, GPO Box 1289K, VIC 3001, Australia.

Email: h.zhang@bom.gov.au

Abstract

This paper compares a number of probabilistic weather forecasting verification approaches. Forecasting skill scores from linear error in probability space and relative operating characteristics are compared with results from an alternative approach that first transforms probabilistic forecasts to yes/no form and then assesses the model forecasting skill. This approach requires a certain departure between the categorical probability from forecast models and its random expectation. The classical contingency table is revised to reflect the “nonapplicable” forecasts in the skill assessment.

The authors present a verification of an Australian seasonal rainfall forecast model hindcasts for the winter and summer seasons over the period from 1900 to 1995. Overall skill scores from different approaches demonstrate similar features. However there are advantages and disadvantages in each of those approaches. Using more than one skill assessment scheme is necessary and is also of practical value in the evaluation of the model forecasts and their applications.

Corresponding author address: Dr. H. Zhang, Bureau of Meteorology Research Centre, GPO Box 1289K, VIC 3001, Australia.

Email: h.zhang@bom.gov.au

Save