Visualizing Multiple Measures of Forecast Quality

Paul J. Roebber Atmospheric Science Group, Department of Mathematical Sciences, University of Wisconsin—Milwaukee, Milwaukee, Wisconsin

Search for other papers by Paul J. Roebber in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

A method for visually representing multiple measures of dichotomous (yes–no) forecast quality (probability of detection, false alarm ratio, bias, and critical success index) in a single diagram is presented. Illustration of the method is provided using performance statistics from two previously published forecast verification studies (snowfall density and convective initiation) and a verification of several new forecast datasets: Storm Prediction Center forecasts of severe storms (nontornadic and tornadic), Hydrometeorological Prediction Center forecasts of heavy precipitation (greater than 12.5 mm in a 6-h period), National Weather Service Forecast Office terminal aviation forecasts (ceiling and visibility), and medium-range ensemble forecasts of 500-hPa height anomalies. The use of such verification metrics in concert with more detailed investigations to advance forecasting is briefly discussed.

Corresponding author address: Prof. Paul J. Roebber, Atmospheric Science Group, Dept. of Mathematical Sciences, University of Wisconsin—Milwaukee, 3200 North Cramer Ave., Milwaukee, WI 53211. Email: roebber@uwm.edu

Abstract

A method for visually representing multiple measures of dichotomous (yes–no) forecast quality (probability of detection, false alarm ratio, bias, and critical success index) in a single diagram is presented. Illustration of the method is provided using performance statistics from two previously published forecast verification studies (snowfall density and convective initiation) and a verification of several new forecast datasets: Storm Prediction Center forecasts of severe storms (nontornadic and tornadic), Hydrometeorological Prediction Center forecasts of heavy precipitation (greater than 12.5 mm in a 6-h period), National Weather Service Forecast Office terminal aviation forecasts (ceiling and visibility), and medium-range ensemble forecasts of 500-hPa height anomalies. The use of such verification metrics in concert with more detailed investigations to advance forecasting is briefly discussed.

Corresponding author address: Prof. Paul J. Roebber, Atmospheric Science Group, Dept. of Mathematical Sciences, University of Wisconsin—Milwaukee, 3200 North Cramer Ave., Milwaukee, WI 53211. Email: roebber@uwm.edu

Save
  • Brooks, H. E., and Doswell C. A. III, 1996: A comparison of measures-oriented and distributions-oriented approaches to forecast verification. Wea. Forecasting, 11 , 288303.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Doswell C. A. III, , 2004: Weather forecasting by humans—Heuristics and decision making. Wea. Forecasting, 19 , 11151126.

  • Doswell C. A. III, , Davies-Jones R. , and Keller D. L. , 1990: On summary measures of skill in rare event forecasting based on contingency tables. Wea. Forecasting, 5 , 576585.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fowle, M. A., and Roebber P. J. , 2003: Short-range (0–48 h) numerical prediction of convective occurrence, mode, and location. Wea. Forecasting, 18 , 782794.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gribbin, J. R., and Gribbin M. , 2004: FitzRoy: The Remarkable Story of Darwin’s Captain and the Invention of the Weather Forecast. Yale University Press, 336 pp.

    • Search Google Scholar
    • Export Citation
  • Hales, J. E., 1988: Improving the watch/warning system through the use of significant event data. Preprints, 15th Conf. on Severe Local Storms, Baltimore, MD, Amer. Meteor. Soc., 165–168.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., Whitaker J. S. , and Mullen S. L. , 2006: Reforecasts: An important dataset for improving weather predictions. Bull. Amer. Meteor. Soc., 87 , 3346.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hammond, K. R., 1996: Human Judgment and Social Policy. Oxford University Press, 436 pp.

  • Jolliffe, I. T., and Stephenson D. B. , Eds. 2003: Forecast Verification. A Practitione’′s Guide in Atmospheric Science. John Wiley and Sons, 240 pp.

    • Search Google Scholar
    • Export Citation
  • Katz, R. W., and Murphy A. H. , Eds. 1997: Economic Value of Weather and Climate Forecasts. Cambridge University Press, 222 pp.

  • Lambert, S. J., and Boer G. J. , 2001: CMIP1 evaluation and intercomparison of coupled climate models. Climate Dyn., 17 , 83106.

  • Morss, R. E., Lazo J. K. , Brown B. G. , Brooks H. E. , Ganderton P. T. , and Mills B. N. , 2008: Societal and economic research and applications for weather forecasts: Priorities for the North American THORPEX program. Bull. Amer. Meteor. Soc., 89 , 335346.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8 , 281293.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., and Winkler R. L. , 1987: A general framework for forecast verification. Mon. Wea. Rev., 115 , 13301338.

  • Murphy, A. H., Brown B. G. , and Chen Y-S. , 1989: Diagnostic verification of temperature forecasts. Wea. Forecasting, 4 , 485501.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ramage, C. S., 1993: Forecasting in meteorology. Bull. Amer. Meteor. Soc., 74 , 18631871.

  • Roebber, P. J., Bruening S. L. , Schultz D. M. , and Cortinas J. V. Jr., 2003: Improving snowfall forecasting by diagnosing snow density. Wea. Forecasting, 18 , 264287.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Roebber, P. J., Butt M. R. , Reinke S. J. , and Grafenauer T. J. , 2007: Real-time forecasting of snowfall using a neural network. Wea. Forecasting, 22 , 676684.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Schaefer, J. T., 1990: The critical success index as an indicator of warning skill. Wea. Forecasting, 5 , 570575.

  • Stephenson, D. B., 2000: Use of the “odds ratio” for diagnosing forecast skill. Wea. Forecasting, 15 , 221232.

  • Taylor, K. E., 2001: Summarizing multiple aspects of model performance in a single diagram. J. Geophys. Res., 106 , 71837192.

  • Verbout, S. M., Brooks H. E. , Leslie L. M. , and Schultz D. M. , 2006: Evolution of the U.S. tornado database: 1954–2004. Wea. Forecasting, 21 , 8693.

    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2006: Statistical Methods in the Atmospheric Sciences. 2nd ed. Academic Press, 627 pp.

  • Willis, E. P., and Hooke W. H. , 2006: Cleveland Abbe and American meteorology, 1871–1901. Bull. Amer. Meteor. Soc., 87 , 315326.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 5155 1779 138
PDF Downloads 3986 1158 78