• Brooks, H. E., Kay M. , and Hart J. A. , 1998: Objective limits on forecasting skill of rare events. Preprints, 19th Conf. on Severe Local Storms, Minneapolis, MN, Amer. Meteor. Soc., 552–555.

  • Brown, B. G., Thompson G. , Bruintjes R. T. , Bullock R. , and Kane T. , 1997: Intercomparison of in-flight icing algorithms. Part II: Statistical verification results. Wea. Forecasting, 12, 890914.

    • Search Google Scholar
    • Export Citation
  • Davis, C., and Carr F. , 2000: Summary of the 1998 workshop on mesoscale model verification. Bull. Amer. Meteor. Soc., 81, 809819.

  • Doswell, C. A. III, Davies-Jones R. , and Keller D. L. , 1990: On summary measures of skill in rare event forecasting based on contingency tables. Wea. Forecasting, 5, 576585.

    • Search Google Scholar
    • Export Citation
  • Ebert, E. E., Wilson L. J. , Brown B. G. , Nurmi P. , Brooks H. E. , Bally J. , and Jaeneke M. , 2004: Verification of nowcasts from the WWRP Sydney 2000 Forecast Demonstration Project. Wea. Forecasting, 19, 7396.

    • Search Google Scholar
    • Export Citation
  • Gilbert, G. K., 1884: Finley’s tornado predictions. Amer. Meteor. J., 1, 166172.

  • Hitchens, N. M., and Brooks H. E. , 2012: Evaluation of the Storm Prediction Center’s day 1 convective outlooks. Wea. Forecasting, 27, 15801585.

    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8, 281293.

    • Search Google Scholar
    • Export Citation
  • Peirce, C. S., 1884: The numerical measure of the success of predictions. Science, 4, 453454.

  • Roebber, P. J., 2009: Visualizing multiple measures of forecast quality. Wea. Forecasting, 24, 601608.

  • Silverman, B. W., 1986: Density Estimation for Statistics and Data Analysis. Chapman and Hall, 175 pp.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 757 433 26
PDF Downloads 542 302 17

Objective Limits on Forecasting Skill of Rare Events

View More View Less
  • 1 NOAA/OAR/National Severe Storms Laboratory, Norman, Oklahoma
  • | 2 Cooperative Institute for Mesoscale Meteorological Studies, University of Oklahoma, and NOAA/NWS/Storm Prediction Center, Norman, Oklahoma
Restricted access

Abstract

A method for determining baselines of skill for the purpose of the verification of rare-event forecasts is described and examples are presented to illustrate the sensitivity to parameter choices. These “practically perfect” forecasts are designed to resemble a forecast that is consistent with that which a forecaster would make given perfect knowledge of the events beforehand. The Storm Prediction Center’s convective outlook slight risk areas are evaluated over the period from 1973 to 2011 using practically perfect forecasts to define the maximum values of the critical success index that a forecaster could reasonably achieve given the constraints of the forecast, as well as the minimum values of the critical success index that are considered the baseline for skillful forecasts. Based on these upper and lower bounds, the relative skill of convective outlook areas shows little to no skill until the mid-1990s, after which this value increases steadily. The annual frequency of skillful daily forecasts continues to increase from the beginning of the period of study, and the annual cycle shows maxima of the frequency of skillful daily forecasts occurring in May and June.

Current affiliation: Department of Geography, Ball State University, Muncie, Indiana.

Current affiliation: AvMet Applications, Inc., Reston, Virginia.

Corresponding author address: Dr. Nathan M. Hitchens, Department of Geography, Ball State University, Muncie, IN 47306. E-mail: nmhitchens@bsu.edu

Abstract

A method for determining baselines of skill for the purpose of the verification of rare-event forecasts is described and examples are presented to illustrate the sensitivity to parameter choices. These “practically perfect” forecasts are designed to resemble a forecast that is consistent with that which a forecaster would make given perfect knowledge of the events beforehand. The Storm Prediction Center’s convective outlook slight risk areas are evaluated over the period from 1973 to 2011 using practically perfect forecasts to define the maximum values of the critical success index that a forecaster could reasonably achieve given the constraints of the forecast, as well as the minimum values of the critical success index that are considered the baseline for skillful forecasts. Based on these upper and lower bounds, the relative skill of convective outlook areas shows little to no skill until the mid-1990s, after which this value increases steadily. The annual frequency of skillful daily forecasts continues to increase from the beginning of the period of study, and the annual cycle shows maxima of the frequency of skillful daily forecasts occurring in May and June.

Current affiliation: Department of Geography, Ball State University, Muncie, Indiana.

Current affiliation: AvMet Applications, Inc., Reston, Virginia.

Corresponding author address: Dr. Nathan M. Hitchens, Department of Geography, Ball State University, Muncie, IN 47306. E-mail: nmhitchens@bsu.edu
Save