This research was performed while the first author held a National Research Council Research Associateship Award at the National Severe Storms Laboratory. The impetus for the origins of the research date back to conversations the second author had with Allan Murphy in the mid-1990s. The constructive comments and suggestions made by the three anonymous reviewers helped improve the manuscript.
Brooks, H. E., Kay M. , and Hart J. A. , 1998: Objective limits on forecasting skill of rare events. Preprints, 19th Conf. on Severe Local Storms, Minneapolis, MN, Amer. Meteor. Soc., 552–555.
Brown, B. G., Thompson G. , Bruintjes R. T. , Bullock R. , and Kane T. , 1997: Intercomparison of in-flight icing algorithms. Part II: Statistical verification results. Wea. Forecasting, 12, 890–914.
Davis, C., and Carr F. , 2000: Summary of the 1998 workshop on mesoscale model verification. Bull. Amer. Meteor. Soc., 81, 809–819.
Doswell, C. A. III, Davies-Jones R. , and Keller D. L. , 1990: On summary measures of skill in rare event forecasting based on contingency tables. Wea. Forecasting, 5, 576–585.
Ebert, E. E., Wilson L. J. , Brown B. G. , Nurmi P. , Brooks H. E. , Bally J. , and Jaeneke M. , 2004: Verification of nowcasts from the WWRP Sydney 2000 Forecast Demonstration Project. Wea. Forecasting, 19, 73–96.
Hitchens, N. M., and Brooks H. E. , 2012: Evaluation of the Storm Prediction Center’s day 1 convective outlooks. Wea. Forecasting, 27, 1580–1585.
Murphy, A. H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8, 281–293.
Silverman, B. W., 1986: Density Estimation for Statistics and Data Analysis. Chapman and Hall, 175 pp.
Note that there may be a large number of metrics used to describe accuracy and skill depending upon the particular forecasting situation.
The term “practically perfect” draws on the usage “practical zero,” in which a person offering a judgment on the probability of a very unlikely event may describe it as zero, even though they do not think the probability is exactly zero. The probability is sufficiently low to be regarded as zero in typical applications. Similarly, the “practically” in practically perfect does not mean that the forecast is almost perfect, but that the forecast is as good as could be expected in typical practice.
In this study 365-day running means are computed by constructing a 2 × 2 table that sums all 365 forecasts centered on each day. In the case of maximum CSI from PP forecasts, the 2 × 2 table associated with each day’s maximum CSI value is used in the construction of the table for the 365-day period.
The term “all forecast days” includes days when an outlook was issued and no reports were recorded (“false alarm”), and days when no outlook was issued but reports were recorded (“missed events”). In the latter scenario the area of the upper bound must be at least as large as the smallest regular outlook area (~64 000 km2).