• Accadia, C., , Mariani S. , , Casaioli M. , , Lavagnini A. , , and Speranza A. , 2003: Sensitivity of precipitation forecast skill scores to bilinear interpolation and a simple nearest-neighbor average method on high-resolution verification grids. Wea. Forecasting, 18 , 918932.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Brooks, H. E., , and Doswell C. A. III, 1996: A comparison of measures-oriented and distributions-oriented approaches to forecast verification. Wea. Forecasting, 11 , 288303.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Doswell C. A. III, , , Davies-Jones R. , , and Keller D. L. , 1990: On summary measures of skill in rare event forecasting based on contingency tables. Wea. Forecasting, 5 , 576585.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ebert, E. E., , and McBride J. L. , 2000: Verification of precipitation in weather systems: Determination of systematic errors. J. Hydrol., 239 , 179202.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gallus, W. A. J., 2002: Impact of verification grid-box size on warm-season QPF skill measures. Wea. Forecasting, 17 , 12961302.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gandin, L. S., , and Murphy A. , 1992: Equitable skill scores for categorical forecasts. Mon. Wea. Rev., 120 , 361370.

  • Gilbert, G. F., 1884: Finley’s tornado predictions. Amer. Meteor. J., 1 , 166172.

  • Gong, X., , Barnston A. G. , , and Ward M. N. , 2003: The effect of spatial aggregation on the skill of seasonal precipitation forecasts. J. Climate, 16 , 30593071.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., 1999: Hypothesis tests for evaluating numerical precipitation forecasts. Wea. Forecasting, 14 , 155167.

  • Hoffman, R. N., , Liu Z. , , Louis J-F. , , and Grassoti C. , 1995: Distortion representation of forecast errors. Mon. Wea. Rev., 123 , 27582770.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Marzban, C., 1998: Scalar measures of performance in rare event situations. Wea. Forecasting, 13 , 753763.

  • Mason, I., 1989: Dependence of the critical success index on sample climate and threshold probability. Aust. Meteor. Mag., 37 , 7581.

  • Mass, C. F., , Ovens D. , , Westrick K. , , and Colle B. A. , 2002: Does increasing horizontal resolution produce more skillful forecasts? Bull. Amer. Meteor. Soc., 83 , 407430.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mesinger, F., 1996: Improvements in quantitative precipitation forecasts with the Eta regional model at the National Centers for Environmental Prediction: The 48-km upgrade. Bull. Amer. Meteor. Soc., 77 , 26372650.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mesinger, F., , and Brill K. , 2004: Bias normalized precipitation scores. Preprints, 17th Conf. on Probability and Statistics, Seattle, WA, Amer. Meteor. Soc., CD-ROM, J12.6.

  • Murphy, A. H., 1991: Forecast verification: Its complexity and dimensionality. Mon. Wea. Rev., 119 , 15901601.

  • Murphy, A. H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8 , 281293.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1996: General decompositions of MSE-based skill scores: Measures of some basic aspects of forecast quality. Mon. Wea. Rev., 124 , 23532369.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., , and Winkler R. L. , 1987: A general framework for forecast verification. Mon. Wea. Rev., 115 , 13301338.

  • Nachamkin, J. E., 2004: Mesoscale verification using meteorological composites. Mon. Wea. Rev., 132 , 941955.

  • Peirce, C. S., 1884: The numerical measure of the success of predictions. Science, 4 , 453454.

  • Richardson, D. S., 2000: Skill and relative economic value of the ECMWF Ensemble Prediction System. Quart. J. Roy. Meteor. Soc., 126 , 649667.

  • Rogers, E., , Black T. , , Ferrier B. , , Lin Y. , , Parrish D. , , and DiMego G. , 2001: Changes to the NCEP Meso Eta Analysis and Forecast System: Increase in resolution, new cloud microphysics, modified precipitation assimilation, modified 3DVAR analysis. NWS Tech. Procedures Bull. 488, 15 pp. [Available online at http://www.emc.ncep.noaa.gov/mmb/mmbpll/eta12tpb/ or from Office of Meteorology, National Weather Service, 1325 East–West Highway, Silver Spring, MD 20910.].

  • Schaefer, J. T., 1990: The critical success index as an indicator of warning skill. Wea. Forecasting, 5 , 570575.

  • Stephenson, D. B., 2000: Use of the “odds ratio” for diagnosing forecast skill. Wea. Forecasting, 15 , 221232.

  • Thornes, J. E., , and Stephenson D. B. , 2001: How to judge the quality and value of weather forecast products. Meteor. Appl., 8 , 307314.

  • Tustison, B., , Harris D. , , and Foufoula-Georgiou E. , 2001: Scale issues in verification of precipitation forecasts. J. Geophys. Res., 106 , 1177511784.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Weisstein, E. W., cited. 2005: Circle–circle intersection. Mathworld, Wolfram. [Available online at http://mathworld.wolfram.com/Circle-CircleIntersection.html.].

  • Weygandt, S. S., , Loughe A. F. , , Benjamin S. G. , , and Mahoney J. L. , 2004: Scale sensitivities in model precipitation skill scores during IHOP. Preprints, 22d Conf. on Severe Local Storms, Hyannis, MA, Amer. Meteor. Soc., CD-ROM, 16A.8.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 183 183 14
PDF Downloads 152 152 14

Sensitivity of Several Performance Measures to Displacement Error, Bias, and Event Frequency

View More View Less
  • 1 Cooperative Institute for Mesoscale Meteorological Studies, University of Oklahoma, and NOAA/OAR/National Severe Storms Laboratory, Norman, Oklahoma
© Get Permissions Rent on DeepDyve
Restricted access

Abstract

The sensitivity of various accuracy measures to displacement error, bias, and event frequency is analyzed for a simple hypothetical forecasting situation. Each measure is found to be sensitive to displacement error and bias, but probability of detection and threat score do not change as a function of event frequency. On the other hand, equitable threat score, true skill statistic, and odds ratio skill score behaved differently with changing event frequency. A newly devised measure, here called the bias-adjusted threat score, does not change with varying event frequency and is relatively insensitive to bias. Numerous plots are presented to allow users of these accuracy measures to make quantitative estimates of sensitivities that are relevant to their particular application.

Corresponding author address: Michael E. Baldwin, OU/CIMMS, 1313 Halley Circle, Norman, OK 73069. Email: mbaldwin@ou.edu

Abstract

The sensitivity of various accuracy measures to displacement error, bias, and event frequency is analyzed for a simple hypothetical forecasting situation. Each measure is found to be sensitive to displacement error and bias, but probability of detection and threat score do not change as a function of event frequency. On the other hand, equitable threat score, true skill statistic, and odds ratio skill score behaved differently with changing event frequency. A newly devised measure, here called the bias-adjusted threat score, does not change with varying event frequency and is relatively insensitive to bias. Numerous plots are presented to allow users of these accuracy measures to make quantitative estimates of sensitivities that are relevant to their particular application.

Corresponding author address: Michael E. Baldwin, OU/CIMMS, 1313 Halley Circle, Norman, OK 73069. Email: mbaldwin@ou.edu

Save