• Bowman, A. W., , and Azzalini A. , 1997: Applied Smoothing Techniques for Data Analysis: The Kernel Approach with S-Plus Illustrations. Oxford University Press, 208 pp.

    • Search Google Scholar
    • Export Citation
  • Brier, G. W., 1950: Verification of forecasts expressed in terms of probability. Mon. Wea. Rev., 78 , 13.

  • Brooks, H. E., , and Doswell C. A. , 1996: A comparison of measures-oriented and distributions-oriented approaches to forecast verification. Wea. Forecasting, 11 , 288303.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Clemen, R. T., , and Winkler R. L. , 1987: Calibrating and combining precipitation probability forecasts. Probability and Bayesian Statistics, R. Viertl, Ed., Plenum Press, 97–110.

    • Search Google Scholar
    • Export Citation
  • Connor, G. J., , and Woodcock F. , 2000: The application of synoptic stratification to precipitation forecasting in the trade wind regime. Wea. Forecasting, 15 , 276297.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cox, D. R., , and Snell E. J. , 1989: Analysis of Binary Data. 2d ed. Chapman and Hall, 236 pp.

  • Day, G. N., 1985: Extended streamflow forecasting using NWSRFS. J. Water Resour. Plan. Manage., 111 , 157170.

  • Hamill, T. M., 1999: Hypothesis tests for evaluating numerical precipitation forecasts. Wea. Forecasting, 14 , 155167.

  • Hashino, T., , Bradley A. A. , , and Schwartz S. S. , 2002: Verification of probabilistic streamflow forecasts. IIHR Rep. 427, IIHR—Hydroscience and Engineering, Iowa City, IA.

    • Search Google Scholar
    • Export Citation
  • Hosmer D. W. Jr., , , and Lemeshow S. , 1989: Applied Logistic Regression. John Wiley and Sons, 307 pp.

  • Kane, T. L., , and Brown B. G. , 2000: Confidence intervals for some verification measures—A survey of several methods. Preprints, 15th Conf. on Probability and Statistics in the Atmospheric Sciences, Asheville, NC, Amer. Meteor. Soc., 46–49.

    • Search Google Scholar
    • Export Citation
  • Katz, R. W., , Murphy A. H. , , and Winkler R. L. , 1982: Assessing the value of frost forecast to orchardists: A dynamic decision-making approach. J. Appl. Meteor., 21 , 518531.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Krzysztofowicz, R., , and Long D. , 1991: Beta likelihood models of probabilistic forecasts. Int. J. Forecasting, 7 , 4755.

  • Murphy, A. H., 1991: Forecast verification: Its complexity and dimensionality. Mon. Wea. Rev., 119 , 15901601.

  • Murphy, A. H., 1997: Forecast verification. Economic Value of Weather and Climate Forecasts, R. W. Katz and A. H. Murphy, Eds., Cambridge University Press, 19–74.

    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., , and Winkler R. L. , 1987: A general framework for forecast verification. Mon. Wea. Rev., 115 , 13301338.

  • Murphy, A. H., , and Winkler R. L. , 1992: Diagnostic verification of probability forecasts. Int. J. Forecasting, 7 , 435455.

  • Murphy, A. H., , and Wilks D. S. , 1998: A case study of the use of statistical models in forecast verification: Precipitation probability forecasts. Wea. Forecasting, 13 , 795810.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Nottingham, Q. J., , Birch J. B. , , and Bodt B. A. , 2000: Local logistic regression: An application to Army penetration data. J. Stat. Comput. Simulat., 66 (1) 3550.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rajagopalan, B., , and Lall U. , 1995: A kernel estimator for discrete distributions. J. Nonparametric Stat., 4 , 409426.

  • Scott, D. W., 1992: Multivariate Density Estimation: Theory, Practice, and Visualization. Wiley, 317 pp.

  • Seaman, R., , Mason I. , , and Woodcock F. , 1996: Confidence intervals for some performance measures of yes–no forecasts. Aust. Meteor. Mag., 45 , 4953.

    • Search Google Scholar
    • Export Citation
  • Smith, J. A., , Day G. N. , , and Kane M. D. , 1992: Nonparametric framework for long-range streamflow forecasting. J. Water Resour. Plan. Manage., 118 , 8292.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Stephenson, D. B., 2000: Use of the “odds ratio” for diagnosing forecast skill. Wea. Forecasting, 15 , 221232.

  • Thornes, J. E., , and Stephenson D. B. , 2001: How to judge the quality and value of weather forecast products. Meteor. Appl., 8 , 307314.

  • Titterington, D. M., 1980: A comparative study of kernel-based density estimates for categorical data. Technometrics, 22 , 259268.

  • Wang, M-C., , and Van Ryzin J. , 1981: A class of smooth estimators for discrete distribution. Biometrika, 68 , 301309.

  • Wilks, D. S., 1991: Representing serial correlation of meteorological events and forecasts in dynamic decision analytic model. Mon. Wea. Rev., 119 , 16401662.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 1995: Statistical Methods in the Atmospheric Sciences. Academic Press, 467 pp.

  • Wilks, D. S., 2000: Diagnostic verification of the Climate Prediction Center Long-Lead Outlooks, 1995–98. J. Climate, 13 , 23892403.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., , and Murphy A. H. , 1986: A decision-analytic study of the joint value of seasonal precipitation and temperature forecasts in a choice-of-crop problem. Atmos.–Ocean, 24 , 353368.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., , and Shen K. W. , 1991: Threshold relative-humidity duration forecasts for plant-disease prediction. J. Appl. Meteor., 30 , 463477.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., , Pitt R. E. , , and Fick G. W. , 1993: Modeling optimal alfalfa harvest scheduling using short-range weather forecasts. Agric. Syst., 42 , 277305.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Woodcock, F., 1976: The evaluation of yes/no forecasts for scientific and administrative purposes. Mon. Wea. Rev., 104 , 12091214.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zhang, S., , Karunamuni R. J. , , and Jones M. C. , 1999: An improved estimator of the density function at the boundary. J. Amer. Stat. Assoc., 94 , 12311241.

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 172 172 11
PDF Downloads 29 29 7

Distributions-Oriented Verification of Probability Forecasts for Small Data Samples

View More View Less
  • 1 IIHR—Hydroscience and Engineering, and Department of Civil and Environmental Engineering, University of Iowa, Iowa City, Iowa
  • | 2 University of North Carolina Water Resources Research Institute, North Carolina State University, Raleigh, North Carolina
© Get Permissions
Restricted access

Abstract

The distributions-oriented approach to forecast verification uses an estimate of the joint distribution of forecasts and observations to evaluate forecast quality. However, small verification data samples can produce unreliable estimates of forecast quality due to sampling variability and biases. In this paper, new techniques for verification of probability forecasts of dichotomous events are presented. For forecasts of this type, simplified expressions for forecast quality measures can be derived from the joint distribution. Although traditional approaches assume that forecasts are discrete variables, the simplified expressions apply to either discrete or continuous forecasts. With the derived expressions, most of the forecast quality measures can be estimated analytically using sample moments of forecasts and observations from the verification data sample. Other measures require a statistical modeling approach for estimation. Results from Monte Carlo experiments for two forecasting examples show that the statistical modeling approach can significantly improve estimates of these measures in many situations. The improvement is achieved mostly by reducing the bias of forecast quality estimates and, for very small sample sizes, by slightly reducing the sampling variability. The statistical modeling techniques are most useful when the verification data sample is small (a few hundred forecast–observation pairs or less), and for verification of rare events, where the sampling variability of forecast quality measures is inherently large.

Corresponding author address: Allen Bradley, IIHR—Hydroscience and Engineering, University of Iowa, Iowa City, IA 52242. Email: allen-bradley@uiowa.edu

Abstract

The distributions-oriented approach to forecast verification uses an estimate of the joint distribution of forecasts and observations to evaluate forecast quality. However, small verification data samples can produce unreliable estimates of forecast quality due to sampling variability and biases. In this paper, new techniques for verification of probability forecasts of dichotomous events are presented. For forecasts of this type, simplified expressions for forecast quality measures can be derived from the joint distribution. Although traditional approaches assume that forecasts are discrete variables, the simplified expressions apply to either discrete or continuous forecasts. With the derived expressions, most of the forecast quality measures can be estimated analytically using sample moments of forecasts and observations from the verification data sample. Other measures require a statistical modeling approach for estimation. Results from Monte Carlo experiments for two forecasting examples show that the statistical modeling approach can significantly improve estimates of these measures in many situations. The improvement is achieved mostly by reducing the bias of forecast quality estimates and, for very small sample sizes, by slightly reducing the sampling variability. The statistical modeling techniques are most useful when the verification data sample is small (a few hundred forecast–observation pairs or less), and for verification of rare events, where the sampling variability of forecast quality measures is inherently large.

Corresponding author address: Allen Bradley, IIHR—Hydroscience and Engineering, University of Iowa, Iowa City, IA 52242. Email: allen-bradley@uiowa.edu

Save