• Barker, T. W., 1991: The relationship between spread and forecast error in extended-range forecasts. J. Climate, 4, 733742.

  • Buizza, R., 1997: Potential forecast skill of ensemble prediction and spread and skill distributions of the ECMWF ensemble prediction system. Mon. Wea. Rev., 125, 99119.

    • Search Google Scholar
    • Export Citation
  • Carroll, R. J., D. Ruppert, and L. A. Stefanski, 1995: Measurement Error in Nonlinear Models. Chapman & Hall, 305 pp.

  • Casella, G., and R. L. Berger, 2001: Statistical Inference. 2nd ed. Duxbery, 660 pp.

  • Dee, D. P., and A. M. DaSilva, 1999: Maximum-likelihood estimation of forecast and observation error covariance parameters. Part I: Methodology. Mon. Wea. Rev., 127, 18221834.

    • Search Google Scholar
    • Export Citation
  • Dorff, M., and J. Gurland, 1961: Estimation of the parameters of a linear functional relations. J. Roy. Stat. Soc., 23B, 160170.

  • Fuller, W. A., 1980: Properties of some estimators for the errors-in-variables model. Ann. Stat., 8, 407422.

  • Grimit, E. P., 2004: Probabilistic mesoscale forecast error prediction using short-range ensembles. Ph.D. dissertation, University of Washington, 146 pp.

    • Search Google Scholar
    • Export Citation
  • Grimit, E. P., and C. F. Mass, 2002: Initial results of a mesoscale short-range ensemble forecasting system over the Pacific Northwest. Wea. Forecasting, 17, 192205.

    • Search Google Scholar
    • Export Citation
  • Grimit, E. P., and C. F. Mass, 2007: Measuring the ensemble spread-error relationship with a probabilistic approach: Stochastic ensemble results. Mon. Wea. Rev., 135, 203221.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., 2001: Interpretation of rank histograms for verifying ensemble forecasts. Mon. Wea. Rev., 129, 550560.

  • Hamill, T. M., and S. J. Colucci, 1998: Evaluation of Eta-RSM ensemble probabilistic precipitation forecasts. Mon. Wea. Rev., 126, 711724.

    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., 1993: Global and local skill forecasts. Mon. Wea. Rev., 121, 18341846.

  • Kalnay, E., and A. Dalcher, 1987: Forecasting forecast skill. Mon. Wea. Rev., 115, 349356.

  • Kolczynski, W. C., Jr., D. R. Stauffer, S. E. Haupt, and A. Deng, 2009: Ensemble variance calibration for representing meteorological uncertainty for atmospheric transport and dispersion modeling. J. Appl. Meteor. Climatol., 48, 20012021.

    • Search Google Scholar
    • Export Citation
  • Leith, C. E., 1974: Theoretical skill of Monte Carlo forecasts. Mon. Wea. Rev., 102, 409418.

  • Lorenz, E. N., 1963: Deterministic nonperiodic flow. J. Atmos. Sci., 20, 130141.

  • Marzban, C., R. Wang, F. Kong, and S. Leyton, 2011: On the effect of correlations on rank histograms: Reliability of temperature and wind speed forecasts from finescale ensemble reforecasts. Mon. Wea. Rev., 139, 295310.

    • Search Google Scholar
    • Export Citation
  • Murphy, J. M., 1988: The impact of ensemble forecasts on predictability. Quart. J. Roy. Meteor. Soc., 114, 463493.

  • National Research Council, 2003: Tracking and Predicting the Atmospheric Dispersion of Hazardous Material Releases: Implications for Homeland Security. The National Academies Press, 114 pp.

    • Search Google Scholar
    • Export Citation
  • Seidman, A. N., 1981: Averaging techniques in long-range weather forecasting. Mon. Wea. Rev., 109, 13671379.

  • Stensrud, D. J., H. E. Brooks, J. Du, M. S. Tracton, and E. Rogers, 1999: Using ensembles for short-range forecasting. Mon. Wea. Rev., 127, 433446.

    • Search Google Scholar
    • Export Citation
  • Wang, X., and C. H. Bishop, 2003: A comparison of breeding and ensemble transform Kalman filter ensemble forecast schemes. J. Atmos. Sci., 60, 11401158.

    • Search Google Scholar
    • Export Citation
  • Wang, X., and C. H. Bishop, 2005: Improvement of ensemble reliability with a new dressing kernel. Quart. J. Roy. Meteor. Soc., 131, 965986.

    • Search Google Scholar
    • Export Citation
  • Wang, X., C. H. Bishop, and S. J. Julier, 2004: Which is better, an ensemble of positive–negative pairs or a centered spherical simplex ensemble? Mon. Wea. Rev., 132, 15901605.

    • Search Google Scholar
    • Export Citation
  • Weisstein, E. W., 2010: The CRC Concise Encyclopedia of Mathematics. 3rd ed. Chapman & Hall/CRC, 1200 pp.

  • Whitaker, J. S., and A. F. Loughe, 1998: The relationship between ensemble spread and ensemble mean skill. Mon. Wea. Rev., 126, 32923302.

    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2006: Statistical Methods in the Atmospheric Sciences. 2nd ed. Academic Press, 627 pp.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 65 65 65
PDF Downloads 0 0 0

Investigation of Ensemble Variance as a Measure of True Forecast Variance

View More View Less
  • 1 Department of Meteorology, The Pennsylvania State University, University Park, Pennsylvania
  • | 2 Department of Statistics, The Pennsylvania State University, University Park, Pennsylvania
  • | 3 Department of Meteorology, The Pennsylvania State University, University Park, Pennsylvania
Restricted access

Abstract

The uncertainty in meteorological predictions is of interest for applications ranging from economic to recreational to public safety. One common method to estimate uncertainty is by using meteorological ensembles. These ensembles provide an easily quantifiable measure of the uncertainty in the forecast in the form of the ensemble variance. However, ensemble variance may not accurately reflect the actual uncertainty, so any measure of uncertainty derived from the ensemble should be calibrated to provide a more reliable estimate of the actual uncertainty in the forecast. A previous study introduced the linear variance calibration (LVC) as a simple method to determine the ensemble variance to error variance relationship and demonstrated this technique on real ensemble data. The LVC parameters, the slopes, and y intercepts, however, are generally different from the ideal values.

This current study uses a stochastic model to examine the LVC in a controlled setting. The stochastic model is capable of simulating underdispersive and overdispersive ensembles as well as perfectly reliable ensembles. Because the underlying relationship is specified, LVC results can be compared to theoretical values of the slope and y intercept. Results indicate that all types of ensembles produce calibration slopes that are smaller than their theoretical values for ensemble sizes less than several hundred members, with corresponding y intercepts greater than their theoretical values. This indicates that all ensembles, even otherwise perfect ensembles, should be calibrated if the ensemble size is less than several hundred.

In addition, it is shown that an adjustment factor can be computed for inadequate ensemble size. This adjustment factor is independent of the stochastic model and is applicable to any linear regression of error variance on ensemble variance. When applied to experiments using the stochastic model, the adjustment produces LVC parameters near their theoretical values for all ensemble sizes. Although the adjustment is unnecessary when applying LVC, it allows for a more accurate assessment of the reliability of ensembles, and a fair comparison of the reliability for differently sized ensembles.

Corresponding author address: Walter C. Kolczynski Jr., Department of Meteorology, The Pennsylvania State University, University Park, PA 16802. E-mail: wck122@psu.edu

Abstract

The uncertainty in meteorological predictions is of interest for applications ranging from economic to recreational to public safety. One common method to estimate uncertainty is by using meteorological ensembles. These ensembles provide an easily quantifiable measure of the uncertainty in the forecast in the form of the ensemble variance. However, ensemble variance may not accurately reflect the actual uncertainty, so any measure of uncertainty derived from the ensemble should be calibrated to provide a more reliable estimate of the actual uncertainty in the forecast. A previous study introduced the linear variance calibration (LVC) as a simple method to determine the ensemble variance to error variance relationship and demonstrated this technique on real ensemble data. The LVC parameters, the slopes, and y intercepts, however, are generally different from the ideal values.

This current study uses a stochastic model to examine the LVC in a controlled setting. The stochastic model is capable of simulating underdispersive and overdispersive ensembles as well as perfectly reliable ensembles. Because the underlying relationship is specified, LVC results can be compared to theoretical values of the slope and y intercept. Results indicate that all types of ensembles produce calibration slopes that are smaller than their theoretical values for ensemble sizes less than several hundred members, with corresponding y intercepts greater than their theoretical values. This indicates that all ensembles, even otherwise perfect ensembles, should be calibrated if the ensemble size is less than several hundred.

In addition, it is shown that an adjustment factor can be computed for inadequate ensemble size. This adjustment factor is independent of the stochastic model and is applicable to any linear regression of error variance on ensemble variance. When applied to experiments using the stochastic model, the adjustment produces LVC parameters near their theoretical values for all ensemble sizes. Although the adjustment is unnecessary when applying LVC, it allows for a more accurate assessment of the reliability of ensembles, and a fair comparison of the reliability for differently sized ensembles.

Corresponding author address: Walter C. Kolczynski Jr., Department of Meteorology, The Pennsylvania State University, University Park, PA 16802. E-mail: wck122@psu.edu
Save