Comparison of Ensemble-MOS Methods Using GFS Reforecasts

Daniel S. Wilks Department of Earth and Atmospheric Sciences, Cornell University, Ithaca, New York

Search for other papers by Daniel S. Wilks in
Current site
Google Scholar
PubMed
Close
and
Thomas M. Hamill NOAA/Earth System Research Laboratory, Boulder, Colorado

Search for other papers by Thomas M. Hamill in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Three recently proposed and promising methods for postprocessing ensemble forecasts based on their historical error characteristics (i.e., ensemble-model output statistics methods) are compared using a multidecadal reforecast dataset. Logistic regressions and nonhomogeneous Gaussian regressions are generally preferred for daily temperature, and for medium-range (6–10 and 8–14 day) temperature and precipitation forecasts. However, the better sharpness of medium-range ensemble-dressing forecasts sometimes yields the best Brier scores even though their calibration is somewhat worse. Using the long (15 or 25 yr) training samples that are available with these reforecasts improves the accuracy and skill of these probabilistic forecasts to levels that are approximately equivalent to gains of 1 day of lead time, relative to using short (1 or 2 yr) training samples.

Corresponding author address: Dr. Daniel S. Wilks, Department of Earth and Atmospheric Sciences, Cornell University, Bradfield Hall, Ithaca, NY 14853. Email: dsw5@cornell.edu

Abstract

Three recently proposed and promising methods for postprocessing ensemble forecasts based on their historical error characteristics (i.e., ensemble-model output statistics methods) are compared using a multidecadal reforecast dataset. Logistic regressions and nonhomogeneous Gaussian regressions are generally preferred for daily temperature, and for medium-range (6–10 and 8–14 day) temperature and precipitation forecasts. However, the better sharpness of medium-range ensemble-dressing forecasts sometimes yields the best Brier scores even though their calibration is somewhat worse. Using the long (15 or 25 yr) training samples that are available with these reforecasts improves the accuracy and skill of these probabilistic forecasts to levels that are approximately equivalent to gains of 1 day of lead time, relative to using short (1 or 2 yr) training samples.

Corresponding author address: Dr. Daniel S. Wilks, Department of Earth and Atmospheric Sciences, Cornell University, Bradfield Hall, Ithaca, NY 14853. Email: dsw5@cornell.edu

Save
  • Barkmeijer, J., M. van Gijzen, and F. Bouttier, 1998: Singular vectors and estimates of the analysis error covariance metric. Quart. J. Roy. Meteor. Soc., 124 , 16951713.

    • Search Google Scholar
    • Export Citation
  • Barkmeijer, J., R. Buizza, and T. N. Palmer, 1999: 3D-Var Hessian singular vectors and their potential use in the ECMWF ensemble prediction system. Quart. J. Roy. Meteor. Soc., 125 , 23332351.

    • Search Google Scholar
    • Export Citation
  • Buizza, R., D. S. Richardson, and T. N. Palmer, 2003: Benefits of increased resolution in the ECMWF ensemble system and comparisons with poor-man’s ensembles. Quart. J. Roy. Meteor. Soc., 129 , 12691288.

    • Search Google Scholar
    • Export Citation
  • Buizza, R., P. L. Houtekamer, Z. Toth, G. Pellerin, M. Wei, and Y. Zhu, 2005: A comparison of the ECMWF, MSC, and NCEP global ensemble prediction systems. Mon. Wea. Rev., 133 , 10761097.

    • Search Google Scholar
    • Export Citation
  • Carter, G. M., J. P. Dallavalle, and H. R. Glahn, 1989: Statistical forecasts based on the National Meteorological Center’s numerical weather prediction system. Wea. Forecasting, 4 , 401412.

    • Search Google Scholar
    • Export Citation
  • Clark, M. P., and L. E. Hay, 2004: Use of medium-range numerical weather prediction model output to produce forecasts of streamflow. J. Hydrometeor., 5 , 1532.

    • Search Google Scholar
    • Export Citation
  • Eckel, F. A., and M. K. Walters, 1998: Calibrated probabilistic quantitative precipitation forecasts based on the MRF ensemble. Wea. Forecasting, 13 , 11321147.

    • Search Google Scholar
    • Export Citation
  • Ehrendorfer, M., and A. H. Murphy, 1988: Comparative evaluation of weather forecasting systems: Sufficiency, quality, and accuracy. Mon. Wea. Rev., 116 , 17571770.

    • Search Google Scholar
    • Export Citation
  • Epstein, E. S., 1969: A scoring system for probability forecasts of ranked categories. J. Appl. Meteor., 8 , 985987.

  • Fortin, V., A-C. Favre, and M. Saïd, 2006: Probabilistic forecasting from ensemble prediction systems: Improving upon the best-member method by using a different weight and dressing kernel for each member. Quart. J. Roy. Meteor. Soc., 132 , 13491369.

    • Search Google Scholar
    • Export Citation
  • Gangopadhyay, S., M. P. Clark, B. Rajagopalan, K. Werner, and D. Brandon, 2004: Effects of spatial and temporal aggregation on the accuracy of statistically downscaled precipitation estimates in the Upper Colorado River basin. J. Hydrometeor., 5 , 11921206.

    • Search Google Scholar
    • Export Citation
  • Glahn, H. R., and D. A. Lowry, 1972: The use of model output statistics (MOS) in objective weather forecasting. J. Appl. Meteor., 11 , 12031211.

    • Search Google Scholar
    • Export Citation
  • Gneiting, T., A. E. Raftery, A. H. Westveld III, and T. Goldman, 2005: Calibrated probabilistic forecasting using ensemble Model Output Statistics and minimum CRPS estimation. Mon. Wea. Rev., 133 , 10981118.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., and S. J. Colucci, 1997: Verification of Eta-RSM short-range ensemble forecasts. Mon. Wea. Rev., 125 , 13121327.

  • Hamill, T. M., and S. J. Colucci, 1998: Evaluation of Eta-RSM ensemble probabilistic precipitation forecasts. Mon. Wea. Rev., 126 , 711724.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., and J. Juras, 2007: Measuring forecast skill: Is it real skill or is it the varying climatology? Quart. J. Roy. Meteor. Soc., in press.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., and J. S. Whitaker, 2006: Probabilistic quantitative precipitation forecasts based on reforecast analogs: Theory and application. Mon. Wea. Rev., 134 , 32093229.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., C. Snyder, and R. E. Morss, 2000: A comparison of probabilistic forecasts from bred, singular vector, and perturbed observation ensembles. Mon. Wea. Rev., 128 , 18351851.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., C. Snyder, and J. S. Whitaker, 2003: Ensemble forecasts and the properties of flow-dependent analysis-error covariance singular vectors. Mon. Wea. Rev., 131 , 17411758.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., J. S. Whitaker, and X. Wei, 2004: Ensemble reforecasting: Improving medium-range forecast skill using retrospective forecasts. Mon. Wea. Rev., 132 , 14341447.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., J. S. Whitaker, and S. L. Mullen, 2006: Reforecasts: An important new dataset for improving weather predictions. Bull. Amer. Meteor. Soc., 87 , 3346.

    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., L. Lefaivre, and J. Derome, 1996: The RPN ensemble prediction system. Proc. ECMWF Seminar on Predictability, Vol. II, Reading, United Kingdom, ECMWF, 121–146. [Available from ECMWF, Shinfield Park, Reading, Berkshire RG2 9AX, United Kingdom.].

  • Krishnamurti, T. N., C. M. Kishtawal, T. E. LaRow, D. R. Bachiochi, Z. Zhan, C. E. Williford, S. Gadgil, and S. Surendran, 1999: Improved weather and seasonal climate forecasts from a multimodel superensemble. Science, 285 , 15481550.

    • Search Google Scholar
    • Export Citation
  • Lorenz, E. N., 1996: Predictability: A problem partly solved. Proc. ECMWF Seminar on Predictability, Vol. I, Reading, United Kingdom, ECMWF, 1–18. [Available from ECMWF, Shinfield Park, Reading, Berkshire RG2 9AX, United Kingdom.].

  • Lorenz, E. N., and K. A. Emanuel, 1998: Optimal sites for supplementary weather observations: Simulations with a small model. J. Atmos. Sci., 55 , 399414.

    • Search Google Scholar
    • Export Citation
  • Molteni, F., R. Buizza, T. N. Palmer, and T. Petroliagis, 1996: The ECMWF ensemble prediction system: Methodology and validation. Quart. J. Roy. Meteor. Soc., 122 , 73119.

    • Search Google Scholar
    • Export Citation
  • Mullen, S. L., and R. Buizza, 2002: The impact of horizontal resolution and ensemble size on probabilistic forecasts of precipitation by the ECMWF ensemble prediction system. Wea. Forecasting, 17 , 173191.

    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1973: A new vector partition of the probability score. J. Appl. Meteor., 12 , 595600.

  • Palmer, T. N., 2001: A nonlinear dynamical perspective on model error: A proposal for non-local stochastic-dynamic parametrization in weather and climate prediction models. Quart. J. Roy. Meteor. Soc., 127 , 279304.

    • Search Google Scholar
    • Export Citation
  • Raftery, A. E., T. Gneiting, F. Balabdaoui, and M. Polakowski, 2005: Using Bayesian model averaging to calibrate forecast ensembles. Mon. Wea. Rev., 133 , 11551174.

    • Search Google Scholar
    • Export Citation
  • Roulston, M. S., and L. A. Smith, 2003: Combining dynamical and statistical ensembles. Tellus, 55A , 1630.

  • Stephenson, D. B., C. A. S. Coelho, M. Balmaseda, and F. J. Doblas-Reyes, 2005: Forecast assimilation: A unified framework for the combination of multi-model weather and climate predictions. Tellus, 57A , 253264.

    • Search Google Scholar
    • Export Citation
  • Sutton, C. J., T. M. Hamill, and T. T. Warner, 2006: Will perturbing soil moisture improve warm-season ensemble forecasts? A proof of concept. Mon. Wea. Rev., 134 , 31743189.

    • Search Google Scholar
    • Export Citation
  • Szunyogh, I., and Z. Toth, 2002: The effect of increased horizontal resolution on the NCEP global ensemble mean forecasts. Mon. Wea. Rev., 130 , 11251143.

    • Search Google Scholar
    • Export Citation
  • Toth, Z., and E. Kalnay, 1993: Ensemble forecasting at NMC: The generation of perturbations. Bull. Amer. Meteor. Soc., 74 , 23172330.

  • Toth, Z., and E. Kalnay, 1997: Ensemble forecasting at NCEP and the breeding method. Mon. Wea. Rev., 125 , 32973319.

  • Vislocky, R. L., and J. M. Fritsch, 1995: Improved model output statistics forecasts through model consensus. Bull. Amer. Meteor. Soc., 76 , 11571164.

    • Search Google Scholar
    • Export Citation
  • Vislocky, R. L., and J. M. Fritsch, 1997: Performance of an advanced MOS system in the 1996–97 national collegiate weather forecasting contest. Bull. Amer. Meteor. Soc., 78 , 28512857.

    • Search Google Scholar
    • Export Citation
  • Wang, X., and C. H. Bishop, 2003: A comparison of breeding and ensemble transform Kalman filter ensemble forecast schemes. J. Atmos. Sci., 60 , 11401158.

    • Search Google Scholar
    • Export Citation
  • Wang, X., and C. H. Bishop, 2005: Improvement of ensemble reliability with a new dressing kernel. Quart. J. Roy. Meteor. Soc., 131 , 965986.

    • Search Google Scholar
    • Export Citation
  • Weisman, M. L., W. C. Skamarock, and J. B. Klemp, 1997: The resolution dependence of explicitly modeled convective systems. Mon. Wea. Rev., 125 , 527548.

    • Search Google Scholar
    • Export Citation
  • Whitaker, J. S., F. Vitart, and X. Wei, 2006: Improving week-2 forecasts with multimodel reforecast ensembles. Mon. Wea. Rev., 134 , 22792284.

    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2005: Effects of stochastic parameterization in the Lorenz ’96 system. Quart. J. Roy. Meteor. Soc., 131 , 389407.

  • Wilks, D. S., 2006a: Comparison of ensemble-MOS methods in the Lorenz ’96 setting. Meteor. Appl., 13 , 243256.

  • Wilks, D. S., 2006b: Statistical Methods in the Atmospheric Sciences. 2d ed. Academic Press, 627 pp.

  • Wilson, L. J., and M. Valée, 2002: The Canadian updateable model output statistics (UMOS) system: Design and development tests. Wea. Forecasting, 17 , 206222.

    • Search Google Scholar
    • Export Citation
  • Wilson, L. J., and M. Valée, 2003: The Canadian updateable model output statistics (UMOS) system: Validation against perfect prog. Wea. Forecasting, 18 , 288302.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 1433 303 19
PDF Downloads 983 225 17