• Anderson, J. L., 1996: A method for producing and evaluating probabilistic forecasts from ensemble model integrations. J. Climate, 9 , 15181530.

    • Search Google Scholar
    • Export Citation
  • Buizza, R., , P. L. Houtekamer, , Z. Toth, , G. Pellerin, , M. Wei, , and Y. Zhu, 2005: A comparison of the ECMWF, MSC, and NCEP global ensemble prediction systems. Mon. Wea. Rev., 133 , 10761097.

    • Search Google Scholar
    • Export Citation
  • Dempster, A. P., , N. M. Laird, , and D. B. Rubin, 1977: Maximum likelihood for incomplete data via the EM algorithm (with discussion). J. Roy. Stat. Soc. Ser. B, 39 , 138.

    • Search Google Scholar
    • Export Citation
  • Dirren, S., , R. D. Torn, , and G. J. Hakim, 2007: A data assimilation case study using a limited-area ensemble Kalman filter. Mon. Wea. Rev., 135 , 14551473.

    • Search Google Scholar
    • Export Citation
  • Doblas-Reyes, F. J., , R. Hagedorn, , and T. N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting II: Calibration and combination. Tellus, 57A , 234252.

    • Search Google Scholar
    • Export Citation
  • Eckel, F. A., , and C. F. Mass, 2005: Effective mesoscale, short-range ensemble forecasting. Wea. Forecasting, 20 , 328350.

  • Evensen, G., 1994: Sequential data assimilation with a non-linear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res., 99 , 1014310162.

    • Search Google Scholar
    • Export Citation
  • Fraley, C., , A. E. Raftery, , T. Gneiting, , and J. M. Sloughter, 2007: EnsembleBMA: An R package for probabilistic forecasting using ensembles and Bayesian model averaging. Tech. Rep. 516, Department of Statistics, University of Washington, 17 pp.

    • Search Google Scholar
    • Export Citation
  • Gneiting, T., , and A. E. Raftery, 2007: Strictly proper scoring rules, prediction, and estimation. J. Amer. Stat. Assoc., 102 , 359378.

    • Search Google Scholar
    • Export Citation
  • Gneiting, T., , F. Balabdaoui, , and A. E. Raftery, 2007: Probabilistic forecasts, calibration and sharpness. J. Roy. Stat. Soc. Ser. B, 69 , 243268.

    • Search Google Scholar
    • Export Citation
  • Grimit, E. P., , and C. F. Mass, 2002: Initial results for a mesoscale short-range ensemble forecasting system over the Pacific Northwest. Wea. Forecasting, 17 , 192205.

    • Search Google Scholar
    • Export Citation
  • Grimit, E. P., , T. Gneiting, , V. J. Berrocal, , and N. A. Johnson, 2006: The continuous ranked probability score for circular variables and its application to mesoscale forecast ensemble verification. Quart. J. Roy. Meteor. Soc., 132 , 32093220.

    • Search Google Scholar
    • Export Citation
  • Hagedorn, R., , F. J. Doblas-Reyes, , and T. N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting I: Basic concept. Tellus, 57A , 219233.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., 2001: Interpretation of rank histograms for verifying ensemble forecasts. Mon. Wea. Rev., 129 , 550560.

  • Hamill, T. M., 2005: Ensemble-based atmospheric data assimilation: A tutorial. Predictability of Weather and Climate, T. Palmer and R. Hagedorn, Eds., Cambridge University Press, 124–156.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., 2007: Comments on “Calibrated surface temperature forecasts from the Canadian ensemble prediction system using Bayesian model averaging”. Mon. Wea. Rev., 135 , 42264230.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., , and S. J. Colucci, 1997: Verification of Eta-RSM short-range ensemble forecasts. Mon. Wea. Rev., 125 , 13121327.

  • Heizenreder, D., , S. Trepte, , and M. Denhard, 2005: SRNWP-PEPS: A regional multi-model ensemble in Europe. Euro. Forecaster, 11 , 2935.

  • Hersbach, H., 2000: Decomposition of the continuous ranked probability score for ensemble predictions. Wea. Forecasting, 15 , 559570.

  • Joslyn, S., , and D. W. Jones, 2008: Strategies in naturalistic decision-making: A cognitive task analysis of naval weather forecasting. Naturalistic Decision Making and Macrocognition, J. M. Schraagen et al., Eds., Ashgate Publishing, 183–202.

    • Search Google Scholar
    • Export Citation
  • McCandless, T. C., , S. E. Haupt, , and G. Young, 2009: Replacing missing data for ensemble systems. Preprints, Seventh Conf. on Artificial Intelligence and Its Applications to the Environmental Sciences, Phoenix, AZ, Amer. Meteor. Soc., 1.2. [Available online at http://ams.confex.com/ams/89annual/techprogram/paper_150305.htm].

    • Search Google Scholar
    • Export Citation
  • McLachlan, G. J., , and T. Krishnan, 1997: The EM Algorithm and Extensions. Wiley, 274 pp.

  • Molteni, F., , R. Buizza, , T. N. Palmer, , and T. Petroliagis, 1996: The ECMWF ensemble system: Methodology and validation. Quart. J. Roy. Meteor. Soc., 122 , 73119.

    • Search Google Scholar
    • Export Citation
  • Park, Y., , R. Buizza, , and M. Leutbecher, 2008: TIGGE: Preliminary results on comparing and combining ensembles. Quart. J. Roy. Meteor. Soc., 134 , 20292050.

    • Search Google Scholar
    • Export Citation
  • Raftery, A. E., , T. Gneiting, , F. Balabdaoui, , and M. Polakowski, 2005: Using Bayesian model averaging to calibrate forecast ensembles. Mon. Wea. Rev., 133 , 11551174.

    • Search Google Scholar
    • Export Citation
  • Schafer, J. L., 1997: Analysis of Incomplete Multivariate Data by Simulation. Chapman and Hall, 430 pp.

  • Sloughter, J. M., , A. E. Raftery, , T. Gneiting, , and C. Fraley, 2007: Probabilistic quantitative precipitation forecasting using Bayesian model averaging. Mon. Wea. Rev., 135 , 32093220.

    • Search Google Scholar
    • Export Citation
  • Sloughter, J. M., , T. Gneiting, , and A. E. Raftery, 2009: Probabilistic wind speed forecasting using ensembles and Bayesian model averaging. J. Amer. Stat. Assoc., in press.

    • Search Google Scholar
    • Export Citation
  • Talagrand, O., , R. Vautard, , and B. Strauss, 1997: Evaluation of probabilistic prediction systems. Proc. Workshop on Predictability, Reading, United Kingdom, European Centre for Medium-Range Weather Forecasts, 1–25.

    • Search Google Scholar
    • Export Citation
  • Torn, R. D., , and G. J. Hakim, 2008: Performance characteristics of a pseudo-operational ensemble Kalman filter. Mon. Wea. Rev., 136 , 39473963.

    • Search Google Scholar
    • Export Citation
  • Torn, R. D., , G. J. Hakim, , and C. Snyder, 2006: Boundary conditions for limited-area ensemble Kalman filters. Mon. Wea. Rev., 134 , 24902502.

    • Search Google Scholar
    • Export Citation
  • Toth, Z., , and E. Kalnay, 1993: Ensemble forecasting at the NMC: The generation of perturbations. Bull. Amer. Meteor. Soc., 74 , 23172330.

    • Search Google Scholar
    • Export Citation
  • Vrugt, J. A., , C. G. H. Diks, , and M. P. Clark, 2008: Ensemble Bayesian model averaging using Markov chain Monte Carlo sampling. Environ. Fluid Mech., 134 , 117.

    • Search Google Scholar
    • Export Citation
  • Weigel, A. P., , M. A. Liniger, , and C. Appenzeller, 2008: Can multi-model combination really enhance the prediction skill of forecasts? Quart. J. Roy. Meteor. Soc., 134 , 241260.

    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2006: Statistical Methods in the Atmospheric Sciences. 2nd ed. Academic Press, 648 pp.

  • Wilson, L. J., , S. Beauregard, , A. E. Raftery, , and R. Verret, 2007a: Calibrated surface temperature forecasts from the Canadian ensemble prediction system using Bayesian model averaging. Mon. Wea. Rev., 135 , 13641385.

    • Search Google Scholar
    • Export Citation
  • Wilson, L. J., , S. Beauregard, , A. E. Raftery, , and R. Verret, 2007b: Reply. Mon. Wea. Rev., 135 , 42314236.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 118 118 30
PDF Downloads 87 87 25

Calibrating Multimodel Forecast Ensembles with Exchangeable and Missing Members Using Bayesian Model Averaging

View More View Less
  • 1 University of Washington, Seattle, Washington
  • | 2 Universitat Heidelberg, Heidelberg, Germany
© Get Permissions
Restricted access

Abstract

Bayesian model averaging (BMA) is a statistical postprocessing technique that generates calibrated and sharp predictive probability density functions (PDFs) from forecast ensembles. It represents the predictive PDF as a weighted average of PDFs centered on the bias-corrected ensemble members, where the weights reflect the relative skill of the individual members over a training period.

This work adapts the BMA approach to situations that arise frequently in practice; namely, when one or more of the member forecasts are exchangeable, and when there are missing ensemble members. Exchangeable members differ in random perturbations only, such as the members of bred ensembles, singular vector ensembles, or ensemble Kalman filter systems. Accounting for exchangeability simplifies the BMA approach, in that the BMA weights and the parameters of the component PDFs can be assumed to be equal within each exchangeable group. With these adaptations, BMA can be applied to postprocess multimodel ensembles of any composition.

In experiments with surface temperature and quantitative precipitation forecasts from the University of Washington mesoscale ensemble and ensemble Kalman filter systems over the Pacific Northwest, the proposed extensions yield good results. The BMA method is robust to exchangeability assumptions, and the BMA postprocessed combined ensemble shows better verification results than any of the individual, raw, or BMA postprocessed ensemble systems. These results suggest that statistically postprocessed multimodel ensembles can outperform individual ensemble systems, even in cases in which one of the constituent systems is superior to the others.

Corresponding author address: Chris Fraley, Department of Statistics, University of Washington, Box 354322, Seattle, WA 98195-4322. Email: fraley@stat.washington.edu

Abstract

Bayesian model averaging (BMA) is a statistical postprocessing technique that generates calibrated and sharp predictive probability density functions (PDFs) from forecast ensembles. It represents the predictive PDF as a weighted average of PDFs centered on the bias-corrected ensemble members, where the weights reflect the relative skill of the individual members over a training period.

This work adapts the BMA approach to situations that arise frequently in practice; namely, when one or more of the member forecasts are exchangeable, and when there are missing ensemble members. Exchangeable members differ in random perturbations only, such as the members of bred ensembles, singular vector ensembles, or ensemble Kalman filter systems. Accounting for exchangeability simplifies the BMA approach, in that the BMA weights and the parameters of the component PDFs can be assumed to be equal within each exchangeable group. With these adaptations, BMA can be applied to postprocess multimodel ensembles of any composition.

In experiments with surface temperature and quantitative precipitation forecasts from the University of Washington mesoscale ensemble and ensemble Kalman filter systems over the Pacific Northwest, the proposed extensions yield good results. The BMA method is robust to exchangeability assumptions, and the BMA postprocessed combined ensemble shows better verification results than any of the individual, raw, or BMA postprocessed ensemble systems. These results suggest that statistically postprocessed multimodel ensembles can outperform individual ensemble systems, even in cases in which one of the constituent systems is superior to the others.

Corresponding author address: Chris Fraley, Department of Statistics, University of Washington, Box 354322, Seattle, WA 98195-4322. Email: fraley@stat.washington.edu

Save