A Comparison of Probabilistic Forecasts from Bred, Singular-Vector, and Perturbed Observation Ensembles

Thomas M. Hamill National Center for Atmospheric Research*, Boulder, Colorado

Search for other papers by Thomas M. Hamill in
Current site
Google Scholar
PubMed
Close
,
Chris Snyder National Center for Atmospheric Research*, Boulder, Colorado

Search for other papers by Chris Snyder in
Current site
Google Scholar
PubMed
Close
, and
Rebecca E. Morss Massachusetts Institute of Technology, Cambridge, Massachusetts

Search for other papers by Rebecca E. Morss in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

The statistical properties of analysis and forecast errors from commonly used ensemble perturbation methodologies are explored. A quasigeostrophic channel model is used, coupled with a 3D-variational data assimilation scheme. A perfect model is assumed.

Three perturbation methodologies are considered. The breeding and singular-vector (SV) methods approximate the strategies currently used at operational centers in the United States and Europe, respectively. The perturbed observation (PO) methodology approximates a random sample from the analysis probability density function (pdf) and is similar to the method performed at the Canadian Meteorological Centre. Initial conditions for the PO ensemble are analyses from independent, parallel data assimilation cycles. Each assimilation cycle utilizes observations perturbed by random noise whose statistics are consistent with observational error covariances. Each member’s assimilation/forecast cycle is also started from a distinct initial condition.

Relative to breeding and SV, the PO method here produced analyses and forecasts with desirable statistical characteristics. These include consistent rank histogram uniformity for all variables at all lead times, high spread/skill correlations, and calibrated, reduced-error probabilistic forecasts. It achieved these improvements primarily because 1) the ensemble mean of the PO initial conditions was more accurate than the mean of the bred or singular-vector ensembles, which were centered on a less-skilful control initial condition—much of the improvement was lost when PO initial conditions were recentered on the control analysis; and 2) by construction, the perturbed observation ensemble initial conditions permitted realistic variations in spread from day to day, while bred and singular-vector perturbations did not. These results suggest that in the absence of model error, an ensemble of initial conditions performs better when the initialization method is designed to produce random samples from the analysis pdf. The perturbed observation method did this much more satisfactorily than either the breeding or singular-vector methods.

The ability of the perturbed observation ensemble to sample randomly from the analysis pdf also suggests that such an ensemble can provide useful information on forecast covariances and hence improve future data assimilation techniques.

Corresponding author address: Dr. Thomas M. Hamill, NCAR/MMM, P.O. Box 3000, Boulder, CO 80301.

Abstract

The statistical properties of analysis and forecast errors from commonly used ensemble perturbation methodologies are explored. A quasigeostrophic channel model is used, coupled with a 3D-variational data assimilation scheme. A perfect model is assumed.

Three perturbation methodologies are considered. The breeding and singular-vector (SV) methods approximate the strategies currently used at operational centers in the United States and Europe, respectively. The perturbed observation (PO) methodology approximates a random sample from the analysis probability density function (pdf) and is similar to the method performed at the Canadian Meteorological Centre. Initial conditions for the PO ensemble are analyses from independent, parallel data assimilation cycles. Each assimilation cycle utilizes observations perturbed by random noise whose statistics are consistent with observational error covariances. Each member’s assimilation/forecast cycle is also started from a distinct initial condition.

Relative to breeding and SV, the PO method here produced analyses and forecasts with desirable statistical characteristics. These include consistent rank histogram uniformity for all variables at all lead times, high spread/skill correlations, and calibrated, reduced-error probabilistic forecasts. It achieved these improvements primarily because 1) the ensemble mean of the PO initial conditions was more accurate than the mean of the bred or singular-vector ensembles, which were centered on a less-skilful control initial condition—much of the improvement was lost when PO initial conditions were recentered on the control analysis; and 2) by construction, the perturbed observation ensemble initial conditions permitted realistic variations in spread from day to day, while bred and singular-vector perturbations did not. These results suggest that in the absence of model error, an ensemble of initial conditions performs better when the initialization method is designed to produce random samples from the analysis pdf. The perturbed observation method did this much more satisfactorily than either the breeding or singular-vector methods.

The ability of the perturbed observation ensemble to sample randomly from the analysis pdf also suggests that such an ensemble can provide useful information on forecast covariances and hence improve future data assimilation techniques.

Corresponding author address: Dr. Thomas M. Hamill, NCAR/MMM, P.O. Box 3000, Boulder, CO 80301.

Save
  • Anderson, J. L., 1996: A method for producing and evaluating probabilistic forecasts from ensemble model integrations. J. Climate,9, 1518–1530.

  • ——, 1997: Impact of dynamical constraints on the selection of initial conditions for ensemble predictions: Low-order perfect model results. Mon. Wea. Rev.,125, 2969–2983.

  • ——, and S. L. Anderson, 1999: A Monte Carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Mon. Wea. Rev.,127, 2741–2758.

  • Barkmeijer, J., M., van Gijzen, and F. Bouttier, 1998: Singular vectors and estimates of the analysis error covariance metric. Quart. J. Roy. Meteor. Soc.,124, 1695–1713.

  • ——, R. Buizza, and T. N. Palmer, 1999: 3D-Var Hessian singular vectors and their use in the ECMWF ensemble prediction system. Quart. J. Roy. Meteor. Soc.,125, 2333–2351.

  • Bergman, K. H., 1979: Multivariate analysis of temperatures and winds using optimum interpolation. Mon. Wea. Rev.,107, 1423–1444.

  • Bishop, C. H., and Z. Toth, 1999: Ensemble transformation and adaptive observations. J. Atmos. Sci.,56, 1748–1765.

  • Brier, G. W., 1950: Verification of forecasts expressed in terms of probabilities. Mon. Wea. Rev.,78, 1–3.

  • Buizza, R., 1994: Sensitivity of optimal unstable structures. Quart. J. Roy. Meteor. Soc.,120, 429–451.

  • ——, and T. N. Palmer, 1995: The singular vector structure of the atmospheric general circulation. J. Atmos. Sci.,52, 1434–1456.

  • ——, M. Miller, and T. N. Palmer, 1999: Stochastic representation of model uncertainties in the ECMWF ensemble prediction system. Quart. J. Roy. Meteor. Soc,125, 2887–2908.

  • Burgers, G., P. J. van Leeuwen, and G. Evensen, 1998: Analysis scheme in the ensemble Kalman filter. Mon. Wea. Rev.,126, 1719–1724.

  • Daley, R., 1991: Atmospheric Data Analysis. Cambridge University Press, 457 pp.

  • Ehrendorfer, M., 1994a: The Liouville equation and its potential usefulness for the prediction of forecast skill. Part I: Theory. Mon. Wea. Rev.,122, 703–713.

  • ——, 1994b: The Liouville equation and its potential usefulness for the prediction of forecast skill. Part II: Applications. Mon. Wea. Rev.,122, 714–728.

  • ——, and J. J. Tribbia, 1997: Optimal prediction of forecast error covariances through singular vectors. J. Atmos. Sci.,54, 286–311.

  • Errico, R. M., 1997: What is an adjoint model? Bull. Amer. Meteor. Soc.,78, 2577–2592.

  • Evensen, G., and P. J. van Leeuwen, 1996: Assimilation of Geosat altimeter data for the Agulhas Current using the ensemble Kalman filter with a quasigeostrophic model. Mon. Wea. Rev.,124, 85–96.

  • Fritsch, J. M., and Coauthors, 1998: Quantitative precipitation forecasts: Report of the eighth prospectus development team, U.S. Weather Research Program. Bull. Amer. Meteor. Soc.,79, 285–299.

  • Hamill, T. M., 1997: Short-range ensemble forecasting using the Eta/RSM models. Ph.D. dissertation, Cornell University, Ithaca, NY, 216 pp. [Available from UMI Dissertation Services, 300 N. Zeeb Rd., P.O. Box 1346, Ann Arbor, MI 48106-1346.].

  • ——, 1999: Hypothesis tests for evaluating numerical precipitation forecasts. Wea. Forecasting,14, 155–167.

  • ——, and S. J. Colucci, 1997: Verification of Eta-RSM short-range ensemble forecasts. Mon. Wea. Rev.,125, 1312–1327.

  • ——, and ——, 1998a: Evaluation of Eta-RSM ensemble probabilistic precipitation forecasts. Mon. Wea. Rev.,126, 711–724.

  • ——, and ——, 1998b: Perturbations to the land surface condition in short-range ensemble forecasts. Preprints, 12th Conf. on Numerical Weather Prediction, Phoenix, AZ, Amer. Meteor. Soc., 273–276.

  • Houtekamer, P. L., 1993: Global and local skill forecasts. Mon. Wea. Rev.,121, 1834–1846.

  • ——, 1995: The construction of optimal perturbations. Mon. Wea. Rev.,123, 2888–2898.

  • ——, and J. Derome, 1995: Methods for ensemble prediction. Mon. Wea. Rev.,123, 2181–2196.

  • ——, and H. L. Mitchell, 1998: Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev.,126, 796–811.

  • ——, and Coauthors, 1996: A system simulation approach to ensemble prediction. Mon. Wea. Rev.,124, 1225–1242.

  • Kalnay, E., and Z. Toth, 1994: Removing growing errors from the analysis cycle. Preprints, 10th Conf. on Numerical Weather Prediction, Portland, OR, Amer. Meteor. Soc., 212–215.

  • ——, and Coauthors, 1996: The NCEP/NCAR 40-Year Reanalysis Project. Bull. Amer. Meteor. Soc.,77, 437–471.

  • Legras, B., and R. Vautard, 1996: A guide to Liapunov vectors. Proc. ECMWF Seminar on Predictablility, Vol. I, Reading, United Kingdom, ECMWF, 143–156.

  • Leith, C. E., 1974: Theoretical skill of Monte Carlo forecasts. Mon. Wea. Rev.,102, 409–418.

  • Lorenz, E. N., 1963: Deterministic nonperiodic flow. J. Atmos. Sci.,20, 131–140.

  • ——, 1965: A study of the predictability of a 28-variable atmospheric model. Tellus,17, 321–333.

  • Mason, I., 1982: A model for assessment of weather forecasts. Aust. Meteor. Mag.,30, 291–303.

  • Mitchell, H. L., and P. L. Houtekamer, 2000: An adaptive ensemble Kalman filter. Mon. Wea. Rev.,128, 416–433.

  • Molteni, F., R. Buizza, T. N. Palmer, and T. Petroliagis, 1996: The ECMWF ensemble prediction system: Methodology and validation. Quart. J. Royal Meteor. Soc.,122, 73–119.

  • Morss, R. E., 1999: Adaptive observations: Idealized sampling strategies for improving numerical weather prediction. Ph.D. thesis, Massachusetts Institute of Technology, Cambridge, MA, 225 pp. [Available from UMI Dissertation Services, 300 N. Zeeb Rd., P.O. Box 1346, Ann Arbor, MI 48106-1346.].

  • Mullen, S. L., and D. P. Baumhefner, 1989: The impact of initial condition uncertainty on numerical simulations of large-scale explosive cyclogenesis. Mon. Wea. Rev.,117, 2800–2821.

  • Parrish, D. F., and J. C. Derber, 1992: The National Meteorological Center’s Spectral Statistical Interpolation Analysis System. Mon. Wea. Rev.,120, 1747–1763.

  • Press, W. H., S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, 1992: Numerical Recipes in Fortran. 2d ed. Cambridge University Press, 963 pp.

  • Rotunno, R., and J.-W. Bao, 1996: A case study of cyclogenesis using a model hierarchy. Mon. Wea. Rev.,124, 1051–1066.

  • Schubert, S. D., and M. Suarez, 1989: Dynamical predictability in a simple general circulation model: Average error growth. J. Atmos. Sci.,46, 353–370.

  • Stanski, H. R., L. J. Wilson, and W. R. Burrows, 1989: Survey of common verification methods in meteorology. Environment Canada Research Rep. 89-5, 114 pp. [Available from Atmospheric Environment Service, Forecast Research Division, 4905 Dufferin St., Downsview, ON M3H 5T4, Canada.].

  • Stensrud, D. J., and J. M. Fritsch, 1994: Mesoscale convective systems in weakly forced large-scale environments. Part III: Numerical simulations and implications for weather forecasting. Mon. Wea. Rev.,122, 2084–2104.

  • Swets, J. A., 1973: The relative operating characteristic in psychology. Science,182, 990–999.

  • Toth, Z., and E. Kalnay, 1993: Ensemble forecasting at NMC: The generation of perturbations. Bull. Amer. Meteor. Soc.,74, 2317–2330.

  • ——, and ——, 1997: Ensemble forecasting at NCEP and the breeding method. Mon. Wea. Rev.,125, 3297–3319.

  • Whitaker, J. S., and A. F. Loughe, 1998: The relationship between ensemble spread and ensemble mean skill. Mon. Wea. Rev.,126, 3292–3302.

  • Wilks, D. S., 1995: Statistical Methods in the Atmospheric Sciences:An Introduction. Academic Press, 467 pp.

  • Wolf, A., J. B. Swift, H. L. Swinney, and J. A. Vastano, 1985: Determining Lyapunov exponents from a time series. Physica D,16, 285–317.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 655 335 30
PDF Downloads 311 111 11