Editorial Type:
Article
Article Type:
Research Article

Controlling Overestimation of Error Covariance in Ensemble Kalman Filters with Sparse Observations: A Variance-Limiting Kalman Filter

Georg A. Gottwald School of Mathematics and Statistics, University of Sydney, Sydney, Australia

Search for other papers by Georg A. Gottwald in
Current site
Google Scholar
PubMed Close
,
Lewis Mitchell School of Mathematics and Statistics, University of Sydney, Sydney, Australia

Search for other papers by Lewis Mitchell in
Current site
Google Scholar
PubMed Close
, and
Sebastian Reich Universität Potsdam, Institut für Mathematik, Potsdam, Germany

Search for other papers by Sebastian Reich in
Current site
Google Scholar
PubMed Close
Print Publication:
01 Aug 2011
DOI:
https://doi.org/10.1175/2011MWR3557.1
Page(s):
2650–2667
Restricted access

Abstract

The problem of an ensemble Kalman filter when only partial observations are available is considered. In particular, the situation is investigated where the observational space consists of variables that are directly observable with known observational error, and of variables of which only their climatic variance and mean are given. To limit the variance of the latter poorly resolved variables a variance-limiting Kalman filter (VLKF) is derived in a variational setting. The VLKF for a simple linear toy model is analyzed and its range of optimal performance is determined. The VLKF is explored in an ensemble transform setting for the Lorenz-96 system, and it is shown that incorporating the information of the variance of some unobservable variables can improve the skill and also increase the stability of the data assimilation procedure.

Corresponding author address: Georg A. Gottwald, School of Mathematics and Statistics, University of Sydney, Sydney, NSW 2006, Australia. E-mail: georg.gottwald@sydney.edu.au

Abstract

The problem of an ensemble Kalman filter when only partial observations are available is considered. In particular, the situation is investigated where the observational space consists of variables that are directly observable with known observational error, and of variables of which only their climatic variance and mean are given. To limit the variance of the latter poorly resolved variables a variance-limiting Kalman filter (VLKF) is derived in a variational setting. The VLKF for a simple linear toy model is analyzed and its range of optimal performance is determined. The VLKF is explored in an ensemble transform setting for the Lorenz-96 system, and it is shown that incorporating the information of the variance of some unobservable variables can improve the skill and also increase the stability of the data assimilation procedure.

Corresponding author address: Georg A. Gottwald, School of Mathematics and Statistics, University of Sydney, Sydney, NSW 2006, Australia. E-mail: georg.gottwald@sydney.edu.au
Save
Cover Monthly Weather Review
Monthly Weather Review

  • Anderson, J. L., 2001: An ensemble adjustment Kalman filter for data assimilation. Mon. Wea. Rev., 129, 2884–2903.

  • Anderson, J. L., 2007: An adaptive covariance inflation error correction algorithm for ensemble filters. Tellus, 59A, 210–224.

  • Anderson, J. L., 2009: Spatially and temporally varying adaptive covariance inflation for ensemble filters. Tellus, 61A, 72–83.

  • Anderson, J. L., and S. L. Anderson, 1999: A Monte Carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Mon. Wea. Rev., 127, 2741–2758.

    • Search Google Scholar
    • Export Citation

  • Bengtsson, L., and Coauthors, 2007: The need for a dynamical climate reanalysis. Bull. Amer. Meteor. Soc., 88, 495–501.

  • Bergemann, K., G. Gottwald, and S. Reich, 2009: Ensemble propagation and continuous matrix factorization algorithms. Quart. J. Roy. Meteor. Soc., 135, 1560–1572.

    • Search Google Scholar
    • Export Citation

  • Bocquet, M., C. A. Pires, and L. Wu, 2010: Beyond Gaussian statistical modeling in geophysical data assimilation. Mon. Wea. Rev., 138, 2997–3023.

    • Search Google Scholar
    • Export Citation

  • Buizza, R., M. Miller, and T. N. Palmer, 1999: Stochastic representation of model uncertainties in the ECMWF Ensemble Prediction System. Quart. J. Roy. Meteor. Soc., 125, 2887–2908.

    • Search Google Scholar
    • Export Citation

  • Charron, M., G. Pellerin, L. Spacek, P. L. Houtekamer, N. Gagnon, H. L. Mitchell, and L. Michelin, 2010: Toward random sampling of model error in the Canadian ensemble prediction system. Mon. Wea. Rev., 138, 1877–1901.

    • Search Google Scholar
    • Export Citation

  • Compo, G. P., and Coauthors, 2011: The twentieth century reanalysis project. Quart. J. Roy. Meteor. Soc., 137, 1–28, doi:10.1002/qj.776.

    • Search Google Scholar
    • Export Citation

  • Durran, D. R., 1999: Numerical Methods for Wave Equations in Geophysical Fluid Dynamics. Springer, 482 pp.

  • Eckermann, S. D., and Coauthors, 2009: High-altitude data assimilation system experiments for the northern summer mesosphere season of 2007. J. Atmos. Sol.-Terr. Phys., 71, 531–551.

    • Search Google Scholar
    • Export Citation

  • Ehrendorfer, M., 2007: A review of issues in ensemble-based Kalman filtering. Meteor. Z., 16, 795–818.

  • Evensen, G., 2006: Data Assimilation: The Ensemble Kalman Filter. Springer, 280 pp.

  • Fisher, M., M. Leutbecher, and G. A. Kelly, 2005: On the equivalence between Kalman smoothing and weak-constraint four-dimensional variational data assimilation. Quart. J. Roy. Meteor. Soc., 131, 3235–3246.

    • Search Google Scholar
    • Export Citation

  • Gardiner, C. W., 2004: Handbook of Stochastic Methods for Physics, Chemistry, and the Natural Sciences. 3rd ed. Springer, 415 pp.

  • Golub, G. H., and C. F. Van Loan, 1996: Matrix Computations. 3rd ed. The Johns Hopkins University Press, 728 pp.

  • Gottwald, G. A., and I. Melbourne, 2005: Testing for chaos in deterministic systems with noise. Physica D, 212, 100–110.

  • Hamill, T. M., and J. S. Whitaker, 2005: Accounting for the error due to unresolved scales in ensemble data assimilation: A comparison of different approaches. Mon. Wea. Rev., 133, 3132–3147.

    • Search Google Scholar
    • Export Citation

  • Hamill, T. M., J. S. Whitaker, and C. Snyder, 2001: Distance-dependent filtering of background covariance estimates in an ensemble Kalman filter. Mon. Wea. Rev., 129, 2776–2790.

    • Search Google Scholar
    • Export Citation

  • Harlim, J., and A. J. Majda, 2010: Catastrophic filter divergence in filtering nonlinear dissipative systems. Comm. Math. Sci., 8, 27–43.

    • Search Google Scholar
    • Export Citation

  • Houtekamer, P. L., and H. L. Mitchell, 1998: Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev., 126, 796–811.

    • Search Google Scholar
    • Export Citation

  • Houtekamer, P. L., and H. L. Mitchell, 2001: A sequential ensemble Kalman filter for atmospheric data assimilation. Mon. Wea. Rev., 129, 123–136.

    • Search Google Scholar
    • Export Citation

  • Houtekamer, P. L., and H. L. Mitchell, 2005: Ensemble Kalman filtering. Quart. J. Roy. Meteor. Soc., 131, 3269–3289.

  • Houtekamer, P. L., H. L. Mitchell, G. Pellerin, M. Buehner, M. Charron, L. Spacek, and B. Hansen, 2005: Atmospheric data assimilation with an ensemble Kalman filter: Results with real observations. Mon. Wea. Rev., 133, 604–620.

    • Search Google Scholar
    • Export Citation

  • Houtekamer, P. L., H. L. Mitchell, and X. Deng, 2009: Model error representation in an operational ensemble Kalman filter. Mon. Wea. Rev., 137, 2126–2143.

    • Search Google Scholar
    • Export Citation

  • Ide, K., P. Courtier, M. Ghil, and A. C. Lorenc, 1997: Unified notation for data assimilation: Operational, sequential and variational. J. Meteor. Soc. Japan, 75, 181–189.

    • Search Google Scholar
    • Export Citation

  • Kalnay, E., 2002: Atmospheric Modeling, Data Assimilation and Predictability. Cambridge University Press, 364 pp.

  • Kepert, J. D., 2004: On ensemble representation of the observation-error covariances in the ensemble Kalman filter. Ocean Dyn., 54, 561–569.

    • Search Google Scholar
    • Export Citation

  • Kepert, J. D., 2009: Covariance localisation and balance in an Ensemble Kalman Filter. Quart. J. Roy. Meteor. Soc., 135, 1157–1176.

    • Search Google Scholar
    • Export Citation

  • Leimkuhler, B., and S. Reich, 2005: Simulating Hamiltonian Dynamics. Cambridge University Press, 379 pp.

  • Li, H., E. Kalnay, and T. Miyoshi, 2009: Simultaneous estimation of covariance inflation and observation errors within an ensemble Kalman filter. Quart. J. Roy. Meteor. Soc., 135, 523–533.

    • Search Google Scholar
    • Export Citation

  • Liu, J., E. J. Fertig, H. Li, E. Kalnay, B. R. Hunt, E. J. Kostelich, I. Szunyogh, and R. Todling, 2008: Comparison between local ensemble transform Kalman filter and PSAS in the NASA finite volume GCM—Perfect model experiments. Nonlinear Processes Geophys., 15, 645–659.

    • Search Google Scholar
    • Export Citation

  • Lorenc, A. C., 2003: The potential of the ensemble Kalman filter for NWP—A comparison with 4DVAR. Quart. J. Roy. Meteor. Soc., 129, 3183–3203.

    • Search Google Scholar
    • Export Citation

  • Lorenz, E. N., 1996: Predictability—A problem partly solved. Predictability, T. Palmer, Ed., European Centre for Medium-Range Weather Forecasts, 1–18.

    • Search Google Scholar
    • Export Citation

  • Lorenz, E. N., and K. A. Emanuel, 1998: Optimal sites for supplementary weather observations: Simulation with a small model. J. Atmos. Sci., 55, 399–414.

    • Search Google Scholar
    • Export Citation

  • Mitchell, H. L., and P. L. Houtekamer, 2000: An adaptive ensemble Kalman filter. Mon. Wea. Rev., 128, 416–433.

  • Neef, L., S. M. Polavarapu, and T. G. Shepherd, 2006: Four-dimensional data assimilation and balanced dynamics. J. Atmos. Sci., 63, 1840–1850.

    • Search Google Scholar
    • Export Citation

  • Orrell, D., and L. Smith, 2003: Visualising bifurcations in high dimensional systems: The spectral bifurcation diagram. Int. J. Bifurcation Chaos, 13, 3015–3028.

    • Search Google Scholar
    • Export Citation

  • Ott, E., B. Hunt, I. Szunyogh, A. Zimin, E. Kostelich, M. Corrazza, E. Kalnay, and J. Yorke, 2004: A local ensemble Kalman filter for atmospheric data assimilation. Tellus, 56A, 415–428.

    • Search Google Scholar
    • Export Citation

  • Pires, C. A., O. Talagrand, and M. Bocquet, 2010: Diagnosis and impacts of non-Gaussianity of innovations in data assimilation. Physica D, 239, 1701–1717.

    • Search Google Scholar
    • Export Citation

  • Polavarapu, S., T. G. Shepherd, Y. Rochon, and S. Ren, 2005: Some challenges of middle atmosphere data assimilation. Quart. J. Roy. Meteor. Soc., 131, 3513–3527.

    • Search Google Scholar
    • Export Citation

  • Sankey, D., S. Ren, S. Polavarapu, Y. Rochon, Y. Nezlin, and S. Beagley, 2007: Impact of data assimilation filtering methods on the mesosphere. J. Geophy. Res., 112, D24104, doi:10.1029/2007JD008885.

    • Search Google Scholar
    • Export Citation

  • Sasaki, Y., 1970: Some basic formalisms on numerical variational analysis. Mon. Wea. Rev., 98, 875–883.

  • Shutts, G. J., 2005: A stochastic kinetic energy backscatter algorithm for use in ensemble prediction systems. Quart. J. Roy. Meteor. Soc., 131, 3079–3102.

    • Search Google Scholar
    • Export Citation

  • Simon, D. J., 2006: Optimal State Estimation: Kalman, H Infinity, and Nonlinear Approaches. John Wiley & Sons, Inc., 552 pp.

  • Szunyogh, I., E. Kostelich, G. Gyarmati, D. J. Patil, B. Hunt, E. Kalnay, E. Ott, and J. Yorke, 2005: Assessing a local ensemble Kalman filter: Perfect model experiments with the National Centers for Environmental Prediction global model. Tellus, 57A, 528–545.

    • Search Google Scholar
    • Export Citation

  • Tippett, M. K., J. L. Anderson, C. H. Bishop, T. M. Hamill, and J. S. Whitaker, 2003: Ensemble square root filters. Mon. Wea. Rev., 131, 1485–1490.

    • Search Google Scholar
    • Export Citation

  • Wang, X., C. H. Bishop, and S. J. Julier, 2004: Which is better, an ensemble of positive–negative pairs or a centered spherical simplex ensemble? Mon. Wea. Rev., 132, 1590–1605.

    • Search Google Scholar
    • Export Citation

  • Whitaker, J. S., G. P. Compo, X. Wei, and T. M. Hamill, 2004: Reanalysis without radiosondes using ensemble data assimilation. Mon. Wea. Rev., 132, 1190–1200.

    • Search Google Scholar
    • Export Citation

  • Whitaker, J. S., G. P. Compo, and J. N. Thépaut, 2009: A comparison of variational and ensemble-based data assimilation systems for reanalysis of sparse observations. Mon. Wea. Rev., 137, 1991–1999.

    • Search Google Scholar
    • Export Citation

  • Wolfram Research, Inc., 2008: Mathematica Version 7.0.Wolfram Research, Inc., Champaign, IL. [Available online at http://www.wolfram.com/mathematica/.]

    • Search Google Scholar
    • Export Citation

  • Zupanski, D., 1997: A general weak constraint applicable to operational 4DVar data assimilation systems. Mon. Wea. Rev., 125, 2274–2292.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 567 113 11
PDF Downloads 413 75 10