Editorial Type:
Article
Article Type:
Research Article

Linear Filtering of Sample Covariances for Ensemble-Based Data Assimilation. Part I: Optimality Criteria and Application to Variance Filtering and Covariance Localization

Benjamin Ménétrier Centre National de Recherches Météorologiques–Groupe d’étude de l’Atmosphère Météorologique, Météo-France/CNRS, Toulouse, France

Search for other papers by Benjamin Ménétrier in
Current site
Google Scholar
PubMed Close
,
Thibaut Montmerle Centre National de Recherches Météorologiques–Groupe d’étude de l’Atmosphère Météorologique, Météo-France/CNRS, Toulouse, France

Search for other papers by Thibaut Montmerle in
Current site
Google Scholar
PubMed Close
,
Yann Michel Centre National de Recherches Météorologiques–Groupe d’étude de l’Atmosphère Météorologique, Météo-France/CNRS, Toulouse, France

Search for other papers by Yann Michel in
Current site
Google Scholar
PubMed Close
, and
Loïk Berre Centre National de Recherches Météorologiques–Groupe d’étude de l’Atmosphère Météorologique, Météo-France/CNRS, Toulouse, France

Search for other papers by Loïk Berre in
Current site
Google Scholar
PubMed Close
Print Publication:
01 May 2015
DOI:
https://doi.org/10.1175/MWR-D-14-00157.1
Page(s):
1622–1643
Restricted access

Abstract

In data assimilation (DA) schemes for numerical weather prediction (NWP) systems, the estimation of forecast error covariances is a key point to get some flow dependency. As shown in previous studies, ensemble data assimilation methods are the most accurate for this task. However, their huge computational cost raises a strong limitation to the ensemble size. Consequently, covariances estimated with small ensembles are affected by random sampling errors. The aim of this study is to develop a theory of covariance filtering in order to remove most of the sampling noise while keeping the signal of interest and then to use it in the DA scheme of a real NWP system. This first part of a two-part study presents the theoretical aspects of such criteria for optimal filtering based on the merging of the theories of optimal linear filtering and of sample centered moments estimation. Its strength relies on the use of sample estimated quantities and filter output only. These criteria pave the way for new algorithms and interesting applications for NWP. Two of them are detailed here: spatial filtering of variances and covariance localization. Results obtained in an idealized 1D analytical framework are shown for illustration. Applications on real forecast error covariances deduced from ensembles at convective scale are discussed in a companion paper.

Corresponding author address: Benjamin Ménétrier, CNRM-GAME/GMAP, 42 avenue G. Coriolis, 31057 Toulouse, France. E-mail: benjamin.menetrier@meteo.fr

Abstract

In data assimilation (DA) schemes for numerical weather prediction (NWP) systems, the estimation of forecast error covariances is a key point to get some flow dependency. As shown in previous studies, ensemble data assimilation methods are the most accurate for this task. However, their huge computational cost raises a strong limitation to the ensemble size. Consequently, covariances estimated with small ensembles are affected by random sampling errors. The aim of this study is to develop a theory of covariance filtering in order to remove most of the sampling noise while keeping the signal of interest and then to use it in the DA scheme of a real NWP system. This first part of a two-part study presents the theoretical aspects of such criteria for optimal filtering based on the merging of the theories of optimal linear filtering and of sample centered moments estimation. Its strength relies on the use of sample estimated quantities and filter output only. These criteria pave the way for new algorithms and interesting applications for NWP. Two of them are detailed here: spatial filtering of variances and covariance localization. Results obtained in an idealized 1D analytical framework are shown for illustration. Applications on real forecast error covariances deduced from ensembles at convective scale are discussed in a companion paper.

Corresponding author address: Benjamin Ménétrier, CNRM-GAME/GMAP, 42 avenue G. Coriolis, 31057 Toulouse, France. E-mail: benjamin.menetrier@meteo.fr
Save
Cover Monthly Weather Review
Monthly Weather Review

  • Anderson, J. L., 2007: Exploring the need for localization in ensemble data assimilation using a hierarchical ensemble filter. Physica D, 230, 99–111, doi:10.1016/j.physd.2006.02.011.

    • Search Google Scholar
    • Export Citation

  • Anderson, J. L., and L. Lei, 2013: Empirical localization of observation impact in ensemble Kalman filters. Mon. Wea. Rev., 141, 4140–4153, doi:10.1175/MWR-D-12-00330.1.

    • Search Google Scholar
    • Export Citation

  • Bannister, R. N., 2008a: A review of forecast error covariance statistics in atmospheric variational data assimilation. I: Characteristics and measurements of forecast error covariances. Quart. J. Roy. Meteor. Soc., 134, 1951–1970, doi:10.1002/qj.339.

    • Search Google Scholar
    • Export Citation

  • Bannister, R. N., 2008b: A review of forecast error covariance statistics in atmospheric variational data assimilation. II: Modelling the forecast error covariance statistics. Quart. J. Roy. Meteor. Soc., 134, 1971–1996, doi:10.1002/qj.340.

    • Search Google Scholar
    • Export Citation

  • Belo Pereira, M., and L. Berre, 2006: The use of an ensemble approach to study the background error covariances in a global NWP model. Mon. Wea. Rev., 134, 2466–2489, doi:10.1175/MWR3189.1.

    • Search Google Scholar
    • Export Citation

  • Berre, L., 2000: Estimation of synoptic and mesoscale forecast error covariances in a limited-area model. Mon. Wea. Rev., 128, 644–667, doi:10.1175/1520-0493(2000)128<0644:EOSAMF>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Berre, L., and G. Desroziers, 2010: Filtering of background error variances and correlations by local spatial averaging: A review. Mon. Wea. Rev., 138, 3693–3720, doi:10.1175/2010MWR3111.1.

    • Search Google Scholar
    • Export Citation

  • Berre, L., O. Pannekoucke, G. Desroziers, S. Stefanescu, B. Chapnik, and L. Raynaud, 2007: A variational assimilation ensemble and the spatial filtering of its error covariances: Increase of sample size by local spatial averaging. Proc. ECMWF Workshop on Flow-Dependent Aspects of Data Assimilation, Reading, United Kingdom, ECMWF, 151–168.

  • Bishop, C. H., and D. Hodyss, 2007: Flow-adaptive moderation of spurious ensemble correlations and its use in ensemble-based data assimilation. Quart. J. Roy. Meteor. Soc., 133, 2029–2044, doi:10.1002/qj.169.

    • Search Google Scholar
    • Export Citation

  • Bishop, C. H., and D. Hodyss, 2009a: Ensemble covariances adaptively localized with ECO-RAP. Part 1: Tests on simple error models. Tellus, 61A, 84–96, doi:10.1111/j.1600-0870.2008.00371.x.

    • Search Google Scholar
    • Export Citation

  • Bishop, C. H., and D. Hodyss, 2009b: Ensemble covariances adaptively localized with ECO-RAP. Part 2: A strategy for the atmosphere. Tellus, 61A, 97–111, doi:10.1111/j.1600-0870.2008.00372.x.

    • Search Google Scholar
    • Export Citation

  • Bonavita, M., L. Isaksen, and E. Holm, 2012: On the use of EDA background error variances in the ECMWF 4D-Var. Quart. J. Roy. Meteor. Soc., 138, 1540–1559, doi:10.1002/qj.1899.

    • Search Google Scholar
    • Export Citation

  • Brousseau, P., L. Berre, F. Bouttier, and G. Desroziers, 2012: Flow-dependent background-error covariances for a convective-scale data assimilation system. Quart. J. Roy. Meteor. Soc., 138, 310–322, doi:10.1002/qj.920.

    • Search Google Scholar
    • Export Citation

  • Buehner, M., 2005: Ensemble-derived stationary and flow-dependent background-error covariances: Evaluation in a quasi-operational NWP setting. Quart. J. Roy. Meteor. Soc., 131, 1013–1043, doi:10.1256/qj.04.15.

    • Search Google Scholar
    • Export Citation

  • Buehner, M., and M. Charron, 2007: Spectral and spatial localization of background-error correlations for data assimilation. Quart. J. Roy. Meteor. Soc., 133, 615–630, doi:10.1002/qj.50.

    • Search Google Scholar
    • Export Citation

  • Caron, J.-F., and L. Fillion, 2010: An examination of background error correlations between mass and rotational wind over precipitation regions. Mon. Wea. Rev., 138, 563–578, doi:10.1175/2009MWR2998.1.

    • Search Google Scholar
    • Export Citation

  • Chatfield, C., 2003: The Analysis of Time Series: An Introduction. Chapman and Hall/CRC, 333 pp.

  • Daley, R., 1991: Atmospheric Data Analysis. Cambridge University Press, 460 pp.

  • Deckmyn, A., and L. Berre, 2005: A wavelet approach to representing background error covariances in a limited-area model. Mon. Wea. Rev., 133, 1279–1294, doi:10.1175/MWR2929.1.

    • Search Google Scholar
    • Export Citation

  • Dee, D., 2005: Bias and data assimilation. Quart. J. Roy. Meteor. Soc., 131, 3323–3343, doi:10.1256/qj.05.137.

  • Dodge, Y., and V. Rousson, 1999: The complications of the fourth central moment. Amer. Stat., 53, 267–269, doi:10.1080/00031305.1999.10474471.

    • Search Google Scholar
    • Export Citation

  • Evensen, G., 1994: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte-Carlo methods to forecast error statistics. J. Geophys. Res., 99, 10 143–10 162, doi:10.1029/94JC00572.

    • Search Google Scholar
    • Export Citation

  • Fisher, M., 2003: Background error covariance modelling. ECMWF Seminar on Recent Developments in Data Assimilation for Atmosphere and Ocean, Reading, United Kingdom, ECMWF, 45–63.

  • Gaspari, G., and S. E. Cohn, 1999: Construction of correlation functions in two and three dimensions. Quart. J. Roy. Meteor. Soc., 125, 723–757, doi:10.1002/qj.49712555417.

    • Search Google Scholar
    • Export Citation

  • Hamill, T. M., and C. Snyder, 2000: A hybrid ensemble Kalman filter/3D variational analysis scheme. Mon. Wea. Rev., 128, 2905–2919, doi:10.1175/1520-0493(2000)128<2905:AHEKFV>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Hamill, T. M., J. S. Whitaker, and C. Snyder, 2001: Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter. Mon. Wea. Rev., 129, 2776–2790, doi:10.1175/1520-0493(2001)129<2776:DDFOBE>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Houtekamer, P. L., and H. L. Mitchell, 1998: Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev., 126, 796–811, doi:10.1175/1520-0493(1998)126<0796:DAUAEK>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Houtekamer, P. L., and H. L. Mitchell, 2001: A sequential ensemble Kalman filter for atmospheric data assimilation. Mon. Wea. Rev., 129, 123–137, doi:10.1175/1520-0493(2001)129<0123:ASEKFF>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Isserlis, L., 1916: On certain probable errors and correlation coefficients of multiple frequency distributions with skew regression. Biometrika, 11, 185–190, doi:10.1093/biomet/11.3.185.

    • Search Google Scholar
    • Export Citation

  • Khintchine, A., 1934: Korrelationstheorie der stationären stochastischen prozesse. Math. Ann., 109, 604–615, doi:10.1007/BF01449156.

  • Lorenc, A. C., 2003: The potential of the ensemble Kalman filter for NWP—A comparison with 4D-Var. Quart. J. Roy. Meteor. Soc., 129, 3183–3203, doi:10.1256/qj.02.132.

    • Search Google Scholar
    • Export Citation

  • Ménétrier, B., and T. Montmerle, 2011: Heterogeneous background-error covariances for the analysis and forecast of fog events. Quart. J. Roy. Meteor. Soc., 137, 2004–2013, doi:10.1002/qj.802.

    • Search Google Scholar
    • Export Citation

  • Ménétrier, B., T. Montmerle, L. Berre, and Y. Michel, 2014: Estimation and diagnosis of heterogeneous flow-dependent background-error covariances at the convective scale using either large or small ensembles. Quart. J. Roy. Meteor. Soc., 140, 2050–2061, doi:10.1002/qj.2267.

    • Search Google Scholar
    • Export Citation

  • Ménétrier, B., T. Montmerle, Y. Michel, and L. Berre, 2015: Linear filtering of sample covariances for ensemble-based data assimilation. Part II: Application to a convective-scale NWP model. Mon. Wea. Rev., 143, 1656–1676, doi:10.1175/MWR-D-14-00156.1.

    • Search Google Scholar
    • Export Citation

  • Michel, Y., 2013: Estimating deformations of random processes for correlation modelling in a limited area model. Quart. J. Roy. Meteor. Soc., 139, 534–547, doi:10.1002/qj.1978.

    • Search Google Scholar
    • Export Citation

  • Michel, Y., and T. Auligné, 2010: Inhomogeneous background error modeling and estimation over Antarctica. Mon. Wea. Rev., 138, 2229–2252, doi:10.1175/2009MWR3139.1.

    • Search Google Scholar
    • Export Citation

  • Monteiro, M., and L. Berre, 2010: A diagnostic study of time variations of regionally averaged background error covariances. J. Geophys. Res., 115, D23203, doi:10.1029/2010JD014095.

    • Search Google Scholar
    • Export Citation

  • Montmerle, T., and L. Berre, 2010: Diagnosis and formulation of heterogeneous background-error covariances at the mesoscale. Quart. J. Roy. Meteor. Soc., 136, 1408–1420, doi:10.1002/qj.655.

    • Search Google Scholar
    • Export Citation

  • Muirhead, R. I., 2005: Aspects of Multivariate Statistical Theory. John Wiley & Sons, 673 pp.

  • Purser, R. J., W.-S. Wu, D. F. Parrish, and N. M. Roberts, 2003: Numerical aspects of the application of recursive filters to variational statistical analysis. Part II: Spatially inhomogeneous and anisotropic general covariances. Mon. Wea. Rev., 131, 1536–1548, doi:10.1175//2543.1.

    • Search Google Scholar
    • Export Citation

  • Raynaud, L., L. Berre, and G. Desroziers, 2009: Objective filtering of ensemble-based background-error variances. Quart. J. Roy. Meteor. Soc., 135, 1177–1199, doi:10.1002/qj.438.

    • Search Google Scholar
    • Export Citation

  • Schur, J., 1911: Bemerkungen zur theorie der beschränkten bilinearformen mit unendlich vielen veränderlichen. J. Reine Angew. Math.,140, 1–28.

  • Varella, H., L. Berre, and G. Desroziers, 2011: Diagnostic and impact studies of a wavelet formulation of background-error correlations in a global model. Quart. J. Roy. Meteor. Soc., 137, 1369–1379, doi:10.1002/qj.845.

    • Search Google Scholar
    • Export Citation

  • Vignat, C., 2012: A generalized Isserlis theorem for location mixtures of Gaussian random vectors. Stat. Probab. Lett., 82, 67–71, doi:10.1016/j.spl.2011.09.008.

    • Search Google Scholar
    • Export Citation

  • Weaver, A. T., and I. Mirouze, 2013: On the diffusion equation and its application to isotropic and anisotropic correlation modelling in variational assimilation. Quart. J. Roy. Meteor. Soc., 139, 242–260, doi:10.1002/qj.1955.

    • Search Google Scholar
    • Export Citation

  • Wiener, N., 1930: Generalized harmonic analysis. Acta Math., 55, 117–258, doi:10.1007/BF02546511.

  • Wiener, N., 1949: Extrapolation, Interpolation, and Smoothing of Stationary Time Series, with Engineering Applications. MIT Press, 163 pp.

  • Wishart, J., 1928: The generalised product moment distribution in samples from a normal multivariate population. Biometrika, 20A, 32–52, doi:10.2307/2331939.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 1894 978 34
PDF Downloads 788 113 17