Filtering of Background Error Variances and Correlations by Local Spatial Averaging: A Review

Loïk Berre CNRM/GAME, Météo-France, CNRS, Toulouse, France

Search for other papers by Loïk Berre in
Current site
Google Scholar
PubMed
Close
and
Gérald Desroziers CNRM/GAME, Météo-France, CNRS, Toulouse, France

Search for other papers by Gérald Desroziers in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

The use of local spatial averaging to estimate and validate background error covariances has received increasing attention recently, in particular in the context of variational data assimilation for global numerical weather prediction. First, theoretical and experimental results are presented to examine spatial structures of sampling noise and signal in ensemble-based variance fields in this context. They indicate that sampling noise tends to be relatively small scale, compared to the signal of interest. This difference in spatial structure motivates the use of spatial averaging techniques.

Based on the usual linear estimation theory, it is shown how this information can be taken into account in order to calculate and apply an objective spatial filter. This kind of approach can also be used to compare and validate ensemble-based variances with innovation-based variances. The use of spatial averaging is even more important for innovation-based variances because local innovations correspond to single error realizations.

Similar ideas can be considered for the estimation of correlation functions. The spatial structures of sampling noise and signal in correlation length scale fields suggest that space-averaging techniques could also be applied to correlation functions. The use of wavelets for this purpose is presented in particular. Connections with related approaches in different contexts such as ensemble Kalman filters and probabilistic forecasting are also discussed.

Corresponding author address: Loïk Berre, CNRM/GAME, Météo-France, CNRS, 42 avenue Coriolis, 31057 Toulouse CEDEX, France. Email: loik.berre@meteo.fr

This article included in the Intercomparisons of 4D-Variational Assimilation and the Ensemble Kalman Filter special collection.

Abstract

The use of local spatial averaging to estimate and validate background error covariances has received increasing attention recently, in particular in the context of variational data assimilation for global numerical weather prediction. First, theoretical and experimental results are presented to examine spatial structures of sampling noise and signal in ensemble-based variance fields in this context. They indicate that sampling noise tends to be relatively small scale, compared to the signal of interest. This difference in spatial structure motivates the use of spatial averaging techniques.

Based on the usual linear estimation theory, it is shown how this information can be taken into account in order to calculate and apply an objective spatial filter. This kind of approach can also be used to compare and validate ensemble-based variances with innovation-based variances. The use of spatial averaging is even more important for innovation-based variances because local innovations correspond to single error realizations.

Similar ideas can be considered for the estimation of correlation functions. The spatial structures of sampling noise and signal in correlation length scale fields suggest that space-averaging techniques could also be applied to correlation functions. The use of wavelets for this purpose is presented in particular. Connections with related approaches in different contexts such as ensemble Kalman filters and probabilistic forecasting are also discussed.

Corresponding author address: Loïk Berre, CNRM/GAME, Météo-France, CNRS, 42 avenue Coriolis, 31057 Toulouse CEDEX, France. Email: loik.berre@meteo.fr

This article included in the Intercomparisons of 4D-Variational Assimilation and the Ensemble Kalman Filter special collection.

Save
  • Anderson, J. L., 2007: Exploring the need for localization in ensemble data assimilation using a hierarchical ensemble filter. Physica D, 230 , 99111.

    • Search Google Scholar
    • Export Citation
  • Andersson, E., and M. Fisher, 1999: Background errors for observed quantities and their propagation in time. Proc. ECMWF Workshop on Diagnosing Data Assimilation Systems, Reading, United Kingdom, ECMWF, 81–90. [Available online at http://www.ecmwf.int/publications/library/do/references/list/17155].

    • Search Google Scholar
    • Export Citation
  • Bannister, R. N., 2008: A review of forecast error covariance statistics in atmospheric variational data assimilation. I: Characteristics and measurements of forecast error covariances. Quart. J. Roy. Meteor. Soc., 134 , 19511970.

    • Search Google Scholar
    • Export Citation
  • Belo Pereira, M., and L. Berre, 2006: The use of an ensemble approach to study the background error covariances in a global NWP model. Mon. Wea. Rev., 134 , 24662489.

    • Search Google Scholar
    • Export Citation
  • Berre, L., 2000: Estimation of synoptic and meso scale forecast error covariances in a limited-area model. Mon. Wea. Rev., 128 , 644667.

    • Search Google Scholar
    • Export Citation
  • Berre, L., S. E. Ştefănescu, and M. Belo Pereira, 2006: The representation of the analysis effect in three error simulation techniques. Tellus, 58A , 196209.

    • Search Google Scholar
    • Export Citation
  • Berre, L., O. Pannekoucke, G. Desroziers, S. E. Ştefănescu, B. Chapnik, and L. Raynaud, 2007: A variational assimilation ensemble and the spatial filtering of its error covariances: Increase of sample size by local spatial averaging. Proc. ECMWF Workshop on Flow-Dependent Aspects of Data Assimilation, Reading, United Kingdom, ECMWF, 151–168. [Available online at http://www.ecmwf.int/publications/library/do/references/list/14092007].

    • Search Google Scholar
    • Export Citation
  • Berre, L., G. Desroziers, L. Raynaud, R. Montroty, and F. Gibier, 2009: Consistent operational ensemble variational assimilation. Extended Abstracts, Fifth WMO Int. Symp. on Data Assimilation, Melbourne, Australia, WMO, N.196.

    • Search Google Scholar
    • Export Citation
  • Berrocal, V. J., A. E. Raftery, and T. Gneiting, 2007: Combining spatial statistical and ensemble information in probabilistic weather forecasts. Mon. Wea. Rev., 135 , 13861402.

    • Search Google Scholar
    • Export Citation
  • Bishop, C. H., and D. Hodyss, 2007: Flow adaptive moderation of spurious ensemble correlations and its use in ensemble-based data assimilation. Quart. J. Roy. Meteor. Soc., 133 , 20292044.

    • Search Google Scholar
    • Export Citation
  • Bishop, C. H., and D. Hodyss, 2009a: Ensemble covariances adaptively localized with ECO-RAP. Part 1: Tests on simple error models. Tellus, 61A , 8496.

    • Search Google Scholar
    • Export Citation
  • Bishop, C. H., and D. Hodyss, 2009b: Ensemble covariances adaptively localized with ECO-RAP. Part 2: A strategy for the atmosphere. Tellus, 61A , 97111.

    • Search Google Scholar
    • Export Citation
  • Boer, G. J., 1983: Homogeneous and isotropic turbulence on the sphere. J. Atmos. Sci., 40 , 154163.

  • Bouttier, F., 1993: The dynamics of error covariances in a barotropic model. Tellus, 45A , 408423.

  • Bouttier, F., 1994: Sur la prévision de la qualité des prévisions météorologiques (The forecast of weather forecast quality). Ph.D. dissertation, Université Paul Sabatier, Toulouse, France, 240 pp. [Available from Université Paul Sabatier, 118 route de Narbonne, 31062 Toulouse CEDEX, France].

  • Buehner, M., 2005: Ensemble-derived stationary and flow-dependent background-error covariances: Evaluation in a quasi-operational NWP setting. Quart. J. Roy. Meteor. Soc., 131 , 10131043.

    • Search Google Scholar
    • Export Citation
  • Buehner, M., and M. Charron, 2007: Spectral and spatial localization of background error correlations for data assimilation. Quart. J. Roy. Meteor. Soc., 133 , 615630.

    • Search Google Scholar
    • Export Citation
  • Courtier, P., and Coauthors, 1998: The ECMWF implementation of three-dimensional variational assimilation (3D-Var). Part I: Formulation. Quart. J. Roy. Meteor. Soc., 124 , 17831807.

    • Search Google Scholar
    • Export Citation
  • Daget, N., A. T. Weaver, and M. A. Balmaseda, 2009: Ensemble estimation of background-error variances in a three-dimensional variational data assimilation system for the global ocean. Quart. J. Roy. Meteor. Soc., 135 , 10711094.

    • Search Google Scholar
    • Export Citation
  • Daley, R., 1991: Atmospheric Data Analysis. Cambridge University Press, 460 pp.

  • Daley, R., 1992: Estimating model-error covariances for application to atmospheric data assimilation. Mon. Wea. Rev., 120 , 17351746.

  • Deckmyn, A., and L. Berre, 2005: A wavelet approach to representing background error covariances in a limited-area model. Mon. Wea. Rev., 133 , 12791294.

    • Search Google Scholar
    • Export Citation
  • Derber, J., and F. Bouttier, 1999: A reformulation of the background error covariance in the ECMWF global data assimilation system. Tellus, 51A , 195221.

    • Search Google Scholar
    • Export Citation
  • Desroziers, G., L. Berre, B. Chapnik, and P. Poli, 2005: Diagnosis of observation, background and analysis error statistics in observation space. Quart. J. Roy. Meteor. Soc., 131 , 33853396.

    • Search Google Scholar
    • Export Citation
  • Drozdov, O., and A. Shepelevskii, 1946: The theory of interpolation in a stochastic field of meteorological elements and its application to meteorological map and network rationalization problems. Trudy Niu Gugms Series, Vol. 1, No. 13, Russian Hydrological and Meteorological Service.

    • Search Google Scholar
    • Export Citation
  • Evensen, G., 1994: Sequential data assimilation with a non-linear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res., 99 , 143162.

    • Search Google Scholar
    • Export Citation
  • Fisher, M., 2003: Background error covariance modelling. Proc. ECMWF Seminar on Recent Developments in Data Assimilation, Reading, United Kingdom, ECMWF, 45–63.

    • Search Google Scholar
    • Export Citation
  • Fisher, M., 2004: On the equivalence between Kalman smoothing and weak-constraint four-dimensional variational data assimilation. ECMWF Tech. Memo. 447, ECMWF, 12 pp.

    • Search Google Scholar
    • Export Citation
  • Fisher, M., and P. Courtier, 1995: Estimating the covariance matrices of analysis and forecast error in variational data assimilation. ECMWF Tech. Memo. 220, ECMWF, 27 pp.

    • Search Google Scholar
    • Export Citation
  • Gandin, L., 1963: Objective Analysis of Meteorological Fields (in Russian). Gridromet, 242 pp.

  • Gaspari, G., and S. E. Cohn, 1999: Construction of correlation functions in two and three dimensions. Quart. J. Roy. Meteor. Soc., 125 , 723757.

    • Search Google Scholar
    • Export Citation
  • Gustafsson, N., L. Berre, S. Hörnquist, X-Y. Huang, M. Lindskog, B. Navasques, K. S. Mogensen, and S. Thorsteinsson, 2001: Three- dimensional variational data assimilation for a limited area model. Part I: General formulation and the background error constraint. Tellus, 53A , 425446.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., J. S. Whitaker, and C. Snyder, 2001: Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter. Mon. Wea. Rev., 129 , 27762790.

    • Search Google Scholar
    • Export Citation
  • Hollingsworth, A., 1987: Objective analysis for numerical weather prediction: Short- and medium-range numerical weather prediction. Proc. WMO/IUGG NWP Symp., Tokyo, Japan, Meteorological Society of Japan, 11–59.

    • Search Google Scholar
    • Export Citation
  • Hollingsworth, A., and P. Lönnberg, 1986: The statistical structure of short-range forecast errors as determined from radiosonde data. Part I: The wind field. Tellus, 38A , 111136.

    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., and H. L. Mitchell, 1998: Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev., 126 , 796811.

    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., and H. L. Mitchell, 2001: A sequential ensemble Kalman filter for atmospheric data assimilation. Mon. Wea. Rev., 129 , 796811.

    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., L. Lefaivre, J. Derome, H. Ritchie, and H. L. Mitchell, 1996: A system simulation approach to ensemble prediction. Mon. Wea. Rev., 124 , 12251242.

    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., H. L. Mitchell, and X. Deng, 2009: Model error representation in an operational ensemble Kalman filter. Mon. Wea. Rev., 137 , 21262143.

    • Search Google Scholar
    • Export Citation
  • Ingleby, N. B., 2001: The statistical structure of forecast errors and its representation in the Met. Office Global 3-D Variational Data Assimilation Scheme. Quart. J. Roy. Meteor. Soc., 127 , 209232.

    • Search Google Scholar
    • Export Citation
  • Isaksen, L., M. Fisher, and J. Berner, 2007: Use of analysis ensembles in estimating flow-dependent background error variance. Proc. ECMWF Workshop on Flow-Dependent Aspects of Data Assimilation, Reading, United Kingdom, ECMWF, 65–86. [Available online at http://www.ecmwf.int/publications/library/do/references/list/14092007].

    • Search Google Scholar
    • Export Citation
  • Kucukkaraca, E., and M. Fisher, 2006: Use of analysis ensembles in estimating flow-dependent background error variances. ECMWF Tech. Memo. 492, ECMWF, 16 pp.

    • Search Google Scholar
    • Export Citation
  • Le Dimet, F-X., and O. Talagrand, 1986: Variational algorithms for analysis and assimilation of meteorological observations: Theoretical aspects. Tellus, 38A , 97110.

    • Search Google Scholar
    • Export Citation
  • Li, H., E. Kalnay, and T. Miyoshi, 2009: Simultaneous estimation of covariance inflation and observation errors within an ensemble Kalman filter. Quart. J. Roy. Meteor. Soc., 135 , 523533.

    • Search Google Scholar
    • Export Citation
  • Lindskog, M., N. Gustafsson, and K. S. Mogensen, 2006: Representation of background error standard deviations in a limited area model data assimilation system. Tellus, 58A , 430444.

    • Search Google Scholar
    • Export Citation
  • Lindskog, M., O. Vignes, N. Gustafsson, T. Landelius, and S. Thorsteinsson, 2007: Background errors in HIRLAM variational data assimilation. Proc. ECMWF Workshop on Flow-Dependent Aspects of Data Assimilation, Reading, United Kingdom, ECMWF, 113–123. [Available online at http://www.ecmwf.int/publications/library/do/references/list/14092007].

    • Search Google Scholar
    • Export Citation
  • Lönnberg, P., and A. Hollingsworth, 1986: The statistical structure of short-range forecast errors as determined from radiosonde data. Part II: The covariance of height and wind errors. Tellus, 38A , 137161.

    • Search Google Scholar
    • Export Citation
  • Lorenc, A. C., 1986: Analysis methods for numerical weather prediction. Quart. J. Roy. Meteor. Soc., 112 , 11771194.

  • Lorenc, A. C., and Coauthors, 2000: The Met. Office global three-dimensional variational data assimilation scheme. Quart. J. Roy. Meteor. Soc., 126 , 29913012.

    • Search Google Scholar
    • Export Citation
  • McNally, A., 2000: Estimates of short-term forecast-temperature error correlations and the implications for radiance-data assimilation. Quart. J. Roy. Meteor. Soc., 126 , 361373.

    • Search Google Scholar
    • Export Citation
  • Monin, A. S., and A. M. Yaglom, 1971: Statistical Fluid Mechanics. Vol. 1, Mechanics of Turbulence, The MIT Press, 782 pp.

  • Pannekoucke, O., 2008: Modélisation des structures locales de covariance des erreurs de prévision à l’aide des ondelettes (Modelling of local covariance structures of forecast errors using wavelets). Ph.D. dissertation, Université Paul Sabatier, 207 pp. [Available from Université Paul Sabatier, 118 route de Narbonne, 31062 Toulouse CEDEX, France].

  • Pannekoucke, O., L. Berre, and G. Desroziers, 2007: Filtering properties of wavelets for local background error correlations. Quart. J. Roy. Meteor. Soc., 133 , 363379.

    • Search Google Scholar
    • Export Citation
  • Pannekoucke, O., L. Berre, and G. Desroziers, 2008: Background-error correlation length-scale estimates and their sampling statistics. Quart. J. Roy. Meteor. Soc., 134 , 497508.

    • Search Google Scholar
    • Export Citation
  • Parrish, D. F., and J. C. Derber, 1992: The National Meteorological Center’s Spectral Statistical Interpolation analysis system. Mon. Wea. Rev., 120 , 17471763.

    • Search Google Scholar
    • Export Citation
  • Rabier, F., A. McNally, E. Andersson, P. Courtier, P. Undén, J. Eyre, A. Hollingsworth, and F. Bouttier, 1998: The ECMWF implementation of three-dimensional variational assimilation (3D-Var). II: Structure functions. Quart. J. Roy. Meteor. Soc., 124 , 18091829.

    • Search Google Scholar
    • Export Citation
  • Raynaud, L., L. Berre, and G. Desroziers, 2008: Spatial averaging of ensemble-based background-error variances. Quart. J. Roy. Meteor. Soc., 134 , 10031014.

    • Search Google Scholar
    • Export Citation
  • Raynaud, L., L. Berre, and G. Desroziers, 2009: Objective filtering of ensemble-based background-error variances. Quart. J. Roy. Meteor. Soc., 135 , 11771199.

    • Search Google Scholar
    • Export Citation
  • Rhodin, A., and H. Anlauf, 2007: Representation of inhomogeneous, non-separable covariances by sparse wavelet-transformed matrices. Proc. ECMWF Workshop on Flow-Dependent Aspects of Data Assimilation, Reading, United Kingdom, ECMWF, 169–183. [Available online at http://www.ecmwf.int/publications/library/do/references/list/14092007].

    • Search Google Scholar
    • Export Citation
  • Rutherford, I. D., 1972: Data assimilation by statistical interpolation of forecast error fields. J. Atmos. Sci., 29 , 809815.

  • Sacher, W., and P. Bartello, 2008: Sampling errors in the ensemble Kalman filtering. Mon. Wea. Rev., 136 , 30353049.

  • Talagrand, O., 1997: Assimilation of observations, an introduction. J. Meteor. Soc. Japan, 75 , 191209.

  • Thépaut, J-N., P. Courtier, G. Belaud, and G. Lemaître, 1996: Dynamical structure functions in a four-dimensional variational assimilation: A case study. Quart. J. Roy. Meteor. Soc., 122 , 535561.

    • Search Google Scholar
    • Export Citation
  • Weaver, A. T., and P. Courtier, 2001: Correlation modelling on the sphere using a generalized diffusion equation. Quart. J. Roy. Meteor. Soc., 127 , 18151846.

    • Search Google Scholar
    • Export Citation
  • Wu, W-S., R. J. Purser, and D. F. Parrish, 2002: Three-dimensional variational analysis with spatially inhomogeneous covariances. Mon. Wea. Rev., 130 , 29052916.

    • Search Google Scholar
    • Export Citation
  • Xu, Q., H. Lu, S. Gao, M. Xue, and M. Tong, 2008: Time-expanded sampling for ensemble Kalman filter: Assimilation experiments with simulated radar observations. Mon. Wea. Rev., 136 , 26512667.

    • Search Google Scholar
    • Export Citation
  • Yang, S. C., E. Kalnay, B. Hunt, and N. E. Bowler, 2009: Weight interpolation for efficient data assimilation with the Local Ensemble Transform Kalman Filter. Quart. J. Roy. Meteor. Soc., 135 , 251262.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 820 318 7
PDF Downloads 298 79 4