Relationships among Four-Dimensional Hybrid Ensemble–Variational Data Assimilation Algorithms with Full and Approximate Ensemble Covariance Localization

Chengsi Liu Center for Analysis and Prediction of Storms, University of Oklahoma, Norman, Oklahoma

Search for other papers by Chengsi Liu in
Current site
Google Scholar
PubMed
Close
and
Ming Xue Center for Analysis and Prediction of Storms, and School of Meteorology, University of Oklahoma, Norman, Oklahoma

Search for other papers by Ming Xue in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Ensemble–variational data assimilation algorithms that can incorporate the time dimension (four-dimensional or 4D) and combine static and ensemble-derived background error covariances (hybrid) are formulated in general forms based on the extended control variable and the observation-space-perturbation approaches. The properties and relationships of these algorithms and their approximated formulations are discussed. The main algorithms discussed include the following: 1) the standard ensemble 4DVar (En4DVar) algorithm incorporating ensemble-derived background error covariance through the extended control variable approach, 2) the 4DEnVar neglecting the time propagation of the extended control variable (4DEnVar-NPC), 3) the 4D ensemble–variational algorithm based on observation space perturbation (4DEnVar), and 4) the 4DEnVar with no propagation of covariance localization (4DEnVar-NPL). Without the static background error covariance term, none of the algorithms requires the adjoint model except for En4DVar. Costly applications of the tangent linear model to localized ensemble perturbations can be avoided by making the NPC and NPL approximations. It is proven that En4DVar and 4DEnVar are mathematically equivalent, while 4DEnVar-NPC and 4DEnVar-NPL are mathematically equivalent. Such equivalences are also demonstrated by single-observation assimilation experiments with a 1D linear advection model. The effects of the non-flow-following or stationary localization approximations are also examined through the experiments.

All of the above algorithms can include the static background error covariance term to establish a hybrid formulation. When the static term is included, all algorithms will require a tangent linear model and an adjoint model. The first guess at appropriate time (FGAT) approximation is proposed to avoid the tangent linear and adjoint models. Computational costs of the algorithms are also discussed.

Corresponding author address: Dr. Ming Xue, Center for Analysis and Prediction of Storms, 120 David Boren Blvd., Norman, OK 73072. E-mail: mxue@ou.edu

This article is included in the Sixth WMO Data Assimilation Symposium Special Collection.

Abstract

Ensemble–variational data assimilation algorithms that can incorporate the time dimension (four-dimensional or 4D) and combine static and ensemble-derived background error covariances (hybrid) are formulated in general forms based on the extended control variable and the observation-space-perturbation approaches. The properties and relationships of these algorithms and their approximated formulations are discussed. The main algorithms discussed include the following: 1) the standard ensemble 4DVar (En4DVar) algorithm incorporating ensemble-derived background error covariance through the extended control variable approach, 2) the 4DEnVar neglecting the time propagation of the extended control variable (4DEnVar-NPC), 3) the 4D ensemble–variational algorithm based on observation space perturbation (4DEnVar), and 4) the 4DEnVar with no propagation of covariance localization (4DEnVar-NPL). Without the static background error covariance term, none of the algorithms requires the adjoint model except for En4DVar. Costly applications of the tangent linear model to localized ensemble perturbations can be avoided by making the NPC and NPL approximations. It is proven that En4DVar and 4DEnVar are mathematically equivalent, while 4DEnVar-NPC and 4DEnVar-NPL are mathematically equivalent. Such equivalences are also demonstrated by single-observation assimilation experiments with a 1D linear advection model. The effects of the non-flow-following or stationary localization approximations are also examined through the experiments.

All of the above algorithms can include the static background error covariance term to establish a hybrid formulation. When the static term is included, all algorithms will require a tangent linear model and an adjoint model. The first guess at appropriate time (FGAT) approximation is proposed to avoid the tangent linear and adjoint models. Computational costs of the algorithms are also discussed.

Corresponding author address: Dr. Ming Xue, Center for Analysis and Prediction of Storms, 120 David Boren Blvd., Norman, OK 73072. E-mail: mxue@ou.edu

This article is included in the Sixth WMO Data Assimilation Symposium Special Collection.

Save
  • Anderson, J. L., 2001: An ensemble adjustment Kalman filter for data assimilation. Mon. Wea. Rev., 129, 28842903, doi:10.1175/1520-0493(2001)129<2884:AEAKFF>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Bishop, C. H., and D. Hodyss, 2007: Flow-adaptive moderation of spurious ensemble correlations and its use in ensemble-based data assimilation. Quart. J. Roy. Meteor. Soc., 133, 20292044, doi:10.1002/qj.169.

    • Search Google Scholar
    • Export Citation
  • Bishop, C. H., and D. Hodyss, 2009: Ensemble covariances adaptively localized with ECO-RAP. Part 1: Tests on simple error models. Tellus, 61A, 8496, doi:10.1111/j.1600-0870.2008.00371.x.

    • Search Google Scholar
    • Export Citation
  • Bishop, C. H., and D. Hodyss, 2011: Adaptive ensemble covariance localization in ensemble 4D-VAR state estimation. Mon. Wea. Rev., 139, 12411255, doi:10.1175/2010MWR3403.1.

    • Search Google Scholar
    • Export Citation
  • Bishop, C. H., B. J. Etherton, and S. J. Majumdar, 2001: Adaptive sampling with the ensemble transform Kalman filter. Part I: Theoretical aspects. Mon. Wea. Rev., 129, 420436, doi:10.1175/1520-0493(2001)129<0420:ASWTET>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Buehner, M., 2005: Ensemble-derived stationary and flow-dependent background error covariances: Evaluation in a quasi-operation NWP setting. Quart. J. Roy. Meteor. Soc., 131, 10131043, doi:10.1256/qj.04.15.

    • Search Google Scholar
    • Export Citation
  • Buehner, M., P. L. Houtekamer, C. Charette, H. L. Mitchell, and B. He, 2010a: Intercomparison of variational data assimilation and the ensemble Kalman filter for global deterministic NWP. Part I: Description and single-observation experiments. Mon. Wea. Rev., 138, 15501566, doi:10.1175/2009MWR3157.1.

    • Search Google Scholar
    • Export Citation
  • Buehner, M., P. L. Houtekamer, C. Charette, H. L. Mitchell, and B. He, 2010b: Intercomparison of variational data assimilation and the ensemble Kalman filter for global deterministic NWP. Part II: One-month experiments with real observations. Mon. Wea. Rev., 138, 15671586, doi:10.1175/2009MWR3158.1.

    • Search Google Scholar
    • Export Citation
  • Buehner, M., J. Morneau, and C. Charette, 2013: Four-dimensional ensemble-variational data assimilation for global deterministic weather prediction. Nonlinear Processes Geophys., 20, 669682, doi:10.5194/npg-20-669-2013.

    • Search Google Scholar
    • Export Citation
  • Burgers, G., P. J. van Leeuwen, and G. Evensen, 1998: Analysis scheme in the ensemble Kalman filter. Mon. Wea. Rev., 126, 17191724, doi:10.1175/1520-0493(1998)126<1719:ASITEK>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Campbell, W. F., C. H. Bishop, and D. Hodyss, 2010: Vertical covariance localization for satellite radiances in ensemble Kalman filters. Mon. Wea. Rev., 138, 282290, doi:10.1175/2009MWR3017.1.

    • Search Google Scholar
    • Export Citation
  • Clayton, A. M., A. C. Lorenc, and D. M. Barker, 2013: Operational implementation of a hybrid ensemble/4D-Var global data assimilation system at the Met Office. Quart. J. Roy. Meteor. Soc., 139, 14451461, doi:10.1002/qj.2054.

    • Search Google Scholar
    • Export Citation
  • Courtier, P., J. N. Thepaut, and A. Hollingsworth, 1994: A strategy for operational implementation of 4D-Var, using an incremental approach. Quart. J. Roy. Meteor. Soc., 120, 13671387, doi:10.1002/qj.49712051912.

    • Search Google Scholar
    • Export Citation
  • Dee, D. P., and S. Uppala, 2009: Variational bias correction of satellite radiance data in the ERA-Interim reanalysis. Quart. J. Roy. Meteor. Soc., 135, 18301841, doi:10.1002/qj.493.

    • Search Google Scholar
    • Export Citation
  • Desroziers, G., J.-T. Camino, and L. Berre, 2014: 4DEnVar: Link with 4D state formulation of variational assimilation and different possible implementations. Quart. J. Roy. Meteor. Soc., 140, 20972110, doi:10.1002/qj.2325.

    • Search Google Scholar
    • Export Citation
  • Evensen, G., 1992: Using the extended Kalman filter with a multi-layer quasi-geostrophic ocean model. J. Geophys. Res., 97, 17 90517 924, doi:10.1029/92JC01972.

    • Search Google Scholar
    • Export Citation
  • Evensen, G., 1994: Sequential data assimilation with a nonlinear quasi-geostrophic model using Montre Carlo methods to forecast error statistics. J. Geophys. Res., 99, 10 14310 162, doi:10.1029/94JC00572.

    • Search Google Scholar
    • Export Citation
  • Evensen, G., 2003: The ensemble Kalman filter: Theoretical formulation and practical implementation. Ocean Dyn., 53, 343367, doi:10.1007/s10236-003-0036-9.

    • Search Google Scholar
    • Export Citation
  • Fairbairn, D., S. R. Pring, A. C. Lorenc, and I. Roulstone, 2014: A comparison of 4D-Var with ensemble data assimilation methods. Quart. J. Roy. Meteor. Soc., 140, 281294, doi:10.1002/qj.2135.

    • Search Google Scholar
    • Export Citation
  • Fisher, M., 2003: Background error covariance modelling. Proc. ECMWF Seminar on Recent Developments in Data Assimilation for Atmosphere and Ocean, Reading, United Kingdom, ECMWF, 4553.

  • Gauthier, P., and J.-N. Thépaut, 2001: Impact of the digital filter as a weak constraint in the preoperational 4DVAR assimilation system of Météo-France. Mon. Wea. Rev., 129, 20892102, doi:10.1175/1520-0493(2001)129<2089:IOTDFA>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Gustafsson, N., 2007: Discussion on ‘4D-Var or EnKF?’ Tellus, 59A, 774777, doi:10.1111/j.1600-0870.2007.00262.x.

  • Hamill, T. M., and C. Snyder, 2000: A hybrid ensemble Kalman filter-3D variational analysis scheme. Mon. Wea. Rev., 128, 29052919, doi:10.1175/1520-0493(2000)128<2905:AHEKFV>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., J. S. Whitaker, and C. Snyder, 2001: Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter. Mon. Wea. Rev., 129, 27762790, doi:10.1175/1520-0493(2001)129<2776:DDFOBE>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., J. S. Whitaker, M. Fiorino, and S. G. Benjamin, 2011a: Global ensemble predictions of 2009’s tropical cyclones initialized with an ensemble Kalman filter. Mon. Wea. Rev., 139, 668688, doi:10.1175/2010MWR3456.1.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., J. S. Whitaker, D. T. Kleist, M. Fiorino, and S. G. Benjamin, 2011b: Predictions of 2010’s tropical cyclones using the GFS and ensemble-based data assimilation methods. Mon. Wea. Rev., 139, 32433247, doi:10.1175/MWR-D-11-00079.1.

    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., and H. L. Mitchell, 1998: Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev., 126, 796811, doi:10.1175/1520-0493(1998)126<0796:DAUAEK>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., and H. L. Mitchell, 2005: Ensemble Kalman filtering. Quart. J. Roy. Meteor. Soc., 131, 32693289, doi:10.1256/qj.05.135.

    • Search Google Scholar
    • Export Citation
  • Hunt, B. R., and Coauthors, 2004: Four-dimensional ensemble Kalman filtering. Tellus, 56A, 273277, doi:10.1111/j.1600-0870.2004.00066.x.

    • Search Google Scholar
    • Export Citation
  • Hunt, B. R., E. Kostelich, and I. Syzunogh, 2007: Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filter. Physica D, 230, 112126, doi:10.1016/j.physd.2006.11.008.

    • Search Google Scholar
    • Export Citation
  • Kalnay, E., H. Li, M. Takemsa, S.-C. Yang, and J. Ballabrera-Poy, 2007a: 4DVAR or ensemble Kalman filter. Tellus, 59A, 758773, doi:10.1111/j.1600-0870.2007.00261.x.

    • Search Google Scholar
    • Export Citation
  • Kalnay, E., H. Li, M. Takemsa, S.-C. Yang, and J. Ballabrera-Poy, 2007b: Response to the discussion on “4-D-Var or EnKF?” by Nils Gustafsson. Tellus, 59A, 778780, doi:10.1111/j.1600-0870.2007.00263.x.

    • Search Google Scholar
    • Export Citation
  • Kleist, D. T., and K. Ide, 2015a: An OSSE-based evaluation of hybrid variational–ensemble data assimilation for the NCEP GFS. Part I: System description and 3D-hybrid results. Mon. Wea. Rev., 143, 433451, doi:10.1175/MWR-D-13-00351.1.

    • Search Google Scholar
    • Export Citation
  • Kleist, D. T., and K. Ide, 2015b: An OSSE-based evaluation of hybrid variational–ensemble data assimilation for the NCEP GFS. Part II: 4D EnVar and hybrid variants. Mon. Wea. Rev., 143, 452470, doi:10.1175/MWR-D-13-00350.1.

    • Search Google Scholar
    • Export Citation
  • Kleist, D. T., D. F. Parrish, J. C. Derber, R. Treadon, W.-S. Wu, and S. Lord, 2009: Introduction of the GSI into the NCEP Global Data Assimilation System. Wea. Forecasting, 24, 16911705, doi:10.1175/2009WAF2222201.1.

    • Search Google Scholar
    • Export Citation
  • Kuhl, D., T. E. Rosmond, C. H. Bishop, J. McClay, and N. L. Baker, 2013: Comparison of hybrid ensemble/4DVar and 4DVar within the NAVDAS-AR data assimilation framework. Mon. Wea. Rev., 141, 27402758, doi:10.1175/MWR-D-12-00182.1.

    • Search Google Scholar
    • Export Citation
  • Le Dimet, F. X., and O. Talagrand, 1986: Variational algorithms for analysis and assimilation of meteorological observations: Theoretical aspects. Tellus, 38A, 97110, doi:10.1111/j.1600-0870.1986.tb00459.x.

    • Search Google Scholar
    • Export Citation
  • Liu, C., and Q. Xiao, 2013: An ensemble-based four-dimensional variational data assimilation scheme. Part III: Antarctic applications with advanced research WRF using real data. Mon. Wea. Rev., 141, 27212739, doi:10.1175/MWR-D-12-00130.1.

    • Search Google Scholar
    • Export Citation
  • Liu, C., and M. Xue, 2013: A unified framework for four-dimensional ensemble-variational hybrid data assimilation: Relationships among ensemble variational algorithms with full and approximate ensemble covariance localization. Proc. Sixth WMO Symp. on Data Assimilation, College Park, MD, WMO.

  • Liu, C., Q. Xiao, and B. Wang, 2008: An ensemble-based four-dimensional variational data assimilation scheme. Part I: Technical formulation and preliminary test. Mon. Wea. Rev., 136, 33633373, doi:10.1175/2008MWR2312.1.

    • Search Google Scholar
    • Export Citation
  • Liu, C., Q.-N. Xiao, and B. Wang, 2009: An ensemble-based four-dimensional variational data assimilation scheme. Part II: Observing system simulation experiments with the Advanced Research WRF (ARW). Mon. Wea. Rev., 137, 16871704, doi:10.1175/2008MWR2699.1.

    • Search Google Scholar
    • Export Citation
  • Liu, Z., and F. Rabier, 2003: The potential of high-density observations for numerical weather prediction: A study with simulated observations. Quart. J. Roy. Meteor. Soc., 129, 30133035, doi:10.1256/qj.02.170.

    • Search Google Scholar
    • Export Citation
  • Lorenc, A. C., 2003: The potential of the ensemble Kalman filter for NWP—A comparison with 4D-Var. Quart. J. Roy. Meteor. Soc., 129, 31833204, doi:10.1256/qj.02.132.

    • Search Google Scholar
    • Export Citation
  • Lorenc, A. C., 2013: Recommended nomenclature for EnVar data assimilation methods. Research Activities in Atmospheric and Oceanic Modeling, WGNE, 2 pp. [Available online at http://www.wcrp-climate.org/WGNE/BlueBook/2013/individual-articles/01_Lorenc_Andrew_EnVar_nomenclature.pdf.]

  • Lorenc, A. C., and Coauthors, 2000: The Met Office global 3-dimensional variational data assimilation scheme. Quart. J. Roy. Meteor. Soc., 126, 29913012, doi:10.1002/qj.49712657002.

    • Search Google Scholar
    • Export Citation
  • Lorenc, A. C., N. E. Bowler, A. M. Clayton, S. R. Pring, and D. Fairbairn, 2015: Comparison of hybrid-4DEnVar and hybrid-4DVar data assimilation methods for global NWP. Mon. Wea. Rev., 143, 212229, doi:10.1175/MWR-D-14-00195.1.

    • Search Google Scholar
    • Export Citation
  • Massart, S., B. Pajot, A. Piacentini, and O. Pannekoucke, 2010: On the merits of using a 3D-FGAT assimilation scheme with an outer loop for atmospheric situations governed by transport. Mon. Wea. Rev., 138, 45094522, doi:10.1175/2010MWR3237.1.

    • Search Google Scholar
    • Export Citation
  • Ota, Y., J. C. Derber, E. Kalnay, and T. Miyoshi, 2013: Ensemble-based observation impact estimates using the NCEP GFS. Tellus, 65A, 20038, doi:10.3402/tellusa.v65i0.20038.

    • Search Google Scholar
    • Export Citation
  • Pan, Y., K. Zhu, M. Xue, X. Wang, M. Hu, S. G. Benjamin, S. S. Weygandt, and J. S. Whitaker, 2014: A GSI-based coupled EnSRF–En3DVar hybrid data assimilation system for the operational rapid refresh model: Tests at a reduced resolution. Mon. Wea. Rev., 142, 37563780, doi:10.1175/MWR-D-13-00242.1.

    • Search Google Scholar
    • Export Citation
  • Parrish, D., and J. Derber, 1992: The National Meteorological Center’s spectral statistical interpolation analysis system. Mon. Wea. Rev., 120, 17471763, doi:10.1175/1520-0493(1992)120<1747:TNMCSS>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Purser, R. J., W.-S. Wu, D. F. Parrish, and N. M. Roberts, 2003a: Numerical aspects of the application of recursive filters to variational statistical analysis. Part I: Spatially homogeneous and isotropic Gaussian covariances. Mon. Wea. Rev., 131, 15241535, doi:10.1175//1520-0493(2003)131<1524:NAOTAO>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Purser, R. J., W.-S. Wu, D. F. Parrish, and N. M. Roberts, 2003b: Numerical aspects of the application of recursive filters to variational statistical analysis. Part II: Spatially inhomogeneous and anisotropic general covariances. Mon. Wea. Rev., 131, 15361548, doi:10.1175//2543.1.

    • Search Google Scholar
    • Export Citation
  • Rabier, F., H. Järvinen, E. Klinker, J.-F. Mahfouf, and A. Simmons, 2000: The ECMWF operational implementation of four-dimensional variational assimilation. I: Experimental results with simplified physics. Quart. J. Roy. Meteor. Soc., 126, 11431170, doi:10.1002/qj.49712656415.

    • Search Google Scholar
    • Export Citation
  • Rawlins, F., S. P. Ballard, K. J. Bovis, A. M. Clayton, D. Li, G. W. Inverarity, A. C. Lorenc, and T. J. Payne, 2007: The Met Office global four-dimensional variational data assimilation scheme. Quart. J. Roy. Meteor. Soc., 133, 347362, doi:10.1002/qj.32.

    • Search Google Scholar
    • Export Citation
  • Sakov, P., G. Evensen, and L. Bertino, 2010: Asynchronous data assimilation with the EnKF. Tellus, 62A, 2429, doi:10.1111/j.1600-0870.2009.00417.x.

    • Search Google Scholar
    • Export Citation
  • Sun, J., and N. A. Crook, 1997: Dynamical and microphysical retrieval from Doppler radar observations using a cloud model and its adjoint. Part I: Model development and simulated data experiments. J. Atmos. Sci., 54, 16421661, doi:10.1175/1520-0469(1997)054<1642:DAMRFD>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Talagrand, O., and P. Courtier, 1987: Variational assimilation of meteorological observations with the adjoint vorticity equation. I: Theory. Quart. J. Roy. Meteor. Soc., 113, 13111328, doi:10.1002/qj.49711347812.

    • Search Google Scholar
    • Export Citation
  • Tanguay, M., L. Fillion, E. Lapalme, and M. Lajoie, 2012: Four-dimensional variational data assimilation for the Canadian Regional Deterministic Prediction System. Mon. Wea. Rev., 140, 15171538, doi:10.1175/MWR-D-11-00160.1.

    • Search Google Scholar
    • Export Citation
  • Wang, S., M. Xue, and J. Min, 2013: A four-dimensional asynchronous ensemble square-root filter (4DEnSRF) algorithm and tests with simulated radar data. Quart. J. Roy. Meteor. Soc., 139, 805819, doi:10.1002/qj.1987.

    • Search Google Scholar
    • Export Citation
  • Wang, X., C. Snyder, and T. M. Hamill, 2007: On the theoretical equivalence of differently proposed ensemble/VAR hybrid analysis schemes. Mon. Wea. Rev., 135, 222227, doi:10.1175/MWR3282.1.

    • Search Google Scholar
    • Export Citation
  • Wang, X., D. Parrish, D. Kleist, and J. Whitaker, 2013: GSI 3DVar-based ensemble–variational hybrid data assimilation for NCEP Global Forecast System: Single-resolution experiments. Mon. Wea. Rev., 141, 40984117, doi:10.1175/MWR-D-12-00141.1.

    • Search Google Scholar
    • Export Citation
  • Whitaker, J. S., and T. M. Hamill, 2002: Ensemble data assimilation without perturbed observations. Mon. Wea. Rev., 130, 19131924, doi:10.1175/1520-0493(2002)130<1913:EDAWPO>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Whitaker, J. S., T. M. Hamill, X. Wei, Y. Song, and Z. Toth, 2008: Ensemble data assimilation with the NCEP Global Forecast System. Mon. Wea. Rev., 136, 463482, doi:10.1175/2007MWR2018.1.

    • Search Google Scholar
    • Export Citation
  • Wu, W.-S., R. J. Purser, and D. F. Parrish, 2002: Three-dimensional variational analysis with spatially inhomogeneous covariances. Mon. Wea. Rev., 130, 29052916, doi:10.1175/1520-0493(2002)130<2905:TDVAWS>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Zhang, M., and F. Zhang, 2012: E4DVar: Coupling an ensemble Kalman filter with four-dimensional variational data assimilation in a limited-area weather prediction model. Mon. Wea. Rev., 140, 587600, doi:10.1175/MWR-D-11-00023.1.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 1731 282 33
PDF Downloads 261 84 16