• Amezcua, J., and P. J. van Leeuwen, 2014: Gaussian anamorphosis in the analysis step of the EnKF: A joint state-variable/observation approach. Tellus, 66A, 23493, https://doi.org/10.3402/tellusa.v66.23493.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Andrews, A., 1968: A square root formulation of the Kalman covariance equations. AIAA J., 6, 11651166, https://doi.org/10.2514/3.4696.

  • Bannister, R. N., 2017: A review of operational methods of variational and ensemble-variational data assimilation. Quart. J. Roy. Meteor., 143, 607633, https://doi.org/10.1002/qj.2982.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bengtsson, T., P. Bickel, and B. Li, 2008: Curse-of-dimensionality revisited: Collapse of the particle filter in very large scale systems. Probability and Statistics: Essays in Honor of David A. Freedman, D. Nolan and T. Speed, Eds., Vol. 2, Institute of Mathematical Statistics, 316–334.

    • Crossref
    • Export Citation
  • Bickel, P., B. Li, and T. Bengtsson, 2008: Sharp failure rates for the bootstrap particle filter in high dimensions. Pushing the Limits of Contemporary Statistics: Contributions in Honor of Jayanta K. Ghosh, B. Clarke and S. Ghosal, Eds., Vol. 3, Institute of Mathematical Statistics, 318–329.

    • Crossref
    • Export Citation
  • Bishop, C. H., 2016: The GIGG-EnKF: Ensemble Kalman filtering for highly skewed non-negative uncertainty distributions. Quart. J. Roy. Meteor. Soc., 142, 13951412, https://doi.org/10.1002/qj.2742.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bocquet, M., and P. Sakov, 2014: An iterative ensemble Kalman smoother. Quart. J. Roy. Meteor. Soc., 140, 15211535, https://doi.org/10.1002/qj.2236.

  • Bonavita, M., P. Lean, and E. Hólm, 2018: Nonlinear effects in 4D-Var. Nonlinear Processes Geophys., 25, 713–729, https://doi.org/10.5194/npg-25-713-2018.

    • Crossref
    • Export Citation
  • Buehner, M., 2005: Ensemble-derived stationary and flow-dependent background-error covariances: Evaluation in a quasi-operational NWP setting. Quart. J. Roy. Meteor. Soc., 131, 10131043, https://doi.org/10.1256/qj.04.15.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Buehner, M., P. L. Houtekamer, C. Charette, H. Mitchell, and B. He, 2010: Intercomparison of variational data assimilation and the ensemble Kalman filter for global deterministic NWP. Part II: One-month experiments with real observations. Mon. Wea. Rev., 138, 15671586, https://doi.org/10.1175/2009MWR3158.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Chen, Y., and D. S. Oliver, 2012: Ensemble randomized maximum likelihood method as an iterative ensemble smoother. Math. Geosci., 44, 126, https://doi.org/10.1007/s11004-011-9376-z.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cohn, S. E., N. Sivakumaran, and R. Todling, 1994: A fixed-lag Kalman smoother for retrospective data assimilation. Mon. Wea. Rev., 122, 28382867, https://doi.org/10.1175/1520-0493(1994)122<2838:AFLKSF>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Courtier, P., J.-N. Thepáut, and A. Hollingsworth, 1994: A strategy for operational implementation of 4D-Var, using an incremental approach. Quart. J. Roy. Meteor. Soc., 120, 13671387, https://doi.org/10.1002/qj.49712051912.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Emerick, A. A., and A. C. Reynolds, 2012: History matching time-lapse seismic data using the ensemble Kalman filter with multiple data assimilations. Comput. Geosci., 16, 639659, https://doi.org/10.1007/s10596-012-9275-5.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Emerick, A. A., and A. C. Reynolds, 2013: Ensemble smoother with multiple data assimilation. Comput. Geosci., 55, 315, https://doi.org/10.1016/j.cageo.2012.03.011.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Errico, R., P. Bauer, and J.-F. Mahfouf, 2007: Issues regarding the assimilation of cloud and precipitation data. J. Atmos. Sci., 64, 37853798, https://doi.org/10.1175/2006JAS2044.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Evensen, G., 1994: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res., 99, 10 14310 162, https://doi.org/10.1029/94JC00572.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Evensen, G., 2018: Analysis of iterative ensemble smoothers for solving inverse problems. Comput. Geosci., 22, 885908, https://doi.org/10.1007/s10596-018-9731-y.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Evensen, G., and P. J. van Leeuwen, 2000: An ensemble Kalman smoother for nonlinear dynamics. Mon. Wea. Rev., 128, 18521867, https://doi.org/10.1175/1520-0493(2000)128<1852:AEKSFN>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fabry, F., and J. Sun, 2010: For how long should what data be assimilated for the mesoscale forecasting of convection and why? Part I: On the propagation of initial condition errors and their implications for data assimilation. Mon. Wea. Rev., 138, 242255, https://doi.org/10.1175/2009MWR2883.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fairbairn, D., S. R. Pring, A. C. Lorenc, and I. Roulstone, 2014: A comparison of 4DVar with ensemble data assimilation methods. Quart. J. Roy. Meteor. Soc., 140, 281294, https://doi.org/10.1002/qj.2135.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fletcher, S., 2010: Mixed Gaussian-lognormal four-dimensional data assimilation. Tellus, 62A, 266287, https://doi.org/10.1111/j.1600-0870.2010.00439.x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fletcher, S., and A. S. Jones, 2014: Multiplicative and additive incremental variational data assimilation for mixed lognormal Gaussian errors. Mon. Wea. Rev., 142, 25212544, https://doi.org/10.1175/MWR-D-13-00136.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gaspari, G., and S. E. Cohn, 1999: Construction of correlation functions in two and three dimensions. Quart. J. Roy. Meteor. Soc., 125, 723757, https://doi.org/10.1002/qj.49712555417.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Geer, A. J., and P. Bauer, 2011: Observation errors in all-sky data assimilation. Quart. J. Roy. Meteor. Soc., 137, 20242037, https://doi.org/10.1002/qj.830.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Geer, A. J., and et al. , 2017: The growing impact of satellite observations sensitive to humidity, cloud and precipitation. Quart. J. Roy. Meteor. Soc., 143, 31893206, https://doi.org/10.1002/qj.3172.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Geer, A. J., and et al. , 2018: All-sky satellite data assimilation at operational weather forecasting centres. Quart. J. Roy. Meteor. Soc., 144, 11911217, https://doi.org/10.1002/qj.3202.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Geer, A. J., S. Migliorini, and M. Matricardi, 2019: All-sky assimilation of infrared radiances sensitive to mid- and upper-tropospheric moisture and cloud. Atmos. Meas. Tech., 12, 49034929, https://doi.org/10.5194/amt-12-4903-2019.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gelb, A., J. F. Kasper, R. A. Nash, C. F. Price, and A. A. Sutherland, 1974: Applied Optimal Estimation. The MIT Press, 374 pp.

  • Gu, Y., and D. S. Oliver, 2007: An iterative ensemble Kalman filter for multiphase fluid flow data assimilation. SPE J., 12, 438446, https://doi.org/10.2118/108438-PA.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., and C. Snyder, 2000: A hybrid ensemble Kalman filter-3D variational analysis scheme. Mon. Wea. Rev., 128, 29052919, https://doi.org/10.1175/1520-0493(2000)128<2905:AHEKFV>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Honda, T., and et al. , 2018: Assimilating all-sky Himawari-8 satellite infrared radiances: A case of Typhoon Soudelor (2015). Mon. Wea. Rev., 146, 213229, https://doi.org/10.1175/MWR-D-16-0357.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., and H. L. Mitchell, 1998: Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev., 126, 796811, https://doi.org/10.1175/1520-0493(1998)126<0796:DAUAEK>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jazwinski, A. H., 1970: Stochastic Processes and Filtering Theory. Academic Press, 376 pp.

  • Kalman, R. E., 1960: A new approach to linear filtering and prediction problems. Trans. ASME–J. Basic Eng., 82, 3545, https://doi.org/10.1115/1.3662552.

    • Crossref
    • Export Citation
  • Kleist, D. T., and K. Ide, 2015: An OSSE-based evaluation of hybrid variational–ensemble data assimilation for the NCEP GFS. Part II: 4DEnVar and hybrid variants. Mon. Wea. Rev., 143, 452470, https://doi.org/10.1175/MWR-D-13-00350.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Liu, C., and Q. Xiao, 2013: Antarctic applications with Advanced Research WRF using real data. Mon. Wea. Rev., 141, 27212739, https://doi.org/10.1175/MWR-D-12-00130.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Liu, C., Q. Xiao, and B. Wang, 2009: An ensemble-based four-dimensional variational data assimilation scheme. Part II: Observing system simulation experiments with Advanced Research WRF (ARW). Mon. Wea. Rev., 137, 16871704, https://doi.org/10.1175/2008MWR2699.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lorenc, A. C., 2003: The potential of the ensemble Kalman filter for NWP: A comparison with 4D-Var. Quart. J. Roy. Meteor. Soc., 129, 31833203, https://doi.org/10.1256/qj.02.132.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lorenz, E. N., 1996: Predictability: A problem partly solved. Proc. Seminar on Predictability, Reading, United Kingdom, ECMWF, https://www.ecmwf.int/node/10829.

  • Lorenz, E. N., and K. A. Emanuel, 1998: Optimal sites for supplementary weather observations: Simulation with a small model. J. Atmos. Sci., 55, 399414, https://doi.org/10.1175/1520-0469(1998)055<0399:OSFSWO>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Minamide, M., and F. Zhang, 2017: Adaptive observation error inflation for assimilating all-sky satellite radiance. Mon. Wea. Rev., 145, 10631081, https://doi.org/10.1175/MWR-D-16-0257.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Morzfeld, M., and D. Hodyss, 2019: Gaussian approximations in filters and smoothers for data assimilation. Tellus, 71A, 127, https://doi.org/10.1080/16000870.2019.1600344.

    • Search Google Scholar
    • Export Citation
  • Neal, R. M., 1996: Sampling from multimodal distributions using tempered transitions. Stat. Comput., 6, 353366, https://doi.org/10.1007/BF00143556.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Okamoto, K., A. P. McNally, and W. Bell, 2014: Progress towards the assimilation of all-sky infrared radiances: An evaluation of cloud effects. Quart. J. Roy. Meteor., 140, 16031614, https://doi.org/10.1002/qj.2242.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Poterjoy, J., 2016: A localized particle filter for high-dimensional nonlinear systems. Mon. Wea. Rev., 144, 5976, https://doi.org/10.1175/MWR-D-15-0163.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Poterjoy, J., and F. Zhang, 2015: Systematic comparison of four-dimensional data assimilation methods with and without a tangent linear model using hybrid background error covariance: E4DVar versus 4DEnVar. Mon. Wea. Rev., 143, 16011621, https://doi.org/10.1175/MWR-D-14-00224.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Poterjoy, J., and J. L. Anderson, 2016: Efficient assimilation of simulated observations in a high-dimensional geophysical system using a localized particle filter. Mon. Wea. Rev., 144, 20072020, https://doi.org/10.1175/MWR-D-15-0322.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Poterjoy, J., F. Zhang, and Y. Weng, 2014: The effects of sampling errors on the EnKF assimilation of inner-core hurricane observations. Mon. Wea. Rev., 142, 16091630, https://doi.org/10.1175/MWR-D-13-00305.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Poterjoy, J., R. A. Sobash, and J. L. Anderson, 2017: Convective-scale data assimilation for the Weather Research and Forecasting Model using the local particle filter. Mon. Wea. Rev., 145, 18971918, https://doi.org/10.1175/MWR-D-16-0298.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Poterjoy, J., L. J. Wicker, and M. Buehner, 2019: Progress toward the application of a localized particle filter for numerical weather prediction. Mon. Wea. Rev., 147, 11071126, https://doi.org/10.1175/MWR-D-17-0344.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Potter, J., 1964: W matrix augmentation. M.I.T. Instrumentation Laboratory Memo. SGA, Massachusetts Institute of Technology, Cambridge, MA, 5–64.

  • Potthast, R., A. Walter, and A. Rhodin, 2019: A localized adaptive particle filter within an operational NWP framework. Mon. Wea. Rev., 147, 345362, https://doi.org/10.1175/MWR-D-18-0028.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Privé, N. C., Y. Xie, J. Woollen, S. E. Koch, R. Atlas, and R. Hood, 2013: Evaluation of the Earth Systems Research Laboratory’s global Observing System Simulation Experiment system. Tellus, 65A, 19011, https://doi.org/10.3402/tellusa.v65i0.19011.

    • Search Google Scholar
    • Export Citation
  • Rommelse, J., 2009: Data assimilation in reservoir management. Ph.D. thesis, Technical University of Delft.

  • Sakov, P., D. S. Oliver, and L. Bertino, 2012: An iterative EnKF for strongly nonlinear systems. Mon. Wea. Rev., 140, 19882004, https://doi.org/10.1175/MWR-D-11-00176.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Skjervheim, J.-A., G. Evensen, J. Hove, and J. G. Vabø, 2011: An ensemble smoother for assisted history matching. SPE Reservoir Simulation Symp., The Woodlands, TX, SPE, SPE-141929-MS, https://doi.org/10.2118/141929-MS.

    • Crossref
    • Export Citation
  • Snyder, C., T. Bengtsson, P. Bickel, and J. Anderson, 2008: Obstacles to high-dimensional particle filtering. Mon. Wea. Rev., 136, 46294640, https://doi.org/10.1175/2008MWR2529.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Stengel, M., P. Unden, M. Lindskog, P. Dahlgren, N. Gustafsson, and R. Bennartz, 2009: Assimilation of SEVIRI infrared radiances with HIRLAM 4D-Var. Quart. J. Roy. Meteor. Soc., 135, 21002109, https://doi.org/10.1002/qj.501.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Thepáut, J.-N., and P. Courtier, 1991: Four-dimensional variational data assimilation using the adjoint of a multilevel primitive-equation model. Quart. J. Roy. Meteor. Soc., 117, 12251254, https://doi.org/10.1002/qj.49711750206.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • van Leeuwen, P. J., and G. Evensen, 1996: Data assimilation and inverse methods in terms of a probabilistic formulation. Mon. Wea. Rev., 124, 28982913, https://doi.org/10.1175/1520-0493(1996)124<2898:DAAIMI>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Vukicevic, T., T. Greenwald, M. Zupanski, D. Zupanski, T. V. Haar, and A. S. Jones, 2004: Mesoscale cloud state estimation from visible and infrared satellite radiances. Mon. Wea. Rev., 132, 30663077, https://doi.org/10.1175/MWR2837.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Whitaker, J., and G. Compo, 2002: An ensemble Kalman smoother for reanalysis. Proc. Symp. on Observations, Data Assimilation and Probabilistic Prediction, Orlando, FL, Amer. Meteor. Soc., 144147.

  • Whitaker, J., and T. M. Hamill, 2002: Ensemble data assimilation without perturbed observations. Mon. Wea. Rev., 130, 19131924, https://doi.org/10.1175/1520-0493(2002)130<1913:EDAWPO>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zhang, F., C. Snyder, and J. Sun, 2004: Impacts of initial estimate and observation availability on convective-scale data assimilation with an ensemble Kalman filter. Mon. Wea. Rev., 132, 12381253, https://doi.org/10.1175/1520-0493(2004)132<1238:IOIEAO>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zhang, F., M. Zhang, and J. A. Hansen, 2009: Coupling ensemble Kalman filter with four-dimensional variational data assimilation. Adv. Atmos. Sci., 26, 18, https://doi.org/10.1007/s00376-009-0001-8.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zhu, Y., and et al. , 2016: All-sky microwave radiance assimilation in NCEP’s GSI analysis system. Mon. Wea. Rev., 144, 47094735, https://doi.org/10.1175/MWR-D-15-0445.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zou, X., F. Weng, B. Zhang, L. Lin, Z. Qin, and V. Tallapragada, 2013: Impacts of assimilation of ATMS data in HWRF on track and intensity forecasts of 2012 four landfall hurricanes. J. Geophys. Res., 118, 1155811576, https://doi.org/10.1002/2013JD020405.

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 115 115 115
Full Text Views 40 40 40
PDF Downloads 51 51 51

Data Assimilation Challenges Posed by Nonlinear Operators: A Comparative Study of Ensemble and Variational Filters and Smoothers

View More View Less
  • 1 a Department of Atmospheric and Oceanic Science, University of Maryland, College Park, College Park, Maryland
  • | 2 b NOAA/Atlantic Oceanographic and Meteorological Laboratory, Miami, Florida
© Get Permissions Rent on DeepDyve
Restricted access

Abstract

The ensemble Kalman filter (EnKF) and the 4D variational method (4DVar) are the most commonly used filters and smoothers in atmospheric science. These methods typically approximate prior densities using a Gaussian and solve a linear system of equations for the posterior mean and covariance. Therefore, strongly nonlinear model dynamics and measurement operators can lead to bias in posterior estimates. To improve the performance in nonlinear regimes, minimization of the 4DVar cost function typically follows multiple sets of iterations, known as an “outer loop,” which helps reduce bias caused by linear assumptions. Alternatively, “iterative ensemble methods” follow a similar strategy of periodically relinearizing model and measurement operators. These methods come with different, possibly more appropriate, assumptions for drawing samples from the posterior density, but have seen little attention in numerical weather prediction (NWP) communities. Last, particle filters (PFs) present a purely Bayesian filtering approach for state estimation, which avoids many of the assumptions made by the above methods. Several strategies for applying localized PFs for NWP have been proposed very recently. The current study investigates intrinsic limitations of current data assimilation methodology for applications that require nonlinear measurement operators. In doing so, it targets a specific problem that is relevant to the assimilation of remotely sensed measurements, such as radar reflectivity and all-sky radiances, which pose challenges for Gaussian-based data assimilation systems. This comparison includes multiple data assimilation approaches designed recently for nonlinear/non-Gaussian applications, as well as those currently used for NWP.

© 2021 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Kenta Kurosawa, kkurosaw@umd.edu

Abstract

The ensemble Kalman filter (EnKF) and the 4D variational method (4DVar) are the most commonly used filters and smoothers in atmospheric science. These methods typically approximate prior densities using a Gaussian and solve a linear system of equations for the posterior mean and covariance. Therefore, strongly nonlinear model dynamics and measurement operators can lead to bias in posterior estimates. To improve the performance in nonlinear regimes, minimization of the 4DVar cost function typically follows multiple sets of iterations, known as an “outer loop,” which helps reduce bias caused by linear assumptions. Alternatively, “iterative ensemble methods” follow a similar strategy of periodically relinearizing model and measurement operators. These methods come with different, possibly more appropriate, assumptions for drawing samples from the posterior density, but have seen little attention in numerical weather prediction (NWP) communities. Last, particle filters (PFs) present a purely Bayesian filtering approach for state estimation, which avoids many of the assumptions made by the above methods. Several strategies for applying localized PFs for NWP have been proposed very recently. The current study investigates intrinsic limitations of current data assimilation methodology for applications that require nonlinear measurement operators. In doing so, it targets a specific problem that is relevant to the assimilation of remotely sensed measurements, such as radar reflectivity and all-sky radiances, which pose challenges for Gaussian-based data assimilation systems. This comparison includes multiple data assimilation approaches designed recently for nonlinear/non-Gaussian applications, as well as those currently used for NWP.

© 2021 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Kenta Kurosawa, kkurosaw@umd.edu
Save