Observation-Dependent Posterior Inflation for the Ensemble Kalman Filter

Daniel Hodyss Naval Research Laboratory, Monterey, California

Search for other papers by Daniel Hodyss in
Current site
Google Scholar
PubMed
Close
,
William F. Campbell Naval Research Laboratory, Monterey, California

Search for other papers by William F. Campbell in
Current site
Google Scholar
PubMed
Close
, and
Jeffrey S. Whitaker NOAA/Earth System Research Laboratory/Physical Sciences Division, Boulder, Colorado

Search for other papers by Jeffrey S. Whitaker in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Ensemble-based Kalman filter (EBKF) algorithms are known to produce posterior ensembles whose variance is incorrect for a variety of reasons (e.g., nonlinearity and sampling error). It is shown here that the presence of sampling error implies that the true posterior error variance is a function of the latest observation, as opposed to the standard EBKF, whose posterior variance is independent of observations. In addition, it is shown that the traditional ensemble validation tool known as the “binned spread-skill” diagram does not correctly identify this issue in the ensemble generation step of the EBKF, leading to an overly optimistic impression of the relationship between posterior variance and squared error. An updated ensemble validation tool is described that reveals the incorrect relationship between mean squared error (MSE) and ensemble variance, and gives an unbiased evaluation of the posterior variances from EBKF algorithms. Last, a new inflation method is derived that accounts for sampling error and correctly yields posterior error variances that depend on the latest observation. The new method has very little computational overhead, does not require access to the observations, and is simple to use in any serial or global EBKF.

Corresponding author address: Dr. Daniel Hodyss, Naval Research Laboratory, Marine Meteorology Division, 7 Grace Hopper Ave., Stop 2, Monterey, CA 93943. E-mail: daniel.hodyss@nrlmry.navy.mil

Abstract

Ensemble-based Kalman filter (EBKF) algorithms are known to produce posterior ensembles whose variance is incorrect for a variety of reasons (e.g., nonlinearity and sampling error). It is shown here that the presence of sampling error implies that the true posterior error variance is a function of the latest observation, as opposed to the standard EBKF, whose posterior variance is independent of observations. In addition, it is shown that the traditional ensemble validation tool known as the “binned spread-skill” diagram does not correctly identify this issue in the ensemble generation step of the EBKF, leading to an overly optimistic impression of the relationship between posterior variance and squared error. An updated ensemble validation tool is described that reveals the incorrect relationship between mean squared error (MSE) and ensemble variance, and gives an unbiased evaluation of the posterior variances from EBKF algorithms. Last, a new inflation method is derived that accounts for sampling error and correctly yields posterior error variances that depend on the latest observation. The new method has very little computational overhead, does not require access to the observations, and is simple to use in any serial or global EBKF.

Corresponding author address: Dr. Daniel Hodyss, Naval Research Laboratory, Marine Meteorology Division, 7 Grace Hopper Ave., Stop 2, Monterey, CA 93943. E-mail: daniel.hodyss@nrlmry.navy.mil
Save
  • Anderson, J. L., 2007: An adaptive covariance inflation error correction algorithm for ensemble filters. Tellus, 59A, 210224, doi:10.1111/j.1600-0870.2006.00216.x.

    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., 2009: Spatially and temporally varying adaptive covariance inflation for ensemble filters. Tellus, 61A, 7283, doi:10.1111/j.1600-0870.2008.00361.x.

    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., and S. L. Anderson, 1999: A Monte Carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Mon. Wea. Rev., 127, 27412758, doi:10.1175/1520-0493(1999)127<2741:AMCIOT>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Evensen, G., 1994: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res., 99, 10 14310 162, doi:10.1029/94JC00572.

    • Search Google Scholar
    • Export Citation
  • Evensen, G., 2003: The ensemble Kalman filter: Theoretical formulation and practical implementation. Ocean Dyn., 53, 343367, doi:10.1007/s10236-003-0036-9.

    • Search Google Scholar
    • Export Citation
  • Furrer, R., and T. Bengtsson, 2007: Estimation of high-dimensional prior and posterior covariance matrices in Kalman filter variants. J. Multivar. Anal., 98, 227255, doi:10.1016/j.jmva.2006.08.003.

    • Search Google Scholar
    • Export Citation
  • Hodyss, D., 2011: Ensemble state estimation for nonlinear systems using polynomial expansions in the innovation. Mon. Wea. Rev., 139, 35713588, doi:10.1175/2011MWR3558.1.

    • Search Google Scholar
    • Export Citation
  • Hodyss, D., and W. F. Campbell, 2013: Square root and perturbed observation ensemble generation techniques in Kalman and quadratic ensemble filtering algorithms. Mon. Wea. Rev., 141, 25612573, doi:10.1175/MWR-D-12-00117.1.

    • Search Google Scholar
    • Export Citation
  • Lee, S., and I. M. Held, 1993: Baroclinic wave packets in models and observations. J. Atmos. Sci., 50, 14131428, doi:10.1175/1520-0469(1993)050<1413:BWPIMA>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Lorenz, E. N., 1963: Deterministic non-periodic flow. J. Atmos. Sci., 20, 130141, doi:10.1175/1520-0469(1963)020<0130:DNF>2.0.CO;2.

  • Menetrier, B., and T. Auligne, 2015: Optimized localization and hybridization to filter ensemble-based covariances. Mon. Wea. Rev., 143, 39313947, doi:10.1175/MWR-D-15-0057.1.

    • Search Google Scholar
    • Export Citation
  • Posselt, D. J., D. Hodyss, and C. H. Bishop, 2014: Errors in ensemble Kalman smoother estimates of cloud microphysical parameters. Mon. Wea. Rev., 142, 16311654, doi:10.1175/MWR-D-13-00290.1.

    • Search Google Scholar
    • Export Citation
  • Sacher, W., and P. Bartello, 2008: Sampling errors in ensemble Kalman filtering. Part I: Theory. Mon. Wea. Rev., 136, 30353049, doi:10.1175/2007MWR2323.1.

    • Search Google Scholar
    • Export Citation
  • Wang, X., and C. H. Bishop, 2003: A comparison of breeding and ensemble transform Kalman filter ensemble forecast schemes. J. Atmos. Sci., 60, 11401158, doi:10.1175/1520-0469(2003)060<1140:ACOBAE>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Whitaker, J. S., and A. F. Loughe, 1998: The relationship between ensemble spread and ensemble mean skill. Mon. Wea. Rev., 126, 32923302, doi:10.1175/1520-0493(1998)126<3292:TRBESA>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Whitaker, J. S., and T. H. Hamill, 2002: Ensemble data assimilation without perturbed observations. Mon. Wea. Rev., 130, 19131924, doi:10.1175/1520-0493(2002)130<1913:EDAWPO>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Whitaker, J. S., and T. H. Hamill, 2012: Evaluating methods to account for system errors in ensemble data assimilation. Mon. Wea. Rev., 140, 30783089, doi:10.1175/MWR-D-11-00276.1.

    • Search Google Scholar
    • Export Citation
  • Zhang, F., C. Snyder, and J. Sun, 2004: Impacts of initial estimate and observation availability on convective-scale data assimilation with an ensemble Kalman filter. Mon. Wea. Rev., 132, 12381253, doi:10.1175/1520-0493(2004)132<1238:IOIEAO>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 348 75 1
PDF Downloads 208 57 2