Evaluating Data Assimilation Algorithms

K. J. H. Law Warwick Mathematics Institute, University of Warwick, Coventry, United Kingdom

Search for other papers by K. J. H. Law in
Current site
Google Scholar
PubMed
Close
and
A. M. Stuart Warwick Mathematics Institute, University of Warwick, Coventry, United Kingdom

Search for other papers by A. M. Stuart in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Data assimilation leads naturally to a Bayesian formulation in which the posterior probability distribution of the system state, given all the observations on a time window of interest, plays a central conceptual role. The aim of this paper is to use this Bayesian posterior probability distribution as a gold standard against which to evaluate various commonly used data assimilation algorithms.

A key aspect of geophysical data assimilation is the high dimensionality and limited predictability of the computational model. This paper examines the two-dimensional Navier–Stokes equations in a periodic geometry, which has these features and yet is tractable for explicit and accurate computation of the posterior distribution by state-of-the-art statistical sampling techniques. The commonly used algorithms that are evaluated, as quantified by the relative error in reproducing moments of the posterior, are four-dimensional variational data assimilation (4DVAR) and a variety of sequential filtering approximations based on three-dimensional variational data assimilation (3DVAR) and on extended and ensemble Kalman filters.

The primary conclusions are that, under the assumption of a well-defined posterior probability distribution, (i) with appropriate parameter choices, approximate filters can perform well in reproducing the mean of the desired probability distribution, (ii) they do not perform as well in reproducing the covariance, and (iii) the error is compounded by the need to modify the covariance, in order to induce stability. Thus, filters can be a useful tool in predicting mean behavior but should be viewed with caution as predictors of uncertainty. These conclusions are intrinsic to the algorithms when assumptions underlying them are not valid and will not change if the model complexity is increased.

Corresponding author address: Kody J. H. Law, Warwick Mathematics Institute, University of Warwick, Coventry CV4 7AL, United Kingdom. E-mail: k.j.h.law@warwick.ac.uk

Abstract

Data assimilation leads naturally to a Bayesian formulation in which the posterior probability distribution of the system state, given all the observations on a time window of interest, plays a central conceptual role. The aim of this paper is to use this Bayesian posterior probability distribution as a gold standard against which to evaluate various commonly used data assimilation algorithms.

A key aspect of geophysical data assimilation is the high dimensionality and limited predictability of the computational model. This paper examines the two-dimensional Navier–Stokes equations in a periodic geometry, which has these features and yet is tractable for explicit and accurate computation of the posterior distribution by state-of-the-art statistical sampling techniques. The commonly used algorithms that are evaluated, as quantified by the relative error in reproducing moments of the posterior, are four-dimensional variational data assimilation (4DVAR) and a variety of sequential filtering approximations based on three-dimensional variational data assimilation (3DVAR) and on extended and ensemble Kalman filters.

The primary conclusions are that, under the assumption of a well-defined posterior probability distribution, (i) with appropriate parameter choices, approximate filters can perform well in reproducing the mean of the desired probability distribution, (ii) they do not perform as well in reproducing the covariance, and (iii) the error is compounded by the need to modify the covariance, in order to induce stability. Thus, filters can be a useful tool in predicting mean behavior but should be viewed with caution as predictors of uncertainty. These conclusions are intrinsic to the algorithms when assumptions underlying them are not valid and will not change if the model complexity is increased.

Corresponding author address: Kody J. H. Law, Warwick Mathematics Institute, University of Warwick, Coventry CV4 7AL, United Kingdom. E-mail: k.j.h.law@warwick.ac.uk
Save
  • Anderson, J., 1996: A method for producing and evaluating probabilistic forecasts from ensemble model integrations. J. Climate, 9, 15181530.

    • Search Google Scholar
    • Export Citation
  • Apte, A., M. Hairer, A. Stuart, and J. Voss, 2007: Sampling the posterior: An approach to non-Gaussian data assimilation. Physica D, 230, 5064.

    • Search Google Scholar
    • Export Citation
  • Apte, A., C. Jones, A. Stuart, and J. Voss, 2008a: Data assimilation: Mathematical and statistical perspectives. Int. J. Numer. Methods Fluids, 56, 10331046.

    • Search Google Scholar
    • Export Citation
  • Apte, A., C. Jones, and A. Stuart, 2008b: A Bayesian approach to Lagrangian data assimilation. Tellus, 60, 336347.

  • Arulampalam, M., S. Maskell, N. Gordon, and T. Clapp, 2002: A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking. IEEE Trans. Signal Process., 50, 174188.

    • Search Google Scholar
    • Export Citation
  • Auvinen, H., J. Bardsley, H. Haario, and T. Kauranne, 2009: Large-scale Kalman filtering using the limited memory BFGS method. Electron. Trans. Numer. Anal., 35, 217233.

    • Search Google Scholar
    • Export Citation
  • Bain, A., and D. Crişan, 2008: Fundamentals of Stochastic Filtering. Springer Verlag, 390 pp.

  • Banks, H., 1992: Computational issues in parameter estimation and feedback control problems for partial differential equation systems. Physica D, 60, 226238.

    • Search Google Scholar
    • Export Citation
  • Banks, H., and K. Kunisch, 1989: Estimation Techniques for Distributed Parameter Systems. Birkhauser, 315 pp.

  • Bengtsson, T., C. Snyder, and D. Nychka, 2003: Toward a nonlinear ensemble filter for high-dimensional systems. J. Geophys. Res., 108, 8775, doi:10.1029/2002JD002900.

    • Search Google Scholar
    • Export Citation
  • Bennett, A., 2002: Inverse Modeling of the Ocean and Atmosphere. Cambridge University Press, 234 pp.

  • Brett, C., A. Lam, K. Law, D. McCormick, M. Scott, and A. Stuart, 2012: Accuracy and stability of filters for dissipative PDEs. Physica D, in press.

    • Search Google Scholar
    • Export Citation
  • Bröcker, J., 2010: On variational data assimilation in continuous time. Quart. J. Roy. Meteor. Soc., 136, 19061919.

  • Brooks, S., and A. Gelman, 1998: General methods for monitoring convergence of iterative simulations. J. Comput. Graph. Stat., 7, 434455.

    • Search Google Scholar
    • Export Citation
  • Bryson, A., and M. Frazier, 1963: Smoothing for linear and nonlinear dynamic systems. U.S. Air Force Tech. Rep. AFB-TDR-63-119, Wright-Patterson Air Force Base, OH, Aeronautical Systems Division, 353–364.

  • Carrassi, A., M. Ghil, A. Trevisan, and F. Uboldi, 2008: Data assimilation as a nonlinear dynamical systems problem: Stability and convergence of the prediction-assimilation system. Chaos, 18, 023112, doi:10.1063/1.2909862.

    • Search Google Scholar
    • Export Citation
  • Chorin, A., and P. Krause, 2004: Dimensional reduction for a Bayesian filter. Proc. Natl. Acad. Sci. USA, 101, 15 01315 017.

  • Chorin, A., M. Morzfeld, and X. Tu, 2010: Implicit particle filters for data assimilation. Commun. Appl. Math. Comput. Sci., 5, 221240.

    • Search Google Scholar
    • Export Citation
  • Cotter, S., M. Dashti, J. Robinson, and A. Stuart, 2009: Bayesian inverse problems for functions and applications to fluid mechanics. Inverse Probl., 25, 115008, doi:10.1088/0266-5611/25/11/115008.

    • Search Google Scholar
    • Export Citation
  • Cotter, S., M. Dashti, and A. Stuart, 2011: Variational data assimilation using targetted random walks. Int. J. Numer. Methods Fluids, 68, 403421.

    • Search Google Scholar
    • Export Citation
  • Courtier, P., and O. Talagrand, 1987: Variational assimilation of meteorological observations with the adjoint vorticity equation. II: Numerical results. Quart. J. Roy. Meteor. Soc., 113, 13291347.

    • Search Google Scholar
    • Export Citation
  • Cox, H., 1964: On the estimation of state variables and parameters for noisy dynamic systems. IEEE Trans. Autom. Control, 9, 512.

  • Cox, S., and P. Matthews, 2002: Exponential time differencing for stiff systems. J. Comput. Phys., 176, 430455.

  • Doucet, A., N. De Freitas, and N. Gordon, 2001: Sequential Monte Carlo Methods in Practice. Springer Verlag, 581 pp.

  • Evensen, G., 2003: The ensemble Kalman filter: Theoretical formulation and practical implementation. Ocean Dyn., 53, 343367.

  • Evensen, G., 2009: Data Assimilation: The Ensemble Kalman Filter. Springer Verlag, 307 pp.

  • Evensen, G., and Coauthors, 1994: Assimilation of Geosat altimeter data for the Agulhas Current using the ensemble Kalman filter with a quasigeostrophic model. Mon. Wea. Rev., 124, 8596.

    • Search Google Scholar
    • Export Citation
  • Fisher, M., M. Leutbecher, and G. Kelly, 2005: On the equivalence between Kalman smoothing and weak-constraint four-dimensional variational data assimilation. Quart. J. Roy. Meteor. Soc., 131, 32353246.

    • Search Google Scholar
    • Export Citation
  • Hamill, T., C. Snyder, and R. Morss, 2000: A comparison of probabilistic forecasts from bred, singular-vector, and perturbed observation ensembles. Mon. Wea. Rev., 128, 18351851.

    • Search Google Scholar
    • Export Citation
  • Harlim, J., and A. Majda, 2008: Filtering nonlinear dynamical systems with linear stochastic models. Nonlinearity, 21, 1281, doi:10.1088/0951-7715/21/6/008.

    • Search Google Scholar
    • Export Citation
  • Harvey, A., 1991: Forecasting, Structural Time Series Models, and the Kalman Filter. Cambridge University Press, 554 pp.

  • Hesthaven, J., S. Gottlieb, and D. Gottlieb, 2007: Spectral Methods for Time-Dependent Problems. Cambridge University Press, 273 pp.

  • Hinze, M., R. Pinnau, M. Ulbrich, and S. Ulbrich, 2009: Optimization with PDE Constraints. Springer, 270 pp.

  • Jazwinski, A., 1970: Stochastic Processes and Filtering Theory. Academic Press, 376 pp.

  • Kaipio, J., and E. Somersalo, 2005: Statistical and Computational Inverse Problems. Springer, 339 pp.

  • Kalman, R., 1960: A new approach to linear filtering and prediction problems. J. Basic Eng., 82, 3545.

  • Kalnay, E., 2003: Atmospheric Modeling, Data Assimilation, and Predictability. Cambridge University Press, 341 pp.

  • Kelley, C., 2003: Solving Nonlinear Equations with Newton’s Method. Vol. 1, Fundamentals of Algorithms, Society for Industrial Mathematics, 104 pp.

  • Lawless, A., N. Nichols, and S. Ballard, 2003: A comparison of two methods for developing the linearization of a shallow-water model. Quart. J. Roy. Meteor. Soc., 129, 12371254.

    • Search Google Scholar
    • Export Citation
  • Lei, J., P. Bickel, and C. Snyder, 2010: Comparison of ensemble Kalman filters under non-Gaussianity. Mon. Wea. Rev., 138, 12931306.

  • Leutbecher, M., 2003: Adaptive observations, the Hessian metric and singular vectors. Proc. ECMWF Seminar on Recent Developments in Data Assimilation for Atmosphere and Ocean, Reading, United Kingdom, ECMWF, 8–12.

  • Liu, N., and D. S. Oliver, 2003: Evaluation of Monte Carlo methods for assessing uncertainty. SPE J., 8, 188195.

  • Lorenc, A., 1986: Analysis methods for numerical weather prediction. Quart. J. Roy. Meteor. Soc., 112, 11771194.

  • Lorenz, E., 1963: Deterministic nonperiodic flow. J. Atmos Sci., 20, 130141.

  • Lorenz, E., 1996: Predictability: A problem partly solved. Proc. Seminar on Predictability, Reading, United Kingdom, ECMWF, 1–18.

  • Majda, A., J. Harlim, and B. Gershgorin, 2010: Mathematical strategies for filtering turbulent dynamical systems. Dyn. Syst., 27, 441486.

    • Search Google Scholar
    • Export Citation
  • Meng, Z., and F. Zhang, 2008: Tests of an ensemble Kalman filter for mesoscale and regional-scale data assimilation. Part IV: Comparison with 3DVAR in a month-long experiment. Mon. Wea. Rev., 136, 36713682.

    • Search Google Scholar
    • Export Citation
  • Miller, R., M. Ghil, and F. Gauthiez, 1994: Advanced data assimilation in strongly nonlinear dynamical systems. J. Atmos. Sci., 51, 10371056.

    • Search Google Scholar
    • Export Citation
  • Nocedal, J., and S. Wright, 1999: Numerical Optimization. Springer Verlag, 636 pp.

  • Palmer, T., R. Gelaro, J. Barkmeijer, and R. Buizza, 1998: Singular vectors, metrics, and adaptive observations. J. Atmos. Sci., 55, 633653.

    • Search Google Scholar
    • Export Citation
  • Quinn, J., and H. Abarbanel, 2010: State and parameter estimation using Monte Carlo evaluation of path integrals. Quart. J. Roy. Meteor. Soc., 136, 18551867.

    • Search Google Scholar
    • Export Citation
  • Saad, Y., 1996: Iterative Methods for Sparse Linear Systems. 1st ed. PWS Publishing, 447 pp.

  • Snyder, T., T. Bengtsson, P. Bickel, and J. Anderson, 2008: Obstacles to high-dimensional particle filtering. Mon. Wea. Rev., 136, 46294640.

    • Search Google Scholar
    • Export Citation
  • Stuart, A., 2010: Inverse problems: A Bayesian perspective. Acta Numer., 19, 451559.

  • Talagrand, O., and P. Courtier, 1987: Variational assimilation of meteorological observations with the adjoint vorticity equation. I: Theory. Quart. J. Roy. Meteor. Soc., 113, 13111328.

    • Search Google Scholar
    • Export Citation
  • Tarantola, A., 2005: Inverse Problem Theory and Methods for Model Parameter Estimation. Society for Industrial Mathematics, 342 pp.

  • Temam, R., 2001: Navier–Stokes Equations: Theory and Numerical Analysis. American Mathematical Society, 408 pp.

  • Tippett, M., J. Anderson, C. Bishop, T. Hamill, and J. Whitaker, 2003: Ensemble square root filters. Mon. Wea. Rev., 131, 14851490.

  • Toth, Z., and E. Kalnay, 1997: Ensemble forecasting at NCEP and the breeding method. Mon. Wea. Rev., 125, 32973319.

  • Trefethen, L., and D. Bau, 1997: Numerical Linear Algebra. Society for Industrial Mathematics, 361 pp.

  • van Leeuwen, P., 2009: Particle filtering in geophysical systems. Mon. Wea. Rev., 137, 40894114.

  • van Leeuwen, P., 2010: Nonlinear data assimilation in geosciences: An extremely efficient particle filter. Quart. J. Roy. Meteor. Soc., 136, 19911999.

    • Search Google Scholar
    • Export Citation
  • Vogel, C., 2002: Computational Methods for Inverse Problems. Society for Industrial Mathematics, 183 pp.

  • Vogel, C., and J. Wade, 1995: Analysis of costate discretizations in parameter estimation for linear evolution equations. SIAM J. Control Optim., 33, 227254.

    • Search Google Scholar
    • Export Citation
  • Zhang, M., and F. Zhang, 2012: E4DVAR: Coupling an ensemble Kalman filter with four-dimensional variational data assimilation in a limited-area weather prediction model. Mon. Wea. Rev., 140, 587600.

    • Search Google Scholar
    • Export Citation
  • Zhang, M., F. Zhang, X. Huang, and X. Zhang, 2010: Intercomparison of an ensemble Kalman filter with three- and four-dimensional variational data assimilation methods in a limited-area model over the month of June 2003. Mon. Wea. Rev., 139, 566572.

    • Search Google Scholar
    • Export Citation
  • Zupanski, D., 1997: A general weak constraint applicable to operational 4DVAR data assimilation systems. Mon. Wea. Rev., 125, 22742292.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 1141 298 45
PDF Downloads 711 141 17