Data Assimilation: A Fully Nonlinear Approach to Ensemble Formation Using Indistinguishable States

Shree Khare National Center for Atmospheric Research, Boulder, Colorado

Search for other papers by Shree Khare in
Current site
Google Scholar
PubMed
Close
and
Leonard A. Smith London School of Economics, London, United Kingdom

Search for other papers by Leonard A. Smith in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Operational forecasting with simulation models involves the melding of observations and model dynamics to determine a set of initial conditions for each forecast. The Kalman filter (KF) provides the optimal closed-form solution to a general linear stochastic (perfect model) case, while the target of the problem has not even been defined in the case of imperfect models. Data assimilation in a nonlinear, perfect-model scenario is considered. It is shown that a new fully nonlinear approach based upon the indistinguishable states (IS) systematically outperforms the ensemble Kalman filter (EnKF). The IS provides an ensemble of initial conditions, consistent with (i) the model dynamics, (ii) the observational noise model, and (iii) the particular observations over a window. It is argued that this is the relevant limit to consider in data assimilation, when the desire is to place high probability density in the vicinity of the target state. The advantages of the IS approach come in part from its ability to provide attractor-balanced ensembles near any attracting manifold the system may evolve on. The use of an EnKF, provides a computationally cheaper alternative that place points in the general vicinity of the target. A low (i.e., 2) dimensional example is used to provide easily visualized evidence for these claims, which are then tested in a higher (i.e., 12) dimensional system. Inasmuch as the IS approach is shown to outperform the EnKF systematically in these perfect-model experiments, it provides an interesting alternative approach when informative ensembles are desired.

Corresponding author address: Shree Khare, NCAR, 1850 Table Mesa Dr., Boulder, CO 80305. E-mail: shree.khare@yahoo.com

Abstract

Operational forecasting with simulation models involves the melding of observations and model dynamics to determine a set of initial conditions for each forecast. The Kalman filter (KF) provides the optimal closed-form solution to a general linear stochastic (perfect model) case, while the target of the problem has not even been defined in the case of imperfect models. Data assimilation in a nonlinear, perfect-model scenario is considered. It is shown that a new fully nonlinear approach based upon the indistinguishable states (IS) systematically outperforms the ensemble Kalman filter (EnKF). The IS provides an ensemble of initial conditions, consistent with (i) the model dynamics, (ii) the observational noise model, and (iii) the particular observations over a window. It is argued that this is the relevant limit to consider in data assimilation, when the desire is to place high probability density in the vicinity of the target state. The advantages of the IS approach come in part from its ability to provide attractor-balanced ensembles near any attracting manifold the system may evolve on. The use of an EnKF, provides a computationally cheaper alternative that place points in the general vicinity of the target. A low (i.e., 2) dimensional example is used to provide easily visualized evidence for these claims, which are then tested in a higher (i.e., 12) dimensional system. Inasmuch as the IS approach is shown to outperform the EnKF systematically in these perfect-model experiments, it provides an interesting alternative approach when informative ensembles are desired.

Corresponding author address: Shree Khare, NCAR, 1850 Table Mesa Dr., Boulder, CO 80305. E-mail: shree.khare@yahoo.com
Save
  • Anderson, J. L., 2001: An ensemble adjustment Kalman filter for data assimilation. Mon. Wea. Rev., 129, 28842903.

  • Anderson, J. L., 2003: A local least squares framework for ensemble filtering. Mon. Wea. Rev., 131, 634642.

  • Anderson, J. L., and S. L. Anderson, 1999: A Monte Carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Mon. Wea. Rev., 127, 27412758.

    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., B. Wyman, S. Zhang, and T. Hoar, 2005: Assimilation of surface pressure observations using an ensemble filter in an idealized global atmospheric prediction system. J. Atmos. Sci., 62, 29252938.

    • Search Google Scholar
    • Export Citation
  • Arnold, C., and C. Dey, 1986: Observation system simulation experiments: Past, present, and future. Bull. Amer. Meteor. Soc., 67, 687695.

    • Search Google Scholar
    • Export Citation
  • Bengtsson, T., C. Snyder, and D. Nychka, 2003: Toward a nonlinear ensemble filter for high-dimensional systems. J. Geophys. Res., 108, 8775, doi:10.1029/2002JD002900.

    • Search Google Scholar
    • Export Citation
  • Bengtsson, T., P. Bickel, and B. Li, 2008: Curse of dimensionality revisited: The collapse of importance sampling in very large scale systems. Probability and Statistics: Essays in Honor of David A. Freedman, D. Nolan and T. Speed, Eds., Vol. 2, Institute of Mathematical Statistics, 316–334.

    • Search Google Scholar
    • Export Citation
  • Bishop, C., C. A. Reynolds, and M. K. Tippett, 2003: Optimization of the fixed global observing network in a simple model. J. Atmos. Sci., 60, 14711489.

    • Search Google Scholar
    • Export Citation
  • Brocker, J., and L. A. Smith, 2007: Scoring probabilistic forecasts: The importance of being proper. Wea. Forecasting, 22, 382388.

  • Daley, R., 1991: Atmospheric Data Analysis. Cambridge University Press, 457 pp.

  • Doucet, A., N. de Freitas, and N. Gordon, 2001: Sequential Monte Carlo Methods in Practice. Springer, 581 pp.

  • Du, H., 2009: Combining statistical methods with dynamical insight to improve nonlinear estimation. Ph.D. thesis, London School of Economics, 190 pp.

  • Evensen, G., 1994: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res., 99, 10 14310 162.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., J. S. Whitaker, and C. Snyder, 2001: Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter. Mon. Wea. Rev., 129, 27762790.

    • Search Google Scholar
    • Export Citation
  • Hansen, J. A., and L. A. Smith, 2001: Probabilistic noise reduction. Tellus, 53A, 587598.

  • Ikeda, K., 1979: Multiple-valued stationary state and its instability of the transmitted light by a ring cavity system. Opt. Commun., 30, 257261.

    • Search Google Scholar
    • Export Citation
  • Jazwinski, A. H., 1970: Stochastic Processes and Filtering Theory. Academic Press, 376 pp.

  • Judd, K., 2003: Nonlinear state estimation, indistinguishable states and the extended Kalman filter. Physica D, 183, 273281.

  • Judd, K., and L. Smith, 2001: Indistinguishable states I. Perfect model scenario. Physica D, 151, 125141.

  • Judd, K., and L. A. Smith, 2004: Indistinguishable states II. The imperfect model scenario. Physica D, 196, 224242.

  • Judd, K., L. A. Smith, and A. Weisheimer, 2004: Gradient free descent: Shadowing, and state estimation using limited derivative information. Physica D, 190, 153166.

    • Search Google Scholar
    • Export Citation
  • Judd, K., C. A. Reynods, T. E. Rosmond, and L. Smith, 2008: The geometry of model error. J. Atmos. Sci., 65, 17491772.

  • Kalman, R. E., 1960: A new approach to linear filtering and prediction problems. Trans. AMSE Ser. D. J. Basic Eng., 82, 3545.

  • Kim, S., G. L. Eyink, J. M. Restrepo, F. J. Alexander, and G. Johnson, 2003: Ensemble filtering for nonlinear dynamics. Mon. Wea. Rev., 131, 25862594.

    • Search Google Scholar
    • Export Citation
  • Lawson, W. G., and J. A. Hansen, 2004: Implications of stochastic and deterministic filters as ensemble-based data assimilation methods in varying regimes of error growth. Mon. Wea. Rev., 132, 19661981.

    • Search Google Scholar
    • Export Citation
  • Leith, C. E., 1974: Theoretical skill of Monte Carlo forecasts. Mon. Wea. Rev., 102, 409418.

  • Lorenc, A. C., 1986: Analysis methods for numerical weather prediction. Quart. J. Roy. Meteor. Soc., 112, 11771194.

  • Lorenz, E. N., 1963: Deterministic nonperiodic flow. J. Atmos. Sci., 20, 130141.

  • Lorenz, E. N., 1996: Predictability: A problem partly solved. Proc. Seminar on Predictability, Vol. 1, Reading, Berkshire, United Kingdom, ECMWF, 1–18.

    • Search Google Scholar
    • Export Citation
  • Lorenz, E. N., 2005: Designing chaotic models. J. Atmos. Sci., 62, 15741587.

  • McSharry, P., and L. A. Smith, 1999: Better nonlinear models from noisy data: Attractors with maximum likelihood. Phys. Rev. Lett., 83, 42854288.

    • Search Google Scholar
    • Export Citation
  • McSharry, P., L. A. Smith, and L. Tarassenko, 2003: Prediction of epileptic seizures: Are nonlinear methods relevant? Nat. Med., 9, 241242.

    • Search Google Scholar
    • Export Citation
  • Pham, D. T., 2001: Stochastic methods for sequential data assimilation in strongly nonlinear systems. Mon. Wea. Rev., 129, 11941207.

  • Ridout, D., and K. Judd, 2002: Convergence properties of gradient descent noise reduction. Physica D, 165, 2647.

  • Smith, L. A., 1996: Accountability and error in ensemble forecasting. Proc. Seminar on Predictability, Vol. 1, Reading, Berkshire, United Kingdom, ECMWF, 351–368.

    • Search Google Scholar
    • Export Citation
  • Smith, L. A., C. Ziehmann, and K. Fraedrich, 1999: Uncertainty dynamics and predictability in chaotic systems. Quart. J. Roy. Meteor. Soc., 125, 28552886.

    • Search Google Scholar
    • Export Citation
  • Tippett, M. K., J. L. Anderson, C. H. Bishop, T. M. Hamill, and J. S. Whitaker, 2003: Ensemble square-root filters. Mon. Wea. Rev., 131, 14851490.

    • Search Google Scholar
    • Export Citation
  • van Leeuwen, P. J., 2001: A variance-minimizing filter for large-scale applications. Mon. Wea. Rev., 131, 20712084.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 140 55 6
PDF Downloads 43 10 3