• Abarbanel, H. D. I., D. R. Creveling, R. Farsian, and M. Kostuk, 2009: Dynamical state and parameter estimation. SIAM J. Appl. Dyn. Syst., 8, 13411381.

    • Search Google Scholar
    • Export Citation
  • Allen, M. B., and E. L. Isaacson, 1998: The secant method. Numerical Analysis for Applied Science, M. B. Allen et al., Eds., John Wiley and Sons, 188–195.

    • Search Google Scholar
    • Export Citation
  • Anderson, B. D. O., and J. B. Moore, 1979: Optimal Filtering. Prentice-Hall, 357 pp.

  • Anderson, J. L., 2001: An ensemble adjustment Kalman filter for data assimilation. Mon. Wea. Rev., 129, 28842903.

  • Anderson, J. L., 2003: A local least squares framework for ensemble filtering. Mon. Wea. Rev., 131, 634642.

  • Anderson, J. L., and S. L. Anderson, 1999: A Monte Carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Mon. Wea. Rev., 127, 27412758.

    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., B. Wyman, S. Q. Zhang, and T. Hoar, 2005: Assimilation of surface pressure observations using an ensemble filter in an idealized global atmospheric prediction system. J. Atmos. Sci., 62, 29252938.

    • Search Google Scholar
    • Export Citation
  • Bennett, A. F., B. S. Chua, and L. M. Leslie, 1996: Generalized inversion of a global numerical weather prediction model. Meteor. Atmos. Phys., 60, 165178.

    • Search Google Scholar
    • Export Citation
  • Berliner, M. L., 1991: Likelihood and Bayesian prediction for chaotic systems. J. Amer. Stat. Assoc., 86, 938952.

  • Bernardo, J. M., 1979: Expected information as expected utility. Ann. Stat., 7, 686690.

  • Bishop, C. H., B. J. Etherton, and S. J. Majumdar, 2001: Adaptive sampling with the ensemble transform Kalman filter. Part I: Theoretical aspects. Mon. Wea. Rev., 129, 420436.

    • Search Google Scholar
    • Export Citation
  • Bouttier, F., 1994: A dynamical estimation of forecast error covariances in an assimilation system. Mon. Wea. Rev., 122, 23762390.

  • Brocker, J., and L. A. Smith, 2007: Scoring probabilistic forecasts: On the importance of being proper. Wea. Forecasting, 22, 382388.

    • Search Google Scholar
    • Export Citation
  • Brocker, J., and L. A. Smith, 2008: From ensemble forecasts to predictive distribution functions. Tellus, 60, 663678.

  • Burgers, G., P. J. V. Leeuwen, and G. Evensen, 1998: Analysis scheme in the ensemble Kalman filter. Mon. Wea. Rev., 126, 17191724.

  • Cohn, S. E., 1997: An introduction to estimation theory. J. Meteor. Soc. Japan, 75, 257288.

  • Courtier, P., J. N. Thepaut, and A. Hollingsworth, 1994: A strategy for operational implementation of 4D-Var, using an incremental approach. Quart. J. Roy. Meteor. Soc., 120, 13671387.

    • Search Google Scholar
    • Export Citation
  • Dimet, F.-X. L., and O. Talagrand, 1986: Variational algorithms for analysis and assimilation of meteorological observations: Theoretical aspects. Tellus, 38A, 97110.

    • Search Google Scholar
    • Export Citation
  • Du, H., 2009: Combining statistical methods with dynamical insight to improve nonlinear estimation. Ph.D. dissertation, London School of Economics and Political Science, 190 pp.

  • Du, H., and L. A. Smith, 2012: Parameter estimation through ignorance. Phys. Rev. E,86, 016213, doi:10.1103/PhysRevE.86.016213.

  • Du, H., and L. A. Smith, 2014: Pseudo-orbit data assimilation. Part II: Assimilation with imperfect models. J. Atmos. Sci.,71, 483–495.

  • Evensen, G., 1994: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res., 99 (C5), 10 14310 162.

    • Search Google Scholar
    • Export Citation
  • Fertig, E. J., J. Harlim, and B. R. Hunt, 2007: A comparative study of 4D-VAR and a 4D ensemble Kalman filter: Perfect model simulations with Lorenz-96. Tellus, 59A, 96100.

    • Search Google Scholar
    • Export Citation
  • Gauthier, P., 1992: Chaos and quadri-dimensional data assimilation: A study based on the Lorenz model. Tellus, 44A, 217.

  • Gelb, A., 1974: Applied Optimal Estimation. MIT Press, 382 pp.

  • Ghil, M., and P. Malanotte-Rizzoli, 1991: Data assimilation in meteorology and oceanography. Advances in Geophysics, Vol. 33, Academic Press, 141–266.

  • Gilmour, I., 1998: Nonlinear model evaluation: ι-shadowing, probabilistic prediction and weather forecasting. Ph.D. dissertation, University of Oxford, 184 pp.

  • Good, I. J., 1952: Rational decisions. J. Roy. Stat. Soc., 14A, 107114.

  • Hagedorn, R., and L. A. Smith, 2009: Communicating the value of probabilistic forecasts with weather roulette. Meteor. Appl., 16, 143155.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., 2006: Ensemble-based atmospheric data assimilation. Predictability of Weather and Climate, T. Palmer and R. Hagedorn, Eds., Cambridge University Press, 124–156.

  • Hamill, T. M., J. S. Whitaker, and C. Snyder, 2001: Distance-dependent filtering of background-error covariance estimates in an ensemble Kalman filter. Mon. Wea. Rev., 129, 27762790.

    • Search Google Scholar
    • Export Citation
  • Haramel, S. M., C. K. R. T. Jones, and J. V. Moloney, 1985: Global dynamical behavior of the optical field in a ring cavity. J. Opt. Soc. Amer., 2B, 552564.

    • Search Google Scholar
    • Export Citation
  • Harlim, J., and A. J. Majda, 2010: Catastrophic filter divergence in filtering nonlinear dissipative systems. Commun. Math. Sci., 8, 2743.

    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., and H. L. Mitchell, 1998: Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev., 126, 796811.

    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., G. Pellerin, M. Buehner, M. Charron, L. Spacek, and B. Hansen, 2005: Atmospheric data assimilation with the ensemble Kalman filter: Results with real observations. Mon. Wea. Rev., 133, 604620.

    • Search Google Scholar
    • Export Citation
  • Ikeda, K., 1979: Multiple valued stationarity state and its instability of the transmitted light by a ring cavity system. Opt. Commun., 30, 257261.

    • Search Google Scholar
    • Export Citation
  • Jazwinski, A. H., 1970: Stochastic Processes and Filtering Theory. Academic Press, 376 pp.

  • Judd, K., and L. A. Smith, 2001: Indistinguishable states I: The perfect model scenario. Physica D, 151, 125141.

  • Judd, K., and L. A. Smith, 2004: Indistinguishable states II: The imperfect model scenario. Physica D, 196, 224242.

  • Judd, K., L. A. Smith, and A. Weisheimer, 2004: Gradient free descent: Shadowing and state estimation with limited derivative information. Physica D, 190, 153166.

    • Search Google Scholar
    • Export Citation
  • Judd, K., C. A. Reynolds, T. E. Rosmond, and L. A. Smith, 2008: The geometry of model error. J. Atmos. Sci., 65, 17491772.

  • Kaipio, J., and E. Somersalo, 2005: Statistical and Computational Inverse Problems. Applied Mathematical Sciences, Vol. 160, Springer, 344 pp.

    • Search Google Scholar
    • Export Citation
  • Kalman, R. E., 1960: A new approach to linear filtering and prediction problems. J. Basic Eng., 82, 3545.

  • Kalnay, E., 2003: Atmospheric Modeling, Data Assimilation and Predictability. Cambridge University Press, 364 pp.

  • Keppenne, C. L., and M. Rienecker, 2002: Initial testing of a massively parallel ensemble Kalman filter with the Poseidon isopycnal ocean general circulation model. Mon. Wea. Rev., 130, 29512965.

    • Search Google Scholar
    • Export Citation
  • Khare, S., and L. Smith, 2011: Data assimilation: A fully nonlinear approach to ensemble formation using indistinguishable states. Mon. Wea. Rev., 139, 20802097.

    • Search Google Scholar
    • Export Citation
  • Lalley, S. P., 2000: Removing the noise from chaos plus noise. Nonlinear Dynamics and Statistics, A. I. Mees, Ed., Birkhäuser, 233–244.

  • Lawson, G., and J. A. Hansen, 2004: Implications of stochastic and deterministic filters as ensemble-based data assimilation methods in varying regimes of error growth. Mon. Wea. Rev., 132, 19661981.

    • Search Google Scholar
    • Export Citation
  • Lermusiaux, P. F. J., and A. R. Robinson, 1999: Data assimilation via error subspace statistical estimation. Part I: Theory and schemes. Mon. Wea. Rev., 127, 13851407.

    • Search Google Scholar
    • Export Citation
  • Leutbecher, M., and T. N. Palmer, 2008: Ensemble forecasting. J. Comput. Phys., 227, 35153539.

  • Lorenc, A. C., 1986: Analysis methods for numerical weather prediction. Quart. J. Roy. Meteor. Soc., 112, 11771194.

  • Lorenz, E. N., 1963: Deterministic nonperiodic flow. J. Atmos. Sci., 20, 130141.

  • Lorenz, E. N., 1965: On the possible reasons for long-period fluctuations. WMO-IUGG symposium on research and development aspects of long-range forecasting, Boulder, Colorado, 1964, World Meteorological Organization Tech. Note 66, 203–211.

  • Lorenz, E. N., 1995: Predictability: A problem partly solved. Proc. Seminar on Predictability, Shinfield Park, United Kingdom, ECMWF, 40–58.

  • Lu, C. G., and G. L. Browning, 1998: The impact of observational and model errors on four-dimensional variational data assimilation. J. Atmos. Sci., 55, 9951011.

    • Search Google Scholar
    • Export Citation
  • McSharry, P. E., and L. A. Smith, 1999: Better nonlinear models from noisy data: Attractors with maximum likelihood. Phys. Rev. Lett., 83, 42854288.

    • Search Google Scholar
    • Export Citation
  • Metropolis, N., and S. Ulam, 1944: The Monte Carlo method. J. Amer. Stat. Assoc., 44, 335341.

  • Miller, R. N., M. Ghil, and F. Gauthiez, 1994: Advanced data assimilation in strongly nonlinear dynamical systems. J. Atmos. Sci., 51, 10371056.

    • Search Google Scholar
    • Export Citation
  • Orrell, D., 1999: A shadow of a doubt: Model error, uncertainty, and shadowing in nonlinear dynamical systems. Ph.D. dissertation, University of Oxford, 199 pp.

  • Pires, C., R. Vautard, and O. Talagrand, 1996: On extending the limits of variational assimilation in nonlinear chaotic systems. Tellus, 48A, 96121.

    • Search Google Scholar
    • Export Citation
  • Pisarenko, V. F., and D. Sornette, 2004: Statistical methods of parameter estimation for deterministically chaotic time series. Phys. Rev. E,69, 036122, doi:10.1103/PhysRevE.69.036122.

  • Raftery, A. E., T. Gneiting, F. Balabdaoui, and M. Polakowski, 2005: Using Bayesian model averaging to calibrate forecast ensembles. Mon. Wea. Rev., 133, 11551174.

    • Search Google Scholar
    • Export Citation
  • Ridout, D., and K. Judd, 2002: Convergence properties of gradient descent noise reduction. Physica D, 165, 2748.

  • Roulston, M. S., and L. A. Smith, 2002: Evaluating probabilistic forecasts using information theory. Mon. Wea. Rev., 130, 16531660.

  • Sakov, P., and P. R. Oke, 2008: Implications of the form of the ensemble transformations in the ensemble square root filters. Mon. Wea. Rev., 136, 10421053.

    • Search Google Scholar
    • Export Citation
  • Schittkowski, K., 1994: Parameter estimation in systems of nonlinear equations. Numer. Math., 68, 129142.

  • Shewchuk, J. R., 1994: An introduction to the conjugate gradient method without the agonizing pain. Carnegie Mellon University Tech. Rep., 58 pp.

  • Smith, L. A., 1996: Accountability and error in ensemble forecasting. Proc. Seminar on Predictability, Shinfield Park, United Kingdom, ECMWF, 351–368.

  • Smith, L. A., 2000: Disentangling uncertainty and error: On the predictability of nonlinear systems. Nonlinear Dynamics and Statistics, A. I. Mees, Ed., Birkhäuser, 31–64.

  • Smith, L. A., 2002: What might we learn from climate forecasts? Proc. Natl. Acad. Sci. USA, 4, 24872492.

  • Smith, L. A., 2006: Predictability past predictability present. Predictability of Weather and Climate, T. Palmer and R. Hagedorn, Eds., Cambridge University Press, 217–250.

  • Smith, L. A., M. C. Cuellar, H. Du, and K. Judd, 2010: Exploiting dynamical coherence: A geometric approach to parameter estimation in nonlinear models. Phys. Lett., 374A, 26182623.

    • Search Google Scholar
    • Export Citation
  • Snyder, C., T. Bengtsson, P. Bickel, and J. Anderson, 2008: Obstacles to high-dimensional particle filtering. Mon. Wea. Rev., 136, 46294640.

    • Search Google Scholar
    • Export Citation
  • Stemler, T., and K. Judd, 2009: A guide to using shadowing filters for forecasting and state estimation. Physica D, 238, 12601273.

  • Stensrud, D. J., and J. W. Bao, 1992: Behaviors of variational and nudging assimilation techniques with a chaotic low-order model. Mon. Wea. Rev., 120, 30163028.

    • Search Google Scholar
    • Export Citation
  • Talagrand, O., and P. Courtier, 1987: Variational assimilation of meteorological observations with the adjoint vorticity equation. I: Theory. Quart. J. Roy. Meteor. Soc., 113, 13111328.

    • Search Google Scholar
    • Export Citation
  • Tarantola, A., 2004: Inverse Problem Theory and Methods for Model Parameter Estimation. SIAM, 352 pp.

  • Tippett, M. K., J. L. Anderson, C. H. Bishop, T. M. Hamill, and J. S. Whitaker, 2003: Ensemble square root filters. Mon. Wea. Rev., 131, 14851490.

    • Search Google Scholar
    • Export Citation
  • Toth, Z., and E. Kalnay, 1993: Ensemble forecasting at NMC: The generation of perturbations. Bull. Amer. Meteor. Soc., 74, 23172330.

  • Voss, H. U., J. Timmer, and J. Kurths, 2004: Nonlinear dynamical system identification from uncertain and indirect measurements. Int. J. Bifurcation Chaos, 14, 19051933.

    • Search Google Scholar
    • Export Citation
  • Wiener, N., 1949: Extrapolation, Interpolation, and Smoothing of Stationary Time Series: With Engineering Applications. MIT Press, 163 pp.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 109 53 11
PDF Downloads 56 24 1

Pseudo-Orbit Data Assimilation. Part I: The Perfect Model Scenario

Hailiang DuCentre for the Analysis of Time Series, London School of Economics and Political Science, London, United Kingdom

Search for other papers by Hailiang Du in
Current site
Google Scholar
PubMed
Close
and
Leonard A. SmithCentre for the Analysis of Time Series, London School of Economics and Political Science, London, United Kingdom

Search for other papers by Leonard A. Smith in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

State estimation lies at the heart of many meteorological tasks. Pseudo-orbit-based data assimilation provides an attractive alternative approach to data assimilation in nonlinear systems such as weather forecasting models. In the perfect model scenario, noisy observations prevent a precise estimate of the current state. In this setting, ensemble Kalman filter approaches are hampered by their foundational assumptions of dynamical linearity, while variational approaches may fail in practice owing to local minima in their cost function. The pseudo-orbit data assimilation approach improves state estimation by enhancing the balance between the information derived from the dynamic equations and that derived from the observations. The potential use of this approach for numerical weather prediction is explored in the perfect model scenario within two deterministic chaotic systems: the two-dimensional Ikeda map and 18-dimensional Lorenz96 flow. Empirical results demonstrate improved performance over that of the two most common traditional approaches of data assimilation (ensemble Kalman filter and four-dimensional variational assimilation).

Corresponding author address: Hailiang Du, Centre for the Analysis of Time Series, London School of Economics and Political Science, Houghton Street, London WC2A 2AE, United Kingdom. E-mail: h.l.du@lse.ac.uk

Abstract

State estimation lies at the heart of many meteorological tasks. Pseudo-orbit-based data assimilation provides an attractive alternative approach to data assimilation in nonlinear systems such as weather forecasting models. In the perfect model scenario, noisy observations prevent a precise estimate of the current state. In this setting, ensemble Kalman filter approaches are hampered by their foundational assumptions of dynamical linearity, while variational approaches may fail in practice owing to local minima in their cost function. The pseudo-orbit data assimilation approach improves state estimation by enhancing the balance between the information derived from the dynamic equations and that derived from the observations. The potential use of this approach for numerical weather prediction is explored in the perfect model scenario within two deterministic chaotic systems: the two-dimensional Ikeda map and 18-dimensional Lorenz96 flow. Empirical results demonstrate improved performance over that of the two most common traditional approaches of data assimilation (ensemble Kalman filter and four-dimensional variational assimilation).

Corresponding author address: Hailiang Du, Centre for the Analysis of Time Series, London School of Economics and Political Science, Houghton Street, London WC2A 2AE, United Kingdom. E-mail: h.l.du@lse.ac.uk
Save