Editorial Type:
Article
Article Type:
Research Article

Data Assimilation via Error Subspace Statistical Estimation.Part I: Theory and Schemes

P. F. J. Lermusiaux Division of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts

Search for other papers by P. F. J. Lermusiaux in
Current site
Google Scholar
PubMed Close
and
A. R. Robinson Division of Engineering and Applied Sciences, Department of Earth and Planetary Sciences, Harvard University,Cambridge, Massachusetts

Search for other papers by A. R. Robinson in
Current site
Google Scholar
PubMed Close
Print Publication:
01 Jul 1999
DOI:
https://doi.org/10.1175/1520-0493(1999)127<1385:DAVESS>2.0.CO;2
Page(s):
1385–1407
Restricted access

Abstract

A rational approach is used to identify efficient schemes for data assimilation in nonlinear ocean–atmosphere models. The conditional mean, a minimum of several cost functionals, is chosen for an optimal estimate. After stating the present goals and describing some of the existing schemes, the constraints and issues particular to ocean–atmosphere data assimilation are emphasized. An approximation to the optimal criterion satisfying the goals and addressing the issues is obtained using heuristic characteristics of geophysical measurements and models. This leads to the notion of an evolving error subspace, of variable size, that spans and tracks the scales and processes where the dominant errors occur. The concept of error subspace statistical estimation (ESSE) is defined. In the present minimum error variance approach, the suboptimal criterion is based on a continued and energetically optimal reduction of the dimension of error covariance matrices. The evolving error subspace is characterized by error singular vectors and values, or in other words, the error principal components and coefficients.

Schemes for filtering and smoothing via ESSE are derived. The data–forecast melding minimizes variance in the error subspace. Nonlinear Monte Carlo forecasts integrate the error subspace in time. The smoothing is based on a statistical approximation approach. Comparisons with existing filtering and smoothing procedures are made. The theoretical and practical advantages of ESSE are discussed. The concepts introduced by the subspace approach are as useful as the practical benefits. The formalism forms a theoretical basis for the intercomparison of reduced dimension assimilation methods and for the validation of specific assumptions for tailored applications. The subspace approach is useful for a wide range of purposes, including nonlinear field and error forecasting, predictability and stability studies, objective analyses, data-driven simulations, model improvements, adaptive sampling, and parameter estimation.

Corresponding author address: Dr. Pierre F. J. Lermusiaux, Division of Engineering and Applied Sciences, Harvard University, 29 Oxford Street, Cambridge, MA 02138.

Email: pierrel@pacific.harvard.edu

Abstract

A rational approach is used to identify efficient schemes for data assimilation in nonlinear ocean–atmosphere models. The conditional mean, a minimum of several cost functionals, is chosen for an optimal estimate. After stating the present goals and describing some of the existing schemes, the constraints and issues particular to ocean–atmosphere data assimilation are emphasized. An approximation to the optimal criterion satisfying the goals and addressing the issues is obtained using heuristic characteristics of geophysical measurements and models. This leads to the notion of an evolving error subspace, of variable size, that spans and tracks the scales and processes where the dominant errors occur. The concept of error subspace statistical estimation (ESSE) is defined. In the present minimum error variance approach, the suboptimal criterion is based on a continued and energetically optimal reduction of the dimension of error covariance matrices. The evolving error subspace is characterized by error singular vectors and values, or in other words, the error principal components and coefficients.

Schemes for filtering and smoothing via ESSE are derived. The data–forecast melding minimizes variance in the error subspace. Nonlinear Monte Carlo forecasts integrate the error subspace in time. The smoothing is based on a statistical approximation approach. Comparisons with existing filtering and smoothing procedures are made. The theoretical and practical advantages of ESSE are discussed. The concepts introduced by the subspace approach are as useful as the practical benefits. The formalism forms a theoretical basis for the intercomparison of reduced dimension assimilation methods and for the validation of specific assumptions for tailored applications. The subspace approach is useful for a wide range of purposes, including nonlinear field and error forecasting, predictability and stability studies, objective analyses, data-driven simulations, model improvements, adaptive sampling, and parameter estimation.

Corresponding author address: Dr. Pierre F. J. Lermusiaux, Division of Engineering and Applied Sciences, Harvard University, 29 Oxford Street, Cambridge, MA 02138.

Email: pierrel@pacific.harvard.edu

Save
Cover Monthly Weather Review
Monthly Weather Review

  • Anderson, J. L., 1996: Selection of initial conditions for ensemble forecasts in a simple perfect model framework. J. Atmos. Sci.,53, 22–36.

  • Austin, J. W., and C. T. Leondes, 1981: Statistically linearized estimation of reentry trajectories. IEEE Trans. Aerosp. Electron. Syst.,17 (1), 54–61.

  • Bendat, J. S., and A. G. Piersol, 1986: Random Data Analysis and Measurement Procedures. John Wiley and Sons, 566 pp.

  • Bengtsson, L., M. Ghil, and E. Kallen, Eds., 1981: Dynamic Meteorology: Data Assimilation Methods. Springer-Verlag, 330 pp.

  • Bennett, A. F., 1992: Inverse methods in physical oceanography. Cambridge Monographs on Mechanics and Applied Mathematics, Cambridge University Press, 346 pp.

  • ——, B. S. Chua, and L. M. Leslie, 1996: Generalized inversion of a global numerical weather prediction model. Meteor. Atmos. Phys.,60 (1–3), 165–178.

  • ——, ——, and ——, 1997: Generalized inversion of a global numerical weather prediction model, II: Analysis and implementation. Meteor. Atmos. Phys.,62 (3–4), 129–140.

  • Bergé, P., Y. Pomeau, and C. Vidal, 1988: L’Ordre dans le Chaos. Vers une Approche Déterministe de la Turbulence. Wiley Interscience, 329 pp.

  • Boguslavskij, I. A., 1988: Filtering and Control. Optimization Software, 380 pp.

  • Brockett, R. W., 1991: Dynamical systems that learn subspaces. Mathematical Systems Theory: The Influence of R. E. Kalman, A. Antoulas, Ed., Springer-Verlag, 579–592.

  • Burgers, G., P. J. van Leeuwen, and G. Evensen, 1998: Analysis scheme in the ensemble Kalman filter. Mon. Wea. Rev.,126, 1719–1724.

  • Catlin, D. E., 1989: Estimation, Control, and the Discrete Kalman Filter. Vol. 71, Applied Mathematical Sciences, Springer-Verlag, 274 pp.

  • Charney, J. G., and G. R. Flierl, 1981: Oceanic analogues of large-scale atmospheric motions. Evolution of Physical Oceanography: Scientific Surveys in Honor of Henry Stommel, B. Warren and G. Wunsch, Eds., The MIT Press, 504–548.

  • Charnock, H., 1981: Air–sea interaction. Evolution of Physical Oceanography: Scientific Surveys in Honor of Henry Stommel, B. Warren and G. Wunsch, Eds., The MIT Press, 482–503.

  • Cho, Y., V. Shin, M. Oh, and Y. Lee, 1996: Suboptimal continuous filtering based on the decomposition of the observation vector. Comput. Math. Appl.,32 (4), 23–31.

  • Cohn, S. E., 1993: Dynamics of short-term univariate forecast error covariances. Mon. Wea. Rev.,121, 3123–3149.

  • ——, and D. F. Parrish, 1991: The behavior of forecast error covariances for a Kalman filter in two dimensions. Mon. Wea. Rev.,119, 1757–1785.

  • Daley, R., 1991: Atmospheric Data Analysis. Cambridge University Press, 457 pp.

  • ——, 1992a: The lagged innovation covariance: A performance diagnostic for atmospheric data assimilation. Mon. Wea. Rev.,120, 178–196.

  • ——, 1992b: Forecast-error statistics for homogeneous and inhomogeneous observation networks. Mon. Wea. Rev.,120, 627–643.

  • ——, 1992c: Estimating model-error covariances for application to atmospheric data assimilation. Mon. Wea. Rev.,120, 1735–1746.

  • Davis, M. H. A., 1977a: Linear Estimation and Stochastic Control. Chapman-Hall, 224 pp.

  • Davis, R. E., 1977b: Techniques for statistical analysis and prediction of geophysical fluid systems. Geophys. Astrophys. Fluid Dyn.,8, 245–277.

  • Dee, D. P., 1990: Simplified adaptive Kalman filtering for large-scale geophysical models. Realization and Modelling in System Theory, M. A. Kaashoek, J. H. van Schuppen, and A. C. M. Ran, Eds., Proceedings of the International Symposium MTNS-89, Vol. 1, Birkhäuser, 567–574.

  • ——, S. E. Cohn, A. Dalcher, and M. Ghil, 1985: An efficient algorithm for estimating noise covariances in distributed systems. IEEE Trans. Control.,AC-30, 1057–1065.

  • Ehrendorfer, M., and R. M. Errico, 1995: Mesoscale predictability and the spectrum of optimal perturbations. J. Atmos. Sci.,52, 3475–3500.

  • Errico, R. M., T. E. Rosmond, and J. S. Goerss, 1993: A comparison of analysis and initialization increments in an operational data-assimilation system. Mon. Wea. Rev.,121, 579–588.

  • Evensen, G., 1993: Open boundary conditions for the extended Kalman filter with a quasi-geostrophic ocean model. J. Geophys. Res.,98, 16 529–16 546.

  • ——, 1994a: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res.,99 (C5), 10 143–10 162.

  • ——, 1994b: Inverse methods and data assimilation in nonlinear ocean models. Physica D,77, 108–129.

  • ——, 1997a: Advanced data assimilation for strongly nonlinear dynamics. Mon. Wea. Rev.,125, 1342–1354.

  • ——, 1997b: Application of ensemble integrations for predictability studies and data assimilation. Monte Carlo Simulations in Oceanography: Proc. Hawaiian Winter Workshop, Honolulu, HI, Office of Naval Research and School of Ocean and Earth Science and Technology, University of Hawaii at Manoa, 11–22.

  • ——, and P. J. van Leeuwen, 1996: Assimilation of Geosat altimeter data for the Agulhas Current using the ensemble Kalman filter with a quasigeostrophic model. Mon. Wea. Rev.,124, 85–96.

  • Farrell, B. F., and A. M. Moore, 1992: An adjoint method for obtaining the most rapidly growing perturbation to the oceanic flows. J. Phys. Oceanogr.,22, 338–349.

  • ——, and P. J. Ioannou, 1996a: Generalized stability theory. Part I: Autonomous operators. J. Atmos. Sci.,53, 2025–2040.

  • ——, and ——, 1996b: Generalized stability theory. Part II: Nonautonomous operators. J. Atmos. Sci.,53, 2041–2053.

  • Foias, C., and R. Teman, 1977: Structure of the set of stationary solutions of the Navier–Stokes equations. Commun. Pure Appl. Math.,30, 149–164.

  • ——, and ——, 1987: The connection between the Navier–Stokes equations, dynamical systems and turbulence. Directions in Partial Differential Equations, M. G. Grandall, P. H. Rabinowitz, and E. E. L. Turner, Eds., Academic Press, 55–73.

  • Fukumori, I., and P. Malanotte-Rizzoli, 1995: An approximate Kalman filter for ocean data assimilation: An example with one idealized Gulf Stream model. J. Geophys. Res.,100, 6777–6793.

  • ——, J. Benveniste, C. Wunsch, and D. B. Haidvogel, 1993: Assimilation of sea surface topography into an ocean circulation model using a steady-state smoother. J. Phys. Oceanogr.,23, 1831–1855.

  • Gamage, N., and W. Blumen, 1993: Comparative analysis of low-level cold fronts: Wavelet, Fourier, and empirical orthogonal function decompositions. Mon. Wea. Rev.,121, 2867–2878.

  • Gelb, A., Ed., 1974: Applied Optimal Estimation. The MIT Press, 374 pp.

  • Ghil, M., 1989: Meteorological data assimilation for oceanographers. Part I: Description and theoretical framework. Dyn. Atmos. Oceans,13 (3–4), 171–218.

  • Hasselmann, K., 1988: PIPs and POPs. A general formalism for the reduction of dynamical systems in terms of principal interaction patterns and principal oscillation patterns. J. Geophys. Res.,93, 11 015–11 021.

  • Horn, R. A., and C. R. Johnson, 1985: Matrix Analysis. Cambridge University Press, 561 pp.

  • ——, and ——, 1991: Topics in Matrix Analysis. Cambridge University Press, 607 pp.

  • Jazwinski, A. H., 1970: Stochastic Processes and Filtering Theory. Academic Press, 376 pp.

  • Jiang, S., and M. Ghil, 1993: Dynamical properties of error statistics in a shallow-water model. J. Phys. Oceanogr.,23, 2541–2566.

  • Kolmogorov, A. N., 1941: Dokl. Akad. Nauk SSSR,30, 301; 32, 16.

  • Le Dimet, F. X., and O. Talagrand, 1986: Variational algorithms for analysis and assimilation of meteorological observations. Tellus,38A, 97–110.

  • Lermusiaux, P. F. J., 1997: Error subspace data assimilation methods for ocean field estimation: Theory, validation and applications. Ph.D. thesis, Harvard University, Cambridge, MA, 402 pp.

  • ——, 1999a: Data assimilation via error subspace statistical estimation. Part II: Middle Atlantic Bight shelfbreak front simulations and ESSE validation. Mon. Wea. Rev.,127, 1408–1432.

  • ——, 1999b: Estimation and study of mesoscale variability in the Strait of Sicily. Dyn. Atmos. Oceans, in press.

  • Lorenc, A. C., 1986: Analysis methods for numerical weather prediction. Quart. J. Roy. Meteor. Soc.,112, 1177–1194.

  • ——, 1992: Iterative analysis using covariance functions and filters. Quart. J. Roy. Meteor. Soc.,118, 569–591.

  • ——, R. S. Bell, and B. Macpherson, 1991: The Meteorological Office analysis correction data assimilation scheme. Quart. J. Roy. Meteor. Soc.,117, 59–89.

  • Lorenz, E. N., 1963: Deterministic nonperiodic flow. J. Atmos. Sci.,20, 130–141.

  • ——, 1965: A study of the predictability of a 28-variable atmospheric model. Tellus,17, 321–333.

  • Lozano, C. J., A. R. Robinson, H. G. Arango, A. Gangopadhyay, N. Q. Sloan, P. J. Haley, and W. G. Leslie, 1996: An interdisciplinary ocean prediction system: Assimilation strategies and structured data models. Modern Appropaches to Data Assimilation in Ocean Modelling, P. Malanotte-Rizzoli, Ed., Elsevier Oceanography Series, Elsevier Science, 413–432.

  • Martel, F., and C. Wunsch, 1993: Combined inversion of hydrography, current meter data and altimetric elevations for the North Atlantic circulation. Manuscripta Geodaetica,18 (4), 219–226.

  • McWilliams, J. C., W. B. Owens, and B. L. Hua, 1986: An objective analysis of the POLYMODE Local Dynamics Experiment. Part I: General formalism and statistical model selection. J. Phys. Oceanogr.,16, 483–504.

  • Miller, A. J., and B. D. Cornuelle, 1999: Forecasts from fits of frontal fluctuations. Dyn. Atmos. Oceans, in press.

  • Miller, R. N., and M. A. Cane, 1989: A Kalman filter analysis of sea level height in the tropical Pacific. J. Phys. Oceanogr.,19, 773–790.

  • ——, M. Ghil, and F. Gauthier, 1994: Data assimilation in strongly nonlinear dynamical systems. J. Atmos. Sci.,51, 1037–1056.

  • ——, E. F. Carter, and S. T. Blue, cited 1998: Data assimilation into nonlinear stochastic models. [Available online at http://tangaroa.oce.orst.edu/stochast.html.].

  • Molteni, F., and T. N. Palmer, 1993: Predictability and finite-time instability of the northern winter circulation. Quart. J. Roy. Meteor. Soc.,119, 269–298.

  • ——, R. Buizza, T. N. Palmer, and T. Petroliagis, 1996: The ECMWF ensemble prediction system: Methodology and validation. Quart. J. Roy. Meteor. Soc.,122, 73–119.

  • Monin, A. S., 1974: Variability of the Oceans. Wiley, 241 pp.

  • Moore, A. M., and B. F. Farrell, 1994: Using adjoint models for stability and predictability analysis. NATO ASI Ser., Vol. 119, 217–239.

  • Mureau, R., F. Molteni, and T. N. Palmer, 1993: Ensemble prediction using dynamically conditioned perturbations. Quart. J. Roy. Meteor. Soc.,119, 299–323.

  • Osborne, A. R., and A. Pastorello, 1993: Simultaneous occurrence of low-dimensional chaos and colored random noise in nonlinear physical systems. Phys. Lett. A,181, 159–171.

  • Parrish, D. F., and S. E. Cohn, 1985: A Kalman filter for a two-dimensional shallow-water model: Formulation and preliminary experiments. Office Note 304, NOAA/NWS/NMC, 64 pp.

  • Penland, C., 1989: Random forcing and forecasting using principal oscillation pattern analysis. Mon. Wea. Rev.,117, 2165–2185.

  • Phillips, N. A., 1986: The spatial statistics of random geostrophic modes and first-guess errors. Tellus,38A, 314–332.

  • Preisendorfer, R. W., 1988: Principal Component Analysis in Meteorology and Oceanography. Elsevier, 426 pp.

  • Rabier, F., E. Klinker, P. Courtier, and A. Hollingsworth, 1996: Sensitivity of forecast errors to initial conditions. Quart. J. Roy. Meteor. Soc.,122, 121–150.

  • Reid, W. T., 1968: Generalized inverses of differential and integral operators. Theory and Applications of Generalized Inverse of Matrices, T. L. Bouillon and P. L. Odell, Eds., Lublock, 1–25.

  • Robinson, A. R., 1989: Progress in Geophysical Fluid Dynamics. Vol. 26, Earth-Science Reviews, Elsevier Science.

  • ——, M. A. Spall, L. J. Walstad, and W. G. Leslie, 1989: Data assimilation and dynamical interpolation in gulfcast experiments. Dyn. Atmos. Oceans,13 (3–4), 301–316.

  • ——, H. G. Arango, A. J. Miller, A. Warn-Varnas, P.-M. Poulain, and W. G. Leslie, 1996a: Real-time operational forecasting on shipboard of the Iceland–Faeroe frontal variability. Bull. Amer. Meteor. Soc.,77, 243–259.

  • ——, ——, A. Warn-Varnas, W. G. Leslie, A. J. Miller, P. J. Haley, and C. J. Lozano, 1996b: Real-time regional forecasting. Modern Approaches to Data Assimilation in Ocean Modeling, P. Malanotte-Rizzoli, Ed., Elsevier Science, 455 pp.

  • ——, J. Sellschopp, A. Warn-Varnas, W. G. Leslie, C. J. Lozano, P. J. Haley Jr., L. A. Anderson, and P. F. J. Lermusiaux, 1997: The Atlantic Ionian Stream, J. Mar. Syst., in press.

  • ——, P. F. J. Lermusiaux, and N. Q. Sloan III, 1998a: Data assimilation. Processes and Methods, K. H. Brink and A. R. Robinson, Eds., The Sea: The Global Coastal Ocean I, Vol. 10, John Wiley and Sons.

  • ——, and Coauthors, 1998b: The Rapid Response 96, 97 and 98 exercises: The Strait of Sicily, Ionian Sea and Gulf of Cadiz. Harvard Open Ocean Model Rep., Rep. in Meteorology and Oceanography 57, 45 pp. [Available from Harvard Oceanography Group, DEAS, 29 Oxford St., Cambridge, MA 02138.].

  • Sasaki, Y., 1970: Some basic formalism in numerical variational analysis. Mon. Wea. Rev.,98, 875–883.

  • Schnur, R., G. Schmitz, N. Grieger, and H. von Storch, 1993: Normal modes of the atmosphere as estimated by principal oscillation patterns and derived from quasigeostrophic theory. J. Atmos. Sci.,50, 2386–2400.

  • Sundqvist, H., Ed., 1993: Special issue on adjoint applications in dynamic meteorology. Tellus,45A (5), 341–569.

  • Tarantola, A., 1987: Inverse Problem Theory: Methods for Data Fitting and Model Parameter Estimation. Elsevier, 613 pp.

  • Teman, R., 1991: Approximation of attractors, large eddy simulations and multiscale methods. Proc. Roy. Soc. London,434A, 23–29.

  • Todling, R., and M. Ghil, 1990: Kalman filtering for a two-layer two-dimensional shallow-water model. Proc. WMO Int. Symp. on Assimilation of Observations in Meteorology and Oceanography, Clermont-Ferrand, France, WMO, 454–459.

  • ——, and S. E. Cohn, 1994: Suboptimal schemes for atmospheric data assimilation based on the Kalman filter. Mon. Wea. Rev.,122, 2530–2557.

  • ——, and M. Ghil, 1994: Tracking atmospheric instabilities with the Kalman filter. Part I: Methodology and one-layer results. Mon. Wea. Rev.,122, 183–204.

  • Toth, Z., and E. Kalnay, 1993: Ensemble forecasting at NMC: The generation of perturbations. Bull. Amer. Meteor. Soc.,74, 2317–2330.

  • Uchino, E., M. Ohta, and H. Takata, 1993: A new state estimation method for a quantized stochastic sound system based on a generalized statistical linearization. J. Sound Vibration,160 (2), 193–203.

  • van Leeuwen, P. J., and G. Evensen, 1996: Data assimilation and inverse methods in terms of a probabilistic formulation. Mon. Wea. Rev.,124, 2898–2913.

  • von Storch, H., and C. Frankignoul, 1998: Empirical modal decomposition in coastal oceanography. Processes and Methods, K. H. Brink and A. R. Robinson, Eds., The Sea: The Global Coastal Ocean I, Vol. 10, John Wiley and Sons, 419–455.

  • ——, I. Bruns, I. Fishcler-Bruns, and K. Hasselmann, 1988: Principal oscillation pattern analysis of the 30- to 60-day oscillation in general circulation model equatorial troposphere. J. Geophys. Res.,93 (D9), 11 022–11 036.

  • Wallace, J. M., C. Smith, and C. S. Bretherton, 1992: Singular value decomposition of wintertime sea surface temperature and 500-mb height anomalies. J. Climate,5, 561–576.

  • Weare, B. C., and J. S. Nasstrom, 1982: Examples of extended empirical orthogonal function analyses. Mon. Wea. Rev.,110, 481–485.

  • West, B. J., and H. J. Mackey, 1991: Geophysical attractors may be only colored noise. J. Appl. Phys.,69 (9), 6747–6749.

  • Wunsch, C., 1988: Transient tracers as a problem in control theory. J. Geophys. Res.,93, 8099–8110.

  • ——, 1996: The Ocean Circulation Inverse Problem. Cambridge University Press, 456 pp.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 438 163 26
PDF Downloads 311 115 18