Editorial Type:
Article
Article Type:
Research Article

Second-Order Information in Data Assimilation

Francois-Xavier Le Dimet Laboratoire de Modélisation et Calcul, IDOPT Project, Université Joseph Fourier, Grenoble, France

Search for other papers by Francois-Xavier Le Dimet in
Current site
Google Scholar
PubMed Close
,
I. M. Navon Department of Mathematics and School of Computational Science and Information Technology, The Florida State University, Tallahassee, Florida

Search for other papers by I. M. Navon in
Current site
Google Scholar
PubMed Close
, and
Dacian N. Daescu Department of Mathematics, The University of Iowa, Iowa City, Iowa

Search for other papers by Dacian N. Daescu in
Current site
Google Scholar
PubMed Close
Print Publication:
01 Mar 2002
DOI:
https://doi.org/10.1175/1520-0493(2002)130<0629:SOIIDA>2.0.CO;2
Page(s):
629–648
Restricted access

Abstract

In variational data assimilation (VDA) for meteorological and/or oceanic models, the assimilated fields are deduced by combining the model and the gradient of a cost functional measuring discrepancy between model solution and observation, via a first-order optimality system. However, existence and uniqueness of the VDA problem along with convergence of the algorithms for its implementation depend on the convexity of the cost function. Properties of local convexity can be deduced by studying the Hessian of the cost function in the vicinity of the optimum. This shows the necessity of second-order information to ensure a unique solution to the VDA problem.

In this paper a comprehensive review of issues related to second-order analysis of the problem of VDA is presented along with many important issues closely connected to it. In particular issues of existence, uniqueness, and regularization through second-order properties are examined. The focus then shifts to second-order information related to statistical properties and to issues related to preconditioning and optimization methods and second-order VDA analysis. Predictability and its relation to the structure of the Hessian of the cost functional is then discussed along with issues of sensitivity analysis in the presence of data being assimilated. Computational complexity issues are also addressed and discussed.

Automatic differentiation issues related to second-order information are also discussed along with the computational complexity of deriving the second-order adjoint.

Finally an application aimed at illustrating the use of automatic differentiation for deriving the second-order adjoint as well as the Hessian/vector product applied to minimizing a cost functional of a meteorological problem using the truncated-Newton method is presented. Results verifying numerically the computational cost of deriving the second-order adjoint as well as results related to the spectrum of the Hessian of the cost functional are displayed and discussed.

Corresponding author address: Dr. I. M. Navon, School of Computational Science and Information Technology, The Florida State University, Tallahassee, FL 32306-4120. Email: navon@csit.fsu.edu

Abstract

In variational data assimilation (VDA) for meteorological and/or oceanic models, the assimilated fields are deduced by combining the model and the gradient of a cost functional measuring discrepancy between model solution and observation, via a first-order optimality system. However, existence and uniqueness of the VDA problem along with convergence of the algorithms for its implementation depend on the convexity of the cost function. Properties of local convexity can be deduced by studying the Hessian of the cost function in the vicinity of the optimum. This shows the necessity of second-order information to ensure a unique solution to the VDA problem.

In this paper a comprehensive review of issues related to second-order analysis of the problem of VDA is presented along with many important issues closely connected to it. In particular issues of existence, uniqueness, and regularization through second-order properties are examined. The focus then shifts to second-order information related to statistical properties and to issues related to preconditioning and optimization methods and second-order VDA analysis. Predictability and its relation to the structure of the Hessian of the cost functional is then discussed along with issues of sensitivity analysis in the presence of data being assimilated. Computational complexity issues are also addressed and discussed.

Automatic differentiation issues related to second-order information are also discussed along with the computational complexity of deriving the second-order adjoint.

Finally an application aimed at illustrating the use of automatic differentiation for deriving the second-order adjoint as well as the Hessian/vector product applied to minimizing a cost functional of a meteorological problem using the truncated-Newton method is presented. Results verifying numerically the computational cost of deriving the second-order adjoint as well as results related to the spectrum of the Hessian of the cost functional are displayed and discussed.

Corresponding author address: Dr. I. M. Navon, School of Computational Science and Information Technology, The Florida State University, Tallahassee, FL 32306-4120. Email: navon@csit.fsu.edu

Save
Cover Monthly Weather Review
Monthly Weather Review

  • Abate, J., C. Bischof, A. Carle, and L. Roh, 1997: Algorithms and design for a second-order automatic differentiation module. Proc. Int. Symp. on Symbolic and Algebraic Computing (ISSAC) ‘97, Maui, HI, Association of Computing Machinery, 149–155.

    • Search Google Scholar
    • Export Citation

  • Alekseev, K. A., and I. M. Navon, 2001: The analysis of an ill-posed problem using multiscale resolution and second order adjoint techniques. Comput. Methods Appl. Mech. Eng, 190 , 1937–1953.

    • Search Google Scholar
    • Export Citation

  • Alekseev, K. A., and I. M. Navon, 2002: On estimation of temperature uncertainty using the second order adjoint problem. Int. J. Comput. Fluid Dyn., in press.

    • Search Google Scholar
    • Export Citation

  • Arian, E., and S. Ta'asan, 1999: Analysis of the Hessian for aerodynamic optimization: inviscid flow. Comput. Fluids, 28 , 853–877.

    • Search Google Scholar
    • Export Citation

  • Averbukh, V. Z., S. Figueroa, and T. Schlick, 1994: Remark on Algorithm-566. ACM Trans. Math Software, 20 , 282–285.

  • Barkmeijer, J., M. van Gijzen, and F. Bouttier, 1998: Singular vectors and estimates of the analysis-error covariance metric. Quart. J. Roy. Meteor. Soc, 124A , 1695–1713.

    • Search Google Scholar
    • Export Citation

  • Barkmeijer, J., R. Buizza, and T. N. Palmer, 1999: 3D-Var Hessian singular vectors and their potential use in the ECMWF Ensemble Prediction System. Quart. J. Roy. Meteor. Soc, 125B , 2333–2351.

    • Search Google Scholar
    • Export Citation

  • Bischof, C. H., 1995: Automatic differentiation, tangent linear models, and (pseudo) adjoints. Proceedings of the Workshop on High-Performance Computing in the Geosciences, F.-X. Le Dimet, Ed., NATO Advanced Science Institutes Series C: Mathematical and Physical Sciences, Vol. 462, Kluwer Academic, 59–80.

    • Search Google Scholar
    • Export Citation

  • Burger, J., J. L. Brizaut, and M. Pogu, 1992: Comparison of two methods for the calculation of the gradient and of the Hessian of the cost functions associated with differential systems. Math. Comput. Simul, 34 , 551–562.

    • Search Google Scholar
    • Export Citation

  • Coleman, T. F., and J. J. More, 1984: Estimation of sparse Hessian matrices and graph-coloring problems. Math. Program, 28 , 243–270.

    • Search Google Scholar
    • Export Citation

  • Coleman, T. F., and J. Y. Cai, 1986: The cyclic coloring problem and estimation of sparse Hessian matrices. SIAM J. Algebra Discrete Math, 7 , 221–235.

    • Search Google Scholar
    • Export Citation

  • Coleman, T. F., and J. Y. Cai, 1985a: Fortran subroutines for estimating sparse Hessian matrices. ACM Trans. Math. Software, 11 (4,) 378–378.

    • Search Google Scholar
    • Export Citation

  • Coleman, T. F., B. S. Garbow, and J. J. More, 1985b: Software for estimating sparse Hessian matrices. ACM Trans. Math. Software, 11 (4,) 363–377.

    • Search Google Scholar
    • Export Citation

  • Courtier, P., 1993: Introduction to numerical weather prediction data assimilation methods. Proc. ECMWF Seminar on Developments in the Use of Satellite Data in Numerical Weather Prediction, Reading, United Kingdom, ECMWF, 189–209.

    • Search Google Scholar
    • Export Citation

  • Courtier, P., J-N. Thepaut, and A. Hollingsworth, 1994: A strategy for operational implementation of 4D-Var, using an incremental approach. Quart J. Roy. Meteor. Soc, 120 , 1367–1388.

    • Search Google Scholar
    • Export Citation

  • Davidon, W. C., 1991: Variable metric method for minimization. SIAM J. Optim, 1 , 1–17.

  • Davidson, E. R., 1975: The iterative calculation of a few of the lowest eigenvalues and corresponding eigenvectors of a large real symmetric matrices. J. Comput. Phys, 17 , 87–94.

    • Search Google Scholar
    • Export Citation

  • Dembo, R. S., and T. Steihaug, 1983: Truncated-Newton algorithms for large-scale unconstrained optimization. Math. Program, 26 , 190–212.

    • Search Google Scholar
    • Export Citation

  • Dembo, R. S., S. C. Eisenstat, and T. Steihaug, 1982: Inexact Newton methods. SIAM J. Numer. Anal, 19 , 400–408.

  • Dixon, L. C. W., 1991: Use of automatic differentiation for calculating Hessians and Newton steps. Automatic Differentiation of Algorithms: Theory, Implementation, and Application, A. Griewank and G. F. Corliss, Eds., SIAM, 115–125.

    • Search Google Scholar
    • Export Citation

  • Ehrendorfer, M., and F. Bouttier, 1998: An explicit low-resolution extended Kalman filter: Implementation and preliminary experimentation. ECMWF Tech. Memo. 259, 27 pp.

    • Search Google Scholar
    • Export Citation

  • Fisher, M., 1998: Development of a simplified Kalman filter. ECMWF Tech. Memo. 260, 16 pp.

  • Fisher, M., and P. Courtier, 1995: Estimating the covariance matrices of analysis and forecast errors in variational data assimilation. ECMWF Tech. Memo. 220, 28 pp.

    • Search Google Scholar
    • Export Citation

  • Forsythe, G. E., and E. G. Strauss, 1955: On best conditioned matrices. Proc. Amer. Math. Soc, 6 , 340–345.

  • Gauthier, P., 1992: Chaos and quadri-dimensional data assimilation: A study based on the Lorentz model. Tellus, 44A , 2–17.

  • Gay, D. M., 1996: More AD of nonlinear AMPL models: Computing Hessian information and exploiting partial separability in computational differentiation: Techniques, applications, and tools. Proceedings in Applied Mathematics, M. Berz et al., Eds., Vol. 89, SIAM, 173–184.

    • Search Google Scholar
    • Export Citation

  • Giering, R., and T. Kaminski, 1998a: Recipes for adjoint code construction. ACM Trans. Math. Software, 24 (4,) 437–474.

  • Giering, R., and T. Kaminski, 1998b: Using TAMC to generate efficient adjoint code: Comparison of automatically generated code for evaluation of first and second order derivatives to hand written code from the Minpack-2 collection. Automatic Differentiation for Adjoint Code Generation, C. Faure, Ed., INRIA Research Rep. 3555, 31–37.

    • Search Google Scholar
    • Export Citation

  • Gilbert, J. C., 1992: Automatic differentiation and iterative processes. Optim. Methods Software, 1 , 13–21.

  • Gill, P. E., and W. Murray, 1979: Newton-type methods for unconstrained and linearly constrained optimization. Math. Program, 28 , 311–350.

    • Search Google Scholar
    • Export Citation

  • Gill, P. E., W. Murray, and M. H. Wright, 1981: Practical Optimization. Academic Press, 401 pp.

  • Grammeltvedt, A., 1969: A survey of finite-difference schemes for the primitive equations for a barotropic fluid. Mon. Wea. Rev, 97 , 387–404.

    • Search Google Scholar
    • Export Citation

  • Griewank, A., 1993: Some bounds on the complexity of gradients, Jacobians, and Hessians. Complexity in Nonlinear Optimization, P. M. Pardalos, Ed., World Scientific, 128–161.

    • Search Google Scholar
    • Export Citation

  • Griewank, A., 2000: Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation. Frontiers in Applied Mathematics, Vol. 19, SIAM, 369 pp.

    • Search Google Scholar
    • Export Citation

  • Griewank, A., 2001: Complexity of gradients, Jacobians and Hessians. Encyclopaedia of Optimization, C. A. Floudas and P. M. Pardalos, Eds., Vol. 1, Kluwer Academic, 290–300.

    • Search Google Scholar
    • Export Citation

  • Griewank, A., and G. F. Corliss, 1991: Automatic Differentiation of Algorithms: Theory, Implementation, and Application. SIAM, 353 pp.

  • Hou, G. J-W., and J. Sheen, 1993: Numerical methods for second order shape sensitivity analysis with application to heat conduction problems. Int. J. Numer. Methods Eng, 36 , 417–435.

    • Search Google Scholar
    • Export Citation

  • Hovland, P., 1995: Using ADIFOR 1.0 to Compute Hessians. Center for Research on Parallel Computation Tech. Rep. CRPC-TR95540-S, Rice University, Houston, TX, 12 pp.

    • Search Google Scholar
    • Export Citation

  • Ide, K., P. Courtier, M. Ghil, and A. Lorenc, 1997: Unified notation for data assimilation: Operational sequential and variational. J. Meteor. Soc Japan, 75B , 71–79.

    • Search Google Scholar
    • Export Citation

  • Jackson, R. H. F., and G. P. McCormic, 1988: Second order sensitivity analysis in factorable programming: Theory and applications. Math. Program, 41 , 1–28.

    • Search Google Scholar
    • Export Citation

  • Kalnay, E., S. K. Park, Z-X. Pu, and J. Gao, 2000: Application of the quasi-inverse method to data assimilation. Mon. Wea. Rev, 128 , 864–875.

    • Search Google Scholar
    • Export Citation

  • Le Dimet, F. X., and I. Charpentier, 1998: Methodes de second order en assimilation de donnees. Equations aux Dérivées Partielles et Applications (Articles Dédiées à Jacques-Louis Lions), Gauthier-Villars, 623–640.

    • Search Google Scholar
    • Export Citation

  • Le Dimet, F. X., H. E. Ngodock, B. Luong, and J. Verron, 1997: Sensitivity analysis in variational data assimilation. J. Meteor. Soc. Japan, 75B , 245–255.

    • Search Google Scholar
    • Export Citation

  • Lehoucq, R. B., D. C. Sorensen, and C. Yang, 1998: ARPACK Users' Guide: Solution of Large-Scale Eigenvalue Problems with Implicitly Restarted Arnoldi Methods. Software, Environments, and Tools, Vol. 6, SIAM, 160 pp.

    • Search Google Scholar
    • Export Citation

  • Liu, D. C., and J. Nocedal, 1989: On the limited memory BFGS method for large scale minimization. Math. Progam, 45 , 503–528.

  • Moré, J. J., B. S. Garbow, and K. E. Hillstrom, 1981: Testing unconstrained optimization software. ACM Trans. Math. Software, 7 , 17–41.

    • Search Google Scholar
    • Export Citation

  • Nash, S. G., 1984a: Newton-type minimization via the Lanczos method. SIAM J. Numer. Anal, 21 , 770–788.

  • Nash, S. G., 1984b: Truncated-Newton methods for large-scale function minimization. Applications of Nonlinear Programming to Optimization and Control, H. E. Rauch, Ed., Pergamon Press, 91–100.

    • Search Google Scholar
    • Export Citation

  • Nash, S. G., 1984c: User's guide for TN/TNBC: Fortran routines for nonlinear optimization. Mathematical Sciences Dept. Tech. Rep. 307, The Johns Hopkins University, 17 pp.

    • Search Google Scholar
    • Export Citation

  • Nash, S. G., 1984d: Solving nonlinear programming problems using truncated-Newton techniques. Numerical Optimization, P. T. Boggs, R. H. Byrd, and R. B. Schnabel, Eds., SIAM, 119–136.

    • Search Google Scholar
    • Export Citation

  • Nash, S. G., 1985: Preconditioning of truncated-Newton methods. SIAM J. Sci. Stat. Comput, 6 , 599–616.

  • Nash, S. G., 2000: A survey of truncated-Newton methods. J. Comput. Appl. Math, 124 , 45–59.

  • Nash, S. G., and A. Sofer, 1989a: Block truncated-Newton methods for parallel optimization. Math. Progam, 45 , 529–546.

  • Nash, S. G., and A. Sofer, 1989b: A parallel line search for Newton type methods in computer science and statistics. Proc. 21st Symp. on the Interface: Computing Science and Statistic, Orlando, FL, American Statistical Association, 134–137.

    • Search Google Scholar
    • Export Citation

  • Nash, S. G., and J. Nocedal, 1991: A numerical study of the limited memory BFGS method and the truncated-Newton method for large scale optimization. SIAM J. Optim, 1 , 358–372.

    • Search Google Scholar
    • Export Citation

  • Nash, S. G., and J. Nocedal, 1996: Linear and Nonlinear Programming. McGraw-Hill, 692 pp.

  • Ngodock, H. E., 1996: Data assimilation and sensitivity analysis. Ph.D. thesis, University Joseph Fourier, Grenoble, France, 213 pp.

  • Nocedal, J., 1980: Updating quasi-Newton matrices with limited storage. Math. Comput, 35 , 773–782.

  • Nocedal, J., and S. J. Wright, 1999: Numerical Optimization. Springer Verlag Series in Operations Research, 656 pp.

  • O'Leary, D. P., 1983: A discrete Newton algorithm for minimizing a function of many variables. Math. Progam, 23 , 20–23.

  • Powell, M. J. D., and P. L. Toint, 1979: Estimation of sparse Hessian matrices. SIAM J. Numer. Anal, 16 , 1060–1074.

  • Pu, Z. X., and E. Kalnay, 1999: Targeting observations with the quasi-inverse linear and adjoint NCEP global models: Performance during FASTEX. Quart. J. Roy. Meteor. Soc, 125 , 3329–3337.

    • Search Google Scholar
    • Export Citation

  • Pu, Z. X., and and Coauthors, 1997: Sensitivity of forecast errors to initial conditions with a quasi-inverse linear method. Mon. Wea. Rev, 125 , 2479–2503.

    • Search Google Scholar
    • Export Citation

  • Rabier, F., and P. Courtier, 1992: Four dimensional assimilation in the presence of baroclinic instability. Quart. J. Roy. Meteor. Soc, 118 , 649–672.

    • Search Google Scholar
    • Export Citation

  • Rabier, F., and and Coauthors, 1997: Recent experimentation on 4D-Var and first results from a simplified Kalman filter. ECMWF Tech. Memo. 240, 42 pp.

    • Search Google Scholar
    • Export Citation

  • Reuther, J. J., 1996: Aerodynamic shape optimization using control theory. Ph.D. dissertation, University of California, Davis, 226 pp.

  • Santosa, F., and W. W. Symes, 1988: Computation of the Hessian for least-squares solutions of inverse problems of reflection seismology. Inverse Problems, 4 , 211–233.

    • Search Google Scholar
    • Export Citation

  • Santosa, F., and W. W. Symes, 1989: An Analysis of Least Squares Velocity Inversion. Geophysical Monogr., Vol. 4, Society of Exploration Geophysicists, 168 pp.

    • Search Google Scholar
    • Export Citation

  • Schlick, T., and A. Fogelson, 1992a: TNPACK—A truncated Newton minimization package for large-scale problems: I. Algorithm and usage. ACM Trans. Math. Software, 18 , 46–70.

    • Search Google Scholar
    • Export Citation

  • Schlick, T., and A. Fogelson, 1992b: TNPACK—A truncated Newton minimization package for large-scale problems: II. Implementation examples. ACM Trans. Math. Software, 18 , 71–111.

    • Search Google Scholar
    • Export Citation

  • Sleijpen, G. L. G., and H. A. van der Vorst, 1996: A Jacobi–Davidson iteration method for linear eigenvalue problems. SIAM J. Matrix Anal, 17A , 401–425.

    • Search Google Scholar
    • Export Citation

  • Symes, W. W., 1990: Velocity inversion: A case study in infinite-dimensional optimization. Math. Program, 48 , 71–102.

  • Symes, W. W., 1991: A differential semblance algorithm for the inverse problem of reflection seismology. Comput. Math. Appl, 22 , (4/5),. 147–178.

    • Search Google Scholar
    • Export Citation

  • Symes, W. W., 1993: A differential semblance algorithm for the inversion of multioffset seismic reflection data. J. Geophys. Res, 98 , (B2),. 2061–2073.

    • Search Google Scholar
    • Export Citation

  • Thacker, W. C., 1989: The role of Hessian matrix in fitting models to measurements. J. Geophys. Res, 94 , 6177–6196.

  • Thepaut, J-N., and P. Moll, 1990: Variational inversion of simulated TOVS radiances using the adjoint technique. Quart. J. Roy. Meteor. Soc, 116 , 1425–1448.

    • Search Google Scholar
    • Export Citation

  • Thepaut, J-N., and P. Courtier, 1991: Four-dimensional variational assimilation using the adjoint of a multilevel primitive equation model. Quart. J. Roy. Meteor. Soc, 117 , 1225–1254.

    • Search Google Scholar
    • Export Citation

  • Veerse, F., 1999: Variable-storage quasi-Newton operators as inverse forecast/analysis error covariance matrices in data assimilation. INRIA Tech. Rep. 3685, 28 pp.

    • Search Google Scholar
    • Export Citation

  • Wang, Z., 1993: Variational data assimilation with 2-D shallow water equations and 3-D FSU global spectral models. Tech. Rep. FSU-SCRI-93T-149, The Florida State University, Tallahassee, FL, 235 pp.

    • Search Google Scholar
    • Export Citation

  • Wang, Z., I. M. Navon, F. X. Le Dimet, and X. Zou, 1992: The second order adjoint analysis: Theory and application. Meteor. Atmos. Phys, 50 , 3–20.

    • Search Google Scholar
    • Export Citation

  • Wang, Z., I. M. Navon, and X. Zou, 1993: The adjoint truncated Newton algorithm for large-scale unconstrained optimization. Tech. Rep. FSU-SCRI-92-170, The Florida State University, Tallahassee, FL, 44 pp.

    • Search Google Scholar
    • Export Citation

  • Wang, Z., I. M. Navon, X. Zou, and F. X. Le Dimet, 1995: A truncated-Newton optimization algorithm in meteorology applications with analytic Hessian/vector products. Comput. Optim. Appl, 4 , 241–262.

    • Search Google Scholar
    • Export Citation

  • Wang, Z., K. K. Droegemeier, L. White, and I. M. Navon, 1997: Application of a new adjoint Newton algorithm to the 3D ARPS storm-scale model using simulated data. Mon. Wea. Rev, 125 , 2460–2478.

    • Search Google Scholar
    • Export Citation

  • Wang, Z., K. K. Droegemeier, and L. White, 1998: The adjoint Newton algorithm for large-scale unconstrained optimization in meteorology applications. Comput. Optim. Appl, 10 , 283–320.

    • Search Google Scholar
    • Export Citation

  • Yang, W., I. M. Navon, and P. Courtier, 1996: A new Hessian preconditioning method applied to variational data assimilation experiments using an adiabatic version of NASA/GEOS-1 GCM. Mon. Wea. Rev, 124 , 1000–1017.

    • Search Google Scholar
    • Export Citation

  • Zou, X., I. M. Navon, F. X. Le Dimet, A. Nouailler, and T. Schlick, 1990: A comparison of efficient large-scale minimization algorithms for optimal control applications in meteorology. Tech. Rep. FSU-SCRI-90-167, The Florida State University, Tallahassee, FL, 44 pp.

    • Search Google Scholar
    • Export Citation

  • Zou, X., I. M. Navon, and F. X. Le Dimet, 1992: Incomplete observations and control of gravity waves in variational data assimilation. Tellus, 44A , 273–296.

    • Search Google Scholar
    • Export Citation

  • Zou, X., I. M. Navon, M. Berger, P. K. H. Phua, T. Schlick, and F. X. Le Dimet, 1993: Numerical experience with limited-memory quasi-Newton methods and truncated Newton methods. SIAM J. Numer. Optim, 3 , 582–608.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 2040 479 63
PDF Downloads 829 106 8