Parallel Direct Solution of the Covariance-Localized Ensemble Square Root Kalman Filter Equations with Matrix Functions

Jeffrey L. Steward University of California, Los Angeles, Los Angeles, California

Search for other papers by Jeffrey L. Steward in
Current site
Google Scholar
PubMed
Close
,
Jose E. Roman Universitat Poltècnica de València, València, Spain

Search for other papers by Jose E. Roman in
Current site
Google Scholar
PubMed
Close
,
Alejandro Lamas Daviña Universitat Poltècnica de València, València, Spain

Search for other papers by Alejandro Lamas Daviña in
Current site
Google Scholar
PubMed
Close
, and
Altuǧ Aksoy Cooperative Institute for Marine and Atmospheric Studies, University of Miami, and Hurricane Research Division, NOAA/AOML, Miami, Florida

Search for other papers by Altuǧ Aksoy in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Recently, the serial approach to solving the square root ensemble Kalman filter (ESRF) equations in the presence of covariance localization was found to depend on the order of observations. As shown previously, correctly updating the localized posterior covariance in serial requires additional effort and computational expense. A recent work by Steward et al. details an all-at-once direct method to solve the ESRF equations in parallel. This method uses the eigenvectors and eigenvalues of the forward observation covariance matrix to solve the difficult portion of the ESRF equations. The remaining assimilation is easily parallelized, and the analysis does not depend on the order of observations. While this allows for long localization lengths that would render local analysis methods inefficient, in theory, an eigenpair-based method scales as the cube number of observations, making it infeasible for large numbers of observations. In this work, we extend this method to use the theory of matrix functions to avoid eigenpair computations. The Arnoldi process is used to evaluate the covariance-localized ESRF equations on the reduced-order Krylov subspace basis. This method is shown to converge quickly and apparently regains a linear scaling with the number of observations. The method scales similarly to the widely used serial approach of Anderson and Collins in wall time but not in memory usage. To improve the memory usage issue, this method potentially can be used without an explicit matrix. In addition, hybrid ensemble and climatological covariances can be incorporated.

© 2018 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Jeffrey L. Steward, jsteward@jifresse.ucla.edu

Abstract

Recently, the serial approach to solving the square root ensemble Kalman filter (ESRF) equations in the presence of covariance localization was found to depend on the order of observations. As shown previously, correctly updating the localized posterior covariance in serial requires additional effort and computational expense. A recent work by Steward et al. details an all-at-once direct method to solve the ESRF equations in parallel. This method uses the eigenvectors and eigenvalues of the forward observation covariance matrix to solve the difficult portion of the ESRF equations. The remaining assimilation is easily parallelized, and the analysis does not depend on the order of observations. While this allows for long localization lengths that would render local analysis methods inefficient, in theory, an eigenpair-based method scales as the cube number of observations, making it infeasible for large numbers of observations. In this work, we extend this method to use the theory of matrix functions to avoid eigenpair computations. The Arnoldi process is used to evaluate the covariance-localized ESRF equations on the reduced-order Krylov subspace basis. This method is shown to converge quickly and apparently regains a linear scaling with the number of observations. The method scales similarly to the widely used serial approach of Anderson and Collins in wall time but not in memory usage. To improve the memory usage issue, this method potentially can be used without an explicit matrix. In addition, hybrid ensemble and climatological covariances can be incorporated.

© 2018 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Jeffrey L. Steward, jsteward@jifresse.ucla.edu
Save
  • Aberson, S. D., A. Aksoy, K. J. Sellwood, T. Vukicevic, and X. Zhang, 2015: Assimilation of high-resolution tropical cyclone observations with an ensemble Kalman filter using HEDAS: Evaluation of 2008–11 HWRF forecasts. Mon. Wea. Rev., 143, 511523, https://doi.org/10.1175/MWR-D-14-00138.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Afanasjew, M., M. Eiermann, O. G. Ernst, and S. Güttel, 2008: Implementation of a restarted Krylov subspace method for the evaluation of matrix functions. Linear Algebra Appl., 429, 22932314, https://doi.org/10.1016/j.laa.2008.06.029.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Aksoy, A., 2013: Storm-relative observations in tropical cyclone data assimilation with an ensemble Kalman filter. Mon. Wea. Rev., 141, 506522, https://doi.org/10.1175/MWR-D-12-00094.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Aksoy, A., S. Lorsolo, T. Vukicevic, K. J. Sellwood, S. D. Aberson, and F. Zhang, 2012: The HWRF Hurricane Ensemble Data Assimilation System (HEDAS) for high-resolution data: The impact of airborne Doppler radar observations in an OSSE. Mon. Wea. Rev., 140, 18431862, https://doi.org/10.1175/MWR-D-11-00212.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Aksoy, A., S. D. Aberson, T. Vukicevic, K. J. Sellwood, S. Lorsolo, and X. Zhang, 2013: Assimilation of high-resolution tropical cyclone observations with an ensemble Kalman filter using NOAA/AOML/HRD’s HEDAS: Evaluation of the 2008–11 vortex-scale analyses. Mon. Wea. Rev., 141, 18421865, https://doi.org/10.1175/MWR-D-12-00194.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., 2001: An ensemble adjustment Kalman filter for data assimilation. Mon. Wea. Rev., 129, 28842903, https://doi.org/10.1175/1520-0493(2001)129<2884:AEAKFF>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., 2003: A local least squares framework for ensemble filtering. Mon. Wea. Rev., 131, 634642, https://doi.org/10.1175/1520-0493(2003)131<0634:ALLSFF>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., 2012: Localization and sampling error correction in ensemble Kalman filter data assimilation. Mon. Wea. Rev., 140, 23592371, https://doi.org/10.1175/MWR-D-11-00013.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., and N. Collins, 2007: Scalable implementations of ensemble filter algorithms for data assimilation. J. Atmos. Oceanic Technol., 24, 14521463, https://doi.org/10.1175/JTECH2049.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Arnoldi, W. E., 1951: The principle of minimized iterations in the solution of the matrix eigenvalue problem. Q. Appl. Math., 9, 1729, https://doi.org/10.1090/qam/42792.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Balay, S., W. D. Gropp, L. C. McInnes, and B. F. Smith, 1997: Efficient management of parallelism in object-oriented numerical software libraries. Modern Software Tools for Scientific Computing, E. Arge, A. M. Bruaset, and H. P. Langtangen, Eds., Springer, 163–202, https://doi.org/10.1007/978-1-4612-1986-6_8.

    • Crossref
    • Export Citation
  • Balay, S., and Coauthors, 2016: PETSc Users Manual. Argonne National Laboratory Tech. Rep. ANL-95/11, Revision 3.7, 272 pp., http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf.

    • Crossref
    • Export Citation
  • Balay, S., and Coauthors, 2017: PETSc: Portable, Extensible Toolkit for Scientific Computation. PETSc/Tao, https://www.mcs.anl.gov/petsc/.

  • Bessho, K., and Coauthors, 2016: An introduction to Himawari-8/9—Japan’s new-generation geostationary meteorological satellites. J. Meteor. Soc. Japan, 94, 151183, https://doi.org/10.2151/jmsj.2016-009.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bishop, C. H., and D. Hodyss, 2009: Ensemble covariances adaptively localized with ECO-RAP. Part 2: A strategy for the atmosphere. Tellus, 61A, 97111, https://doi.org/10.1111/j.1600-0870.2008.00372.x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bishop, C. H., B. J. Etherton, and S. J. Majumdar, 2001: Adaptive sampling with the ensemble transform Kalman filter. Part I: Theoretical aspects. Mon. Wea. Rev., 129, 420436, https://doi.org/10.1175/1520-0493(2001)129<0420:ASWTET>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bishop, C. H., B. Huang, and X. Wang, 2015: A nonvariational consistent hybrid ensemble filter. Mon. Wea. Rev., 143, 50735090, https://doi.org/10.1175/MWR-D-14-00391.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Björck, Å., 1994: Numerics of Gram-Schmidt orthogonalization. Linear Algebra Appl., 197-198, 297316, https://doi.org/10.1016/0024-3795(94)90493-6.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Burgers, G., P. J. van Leeuwen, and G. Evensen, 1998: Analysis scheme in the ensemble Kalman filter. Mon. Wea. Rev., 126, 17191724, https://doi.org/10.1175/1520-0493(1998)126<1719:ASITEK>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Campbell, W. F., C. H. Bishop, and D. Hodyss, 2010: Vertical covariance localization for satellite radiances in ensemble Kalman filters. Mon. Wea. Rev., 138, 282290, https://doi.org/10.1175/2009MWR3017.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Chang, C.-C., S.-C. Yang, and C. Keppenne, 2014: Applications of the mean recentering scheme to improve typhoon track prediction: A case study of Typhoon Nanmadol (2011). J. Meteor. Soc. Japan, 92, 559584, https://doi.org/10.2151/jmsj.2014-604.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Christophersen, H., A. Aksoy, J. Dunion, and K. Sellwood, 2017: The impact of NASA Global Hawk unmanned aircraft dropwindsonde observations on tropical cyclone track, intensity, and structure: Case studies. Mon. Wea. Rev., 145, 18171830, https://doi.org/10.1175/MWR-D-16-0332.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Deadman, E., N. J. Higham, and R. Ralha, 2012: Blocked Schur algorithms for computing the matrix square root. Applied Parallel and Scientific Computing, P. Manninen and P. Öster, Eds., Lecture Notes in Computer Science Series, Vol. 7782, Springer, 171–182, https://doi.org/10.1007/978-3-642-36803-5_12.

    • Crossref
    • Export Citation
  • Eiermann, M., and O. Ernst, 2006: A restarted Krylov subspace method for the evaluation of matrix functions. SIAM J. Numer. Anal., 44, 24812504, https://doi.org/10.1137/050633846.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Eiermann, M., O. Ernst, and S. Güttel, 2011: Deflated restarting for matrix functions. SIAM J. Matrix Anal. Appl., 32, 621641, https://doi.org/10.1137/090774665.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Emanuel, K., and F. Zhang, 2017: The role of inner-core moisture in tropical cyclone predictability and practical forecast skill. J. Atmos. Sci., 74, 23152324, https://doi.org/10.1175/JAS-D-17-0008.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Evensen, G., 1994: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res., 99, 1014310162, https://doi.org/10.1029/94JC00572.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Frayssé, V., L. Giraud, and H. Kharraz-Aroussi, 1998: On the influence of the orthogonalization scheme on the parallel performance of GMRES. Euro-Par’98 Parallel Processing, D. Pritchard and J. Reeve, Eds., Lecture Notes in Computer Science Series, Vol. 1470, Springer, 751–762, https://doi.org/10.1007/BFb0057927.

    • Crossref
    • Export Citation
  • Frommer, A., K. Lund, M. Schweitzer, and D. Szyld, 2017: The Radau–Lanczos method for matrix functions. SIAM J. Matrix Anal. Appl., 38, 710732, https://doi.org/10.1137/16M1072565.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gaspari, G., and S. E. Cohn, 1999: Construction of correlation functions in two and three dimensions. Quart. J. Roy. Meteor. Soc., 125, 723757, https://doi.org/10.1002/qj.49712555417.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Godinez, H. C., and J. D. Moulton, 2012: An efficient matrix-free algorithm for the ensemble Kalman filter. Comput. Geosci., 16, 565575, https://doi.org/10.1007/s10596-011-9268-9.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Golub, G. H., and C. F. Van Loan, 1996: Matrix Computations. Vol. 3. Johns Hopkins University Press, 694 pp.

  • Gopalakrishnan, S. G., F. Marks, X. Zhang, J.-W. Bao, K.-S. Yeh, and R. Atlas, 2011: The experimental HWRF system: A study on the influence of horizontal resolution on the structure and intensity changes in tropical cyclones using an idealized framework. Mon. Wea. Rev., 139, 17621784, https://doi.org/10.1175/2010MWR3535.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., J. S. Whitaker, and C. Snyder, 2001: Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter. Mon. Wea. Rev., 129, 27762790, https://doi.org/10.1175/1520-0493(2001)129<2776:DDFOBE>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hernandez, V., J. E. Roman, and V. Vidal, 2005: SLEPc: A scalable and flexible toolkit for the solution of eigenvalue problems. ACM Trans. Math. Software, 31, 351362, https://doi.org/10.1145/1089014.1089019.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hernandez, V., J. E. Roman, and A. Tomas, 2007: Parallel Arnoldi eigensolvers with enhanced scalability via global communications rearrangement. Parallel Comput., 33, 521540, https://doi.org/10.1016/j.parco.2007.04.004.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Higham, N. J., 1987: Computing real square roots of a real matrix. Linear Algebra Appl., 88–89, 405430, https://doi.org/10.1016/0024-3795(87)90118-2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Higham, N., 2008: Functions of Matrices: Theory and Computation. Other Titles in Applied Mathematics Series, Vol. 104, Society for Industrial and Applied Mathematics, 425 pp.

  • Hochbruck, M., and C. Lubich, 1997: On Krylov subspace approximations to the matrix exponential operator. SIAM J. Numer. Anal., 34, 19111925, https://doi.org/10.1137/S0036142995280572.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., and H. L. Mitchell, 1998: Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev., 126, 796811, https://doi.org/10.1175/1520-0493(1998)126<0796:DAUAEK>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., and H. L. Mitchell, 2001: A sequential ensemble Kalman filter for atmospheric data assimilation. Mon. Wea. Rev., 129, 123137, https://doi.org/10.1175/1520-0493(2001)129<0123:ASEKFF>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., B. He, and H. L. Mitchell, 2014: Parallel implementation of an ensemble Kalman filter. Mon. Wea. Rev., 142, 11631182, https://doi.org/10.1175/MWR-D-13-00011.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hunt, B. R., E. J. Kostelich, and I. Szunyogh, 2007: Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filter. Physica D, 230, 112126, https://doi.org/10.1016/j.physd.2006.11.008.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kalman, R. E., 1960: A new approach to linear filtering and prediction problems. J. Basic Eng., 82, 3545, https://doi.org/10.1115/1.3662552.

  • Keppenne, C. L., and M. M. Rienecker, 2002: Initial testing of a massively parallel ensemble Kalman filter with the Poseidon isopycnal ocean general circulation model. Mon. Wea. Rev., 130, 29512965, https://doi.org/10.1175/1520-0493(2002)130<2951:ITOAMP>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kotsuki, S., S. J. Greybush, and T. Miyoshi, 2017: Can we optimize the assimilation order in the serial ensemble Kalman filter? A study with the Lorenz-96 model. Mon. Wea. Rev., 145, 49774995, https://doi.org/10.1175/MWR-D-17-0094.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Miyoshi, T., and Coauthors, 2016: “Big data assimilation” revolutionizing severe weather prediction. Bull. Amer. Meteor. Soc., 97, 13471354, https://doi.org/10.1175/BAMS-D-15-00144.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Nerger, L., 2015: On serial observation processing in localized ensemble Kalman filters. Mon. Wea. Rev., 143, 15541567, https://doi.org/10.1175/MWR-D-14-00182.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Niño-Ruiz, E. D., A. Sandu, and J. Anderson, 2015: An efficient implementation of the ensemble Kalman filter based on an iterative Sherman–Morrison formula. Stat. Comput., 25, 561577, https://doi.org/10.1007/s11222-014-9454-4.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Niño-Ruiz, E. D., A. Sandu, and X. Deng, 2018: A parallel implementation of the ensemble Kalman filter based on modified Cholesky decomposition. J. Comput. Sci., https://doi.org/10.1016/j.jocs.2017.04.005, in press.

    • Search Google Scholar
    • Export Citation
  • Ott, E., and Coauthors, 2004: A local ensemble Kalman filter for atmospheric data assimilation. Tellus, 56A, 415428, https://doi.org/10.1111/j.1600-0870.2004.00076.x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rogers, R. F., J. A. Zhang, J. Zawislak, H. Jiang, G. R. Alvey, E. J. Zipser, and S. N. Stevenson, 2016: Observations of the structure and evolution of Hurricane Edouard (2014) during intensity change. Part II: Kinematic structure and the distribution of deep convection. Mon. Wea. Rev., 144, 33553376, https://doi.org/10.1175/MWR-D-16-0017.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Saad, Y., 1992: Analysis of some Krylov subspace approximations to the matrix exponential operator. SIAM J. Numer. Anal., 29, 209228, https://doi.org/10.1137/0729014.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sakov, P., and L. Bertino, 2011: Relation between two common localisation methods for the EnKF. Comput. Geosci., 15, 225237, https://doi.org/10.1007/s10596-010-9202-6.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Schmit, T. J., P. Griffith, M. M. Gunshor, J. M. Daniels, S. J. Goodman, and W. J. Lebair, 2017: A closer look at the ABI on the GOES-R series. Bull. Amer. Meteor. Soc., 98, 681698, https://doi.org/10.1175/BAMS-D-15-00230.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Steward, J. L., A. Aksoy, and Z. S. Haddad, 2017: Parallel direct solution of the ensemble square root Kalman filter equations with observation principal components. J. Atmos. Oceanic Technol., 34, 18671884, https://doi.org/10.1175/JTECH-D-16-0140.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Tippett, M. K., J. L. Anderson, C. H. Bishop, T. M. Hamill, and J. S. Whitaker, 2003: Ensemble square root filters. Mon. Wea. Rev., 131, 14851490, https://doi.org/10.1175/1520-0493(2003)131<1485:ESRF>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Van Der Vorst, H. A., 1987: An iterative solution method for solving f(A)x = b, using Krylov subspace information obtained for the symmetric positive definite matrix A. J. Comput. Appl. Math., 18, 249263, https://doi.org/10.1016/0377-0427(87)90020-3.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Vukicevic, T., A. Aksoy, P. Reasor, S. D. Aberson, K. J. Sellwood, and F. Marks, 2013: Joint impact of forecast tendency and state error biases in ensemble Kalman filter data assimilation of inner-core tropical cyclone observations. Mon. Wea. Rev., 141, 29923006, https://doi.org/10.1175/MWR-D-12-00211.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wang, Y., Y. Jung, T. A. Supinie, and M. Xue, 2013: A hybrid MPI–OpenMP parallel algorithm and performance analysis for an ensemble square root filter designed for multiscale observations. J. Atmos. Oceanic Technol., 30, 13821397, https://doi.org/10.1175/JTECH-D-12-00165.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Whitaker, J. S., and T. M. Hamill, 2002: Ensemble data assimilation without perturbed observations. Mon. Wea. Rev., 130, 19131924, https://doi.org/10.1175/1520-0493(2002)130<1913:EDAWPO>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zawislak, J., H. Jiang, G. R. Alvey, E. J. Zipser, R. F. Rogers, J. A. Zhang, and S. N. Stevenson, 2016: Observations of the structure and evolution of Hurricane Edouard (2014) during intensity change. Part I: Relationship between the thermodynamic structure and precipitation. Mon. Wea. Rev., 144, 33333354, https://doi.org/10.1175/MWR-D-16-0018.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zhang, S., M. J. Harrison, A. T. Wittenberg, A. Rosati, J. L. Anderson, and V. Balaji, 2005: Initialization of an ENSO forecast system using a parallelized ensemble filter. Mon. Wea. Rev., 133, 31763201, https://doi.org/10.1175/MWR3024.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 1132 729 294
PDF Downloads 403 70 12