A Quantile-Conserving Ensemble Filter Framework. Part II: Regression of Observation Increments in a Probit and Probability Integral Transformed Space

Jeffrey L. Anderson aNCAR/CISL/TDD/DAReS, Boulder, Colorado

Search for other papers by Jeffrey L. Anderson in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Traditional ensemble Kalman filter data assimilation methods make implicit assumptions of Gaussianity and linearity that are strongly violated by many important Earth system applications. For instance, bounded quantities like the amount of a tracer and sea ice fractional coverage cannot be accurately represented by a Gaussian that is unbounded by definition. Nonlinear relations between observations and model state variables abound. Examples include the relation between a remotely sensed radiance and the column of atmospheric temperatures, or the relation between cloud amount and water vapor quantity. Part I of this paper described a very general data assimilation framework for computing observation increments for non-Gaussian prior distributions and likelihoods. These methods can respect bounds and other non-Gaussian aspects of observed variables. However, these benefits can be lost when observation increments are used to update state variables using the linear regression that is part of standard ensemble Kalman filter algorithms. Here, regression of observation increments is performed in a space where variables are transformed by the probit and probability integral transforms, a specific type of Gaussian anamorphosis. This method can enforce appropriate bounds for all quantities and deal much more effectively with nonlinear relations between observations and state variables. Important enhancements like localization and inflation can be performed in the transformed space. Results are provided for idealized bivariate distributions and for cycling assimilation in a low-order dynamical system. Implications for improved data assimilation across Earth system applications are discussed.

© 2023 American Meteorological Society. This published article is licensed under the terms of the default AMS reuse license. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Jeffrey L. Anderson, jla@ucar.edu

Abstract

Traditional ensemble Kalman filter data assimilation methods make implicit assumptions of Gaussianity and linearity that are strongly violated by many important Earth system applications. For instance, bounded quantities like the amount of a tracer and sea ice fractional coverage cannot be accurately represented by a Gaussian that is unbounded by definition. Nonlinear relations between observations and model state variables abound. Examples include the relation between a remotely sensed radiance and the column of atmospheric temperatures, or the relation between cloud amount and water vapor quantity. Part I of this paper described a very general data assimilation framework for computing observation increments for non-Gaussian prior distributions and likelihoods. These methods can respect bounds and other non-Gaussian aspects of observed variables. However, these benefits can be lost when observation increments are used to update state variables using the linear regression that is part of standard ensemble Kalman filter algorithms. Here, regression of observation increments is performed in a space where variables are transformed by the probit and probability integral transforms, a specific type of Gaussian anamorphosis. This method can enforce appropriate bounds for all quantities and deal much more effectively with nonlinear relations between observations and state variables. Important enhancements like localization and inflation can be performed in the transformed space. Results are provided for idealized bivariate distributions and for cycling assimilation in a low-order dynamical system. Implications for improved data assimilation across Earth system applications are discussed.

© 2023 American Meteorological Society. This published article is licensed under the terms of the default AMS reuse license. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Jeffrey L. Anderson, jla@ucar.edu
Save
  • Amezcua, J., and P. J. Van Leeuwen, 2014: Gaussian anamorphosis in the analysis step of the EnKF: A joint state-variable/observation approach. Tellus, 66A, 23493, https://doi.org/10.3402/tellusa.v66.23493.

    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., 2001: An ensemble adjustment Kalman filter for data assimilation. Mon. Wea. Rev., 129, 28842903, https://doi.org/10.1175/1520-0493(2001)129<2884:AEAKFF>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., 2003: A local least squares framework for ensemble filtering. Mon. Wea. Rev., 131, 634642, https://doi.org/10.1175/1520-0493(2003)131<0634:ALLSFF>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., 2007: Exploring the need for localization in ensemble data assimilation using a hierarchical ensemble filter. Physica D, 230, 99111, https://doi.org/10.1016/j.physd.2006.02.011.

    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., 2009: Spatially and temporally varying adaptive covariance inflation for ensemble filters. Tellus, 61, 7283, https://doi.org/10.1111/j.1600-0870.2008.00361.x.

    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., 2010: A non-Gaussian ensemble filter update for data assimilation. Mon. Wea. Rev., 138, 41864198, https://doi.org/10.1175/2010MWR3253.1.

    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., 2012: Localization and sampling error correction in ensemble Kalman filter data assimilation. Mon. Wea. Rev., 140, 23592371, https://doi.org/10.1175/MWR-D-11-00013.1.

    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., 2019: A nonlinear rank regression method for ensemble Kalman filter data assimilation. Mon. Wea. Rev., 147, 28472860, https://doi.org/10.1175/MWR-D-18-0448.1.

    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., 2020: A marginal adjustment rank histogram filter for non-Gaussian ensemble data assimilation. Mon. Wea. Rev., 148, 33613378, https://doi.org/10.1175/MWR-D-19-0307.1.

    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., 2022: A quantile-conserving ensemble filter framework. Part I: Updating an observed variable. Mon. Wea. Rev., 150, 10611074, https://doi.org/10.1175/MWR-D-21-0229.1.

    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., and S. L. Anderson, 1999: A Monte Carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Mon. Wea. Rev., 127, 27412758, https://doi.org/10.1175/1520-0493(1999)127<2741:AMCIOT>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., and N. Collins, 2007: Scalable implementations of ensemble filter algorithms for data assimilation. J. Atmos. Oceanic Technol., 24, 14521463, https://doi.org/10.1175/JTECH2049.1.

    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., and L. Lei, 2013: Empirical localization of observation impact in ensemble Kalman filters. Mon. Wea. Rev., 141, 41404153, https://doi.org/10.1175/MWR-D-12-00330.1.

    • Search Google Scholar
    • Export Citation
  • Anderson, J. L., T. Hoar, K. Raeder, H. Liu, N. Collins, R. Torn, and A. Avellano, 2009: The data assimilation research testbed: A community facility. Bull. Amer. Meteor. Soc., 90, 12831296, https://doi.org/10.1175/2009BAMS2618.1.

    • Search Google Scholar
    • Export Citation
  • Béal, D., P. Brasseur, J.-M. Brankart, Y. Ourmiéres, and J. Verron, 2010: Characterization of mixing errors in a coupled physical biogeochemical model of the North Atlantic: Implications for nonlinear estimation using Gaussian anamorphosis. Ocean Sci., 6, 247262, https://doi.org/10.5194/os-6-247-2010.

    • Search Google Scholar
    • Export Citation
  • Bertino, L., G. Evensen, and H. Wackernagel, 2003: Sequential data assimilation techniques in oceanography. Int. Stat. Rev., 71, 223241, https://doi.org/10.1111/j.1751-5823.2003.tb00194.x.

    • Search Google Scholar
    • Export Citation
  • Bishop, C. H., 2016: The GIGG‐EnKF: Ensemble Kalman filtering for highly skewed non‐negative uncertainty distributions. Quart. J. Roy. Meteor. Soc., 142, 13951412, https://doi.org/10.1002/qj.2742.

    • Search Google Scholar
    • Export Citation
  • Bocquet, M., C. A. Pires, and L. Wu, 2010: Beyond Gaussian statistical modeling in geophysical data assimilation. Mon. Wea. Rev., 138, 29973023, https://doi.org/10.1175/2010MWR3164.1.

    • Search Google Scholar
    • Export Citation
  • Burgers, G., P. J. van Leeuwen, and G. Evensen, 1998: Analysis scheme in the ensemble Kalman filter. Mon. Wea. Rev., 126, 17191724, https://doi.org/10.1175/1520-0493(1998)126<1719:ASITEK>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Chan, M.-Y., J. L. Anderson, and X. Chen, 2020: An efficient bi-Gaussian ensemble Kalman filter for satellite infrared radiance data assimilation. Mon. Wea. Rev., 148, 50875104, https://doi.org/10.1175/MWR-D-20-0142.1.

    • Search Google Scholar
    • Export Citation
  • Doron, M., P. Brasseur, J.-M. Brankart, S. N. Losa, and A. Melet, 2013: Stochastic estimation of biogeochemical parameters from Globcolour ocean colour satellite data in a North Atlantic 3D ocean coupled physical–biogeochemical model. J. Mar. Syst., 117118, 8195, https://doi.org/10.1016/j.jmarsys.2013.02.007.

    • Search Google Scholar
    • Export Citation
  • Furrer, R., and T. Bengtsson, 2007: Estimation of high-dimensional prior and posterior covariance matrices in Kalman filter variants. J. Multivar. Anal., 98, 227255, https://doi.org/10.1016/j.jmva.2006.08.003.

    • Search Google Scholar
    • Export Citation
  • Gharamti, M. E., A. Samuelsen, L. Bertino, E. Simon, A. Korosov, and U. Daewel, 2017: Online tuning of ocean biogeochemical model parameters using ensemble estimation techniques: Application to a one-dimensional model in the North Atlantic. J. Mar. Syst., 168, 116, https://doi.org/10.1016/j.jmarsys.2016.12.003.

    • Search Google Scholar
    • Export Citation
  • Gharamti, M. E., K. Raeder, J. Anderson, and X. Wang, 2019: Comparing adaptive prior and posterior inflation for ensemble filters using an atmospheric general circulation model. Mon. Wea. Rev., 147, 25352553, https://doi.org/10.1175/MWR-D-18-0389.1.

    • Search Google Scholar
    • Export Citation
  • Grooms, I., 2022: A comparison of nonlinear extensions to the ensemble Kalman filter. Comput. Geosci., 26, 633650, https://doi.org/10.1007/s10596-022-10141-x.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., J. S. Whitaker, and C. Snyder, 2001: Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter. Mon. Wea. Rev., 129, 27762790, https://doi.org/10.1175/1520-0493(2001)129<2776:DDFOBE>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., and H. L. Mitchell, 1998: Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev., 126, 796811, https://doi.org/10.1175/1520-0493(1998)126<0796:DAUAEK>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., and F. Zhang, 2016: Review of the ensemble Kalman filter for atmospheric data assimilation. Mon. Wea. Rev., 144, 44894532, https://doi.org/10.1175/MWR-D-15-0440.1.

    • Search Google Scholar
    • Export Citation
  • Kalman, R. E., 1960: A new approach to linear filtering and prediction problems. J. Basic Eng. Mar., 82, 3545, https://doi.org/10.1115/1.3662552.

    • Search Google Scholar
    • Export Citation
  • Li, L., H. Zhou, H. J. Hendricks Franssen, and J. J. Gómez-Hernández, 2012: Groundwater flow inverse modeling in non-multi-Gaussian media: Performance assessment of the normal-score ensemble Kalman filter. Hydrol. Earth Syst. Sci., 16, 573590, https://doi.org/10.5194/hess-16-573-2012.

    • Search Google Scholar
    • Export Citation
  • Lien, G.-Y., E. Kalnay, and T. Miyoshi, 2013: Effective assimilation of global precipitation: Simulation experiments. Tellus, 65A, 19915, https://doi.org/10.3402/tellusa.v65i0.19915.

    • Search Google Scholar
    • Export Citation
  • Lorenz, E. N., and K. A. Emanuel, 1998: Optimal sites for supplementary weather observations: Simulation with a small model. J. Atmos. Sci., 55, 399414, https://doi.org/10.1175/1520-0469(1998)055<0399:OSFSWO>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Ludwig, V., G. Spreen, and L. T. Pedersen, 2020: Evaluation of a new merged sea-ice concentration dataset at 1 km resolution from thermal infrared and passive microwave satellite data in the Arctic. Remote Sens., 12, 3183, https://doi.org/10.3390/rs12193183.

    • Search Google Scholar
    • Export Citation
  • Pham, D. T., 2001: Stochastic methods for sequential data assimilation in strongly nonlinear systems. Mon. Wea. Rev., 129, 11941207, https://doi.org/10.1175/1520-0493(2001)129<1194:SMFSDA>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Raeder, K., T. J. Hoar, M. El Gharamti, B. K. Johnson, N. Collins, J. L. Anderson, J. Steward, and M. Coady, 2021: A new CAM6 + DART reanalysis with surface forcing from CAM6 to other CESM models. Sci. Rep., 11, 16384, https://doi.org/10.1038/s41598-021-92927-0.

    • Search Google Scholar
    • Export Citation
  • Rostosky, P., G. Spreen, S. Gerland, M. Huntemann, and M. Mech, 2020: Modeling the microwave emission of snow on Arctic sea ice for estimating the uncertainty of satellite retrievals. J. Geophys. Res. Oceans, 125, e2019JC015465, https://doi.org/10.1029/2019JC015465.

    • Search Google Scholar
    • Export Citation
  • Simon, E., and L. Bertino, 2012: Gaussian anamorphosis extension of the DEnKF for combined state parameter estimation: Application to a 1D ocean ecosystem model. J. Mar. Syst., 89, 118, https://doi.org/10.1016/j.jmarsys.2011.07.007.

    • Search Google Scholar
    • Export Citation
  • Sklar, A., 1973: Random variables, joint distribution functions, and copulas. Kybernetika, 9, 449460.

  • van Leeuwen, P. J., 2009: Particle filtering in geophysical systems. Mon. Wea. Rev., 137, 40894114, https://doi.org/10.1175/2009MWR2835.1.

    • Search Google Scholar
    • Export Citation
  • van Leeuwen, P. J., H. R. Künsch, L. Nerger, R. Potthast, and S. Reich, 2019: Particle filters for high‐dimensional geoscience applications: A review. Quart. J. Roy. Meteor. Soc., 145, 23352365, https://doi.org/10.1002/qj.3551.

    • Search Google Scholar
    • Export Citation
  • Zhang, F., C. Snyder, and J. Sun, 2004: Impacts of initial estimate and observation availability on convective-scale data assimilation with an ensemble Kalman filter. Mon. Wea. Rev., 132, 12381253, https://doi.org/10.1175/1520-0493(2004)132<1238:IOIEAO>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 319 319 71
Full Text Views 133 133 28
PDF Downloads 138 138 32