Correcting Biased Observation Model Error in Data Assimilation

Tyrus Berry Department of Mathematical Sciences, George Mason University, Fairfax, Virginia

Search for other papers by Tyrus Berry in
Current site
Google Scholar
PubMed
Close
and
John Harlim Department of Mathematics and Department of Meteorology and Atmospheric Science, The Pennsylvania State University, University Park, Pennsylvania

Search for other papers by John Harlim in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

While the formulation of most data assimilation schemes assumes an unbiased observation model error, in real applications model error with nontrivial biases is unavoidable. A practical example is errors in the radiative transfer model (which is used to assimilate satellite measurements) in the presence of clouds. Together with the dynamical model error, the result is that many (in fact 99%) of the cloudy observed measurements are not being used although they may contain useful information. This paper presents a novel nonparametric Bayesian scheme that is able to learn the observation model error distribution and correct the bias in incoming observations. This scheme can be used in tandem with any data assimilation forecasting system. The proposed model error estimator uses nonparametric likelihood functions constructed with data-driven basis functions based on the theory of kernel embeddings of conditional distributions developed in the machine learning community. Numerically, positive results are shown with two examples. The first example is designed to produce a bimodality in the observation model error (typical of “cloudy” observations) by introducing obstructions to the observations that occur randomly in space and time. The second example, which is physically more realistic, is to assimilate cloudy satellite brightness temperature–like quantities, generated from a stochastic multicloud model for tropical convection and a simple radiative transfer model.

© 2017 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: John Harlim, jharlim@psu.edu

Abstract

While the formulation of most data assimilation schemes assumes an unbiased observation model error, in real applications model error with nontrivial biases is unavoidable. A practical example is errors in the radiative transfer model (which is used to assimilate satellite measurements) in the presence of clouds. Together with the dynamical model error, the result is that many (in fact 99%) of the cloudy observed measurements are not being used although they may contain useful information. This paper presents a novel nonparametric Bayesian scheme that is able to learn the observation model error distribution and correct the bias in incoming observations. This scheme can be used in tandem with any data assimilation forecasting system. The proposed model error estimator uses nonparametric likelihood functions constructed with data-driven basis functions based on the theory of kernel embeddings of conditional distributions developed in the machine learning community. Numerically, positive results are shown with two examples. The first example is designed to produce a bimodality in the observation model error (typical of “cloudy” observations) by introducing obstructions to the observations that occur randomly in space and time. The second example, which is physically more realistic, is to assimilate cloudy satellite brightness temperature–like quantities, generated from a stochastic multicloud model for tropical convection and a simple radiative transfer model.

© 2017 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: John Harlim, jharlim@psu.edu
Save
  • Anderson, J., 2001: An ensemble adjustment Kalman filter for data assimilation. Mon. Wea. Rev., 129, 28842903, doi:10.1175/1520-0493(2001)129<2884:AEAKFF>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Berry, T., and T. Sauer, 2013: Adaptive ensemble Kalman filtering of nonlinear systems. Tellus, 65A, 20331, doi:10.3402/tellusa.v65i0.20331.

  • Berry, T., and J. Harlim, 2016a: Forecasting turbulent modes with nonparametric diffusion models: Learning from noisy data. Physica D, 320, 5776, doi:10.1016/j.physd.2016.01.012.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Berry, T., and J. Harlim, 2016b: Variable bandwidth diffusion kernels. Appl. Comput. Harmonic Anal., 40, 6896, doi:10.1016/j.acha.2015.01.001.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Berry, T., and T. Sauer, 2016: Local kernels and the geometric structure of data. Appl. Comput. Harmonic Anal., 40, 439469, doi:10.1016/j.acha.2015.03.002.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Biello, J., B. Khouider, and A. J. Majda, 2010: A stochastic multicloud model for tropical convection. Commun. Math. Sci., 8, 187216, doi:10.4310/CMS.2010.v8.n1.a10.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bishop, C., B. Etherton, and S. Majumdar, 2001: Adaptive sampling with the ensemble transform Kalman filter. Part I: The theoretical aspects. Mon. Wea. Rev., 129, 420436, doi:10.1175/1520-0493(2001)129<0420:ASWTET>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Chen, Y., F. Weng, Y. Han, and Q. Liu, 2008: Validation of the community radiative transfer model by using CloudSat data. J. Geophys. Res., 113, D00A03, doi:10.1029/2007JD009561.

    • Search Google Scholar
    • Export Citation
  • Coifman, R., and S. Lafon, 2006: Diffusion maps. Appl. Comput. Harmonic Anal., 21, 530, doi:10.1016/j.acha.2006.04.006.

  • Deng, Q., B. Khouider, and A. J. Majda, 2015: The MJO in a coarse-resolution GCM with a stochastic multicloud parameterization. J. Atmos. Sci., 72, 5574, doi:10.1175/JAS-D-14-0120.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Epstein, E. S., 1969: Stochastic dynamic prediction. Tellus, 21, 739759, doi:10.3402/tellusa.v21i6.10143.

  • Evensen, G., 1994: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res., 99, 10 14310 162, doi:10.1029/94JC00572.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gottwald, G. A., and A. Majda, 2013: A mechanism for catastrophic filter divergence in data assimilation for sparse observation networks. Nonlinear Processes Geophys., 20, 705712, doi:10.5194/npg-20-705-2013.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hastie, T., R. Tibshirani, and J. Friedman, 2001: The Elements of Statistical Learning. Vol. 1. Springer Series in Statistics, Springer, 533 pp.

    • Crossref
    • Export Citation
  • Heilliette, S., Y. Rochon, L. Garand, and J. Kaminski, 2013: Assimilation of infrared radiances in the context of observing system simulation experiments. J. Appl. Meteor. Climatol., 52, 10311045, doi:10.1175/JAMC-D-12-0124.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kalnay, E., 2003: Atmospheric Modeling, Data Assimilation, and Predictability. Cambridge University Press, 341 pp.

    • Crossref
    • Export Citation
  • Kalnay, E., and Coauthors, 1996: The NCEP/NCAR 40-Year Reanalysis Project. Bull. Amer. Meteor. Soc., 77, 437471, doi:10.1175/1520-0477(1996)077<0437:TNYRP>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Liou, K.-N., 2002: An Introduction to Atmospheric Radiation. International Geophysics Series, Vol. 84, Academic Press, 583 pp.

  • Lorenz, E., 1996: Predictability—A problem partly solved. Proc. Workshop on Predictability, Reading, United Kingdom, ECMWF, 1–18.

  • Majda, A. J., and J. Harlim, 2012: Filtering Complex Turbulent Systems. Cambridge University Press, 357 pp.

  • McNally, A. P., 2009: The direct assimilation of cloud-affected satellite infrared radiances in the ECMWF 4D-Var. Quart. J. Roy. Meteor. Soc., 135, 12141229, doi:10.1002/qj.426.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • McNally, A. P., P. D. Watts, J. A. Smith, R. Engelen, G. A. Kelly, J. N. Thépaut, and M. Matricardi, 2006: The assimilation of AIRS radiance data at ECMWF. Quart. J. Roy. Meteor. Soc., 132, 935957, doi:10.1256/qj.04.171.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Nyström, E. J., 1930: Über die praktische Auflösung von Integralgleichungenmit Anwendungen auf Randwertaufgaben. Acta Math., 54, 185204, doi:10.1007/BF02547521.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Otkin, J. A., 2010: Clear and cloudy sky infrared brightness temperature assimilation using an ensemble Kalman filter. J. Geophys. Res., 115, D19207, doi:10.1029/2009JD013759.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Reale, O., J. Susskind, R. Rosenberg, E. Brin, E. Liu, L. P. Riishojgaard, J. Terry, and J. C. Jusem, 2008: Improving forecast skill by assimilation of quality-controlled airs temperature retrievals under partially cloudy conditions. Geophys. Res. Lett., 35, L08809, doi:10.1029/2007GL033002.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Saunders, R., M. Matricardi, and P. Brunel, 1999: An improved fast radiative transfer model for assimilation of satellite radiance observations. Quart. J. Roy. Meteor. Soc., 125, 14071425, doi:10.1002/qj.1999.49712555615.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Song, L., J. Huang, A. Smola, and K. Fukumizu, 2009: Hilbert space embeddings of conditional distributions with applications to dynamical systems. Proc. 26th Annual Int. Conf. on Machine Learning, Montreal, QC, Canada, ACM, 961–968.

    • Crossref
    • Export Citation
  • Song, L., K. Fukumizu, and A. Gretton, 2013: Kernel embeddings of conditional distributions: A unified kernel framework for nonparametric inference in graphical models. IEEE Signal Process. Mag., 30, 98111, doi:10.1109/MSP.2013.2252713.

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 801 218 21
PDF Downloads 651 141 9