Observation Quality Control with a Robust Ensemble Kalman Filter

Soojin Roh Department of Statistics, Texas A&M University, College Station, Texas

Search for other papers by Soojin Roh in
Current site
Google Scholar
PubMed
Close
,
Marc G. Genton CEMSE Division, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia

Search for other papers by Marc G. Genton in
Current site
Google Scholar
PubMed
Close
,
Mikyoung Jun Department of Statistics, Texas A&M University, College Station, Texas

Search for other papers by Mikyoung Jun in
Current site
Google Scholar
PubMed
Close
,
Istvan Szunyogh Department of Atmospheric Sciences, Texas A&M University, College Station, Texas

Search for other papers by Istvan Szunyogh in
Current site
Google Scholar
PubMed
Close
, and
Ibrahim Hoteit CEMSE Division, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia

Search for other papers by Ibrahim Hoteit in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Current ensemble-based Kalman filter (EnKF) algorithms are not robust to gross observation errors caused by technical or human errors during the data collection process. In this paper, the authors consider two types of gross observational errors, additive statistical outliers and innovation outliers, and introduce a method to make EnKF robust to gross observation errors. Using both a one-dimensional linear system of dynamics and a 40-variable Lorenz model, the performance of the proposed robust ensemble Kalman filter (REnKF) was tested and it was found that the new approach greatly improves the performance of the filter in the presence of gross observation errors and leads to only a modest loss of accuracy with clean, outlier-free, observations.

Corresponding author address: Marc G. Genton, CEMSE Division, KAUST, Thuwal 23955-6900, Saudi Arabia. E-mail: marc.genton@kaust.edu.sa

Abstract

Current ensemble-based Kalman filter (EnKF) algorithms are not robust to gross observation errors caused by technical or human errors during the data collection process. In this paper, the authors consider two types of gross observational errors, additive statistical outliers and innovation outliers, and introduce a method to make EnKF robust to gross observation errors. Using both a one-dimensional linear system of dynamics and a 40-variable Lorenz model, the performance of the proposed robust ensemble Kalman filter (REnKF) was tested and it was found that the new approach greatly improves the performance of the filter in the presence of gross observation errors and leads to only a modest loss of accuracy with clean, outlier-free, observations.

Corresponding author address: Marc G. Genton, CEMSE Division, KAUST, Thuwal 23955-6900, Saudi Arabia. E-mail: marc.genton@kaust.edu.sa
Save
  • Anderson, E., and H. Järvinen, 1999: Variational quality control. Quart. J. Roy. Meteor. Soc., 125, 697722.

  • Birmiwal, K., and J. Shen, 1993: Optimal robust filtering. Stat. Decis., 11, 101119.

  • Birmiwal, K., and P. Papantoni-Kazakos, 1994: Outlier resistant prediction for stationary processes. Stat. Decis., 12, 395427.

  • Burgers, G., P. J. van Leeuwen, and G. Evensen, 1998: Analysis scheme in the ensemble Kalman filter. Mon. Wea. Rev., 126, 17191724.

  • Calvet, L. E., V. Czellar, and E. Ronchetti, cited 2012: Robust filtering. [Available online at http://ssrn.com/abstract=2123477.]

  • Daley, R., 1991: Atmospheric Data Analysis. Cambridge Atmospheric and Space Science Series, Cambridge University Press, 457 pp.

  • Ershov, A. A., and R. S. Liptser, 1978: Robust Kalman filter in discrete time. IEEE Trans. Autom. Remote Control, 39, 359367.

  • Fahrmeir, L., and H. Kaufmann, 1991: On Kalman filtering, posterior mode estimation and Fisher scoring in dynamic exponential family regression. Metrika, 38, 3760.

    • Search Google Scholar
    • Export Citation
  • Fahrmeir, L., and R. Kunstler, 1999: Penalized likelihood smoothing in robust state space models. Metrika, 49, 173191.

  • Fox, A. J., 1972: Outliers in time series. J. Roy. Stat. Soc., B34, 350363.

  • Genton, M. G., 2003: Breakdown-point for spatially and temporally correlated observations. Developments in Robust Statistics, R. Dutter et al., Eds., Springer, 148–159.

  • Genton, M. G., and A. Lucas, 2003: Comprehensive definitions of breakdown-points for independent and dependent observations. J. Roy. Stat. Soc., B65, 8194.

    • Search Google Scholar
    • Export Citation
  • Genton, M. G., and A. Lucas, 2005: Discussion of “Breakdown and groups” by L. Davies and U. Gather. Ann. Stat., 33, 988993.

  • Hampel, F. R., 1968: Contributions to the theory of robust estimation. Ph.D. thesis, University of California.

  • Harlim, J., and B. R. Hunt, 2007: A non-Gaussian ensemble filter for assimilating infrequent noisy observations. Tellus, 59A, 225237.

    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., and H. L. Mitchell, 1998: Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev., 126, 796811.

    • Search Google Scholar
    • Export Citation
  • Huber, P. J., 1981: Robust Statistics. Wiley, 308 pp.

  • Ingleby, N. B., and A. C. Lorenc, 1993: Bayesian quality control using multivariate normal distributions. Quart. J. Roy. Meteor. Soc., 119, 11951225.

    • Search Google Scholar
    • Export Citation
  • Kalman, R. E., 1960: A new approach to linear filtering and prediction problems. J. Basic Eng., 82, 3445.

  • Kassam, S. A., and H. V. Poor, 1985: Robust techniques for signal processing: A survey. Proc. IEEE, 73, 433481.

  • Lorenz, E. N., and K. A. Emanuel, 1998: Optimal sites for supplementary weather observations: Simulation with a small model. J. Atmos. Sci., 55, 399414.

    • Search Google Scholar
    • Export Citation
  • Luo, X., and I. Hoteit, 2011: Robust ensemble filtering and its relation to covariance inflation in the ensemble Kalman filter. Mon. Wea. Rev., 139, 39383953.

    • Search Google Scholar
    • Export Citation
  • Maronna, A., R. D. Martin, and V. J. Yohai, 2006: Robust Statistics: Theory and Methods. Wiley, 436 pp.

  • Martin, R. D., and A. E. Raftery, 1987: Robustness, computation and non-Euclidean models. J. Amer. Stat. Assoc., 82, 10441050.

  • Meinhold, R. J., and N. D. Singpurwalla, 1983: Understanding the Kalman filter. Amer. Stat., 37, 123127.

  • Meinhold, R. J., and N. D. Singpurwalla, 1989: Robustification of Kalman filter models. J. Amer. Stat. Assoc., 84, 479486.

  • Naveau, P., M. G. Genton, and X. Shen, 2005: A skewed Kalman filter. J. Multivariate Anal., 95, 382400.

  • Ruckdeschel, P., 2010: Optimally robust Kalman filtering. Berichte des Fraunhofer ITWM 185, 53 pp.

  • Schick, I. C., and S. K. Mitter, 1994: Robust recursive estimation in the presence of heavy-tailed observation noise. Ann. Stat., 22, 10451080.

    • Search Google Scholar
    • Export Citation
  • Schlee, F. H., C. J. Standish, and N. F. Toda, 1967: Divergence in the Kalman filter. Amer. Inst. Aeronaut. Astronaut. J., 5, 11141120.

    • Search Google Scholar
    • Export Citation
  • Stockinger, N., and R. Dutter, 1987: Robust time series analysis: A survey. Kybernetika, 23, 388.

  • Szunyogh, I., E. J. Kostelich, G. Gyarmati, E. Kalnay, B. R. Hunt, E. Ott, E. Satterfield, and J. A. Yorke, 2008: A local ensemble transform Kalman filter data assimilation system for the NCEP global model. Tellus, 60, 113130.

    • Search Google Scholar
    • Export Citation
  • Tavolato, C., and L. Isaksen, 2010: Huber norm quality control in the IFS. ECMWF Newsletter, No. 122, ECMWF, Reading, United Kingdom, 27–31.

  • Tukey, J. W., 1970: Exploratory Data Analysis. Vol. 1. Addison-Wesley, 688 pp.

  • West, M., 1981: Robust sequential approximate Bayesian estimation. J. Roy. Stat. Soc., B43, 157166.

  • West, M., 1983: Generalized linear models: Scale parameters, outlier accommodation and prior distributions. Bayesian Stat., 2, 531558.

    • Search Google Scholar
    • Export Citation
  • West, M., 1984: Outlier models and prior distributions in Bayesian linear regression. J. Roy. Stat. Soc., B46, 431439.

  • Whitaker, J. S., and T. M. Hamill, 2002: Ensemble data assimilation without perturbed observations. Mon. Wea. Rev., 130, 19131924.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 978 548 51
PDF Downloads 304 83 8