Editorial Type:
Article
Article Type:
Research Article

Adaptive Sampling with the Ensemble Transform Kalman Filter. Part I: Theoretical Aspects

Craig H. Bishop The Pennsylvania State University, University Park, Pennsylvania

Search for other papers by Craig H. Bishop in
Current site
Google Scholar
PubMed Close
,
Brian J. Etherton The Pennsylvania State University, University Park, Pennsylvania

Search for other papers by Brian J. Etherton in
Current site
Google Scholar
PubMed Close
, and
Sharanya J. Majumdar The Pennsylvania State University, University Park, Pennsylvania

Search for other papers by Sharanya J. Majumdar in
Current site
Google Scholar
PubMed Close
Print Publication:
01 Mar 2001
DOI:
https://doi.org/10.1175/1520-0493(2001)129<0420:ASWTET>2.0.CO;2
Page(s):
420–436
Restricted access

Abstract

A suboptimal Kalman filter called the ensemble transform Kalman filter (ET KF) is introduced. Like other Kalman filters, it provides a framework for assimilating observations and also for estimating the effect of observations on forecast error covariance. It differs from other ensemble Kalman filters in that it uses ensemble transformation and a normalization to rapidly obtain the prediction error covariance matrix associated with a particular deployment of observational resources. This rapidity enables it to quickly assess the ability of a large number of future feasible sequences of observational networks to reduce forecast error variance. The ET KF was used by the National Centers for Environmental Prediction in the Winter Storm Reconnaissance missions of 1999 and 2000 to determine where aircraft should deploy dropwindsondes in order to improve 24–72-h forecasts over the continental United States. The ET KF may be applied to any well-constructed set of ensemble perturbations.

The ET KF technique supercedes the ensemble transform (ET) targeting technique of Bishop and Toth. In the ET targeting formulation, the means by which observations reduced forecast error variance was not expressed mathematically. The mathematical representation of this process provided by the ET KF enables such things as the evaluation of the reduction in forecast error variance associated with individual flight tracks and assessments of the value of targeted observations that are distributed over significant time intervals. It also enables a serial targeting methodology whereby one can identify optimal observing sites given the location and error statistics of other observations. This allows the network designer to nonredundantly position targeted observations. Serial targeting can also be used to greatly reduce the computations required to identify optimal target sites. For these theoretical and practical reasons, the ET KF technique is more useful than the ET technique. The methodology is illustrated with observation system simulation experiments involving a barotropic numerical model of tropical cyclonelike vortices. These include preliminary empirical tests of ET KF predictions using ET KF, 3DVAR, and hybrid data assimilation schemes—the results of which look promising. To concisely describe the future feasible sequences of observations considered in adaptive sampling problems, an extension to Ide et al.’s unified notation for data assimilation is suggested.

Corresponding author address: Craig H. Bishop, Department of Meteorology, The Pennsylvania State University, 520 Walker Building, University Park, PA 16802.

Abstract

A suboptimal Kalman filter called the ensemble transform Kalman filter (ET KF) is introduced. Like other Kalman filters, it provides a framework for assimilating observations and also for estimating the effect of observations on forecast error covariance. It differs from other ensemble Kalman filters in that it uses ensemble transformation and a normalization to rapidly obtain the prediction error covariance matrix associated with a particular deployment of observational resources. This rapidity enables it to quickly assess the ability of a large number of future feasible sequences of observational networks to reduce forecast error variance. The ET KF was used by the National Centers for Environmental Prediction in the Winter Storm Reconnaissance missions of 1999 and 2000 to determine where aircraft should deploy dropwindsondes in order to improve 24–72-h forecasts over the continental United States. The ET KF may be applied to any well-constructed set of ensemble perturbations.

The ET KF technique supercedes the ensemble transform (ET) targeting technique of Bishop and Toth. In the ET targeting formulation, the means by which observations reduced forecast error variance was not expressed mathematically. The mathematical representation of this process provided by the ET KF enables such things as the evaluation of the reduction in forecast error variance associated with individual flight tracks and assessments of the value of targeted observations that are distributed over significant time intervals. It also enables a serial targeting methodology whereby one can identify optimal observing sites given the location and error statistics of other observations. This allows the network designer to nonredundantly position targeted observations. Serial targeting can also be used to greatly reduce the computations required to identify optimal target sites. For these theoretical and practical reasons, the ET KF technique is more useful than the ET technique. The methodology is illustrated with observation system simulation experiments involving a barotropic numerical model of tropical cyclonelike vortices. These include preliminary empirical tests of ET KF predictions using ET KF, 3DVAR, and hybrid data assimilation schemes—the results of which look promising. To concisely describe the future feasible sequences of observations considered in adaptive sampling problems, an extension to Ide et al.’s unified notation for data assimilation is suggested.

Corresponding author address: Craig H. Bishop, Department of Meteorology, The Pennsylvania State University, 520 Walker Building, University Park, PA 16802.

Save
Cover Monthly Weather Review
Monthly Weather Review

  • Anderson, J. L., and S. L. Anderson, 1999: A Monte Carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Mon. Wea. Rev.,127, 2741–2758.

  • Baker, N. L., and R. Daley, 2000: Observation and background adjoint sensitivity in the adaptive observation–targeting problem. Quart. J. Roy. Meteor. Soc.,126, 1431–1454.

  • Bergot, T., 1999: Adaptive observations during FASTEX: A systematic survey of upstream flights. Quart. J. Royal Meteor. Soc.,125, 3271–3298.

  • ——, G. Hello, A. Joly, and S. Malardel, 1999: Adaptive observations: A feasibility study. Mon. Wea. Rev.,127, 743–765.

  • Berliner, L. M., Q. Lu, and C. Snyder, 1999: Statistical design for adaptive weather observations. J. Atmos. Sci.,56, 2536–2552.

  • Bierman, G. J., 1977: Factorization Methods for Discrete Sequential Estimation. Academic Press, 241 pp.

  • Bishop, C. H., and Z. Toth, 1999: Ensemble transformation and adaptive observations. J. Atmos. Sci.,56, 1748–1765.

  • Buizza, R., and A. Montani, 1999: Targeting observations using singular vectors. J. Atmos. Sci.,56, 2965–2985.

  • ——, R. Gelaro, F. Molteni, and T. N. Palmer, 1997: The impact of increased resolution on predictability studies with singular vectors. Quart. J. Roy. Meteor. Soc.123, 1007–1033.

  • Cohn, S. E., 1997: An introduction to estimation theory. J. Meteor. Soc. Japan.,75, 257–288.

  • ——, N. S. Sivakumaran, and R. Todling, 1994: A fixed lag Kalman smoother for retrospective data assimilation. Mon. Wea. Rev.,122, 2838–2867.

  • ——, A. da Silva, J. Guo, M. Sienkiewicz, and D. Lamich, 1998: Assessing the effects of data selection with the DAO Physical Space Statistical Analysis System. Mon. Wea. Rev.,126, 2913–2926.

  • Courtier, P., and Coauthors, 1998: The ECMWF implementation of three-dimensional variational assimilation (3D-Var). I: Formulation. Quart. J. Roy. Meteor. Soc.,124, 1783–1809.

  • Daley, R., 1991: Atmospheric Data Analysis. Vol. 2. Cambridge University Press, 457 pp.

  • da Silva, A., and J. Guo, 1996: Documentation of the Physical Space Statistical Analysis System (PSAS). Part I: The conjugate gradient solver version PSAS-1.00. DAO Office Note 96-02, 66 pp. [Available from Data Assimilation Office, Goddard Space Flight Center, Greenbelt, MD 20771; also available online at http://dao.gsfc.nasa.gov/subpages/officenotes.htmlm.].

  • Dee, D. P., 1995: On-line estimation of error covariance parameters for atmospheric data assimilation. Mon. Wea. Rev.,123, 1128–1196.

  • Desroziers, G., B. Pouponneau, J.-N. Thépaut, M. Janisková, and F. Veersé, 1999: Four-dimensional variational analyses of FASTEX situations using special observations. Quart. J. Roy. Meteor. Soc.,125, 3339–3358.

  • Ehrendorfer, M., and J. J. Tribbia, 1997: Optimal prediction of forecast error covariances through singular vectors. J. Atmos. Sci.,54, 286–311.

  • Evensen, G., and P. J. van Leeuwen, 1996: Assimilation of Geosat altimeter data for the Agulhas Current using the ensemble Kalman filter with a quasigeotrophic model. Mon. Wea. Rev.,124, 85–96.

  • Fischer, C., A. Joly, and F. Lalaurette, 1998: Error growth and Kalman filtering within an idealized baroclinic flow. Tellus,50A, 596–615.

  • Gelaro, R., R. Buizza, T. N. Palmer, and E. Klinker, 1998: Sensitivity analysis of forecast errors and the construction of optimal perturbations using singular vectors. J. Atmos. Sci.,55, 1012–1037.

  • ——, R. Langland, G. D. Rohaly, and T. E. Rosmond, 1999: An assessment of the singular vector approach to targeted observations using the FASTEX data set. Quart. J. Roy. Meteor. Soc.,125, 3299–3328.

  • Hamill, T. M., and C. Snyder, 2000: A hybrid ensemble Kalman filter–3D-Variational analysis scheme. Mon. Wea. Rev.,128, 2905–2919.

  • ——, ——, and R. E. Morss, 2000: A comparison of probabilistic forecasts from bred, singular-vector, and perturbed observation ensembles. Mon. Wea. Rev.,128, 1835–1851.

  • Houtekamer, P. L., and H. L. Mitchell, 1998: Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev.,126, 796–811.

  • ——, L. Lefaivre, J. Derome, H. Ritchie, and H. L. Mitchell, 1996:A system simulation approach to ensemble prediction. Mon. Wea. Rev.,124, 1225–1242.

  • Ide, K., P. Courtier, M. Ghil, and A. C. Lorenc, 1997: Unified notation for data assimilation: Operational, sequential and variational. J. Meteor. Soc. Japan.,75 (1B), 181–189.

  • Joly, A., and Coauthors, 1997: The Fronts and Atlantic Storm-Track Experiment (FASTEX): Scientific objectives and experimental design. Bull. Amer. Meteor. Soc.,78, 1917–1940.

  • Langland, R. H., and G. D. Rohaly, 1996: Adjoint-based targeting of observations for FASTEX cyclones. Preprints, Seventh Conf. on Mesoscale Processes, Phoenix, AZ, Amer. Meteor. Soc., 369–371.

  • ——, and Coauthors, 1999: The North Pacific Experiment (NORPEX-98) Targeted observations for improved weather forecasts. Bull. Amer. Meteor. Soc.,80, 1363–1384.

  • Lorenz, E. N., and K. A. Emanuel, 1998: Optimal sites for supplementary observation sites: Simulation with a small model. J. Atmos. Sci.,55, 399–414.

  • Molteni, F., R. Buizza, T. N. Palmer, and T. Petroliagis, 1996: The ECMWF ensemble prediction system. Methodology and validation. Quart. J. Roy. Meteor. Soc.,122, 73–120.

  • Morss, R. E., 1999: Adaptive observations: Idealized sampling strategies for improving numerical weather prediction. Ph.D. thesis, Massachusetts Institute of Technology, 225 p. [Available from Dept. of Earth, Atmospheric and Planetary Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139.].

  • ——, K. A. Emanuel, and C. Snyder, 2001: Idealized adaptive observation strategies for improving numerical weather prediction. J. Atmos. Sci.,58, 210–232.

  • Palmer, T. N., R. Gelaro, J. Barkmeijer, and R. Buizza, 1998: Singular vectors, metrics, and adaptive observations. J. Atmos. Sci.,55, 633–653.

  • Parrish, D. F., and J. C. Derber, 1992: The National Meteorological Center’s spectral statistical interpolation analysis scheme. Mon. Wea. Rev.,120, 1747–1763.

  • Pu, Z., and E. Kalnay, 1999: Targeting observations with the quasi-linear inverse and adjoint and adjoint NCEP global models: Performance during FASTEX. Quart. J. Roy. Meteor. Soc.,125, 3329–3338.

  • Rabier, F., H. Jarvinen, E. Klinker, J.-F. Mahfouf, and A. Simmons, 2000: The ECMWF operational implementation of four-dimensional variational assimilation. I: Experimental results with simplified physics. Quart. J. Roy. Meteor. Soc.,126, 1143–1170.

  • Smith, L. A., 2000: Disentangling uncertainty and error: On the predictability of nonlinear systems. Nonlinear Dynamics and Statistics, A. Mees, Ed., Birkhauser, 31–64.

  • Snyder, C., 1996: Summary of an informal workshop on adaptive observations and FASTEX. Bull. Amer. Meteor. Soc.,77, 953–965.

  • Szunyogh, I., Z. Toth, K. A. Emanuel, C. H. Bishop, C. Snyder, R. E. Morss, J. Woolen, and T. Marchok, 1999: Ensemble based targeting experiments during FASTEX: The impact of dropsonde data from the Lear jet. Quart. J. Roy. Meteor. Soc.,125, 3189–3218.

  • ——, ——, R. E. Morss, S. J. Majumdar, B. J. Etherton, and C. H. Bishop, 2000: The effect of targeted dropsonde observations during the 1999 Winter Storm Reconnaissance program. Mon. Wea. Rev.,128, 3520–3537.

  • Todling, R., S. E. Cohn, and N. S. Sivakumaran, 1998: Suboptimal schemes for retrospective data assimilation based on the fixed-lag Kalman smoother. Mon. Wea. Rev.,126, 2274–2286.

  • Toth, Z., and E. Kalnay, 1993: Ensemble forecasting at NMC: The generation of perturbations. Bull. Amer. Meteor. Soc.,74, 2317–2330.

  • ——, and ——, 1997: Ensemble forecasting at NCEP and the breeding method. Mon. Wea. Rev.,126, 3292–3302.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 6587 2679 108
PDF Downloads 4181 899 42