Mobile observation platforms, such as dropwindsonde-equipped aircraft, allow observations to be placed in atmospheric regions where they are likely to improve forecasts of significant weather. For example, if a weather forecast was predicting torrential rain for California in three days, one might decide to deploy dropwindsondes somewhere upstream of California in an effort to obtain a more accurate 2-day forecast of the event. In this example, we would call California the verification region while the dropwindsonde deployment site(s) would be called the target site(s). The time(s) at which the dropwindsondes were launched would be called the targeting time(s) while the time at which the heavy rain was expected over California would be called the verification time. The problem of determining where the dropwindsondes should be deployed is an example of the type of adaptive sampling problems we wish to consider here.
More generally, one may consider an observational network to be composed of a routine component and an adaptive component. The routine component comprises observations from the fixed rawinsonde network, satellite measurements, and other measurements that are routinely taken. The adaptive component gives the network designer a choice of some finite but possibly large number Q of future sequences of observational networks; for example, the network designer might have to choose between Q different possible flight paths that could be taken by a dropwindsonde-equipped plane. The purpose of this paper is to describe how the ensemble transform Kalman filter (ET KF) can be used to identify which of these Q feasible future sequences of observational resources would minimize forecast error variance over some prespecified verification region at some prespecified verification time.
Other objective strategies for optimally deploying adaptive observations include the singular vector technique (Palmer et al. 1998; Gelaro et al. 1999; Buizza and Montani 1999; Bergot et al. 1999), the gradient technique (Langland and Rohaly 1996), the inverse model technique (Pu and Kalnay 1999), the ensemble spread technique (Lorenz and Emanuel 1998), and the ensemble transform technique (Bishop and Toth 1999). With the exception of the ensemble spread technique, all of these techniques attempt to identify target regions that are dynamically connected to the verification region. In order to account for the fact that large-amplitude but slowly amplifying initial condition errors may sometimes damage a forecast as much as small-amplitude rapidly growing initial condition errors, targeting techniques also need to incorporate information about the probability density of analysis errors.
In principle, the singular vector technique (Palmer et al. 1998) accounts for the analysis error statistics of the routine observational network by identifying structures that evolve into the leading eigenvectors of the forecast error covariance matrix associated with the routine observational network and the localized verification region. The singular vector technique does not, however, quantitatively account for the effect of specific deployments of supplemental observations on prediction or analysis error statistics. The need to incorporate such information has been stressed by Berliner et al. (1999) and Baker and Daley (2000).
Among other things, Baker and Daley acknowledge that both the adjoint and singular vector targeting techniques identify target sites in regions where the vector gradient of a forecast aspect with respect to the analysis is large (cf. Buizza et al. 1997; Gelaro et al. 1998). They go on to show that in cases where the spatial scale of such analysis sensitivity vectors is much smaller than the correlation length scale assumed by the data assimilation scheme, the analysis sensitivity vectors poorly represent the sensitivity of the forecast aspect to observations. They point out that targeting strategies would be more reliable if they were based on observation sensitivity rather than analysis sensitivity.
The ET KF directly provides observation sensitivity. Moreover, it provides quantitative estimates of the reduction in forecast error variance likely to result from each of the possible future sequences of observational networks. However, ET KF observation sensitivities only strictly pertain to data assimilation schemes that assume the same forecast or first-guess error covariance matrix as that assumed by the ET KF. Consequently, its observation sensitivities are unreliable when the error covariances assumed by the data assimilation scheme used to assimilate the data are very different from those assumed by the ET KF. Mismatches between the error statistics assumed by the ET KF and those assumed by the actual assimilating scheme can, in principle, lead to many of the same difficulties discussed by Baker and Daley. Gross mismatches between the ET KF and the assimilating scheme could be avoided, however, by incorporating the ET KF within ensemble-based data assimilation schemes such as those that have been suggested by Evensen and van Leeuwen (1996), Houtekamer and Mitchell (1998), Anderson and Anderson (1999), and Hamill and Snyder (2000).
Targeted observations taken during the Fronts and Atlantic Storm-Track Experiment (FASTEX; Snyder 1996; Joly et al. 1997), the North Pacific Experiment (Langland et al. 1999), and the National Centers for Environmental Prediction’s (NCEP’s) Winter Storm Reconnaissance missions of 1999 and 2000 have shown that, on average, targeted observations improve forecast skill (Bergot et al. 1999; Szunyogh et al. 1999, 2000). They have also shown that targeted observations occasionally degrade forecast skill. See Morss (1999) for an investigation of the conditions under which such degradations are likely to occur. Within the context of simple models containing data voids of one sort or another, Fischer et al. (1998), Morss et al. (2001), and Lorenz and Emanuel (1998) using Kalman filter, 3DVAR and direct insertion data assimilation schemes, respectively, have all shown that targeted observations reduce forecast error more than fixed observations. Work by Fischer et al. (1998), Bergot (2000, manuscript submitted to Quart. J. Roy. Meteor. Soc.), and Rabier et al. (2000) indicates that the usefulness of targeted observations is significantly enhanced when more sophisticated data assimilation schemes are used to assimilate the data.
In section 2, some general background on Kalman filters and how they can be used in targeting is given. In this discussion, we develop a notation for describing the future feasible sequences of observational networks that one must consider in addressing adaptive sampling problems. In section 3, the characteristics that distinguish the ET KF from other ensemble Kalman filters are outlined. The use of the ET KF as a targeting tool is illustrated in section 4. Concluding remarks follow in section 5.
2. Kalman filters and the optimal adaptive network design problem
Estimation theory (Cohn 1997) not only provides minimum error variance estimates of the state of the atmosphere x(t|ti) at the time t given observations up to and including those taken at the time ti, it also provides the covariance matrix
Future observational networks are made up of a routine component, an adaptive component, or both. The routine component consists of all of those aspects of the observational network that cannot be varied at will such as the fixed rawinsonde network and the temporally varying sets of observations available from satellites. Although certain aspects of the routine observational network such as the location of future cloud track wind vectors are difficult to accurately predict, for simplicity we shall assume that the future states of the routine component of the observational network can be known before the first future observing time ti+1. The adaptive component comprises those aspects of the observational network that can be varied at will such as the flight tracks taken by reconnaissance planes or the times and locations at which cloud track wind satellite algorithms are put into high-resolution rapid-scan mode (C. Velden 1998, personal communication). For simplicity we shall also assume that the future possible states of the adaptive component of the observational network are known.
Assuming that ti is an analysis time in the recent past, let us suppose that there are a series of discrete future times ti+m, m = 1, 2, . . . , M at which one of Ji+m feasible observational networks might be deployed. At some of these times, there might only be one feasible observational network (Ji+m = 1), the “routine” observational network. At other times adaptive observing facilities (e.g., reconnaissance planes) might be available to supplement the routine observational network. Since any feasible observational network chosen at the time ti+m could be combined with any of the feasible networks available at ti+m+1, there could be as many as
To distinguish between the Q future possible sequences of observational networks, we let
Generally, adaptive sampling strategies must be able to identify the “optimal” deployment of observational resources by some decision time td where ti < td < ti+M in order to leave time to make deployment arrangements such as sending reconnaissance aircraft to their optimal space–time locations; cf. Fig. 1. In terms of forecast error variance, the adaptive observational network design problem in its most general form may be stated as follows. Using the forecast x(t|Hi), determine which of the future possible sequences of observational networks minimizes the forecast error variance of the forecast x(ti+V|
Using induction, it is straightforward to show that Eqs. (2a) and (2b) could be used to predict
Computational resources are currently not nearly sufficient to run a full Kalman filter for an operational weather forecasting system for one distinct observational network let alone Q distinct observational resources. But suppose one had access to a very large ensemble of forecasts running from ti to ti+V; by subtracting each of the ensemble members from the ensemble mean, one would obtain a vector subspace of ensemble perturbations. These ensemble perturbations could be linearly combined to create a huge variety of differently structured perturbations in the same way that sine and cosine basis functions can be added together to create a huge variety of functions. Transforming the perturbations into orthonormal vectors and attaching variances to each of the direction vectors enables one to use the ensemble to describe error covariance within the vector subspace of the ensemble perturbations. The basic idea behind the ET KF is to define these transformations so that the transformed perturbations will describe the forecast error covariance one would obtain if certain sequences of observations were taken in the near future. This approach reduces the computational expense of having to propagate Q covariance matrices forward in time to the computational expense of running a single nonlinear ensemble forecast from ti to ti+V and performing Q × M ensemble transformations. This combined with serial observation processing techniques makes the ET KF a practical tool for distinguishing between effective and ineffective future deployments of observational resources.
3. Optimal network design with the ET KF
a. Ensemble transformation
The estimate (3) does not allow for the inclusion of a parameterized form of model error covariance
Both Evensen and Van Leeuwen (1996) and Houtekamer and Mitchell (1998) have commented that the significant computational expense of ensemble Kalman filters is the inversion of the innovation or residual covariance matrix (
The ensemble Kalman filter formulation of Evensen and Van Leeuwen (1996) solves the size problem by only using an intelligently chosen subset of observations. They solve the ill-conditioning problem by means of a singular value decomposition of (
Houtekamer and Mitchell’s (1998) ensemble Kalman filter formulation simultaneously solves the size and ill-conditioning problem by using a cutoff radius beyond which covariances between variables are assumed to be zero. The cutoff radius also serves to guard against spurious long-distance error correlations that arise when small ensembles are used in the estimation of
Further reductions in computational expense are obtained by making appropriate use of serial processing theory. The basic idea is to first estimate the effect of the routine component of the observational network on error covariance before estimating the effect of each of the feasible deployments of the adaptive component of the observational network. This serial processing approach is described in appendix B.
4. A simple test of the ET KF
a. The “agency” ensemble and the “truth” run
A barotropic numerical model was used to perform a simple observation system simulation experiment (OSSE). In order to obtain some interesting structures in the truth run, tropical cyclonelike vortices (cf. Fig. 2) were grown from a barotropically unstable initial state. This unstable state featured a barotropically unstable strip of high cyclonic (positive) vorticity reminiscent of the intertropical convergence zone (ITCZ) and a broad barotropically unstable midlatitude vortex strip. Broad strips of weak anticyclonic (negative) vorticity lay between the cyclonic strips. The size of the square domain was 6400 km × 6400 km.
The meridional gradient of the Coriolis parameter β was set equal to its actual value at 20°N. The model included a vortex-preserving relaxation scheme that used a zonally uniform nudging term to keep the zonal average of vorticity at each latitude line close to the initial zonal average of vorticity.
To initiate barotropic instability, random noise was added to the initially zonally invariant shear flow and the model was run with a horizontal grid spacing of 50 km. Amplifying barotropic waves grew on the ITCZ-like shear zone and subsequently rolled up into coherent vortices. Once vortex rollup was complete, the model fields were truncated to a horizontal resolution of 100 km and saved.
The initial condition for the base member of our hypothetical forecasting agency’s ensemble forecast was obtained by adding a random perturbation to the saved field. An additional 64 agency initial conditions for a 65-member ensemble were obtained by adding the 64 largest scale-orthogonal sine and cosine perturbations that are periodic on our chosen domain to the base member. The perturbations were all given the same initial streamfunction amplitude of 1414 m2 s−1. With this streamfunction amplitude, the largest scale perturbation had a wind maximum of 0.2 m s−1 while the smallest scale wind perturbation had a wind maximum of 0.8 m s−1. For comparison, the maximum wind in the agency’s base member was 25 m s−1. All of the agency’s initial conditions were integrated forward in time at a horizontal resolution of 100 km.
In real atmosphere applications, model error makes forecasts systematically different to reality. Such systematic differences make it impossible for ensemble perturbations to represent unbiased realizations of forecast error (Smith 2000). Some models systematically overestimate the speed of coherent features such as low pressure systems moving across significant topography (J. M. Fritsch 1999, personal communication). In order to emulate this type of error in our first-guess field, the “truth run” shown in Fig. 2 is made to be 24 h ahead of the agency’s forecasts. To achieve this, we take the agency’s base member, subtract the random perturbation by which it differs from the saved fields, integrate the resulting field for 24 h and then use this field as the initial condition for the truth run. Since the truth run is also run at 100 km, the models used for the truth and agency runs are effectively equivalent. Consequently, one can only count on the agency’s forecasts being systematically different to the truth for a finite number of data assimilation cycles. That need not concern us here, as we shall only be analyzing a single cycle.
To further increase the difference between the agency’s forecast and the truth run, the agency’s forecast was run for 48 h before any observations were taken. For simplicity, in this experiment, we assume that there is just one targeting time. Consequently, M = 1 so that the targeting time ti+M = ti+1 = (ti + 48) h. The mean
The vorticity variance of the ensemble
b. Selection of two observation sites
For simplicity, suppose that just two targeted vorticity observations at ti+1 = (ti + 48) h could be used to sample the first-guess error field shown in Fig. 4a at any of the 4096 grid points defining the model state. (We observe vorticity simply because it is relatively easy to understand how the vorticity field evolves through time.) To achieve the observational objective of minimizing the 48-h forecast error variance for a verification region covering the whole domain at a verification time ti+v = ti+1 + 48 = ti + 96 h, where should the observations be placed?
To answer this question, the ET KF could be used to assess the error reducing effectiveness of all 4096!/(4094!2!) ≅ 8.4 × 106 distinct possible combinations of observations. However, in order to illustrate how the serial observation processing discussed in appendix B can greatly reduce the computational burden of identifying optimal combinations of adaptive observational resources, we will find a suboptimal solution to the problem by first identifying the best site for a single observation of vorticity. The site for the second vorticity observation will be obtained by using the serial assimilation theory discussed in appendix B to find the best site for an observation given that the best site for a single observation was also observed. This serial processing approach only requires the evaluation of 8192 future feasible observational networks.
Specifically, Eq. (17b) together with the ti+v = 96 h ensemble perturbations were used to find which of the 4096 grid points should be observed at ti+1 = 48 h in order to maximize the trace of the signal covariance matrix,
The location of the second observation site was obtained by constructing a map of the trace of the signal covariance matrix as a function of the location of the second observation given that the an observation was also to be taken at 8°N and 68°W using the serial processing approach described in section 3. This map is shown in Fig. 5b. It indicates that there is little point in taking additional observations near the vortex that was sampled by the first observation and that the best site for the second observation is at 12°N, 83°W. Note that the two sites selected by the ET KF are intuitively reasonable given that Fig. 2b indicates that, at the verification time ti+2 = 96 h, the two vortices sampled rotate around each other as if they were in the initial stages of a vortex pairing instability. Note also that Figs. 5a and 5b are the ET KF counterparts of Baker and Daley’s (2000) single observation and marginal observation sensitivity maps, respectively.
c. Effects of assimilating the observations
1) Data assimilation with the ET KF
To assimilate the observations with the ET KF, a normalized observational operator
With the observational operator
2) Quasi-isotropic error correlation data assimilation (3DVAR)
The analysis increment obtained by assimilating the two observations with this formulation is shown in Fig. 4c. Figures 6 and 7 show that the forecast for the verification time made from the analysis increment based on the isotropic error correlation function (22b) is not nearly as good as the improvement obtained from the ET KF assimilation.
3) Hamill and Snyder–type hybrid ensemble Kalman filter
d. Additional observations required to make 3DVAR respond like an ensemble KF
We have seen that ET KF analysis increments are strikingly different to three-dimensional variational (3DVAR) increments that assume isotropic error correlations. If the ET KF is used to position observations that will be assimilated by 3DVAR it may be desirable to choose a targeting strategy that will increase the chances of the 3DVAR increment having the same qualitative features as a corresponding ET KF increment for a number of reasons. First, the ET KF will be trustworthier if the 3DVAR increment is similar to the corresponding ET KF increment. Second, in our experiments, the ET KF increments reduce forecast error more than 3DVAR increments.
Figure 4c suggests that such 3DVAR increments may be loosely thought of as being composed of overlaying bull’s-eye patterns. The general features of an ET KF increment may be anticipated before observations are actually taken by examining a plot of the variances listed on the diagonal of the signal covariance matrix
Figure 8 shows an example of such a plot for the observation sites selected in section 4b. A comparison of Fig. 8 with Fig. 4b shows that the signal variance field has similar qualitative features to the actual ET KF analysis increment. To give the 3DVAR increment a chance of attaining the qualitative features of an ET KF increment, additional observations would need to be taken to the south of vortex B and the north of vortex C, respectively. The analysis increment obtained by assimilating all four observations with 3DVAR is shown in Fig. 4d. Figures 6 and 7 show that the skill of the forecast that results from this four-observation 3DVAR analysis is substantially better than the skill of the two-observation 3DVAR forecast but still slightly less skilful than the two-observation ET KF forecast.
e. Comparison of ET KF ranking of observation sites with empirically determined ranking
As noted in section 3, in the simple targeting situations considered here, the signal variance at the verification time is equal to the reduction in error variance at the verification time when error covariances are accurately specified. Figure 5a shows a map of the signal variance at the verification time as a function of the location of a single observation. By actually assimilat- ing an observation at each of the sites and integrating the resulting analysis forward in time, one can build maps of the squared signal magnitude at the verification time as a function of the observation location. These maps are similar to those introduced by Morss (1999) for a 3DVAR scheme. Figure 9 shows such maps for the three data assimilation schemes mentioned in section 4c. Each point on these contour maps gives the squared magnitude of a single signal realization as a function of the observation site. In contrast, each point on the contour map shown in Fig. 5a gives the expected or mean squared signal magnitude as a function of observation site.
While the ET KF data assimilation scheme map (Fig. 9a) is most similar to Fig. 5a, both the hybrid Ens KF and the 3DVAR data assimilation schemes give patterns that are also qualitatively similar to Fig. 5a. Thus, in this case, the ET KF successfully anticipated the qualitative features of signal squared amplitude as a function of observation site for a variety of data assimilation schemes. Note also that the contour interval for Fig. 9c is one-fifth of that used in Figs. 9a and 9b. This reflects the fact that 3DVAR increments cover less geographical area than the other types of increments.
When error covariances are perfectly specified in the data assimilation scheme, taking observations always reduces forecast error variance. Reducing forecast error variance is, however, a very different thing than reducing forecast error in every single case. When error covariances are imperfectly specified, the signal variance associated with observations will not necessarily be equal to the reduction in forecast error variance.
With these points in mind, compare Fig. 5a with Figs. 10a and 10b, which show plots of the reduction in squared forecast vorticity error as a function of the location of a single error free observation that was assimilated using the ET KF and hybrid Ens KF, respectively. They show that, as might have been hoped, significant forecast improvements result from placing observations near vortices B and C in the manner indicated by the reduction in forecast error variance plot shown in Fig. 5a. Note that while Fig. 5a indicates that observing vortex C would reduce expected forecast error variance slightly more than observing vortex B, Figs. 10a and 10b show that observations of vortex B reduced forecast error more than observations of vortex C. This fact alone does not mean that the information in Fig. 5a is incorrect. Figures 10a and 10b represent individual realizations of forecast error reduction whereas Fig. 5a represents the average reduction in forecast error variance over many different realizations of first-guess and observation error.
The contour interval in Fig. 10c is half of that used in Figs. 10b and 10c. Thus, assimilating a single observation with the 3DVAR scheme has much less impact on the forecast than assimilating with either the hybrid or ET KF schemes. For all of the schemes, there are a considerable number of sites where assimilating a single observation degrades forecast accuracy. Interestingly, Fig. 11 shows that many of the observation sites that led to improved analyses went on to produce forecasts of degraded accuracy, for example, observation sites near 10°N, 55°W. An attractive feature of the hybrid scheme is that while the large forecast improvements it yields from observations near vortices B and C are similar to those produced by the ET KF, it produces smaller and less frequent forecast degradations than those produced by the ET KF.
To assess the extent that improvements outweigh degradations, Fig. 12 shows the horizontal average of the reductions in enstrophy error from single observations. It indicates that forecast improvements outweigh forecast degradations for all of the data assimilation schemes and that the improvement is larger for the hybrid Ens KF scheme than it is for the other two schemes. In considering this diagram, one should note that the correlation length scale D required by the 3DVAR data assimilation scheme was chosen by searching for the correlation length scale D that maximized the mean forecast improvement delivered by 3DVAR. Thus, the 3DVAR result shown in Fig. 12 represents an upper bound on what could be achieved with 3DVAR. In contrast, the ensemble covariances used by the ET KF and hybrid schemes were deliberately degraded by making the agency’s first-guess field lag the truth by 24 h. Despite this both schemes outperformed 3DVAR. Finally, we note that the hybrid’s α was tuned to maximize the mean forecast improvement produced by the hybrid scheme.
5. Concluding remarks
Forecast signal variance has been defined to be the variance of forecast signals that could be imparted by targeted observations. If the error covariances in the data assimilation scheme are accurately specified, forecast signal variance corresponds to the reduction in forecast error variance. To test the ET KF’s predictions of the forecast signal variance associated with a single future observation, OSSEs were performed. In these OSSEs, maps of the ET KF–predicted forecast signal variance as a function of the location of a single observation were compared with empirically determined maps of squared forecast signal and maps of the reduction in squared forecast error as a function of observation sites. The empirical maps were produced by performing separate assimilation and forecast cycles for every possible location of a single observation using an ET KF, a 3DVAR, and a hybrid ensemble Kalman filter data assimilation scheme.
For all of the data assimilation schemes tested, the maps of ET KF–predicted signal variance as a function of observation site were found to capture the same qualitative features as the empirically determined maps of squared forecast signal as a function of observation site. The degree of correspondence between these empirically determined squared forecast signal maps and corresponding empirically determined reduction in squared forecast error maps was higher for the ensemble-based data assimilation schemes than it was for the 3DVAR data assimilation scheme. This finding indicates that the error covariance model of the ensemble-based data assimilation schemes was superior to the 3DVAR covariance model. It is also consistent with the fact that the overall forecast error reduction produced by the ensemble-based data assimilation schemes was significantly greater than that produced by the 3DVAR data assimilation scheme.
The ET KF was better able to anticipate the signals produced by ensemble-based data assimilation schemes than 3DVAR. As such, the ET KF is likely to be a more reliable targeting tool in cases where data assimilation schemes capable of inferring flow-dependent error statistics are used to assimilate targeted data. The ensemble-based data assimilation schemes of Houtekamer and Mitchell (1998), Anderson and Anderson (1999), and Hamill and Snyder (2000) fall into this category as do 4DVAR schemes such as those discussed by Desroziers et al. (1999) and Rabier et al. (2000). Via dynamical constraints, 4DVAR schemes produce analysis increments qualitatively similar to those that would be obtained from a scheme that incorporated flow-dependent error covariances similar to those used by the ET KF. A detailed analysis of the importance of flow-dependent error covariances to assimilating targeted data can be found in Fischer et al. (1998).
To improve the performance of 3DVAR in targeting experiments, we have suggested that additional observations be placed at the maxima of the appropriate ET KF signal variance field. Whether such a strategy would be effective in a more sophisticated model or in a field program is a subject for future research.
The 3DVAR scheme used in this paper assumed isotropic first-guess error correlations qualitatively similar to those assumed by the data assimilation schemes used in the assessments of adaptive observations made by Szunyogh et al. (1999a,b), Langland et al. (1999), Bergot (1999), and Morss et al. (2001). Our finding that ensemble-based data assimilation schemes are much better than 3DVAR at extracting forecast improvements from a couple of irregularly distributed observations suggests that the positive effects of adaptive observations on forecast accuracy reported by these authors might be greatly enhanced by more sophisticated data assimilation schemes. Indeed, work by Fischer et al. (1998), Bergot (2000), and Rabier et al. (2000) indicates that the positive effect of FASTEX targeted observations is much greater when a 4DVAR data assimilation scheme is used.
The authors wish to express their gratitude to Arlindo da Silva of NASA’s Data Assimilation Office for his encouragement and for providing us with the barotropic vorticity equation model used in this study and also to Kerry Emanuel of MIT for suggesting the vortex preserving relaxation scheme. The encouragement, comments, and criticisms of Istvan Szunyogh and Zoltan Toth from the National Centers for Environmental Prediction are also appreciated. The authors gratefully acknowledge the financial support of National Science Foundation Grants ATM-96-12502 and ATM-98-14376.
Anderson, J. L., and S. L. Anderson, 1999: A Monte Carlo implementation of the nonlinear filtering problem to produce ensemble assimilations and forecasts. Mon. Wea. Rev.,127, 2741–2758.
Baker, N. L., and R. Daley, 2000: Observation and background adjoint sensitivity in the adaptive observation–targeting problem. Quart. J. Roy. Meteor. Soc.,126, 1431–1454.
Bergot, T., 1999: Adaptive observations during FASTEX: A systematic survey of upstream flights. Quart. J. Royal Meteor. Soc.,125, 3271–3298.
——, G. Hello, A. Joly, and S. Malardel, 1999: Adaptive observations: A feasibility study. Mon. Wea. Rev.,127, 743–765.
Berliner, L. M., Q. Lu, and C. Snyder, 1999: Statistical design for adaptive weather observations. J. Atmos. Sci.,56, 2536–2552.
Bierman, G. J., 1977: Factorization Methods for Discrete Sequential Estimation. Academic Press, 241 pp.
Bishop, C. H., and Z. Toth, 1999: Ensemble transformation and adaptive observations. J. Atmos. Sci.,56, 1748–1765.
Buizza, R., and A. Montani, 1999: Targeting observations using singular vectors. J. Atmos. Sci.,56, 2965–2985.
——, R. Gelaro, F. Molteni, and T. N. Palmer, 1997: The impact of increased resolution on predictability studies with singular vectors. Quart. J. Roy. Meteor. Soc.123, 1007–1033.
Cohn, S. E., 1997: An introduction to estimation theory. J. Meteor. Soc. Japan.,75, 257–288.
——, N. S. Sivakumaran, and R. Todling, 1994: A fixed lag Kalman smoother for retrospective data assimilation. Mon. Wea. Rev.,122, 2838–2867.
——, A. da Silva, J. Guo, M. Sienkiewicz, and D. Lamich, 1998: Assessing the effects of data selection with the DAO Physical Space Statistical Analysis System. Mon. Wea. Rev.,126, 2913–2926.
Courtier, P., and Coauthors, 1998: The ECMWF implementation of three-dimensional variational assimilation (3D-Var). I: Formulation. Quart. J. Roy. Meteor. Soc.,124, 1783–1809.
Daley, R., 1991: Atmospheric Data Analysis. Vol. 2. Cambridge University Press, 457 pp.
da Silva, A., and J. Guo, 1996: Documentation of the Physical Space Statistical Analysis System (PSAS). Part I: The conjugate gradient solver version PSAS-1.00. DAO Office Note 96-02, 66 pp. [Available from Data Assimilation Office, Goddard Space Flight Center, Greenbelt, MD 20771; also available online at http://dao.gsfc.nasa.gov/subpages/officenotes.htmlm.].
Dee, D. P., 1995: On-line estimation of error covariance parameters for atmospheric data assimilation. Mon. Wea. Rev.,123, 1128–1196.
Desroziers, G., B. Pouponneau, J.-N. Thépaut, M. Janisková, and F. Veersé, 1999: Four-dimensional variational analyses of FASTEX situations using special observations. Quart. J. Roy. Meteor. Soc.,125, 3339–3358.
Ehrendorfer, M., and J. J. Tribbia, 1997: Optimal prediction of forecast error covariances through singular vectors. J. Atmos. Sci.,54, 286–311.
Evensen, G., and P. J. van Leeuwen, 1996: Assimilation of Geosat altimeter data for the Agulhas Current using the ensemble Kalman filter with a quasigeotrophic model. Mon. Wea. Rev.,124, 85–96.
Fischer, C., A. Joly, and F. Lalaurette, 1998: Error growth and Kalman filtering within an idealized baroclinic flow. Tellus,50A, 596–615.
Gelaro, R., R. Buizza, T. N. Palmer, and E. Klinker, 1998: Sensitivity analysis of forecast errors and the construction of optimal perturbations using singular vectors. J. Atmos. Sci.,55, 1012–1037.
——, R. Langland, G. D. Rohaly, and T. E. Rosmond, 1999: An assessment of the singular vector approach to targeted observations using the FASTEX data set. Quart. J. Roy. Meteor. Soc.,125, 3299–3328.
Hamill, T. M., and C. Snyder, 2000: A hybrid ensemble Kalman filter–3D-Variational analysis scheme. Mon. Wea. Rev.,128, 2905–2919.
——, ——, and R. E. Morss, 2000: A comparison of probabilistic forecasts from bred, singular-vector, and perturbed observation ensembles. Mon. Wea. Rev.,128, 1835–1851.
Houtekamer, P. L., and H. L. Mitchell, 1998: Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev.,126, 796–811.
——, L. Lefaivre, J. Derome, H. Ritchie, and H. L. Mitchell, 1996:A system simulation approach to ensemble prediction. Mon. Wea. Rev.,124, 1225–1242.
Ide, K., P. Courtier, M. Ghil, and A. C. Lorenc, 1997: Unified notation for data assimilation: Operational, sequential and variational. J. Meteor. Soc. Japan.,75 (1B), 181–189.
Joly, A., and Coauthors, 1997: The Fronts and Atlantic Storm-Track Experiment (FASTEX): Scientific objectives and experimental design. Bull. Amer. Meteor. Soc.,78, 1917–1940.
Langland, R. H., and G. D. Rohaly, 1996: Adjoint-based targeting of observations for FASTEX cyclones. Preprints, Seventh Conf. on Mesoscale Processes, Phoenix, AZ, Amer. Meteor. Soc., 369–371.
——, and Coauthors, 1999: The North Pacific Experiment (NORPEX-98) Targeted observations for improved weather forecasts. Bull. Amer. Meteor. Soc.,80, 1363–1384.
Lorenz, E. N., and K. A. Emanuel, 1998: Optimal sites for supplementary observation sites: Simulation with a small model. J. Atmos. Sci.,55, 399–414.
Molteni, F., R. Buizza, T. N. Palmer, and T. Petroliagis, 1996: The ECMWF ensemble prediction system. Methodology and validation. Quart. J. Roy. Meteor. Soc.,122, 73–120.
Morss, R. E., 1999: Adaptive observations: Idealized sampling strategies for improving numerical weather prediction. Ph.D. thesis, Massachusetts Institute of Technology, 225 p. [Available from Dept. of Earth, Atmospheric and Planetary Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139.].
——, K. A. Emanuel, and C. Snyder, 2001: Idealized adaptive observation strategies for improving numerical weather prediction. J. Atmos. Sci.,58, 210–232.
Palmer, T. N., R. Gelaro, J. Barkmeijer, and R. Buizza, 1998: Singular vectors, metrics, and adaptive observations. J. Atmos. Sci.,55, 633–653.
Parrish, D. F., and J. C. Derber, 1992: The National Meteorological Center’s spectral statistical interpolation analysis scheme. Mon. Wea. Rev.,120, 1747–1763.
Pu, Z., and E. Kalnay, 1999: Targeting observations with the quasi-linear inverse and adjoint and adjoint NCEP global models: Performance during FASTEX. Quart. J. Roy. Meteor. Soc.,125, 3329–3338.
Rabier, F., H. Jarvinen, E. Klinker, J.-F. Mahfouf, and A. Simmons, 2000: The ECMWF operational implementation of four-dimensional variational assimilation. I: Experimental results with simplified physics. Quart. J. Roy. Meteor. Soc.,126, 1143–1170.
Smith, L. A., 2000: Disentangling uncertainty and error: On the predictability of nonlinear systems. Nonlinear Dynamics and Statistics, A. Mees, Ed., Birkhauser, 31–64.
Snyder, C., 1996: Summary of an informal workshop on adaptive observations and FASTEX. Bull. Amer. Meteor. Soc.,77, 953–965.
Szunyogh, I., Z. Toth, K. A. Emanuel, C. H. Bishop, C. Snyder, R. E. Morss, J. Woolen, and T. Marchok, 1999: Ensemble based targeting experiments during FASTEX: The impact of dropsonde data from the Lear jet. Quart. J. Roy. Meteor. Soc.,125, 3189–3218.
——, ——, R. E. Morss, S. J. Majumdar, B. J. Etherton, and C. H. Bishop, 2000: The effect of targeted dropsonde observations during the 1999 Winter Storm Reconnaissance program. Mon. Wea. Rev.,128, 3520–3537.
Todling, R., S. E. Cohn, and N. S. Sivakumaran, 1998: Suboptimal schemes for retrospective data assimilation based on the fixed-lag Kalman smoother. Mon. Wea. Rev.,126, 2274–2286.
Toth, Z., and E. Kalnay, 1993: Ensemble forecasting at NMC: The generation of perturbations. Bull. Amer. Meteor. Soc.,74, 2317–2330.
——, and ——, 1997: Ensemble forecasting at NCEP and the breeding method. Mon. Wea. Rev.,126, 3292–3302.
Extension to Unified Data Assimilation Notation
Ide et al.’s (1997) unified data assimilation notation does not provide a symbolic means for representing the error covariances and estimates associated with future feasible sequences of observational networks. To suitably extend Ide et al.’s nomenclature, in section 2, we borrowed from Cohn et al.’s (1994) and Todling et al.’s (1998) conditional subscript notation. This subscript notation made explicit the fact that every estimate of the state of the atmosphere and its corresponding error covariance matrix is conditional on some temporal sequence of observations. Their notation specified the last observations used in making an estimate of the state of the atmosphere; for example, the subscripts on
Serial Observation Processing
The effect on error covariance of serially assimilating mutually noncovariant subsets of observational data is equivalent to assimilating all of the observations all at once (cf. Bierman 1977; Berliner et al. 1999). Two proofs of this fact, a lengthy one in terms of the rank deficient matrices used in this paper and a shorter one based on Bayes’ theorem can be obtained from the authors of this paper. In the context of targeting, serial observation processing is particularly useful when all of the targeted observations are taken at one time ti+M and/or when there are many reconnaissance aircraft that one must simultaneously deploy.
In cases where all of the targeted observations are taken at just one time ti+M and only the routine component of the observational network is used between ti and ti+M, the analysis error covariance matrix at
Serial processing of observations may also be used to rapidly find a highly skillful deployment of observational resources when there are just too many possible deployments of adaptive observations to assess all of them individually. For example, suppose one had 10 dropwindsonde-equipped planes with which to deploy targeted observations and that each of those planes was capable of 103 distinct deployments of dropwindsondes. The number of feasible observational deployments would then be equal to 103!/[(103 − 10)!10!] = 2.6 × 1023. The individual evaluation of the merits of such a large number of observational networks would be impractical. Serial processing of observations offers a computationally tractable alternative. First, the ET KF is used to determine how to deploy just one of the planes if only one of the planes were available. The analysis error covariance matrix associated with this particular deployment is then used as the prior error covariance matrix to determine how to deploy the second plane given that the first plane was to be deployed under the assumption that it was the only plane being deployed. One then repeats this procedure until flight paths for all 10 planes have been determined. Such a calculation would only require the assessment of 104 individual deployments. Although this serial approach to deciding how to deploy the 10 planes does not necessarily yield the optimal deployment of the 10 planes, it does provide an objective framework for selecting 10 nonredundant observation deployments.
The application of Dee’s online estimation of a single scalar coefficient to the ET KF did not affect the ET KF increment very much.