a. Kalman filtering for data assimilation

The Kalman filter (KF; Kalman 1960; Kalman and Bucy 1961) is a method of estimating the state of a system. There are two components of a KF: a first step where the estimate of the state of the system and an estimate of the uncertainty of that state estimate are adjusted to new observations. The second step is a forecast step where the updated state and the uncertainty estimate are propagated forward in time. The KF estimates the state of a system by interpolating the differences between observations and the first-guess field using two error covariance matrices. These two matrices are a background (or first guess) error covariance matrix and an observation error covariance matrix. These covariance matrices are often written as 𝗣 and 𝗥, respectively. The process of combining a first-guess field and observations into a new estimate of the state of a system is known as data assimilation.

A KF in which the errors of the first-guess field and the errors of the observational data are assumed to be Gaussian is an extended Kalman filter (EKF). The EKF is an extension of the KF for nonlinear dynamics, where the error covariance is propagated forward in time using the tangent linear of the deterministic evolution of the control forecast. The EKF also accounts for nonlinear observation operators. A KF that uses ensembles to estimate the background error covariance matrix is known as an ensemble Kalman filter (EnKF). Evensen (2003) provides a thorough review of recent work in data assimilation using many different forms of ensemble Kalman filters. Some EnKFs add random perturbations to the observed values, and also use a different set of such perturbations when updating the state of each ensemble member. These are known as “stochastic” ensemble Kalman filters, examples of which are given in Houtekamer and Mitchell (1998) and Burgers et al. (1998). An ensemble Kalman Filter that does not stochastically account for errors of the observations is the ensemble square root filter (EnSRF). For a review of EnSRFs, see Tippett et al. (2003). As with all EnKFs, the EnSRF estimates the background error covariance matrix as the product of a matrix of ensemble perturbations with its transpose, and thus the background error covariance used in EnKFs is a sample covariance.

b. Targeting

In addition to data assimilation, ensembles have been used for selecting the locations for adaptive, or targeted, observations. Ensemble techniques have been used to estimate forecast uncertainty (e.g., Lorenz and Emanuel 1998; Morss et al. 2001; Hansen and Smith 2000). The ensemble transform technique (Bishop and Toth 1999) and the subsequent ensemble transform Kalman filter (ETKF; Bishop et al. 2001) use information from ensembles to identify regions that, when sampled, would lead to forecast improvements. The ETKF produces an error covariance matrix associated with a first-guess field (an estimate of the uncertainty of the first-guess field) and the change to this error covariance matrix that would result from observations at a given location. The ETKF then propagates this information forward in time. Hamill and Snyder (2002) took a similar approach, but their work focused on reducing analysis uncertainty whereas Bishop et al. (2001) focused on reducing future forecast uncertainty.

c. Rank deficiency

Ensemble-based error statistics are nearly always rank deficient. The number of members in an ensemble is usually several orders of magnitude lower than the number of degrees of freedom in the forecast model. As a result of this rank deficiency, the matrix does not contain smaller pieces of independent information than the order of the matrix. Long-distance error correlations are more likely to be spurious than to be actual. The localization of ensemble-based error statistics is a means of mitigating these spurious correlations. One approach to covariance localization is to limit the number of observations that can have an impact on a particular state variable at a particular location. Houtekamer and Mitchell (1998) and Evensen (2003), for example, use a cutoff radius so that observations are not assimilated beyond a certain distance for a grid point. Another approach is to localize the ensemble-based background error statistics using a localization function. Taking an element by element multiplication of a localization function with the ensemble-based background error covariance matrix (a Schur product or Hadamard product) produces a localized covariance matrix. This approach, used in Houtekamer (2001) and Hamill et al. (2001), is what we shall use.

d. Model error

An ensemble-based set of error statistics for the first-guess field is inherently on the subspace of the model attractor. As a result of the model not representing the system exactly, the ensemble-based covariance matrix may not represent the actual first-guess error statistics. For this reason, a “model error” term should be included in the background error covariance matrix (Dee 1995). In practice, this model error covariance matrix is difficult to estimate.

A commonplace approach for dealing with model error is to simply inflate the covariances from the ensemble such that the amplitude of the ensemble-based background error covariance matrix is as large as the sum of the error covariances on and off the ensemble subspace. While this approach ensures that the relative sizes of the background error covariance matrix and observation error covariance matrix are correct, this approach does nothing to represent background errors off of the model subspace. Consider the eigenvector/eigenvalue decomposition of the background error covariance matrix. Covariance inflation adjusts the trace of the background error covariance matrix (the sum of all the eigenvalues of that matrix) to ensure that the value is representative of the magnitudes of the errors, but covariance inflation does nothing to change the eigenvectors—the directions of error spanned by the background error covariance

An alternative approach to covariance localization is to add together the ensemble-based background error covariance matrix with a model error covariance matrix. Houtekamer et al. (2005) and Hamill and Snyder (2000) took this approach to incorporating error structures that were not on the ensemble subspace into the background error covariance matrix. In both cases, the authors used a three-dimensional variational data assimilation (3DVAR) like, isotropic, covariance matrix. This covariance matrix may or may not accurately represent model error, but the model error statistics are very difficult to know, and using a quasi-isotropic covariance matrix to represent the model error results in more accurate analyses and forecasts than if this term was neglected.

e. Preemptive forecasting

In data assimilation, an EnKF uses ensemble-generated error statistics to produce an increment to the first-guess field. In targeting, the ETKF (Bishop et al. 2001) uses ensemble-generated error statistics to estimate the likely impact of observations on the error statistics of future forecasts. Combining these two methodologies, the change to a future forecast resulting from current observations can be calculated using ensemble-based error statistics. In the same manor that adding in increment to the current forecast generates a new analysis, adding a propagated increment to a future forecast results in an updated, or preemptive, forecast. This technique is not dissimilar from the Kalman smoother (Evensen and van Leeuwen 2000) or a four-dimensional ensemble Kalman filter (Hunt et al. 2004), which also uses covariances at more than one time. Just like covariances in space allow observations to update model states that are far from the observation location in space, covariances in time allow observations to update model states that are far from the observation location in time.

The most compelling reason for producing a preemptive forecast is speed. Given that the ensemble of forecasts already exists, producing a preemptive forecast takes no longer to produce than it takes to run a data assimilation scheme. This is less time than it would take to run the data assimilation scheme and then integrate the forecast model out to the desired forecast verification time. Assuming that preemptive forecasts are of similar skill to the updated model forecasts, preemptive forecasts can provide additional time for weather-related decision making.

In section two of this paper, the experimental design and theory of the preemptive forecast is presented. Results from the observing system simulation experiment are given in section 3, and section 4 states conclusions from the work.

a. Experimental framework

The same simple model framework used in Bishop et al. (2001), Etherton and Bishop (2004), and Bishop et al. (2006) was used to construct an observing system simulation experiment (OSSE). In this OSSE, a doubly periodic barotropic vorticity model was used to depict the state of a two-dimensional turbulent flow. The turbulent flow, also known as the nature run, was a 300-day integration of this barotropic model. An example of this turbulent flow, the vorticity field on day 150 of our 300-day nature run, is shown in Fig. 1. The governing equation for the model is

where Ψ is the streamfunction, δ and n are parameters associated with numerical diffusion, κ is a relaxation parameter, and β is the planetary vorticity gradient. The model covers a 32 × 32 gridpoint domain, for 1024 total grid points. A relaxation scheme relaxes the vorticity values at grid points by a quantity proportional to the difference between the zonally averaged vorticity, ∇²Ψ, and a reference zonal average of vorticity, ∇²Ψ_r. In this manner, local vorticity extrema are not preferentially eroded, and meridional gradients of zonally averaged vorticity are maintained.

A fictitious forecast agency is charged with predicting the state of this turbulent flow every 24 h, and uses the same barotropic vorticity model as was used to make the nature run to make its predictions. However, the vorticity relaxation used by the agency model is not the same as what was used for the nature run. Instead of relaxing back to the time-invariant barotropically unstable state that the nature run relaxes toward, the agency model relaxes toward a zonal state whose meridional vorticity distribution is given by the left-most grid points of the model at the beginning of each forecast. Consequently, while the state relaxed to by the truth run is always the same, the state relaxed to by the agency forecast model varies randomly from forecast to forecast. Our experiment therefore was not a perfect-model experiment by virtue of this difference.

On each day of the 300-day simulation, observations of vorticity were taken and used to make a new estimate of the state. Vorticity observations were simulated by adding random values (representing observational errors) from a Gaussian distribution to values of the nature run. Shown in Fig. 1, there are 72 observations in the domain, and the root-mean-squared observational error was taken to be 1.4 × 10⁻⁷ s⁻¹, which is approximately 1/100 of the average 24-h forecast error (2.3 × 10⁻⁵ s⁻¹). Observations were assimilated using either a hybrid EnKF or a pure EnKF. To generate the ensemble for each of the 300 days, 64 initial perturbations were generated using the ETKF spherical simplex ensemble generation scheme described in Wang et al. (2004), and integrating those initial conditions for 48 h using the agency forecast model.

While many different sets of error statistics were used for data assimilation (as will be outlined in the following subsections), it is the analysis made using the hybrid EnKF, using an isotropic 𝗕 matrix where no model error term was included, that serves as the official agency analysis. From this official analysis is produced the new agency forecast, and perturbations generated by the ETKF are added to this official analysis to produce the initial conditions for the 64 ensemble members. Thus, for each of the 300 days, there is only one 64-member forecast ensemble.

1) Hybrid ensemble Kalman filter

As given in Cohn (1997), the following equation,

is used to combine observations of vorticity, y, with the first-guess field (in our experiment, a control run) x^f, to produce the new analysis, x^a. The fields y, x ^f, and x^a are all valid at time t_o, the observation time. The observation operator 𝗛 is a simple mapping in our OSSE as observations are taken at model grid points (the 𝗛 matrix elements are all either 0 or 1). The matrix 𝗥 denotes the corresponding observation error covariance matrix. Observation errors are assumed to be uncorrelated, and thus, the 𝗥 matrix was diagonal, with values along the diagonal equal to 1.4 × 10⁻⁷ s⁻¹ squared, approximately 2.0 × 10⁻¹⁴ s⁻².

For the cycling forecast model, the hybrid analysis scheme was used to assimilate the 72 observations. Analyses produced from the hybrid scheme are the “official” analysis from the forecast agency, and model integrations of this analysis serve as the official deterministic forecast from the agency. The hybrid scheme (Hamill and Snyder 2000) approximates the forecast error covariance matrix 𝗣^f, from Eq. (2), with a mix of parameterized covariances, 𝗕^f, and flow-dependent, ensemble-based covariances, 𝗙^f. The forecast error covariance matrix 𝗣^f from Hamill and Snyder (2000) is given by

where α is a constant, chosen to be 0.5, the value that produced the smallest forecast errors for our system (as used in Etherton and Bishop 2004). Here, 𝗕^f is a 3DVAR-like time-invariant forecast error covariance matrix based on an isotropic correlation function. As in Bishop et al. (2001), Etherton and Bishop (2004), and Bishop et al. (2006), the covariance c_ij between vorticity errors at the ith and jth grid points is given by

where D defines the correlation length scale, r_ij gives the separation distance between the ith and jth grid points, and the constant A gives the forecast error variance. Note that this function precisely describes the structure of the analysis increment that would be obtained from an observation of vorticity at the ith grid point using the isotropic error covariance model. This particular vorticity structure is proportional to the Laplacian of exp[ln(0.1)(r_ij/D)² and hence is consistent with an exponential correlation model for the streamfunction. The factor ln(0.1) simply ensures that the streamfunction error correlation between grid points separated by distance D is 0.1. The parameters A and D were independently tuned to minimize the 24-h forecast error variance over a 50-day period.

The flow-dependent, ensemble-based covariance matrix, 𝗙^f, is

where each column of the matrix 𝗫 is proportional to the difference between an ensemble member forecast and the control forecast (not the ensemble mean) at the time at which the observations are to be assimilated. As the perturbations are not taken with respect to the ensemble mean, there is a bias in the estimate of 𝗙^f(t_o, t_o). However, in a nonlinear system, any ensemble-based Kalman update will introduce a bias, and so the bias we introduce by taking perturbations about the control run as opposed to the ensemble mean is suboptimal but not fatal to our approach. The parameter λ is a rescaling factor, calculated from the previous 5 days’ assimilation cycles such that the trace of 𝗙^f at the observing sites added to the trace of 𝗥 equals the squared innovation vector, [y − 𝗛x^f(t_o)]. The 𝗕^f matrix is also rescaled such that the trace of the rescaled 𝗕^f and the trace of 𝗙^f are the same.

2) Pure ensemble Kalman filter

Data assimilation can be done using only ensemble-based error statistics, the 𝗙^f matrix alone [setting alpha to 1 in Eq. (3)]. In addition, when using the ensemble Kalman filter for preemptive forecasting, we do not incorporate the 𝗕^f matrix into the background error covariance matrix. For both the purposes of data assimilation and preemptive forecasting, for which the ensemble-based covariance matrix must now stand on its own, we further improve it.

In addition to rescaling the magnitudes of the covariances, it is critical to apply a Schur product of the ensemble-based background error covariance matrix, 𝗙^f, with a localization function. The function given in Eq. 4.10 of Gaspari and Cohn (1999) serves this purpose. The rescaling function is represented as ρ(t_o, t_o), where t_o is the observation time. Figure 2a shows the localization function for an observation taken at grid point (11, 15).

Recall that while the state relaxed to in the nature run is always the same, the state relaxed to by the agency forecast model varies for each forecast. The state that the agency model relaxes to generally has weaker vorticity gradients than the state relaxed to by the nature run. It is thus necessary to add in a term representing the model error to the background error covariance matrix. This term is symbolized by 𝗤^f(t_o, t_o) (the specifics of how we calculate 𝗤^f can be found in section 2b). The equation for the rescaled and localized ensemble-based background error covariance matrix including a model error term is

The parameter γ is a rescaling factor, such that the sum of the traces of (γ)ρ(t_o, t_o) · 𝗛𝗫(t_o)𝗫(t_o)^T𝗛^T, 𝗛𝗤^f(t_o, t_o)𝗛^T, and R equals the squared innovation vector, [y − 𝗫x^f(t_o)]. In this approach, the magnitudes of the 𝗤^f and 𝗥 matrices are fixed and constant every day, and it is the ensemble-based background error covariance matrix that is rescaled. Testing was done to determine the optimal length scale for the correlation function ρ(t_o, t_o). Lengths between 500 and 2000 km were used in ρ(t_o, t_o), and 300 days of forecasts were made from EnKF-produced analyses. The average vorticity error of those 300 forecasts was calculated, and the length scale of 1000 km produced the smallest forecast errors.

Note that regardless of whether or not 𝗤^f or 𝗕^f is used in the Kalman filter when making an analysis, the ensemble-based prediction error covariance matrix, 𝗙^f(t₁, t₂) = λ{[1/K]𝗫(t₁)𝗫(t₂)^T}, is always the same. This results in an inconsistency, as each method of data assimilation should result in the generation of its own unique ensemble, but in our experiments all KFs use the same set of ensemble members, with only one configuration cycling the ensemble.

b. Preemptive forecast theory: The ensemble Kalman filter

A preemptive forecast is made if the signal, x^a(t_o) − x^f(t_o) from Eq. (2), is propagated to some future time, t, and added to the control forecast at that time, x^f(t). This is accomplished by operating on both sides of (2) with the true autonomous nonlinear operator M(t, t_o). Operating on Eq. (2) using M(t, t_o) yields

Substituting the ensemble-based background error covariance matrix from (6) into (7) yields

This is the equation to produce the analysis increment using a background error covariance matrix consisting of an ensemble-based covariance matrix localized using the Schur product and a model error covariance matrix.

This increment can be propagated forward in time using the ensemble perturbations valid at any future verification time, t_υ. Propagation of 𝗣^f(t_o, t_o) by the true autonomous nonlinear operator M(t_υ, t_o) results in

To propagate 𝗫(t_o) forward in time, we simply integrate the forecast model from t_a to t_υ for each of the ensemble members and the control run. To propagate the localization function forward in time, we expand the radius of the localization function (by trying several values and selecting the radius that produces the most accurate forecasts) to account for the movements of error structures in the flow. In the propagation of 𝗣^f(t_o, t_o) by M(t_υ, t_o), note that both ρ(t_o, t_o) and 𝗫(t_o) 𝗫(t_o)^T are propagated forward in time separately and then combined using a Schur product. If one thinks of the propagator M as a matrix 𝗠, then it looks like 𝗠[ρ(t_o, t_o) · 𝗫(t_o)𝗫(t_o)^T] = [𝗠ρ(t_o, t_o)] · [𝗠X(t_o)X(t_o)^T]. However, the true propagator is nonlinear, and operates on ρ(t_o, t_o) and 𝗫(t_o). If one thinks of the propagator as an operator, it is more like (𝗕 · 𝗖)^M = 𝗕^M · 𝗖^M, with M operating on both matrices independently.

Note that the localization function, ρ(t_o, t_o), is not the same as ρ(t_υ, t_o). It is expected that for longer lead times, the distance for which ensemble-based error correlations are valid will increase, as the structures of the error are moving in the flow. It is also possible that the structure of the localization function would be different when the error statistics correlate two different times. However, for this simple experiment, we chose to use the same function for ρ(t_υ, t_o) as for ρ(t_o, t_o), but we have increased the length scale to 1400 km. The value of 1400 km was chosen by testing a number of different length scales, and choosing the one that gave the smallest forecast errors. The 1400-km localization for an observation taken at grid point (11, 15) is shown in Fig. 2b.

The other covariance matrix in Eq. (9) is the model error covariance matrix, 𝗤^f. In general, model error statistics are not known, but are assumed to be quasi-isotropic. In this simple model experiment, because we know exactly the model error, we were able to calculate a climatological model error covariance matrix. For each day of the nature run, we were able to run the barotropic model twice, once using the same relaxation as the nature run and once using the agency forecast model. Taking the differences between these forecasts yields the model error for each set of ensemble initial conditions. Each difference represents one realization of the model error. By placing each realization in the column of a matrix, and multiplying that matrix of realizations by its transpose, and dividing by the number of cases, an “unobtainable” model error covariance matrix is calculated. We do not localize this error covariance matrix. The barotropic turbulent flow is completely described with the 1024-element state vector. The unobtainable model error covariance matrix comes from a set of 300 samples and, thus, captures the majority of the directions of the error. This is in contrast to the ensemble-based background error covariance matrix, formed using 64 ensemble members, which is more likely to need localization.

Figure 3 shows the correlation of the errors at all model grid points with errors at grid point (11, 15). Figure 3a shows the correlations using the isotropic covariance matrix, while Fig. 3b shows the model error covariance matrix from the unobtainable method. It has been assumed that the model error statistics are isotropic in nature, but at least for this simple barotropic model, where the source of the model error is the vorticity relaxation scheme, this is not the case. Model errors at grid point (11, 15) are positively correlated with areas immediately around this site as well as locations in the same latitude band. Model errors at grid point (11, 15) are negatively correlated with latitude bands farther to the north and south. Given the initial vorticity field used to initialize the model with two bands of positive vorticity positioned in the latitudes typically associated with the ITCZ and the midlatitude storm track, and areas of negative vorticity elsewhere, this structure of model errors agrees well with the dynamics of the system.

The propagated background error covariance matrix, 𝗣^f(t_υ, t_o), when incorporated into Eq. (8), and also propagating the left-hand side of (8), then yields

This is the equation used to produce the preemptive forecast, x(t_υ).

c. Experimental design

Five 24-h model forecasts were made for each day’s analysis. These five 24-h forecasts were compared to the forecast made without assimilating any observations—a 48-h forecast valid at the same time. These six model integrations are compared with three preemptive forecasts.

Figure 4 shows a timeline of the forecast process. At the initial time, t_i, 64 initial conditions are generated [using the ETKF spherical simplex scheme from Wang et al. (2004)] and integrated by the forecast model for 48 h to the verification time, t_υ. At the observation time, t_o, 24 h, 72 observations are taken. These observations are used to produce new analyses and also as input for preemptive forecasts. New analyses are made using the five different methods: 1) a hybrid ensemble Kalman filter with no covariance localization; 2) a hybrid ensemble Kalman filter with covariance localization; 3) a pure ensemble Kalman filter, with no covariance localization nor an additive model error term; 4) a pure ensemble Kalman filter with covariance localization, but no additive model error term; and 5) a pure ensemble Kalman filter with covariance localization and an explicit model error term, mentioned in the paragraph above. These five new analyses are integrated an additional 24 h to the verification time. Preemptive forecasts are made using three different covariance matrices, propagating the impact of observations at t_o to t_υ. These three covariance matrices were 1) a pure ensemble Kalman filter, with no covariance localization nor an additive model error term; 2) a pure ensemble Kalman filter with covariance localization, but no additive model error term; and 3) a pure ensemble Kalman filter with covariance localization and an explicit model error term.

Thus, for each of the 300 days of the simulation, there are nine different forecasts valid at the verification time. One is a 48-h forecast. There are five 24-h forecasts, all initialized from different analyses. There are also three preemptive forecasts, which use the data from the observations at t_o, but do not require any model integrations, using the existing 48-h integrations to produce a forecast. While there are many different sets of error statistics used for data assimilation, it is only the analysis generated using the hybrid EnKF, and the ensemble members generated by adding ETKF generated perturbations to this analysis, that are used for the next forecast cycle. All other analyses and forecasts from those analyses are used for comparative purposes only.