a. Joint state–observation space and Bayesian framework

One implication of (2) is that subsets of observations with independent (observational error) distributions can be assimilated sequentially. Let y^o be composed of s subsets of observations, y^o = { y^o₁, y^o₂, … , y^o_s}, where the distribution of the observational errors for observations in subset i is independent of the distribution for the observations in subset j, for i ≠ j. Then

View Expanded

In particular, if the individual scalar observations in y^o have mutually independent error distributions, they can be assimilated sequentially in any order without changing the result in (2). This allows sequential assimilation with observational error distributions represented as Gaussians as long as the observational error covariance matrix is diagonal. This was pointed out by Houtekamer and Mitchell (2001) in the ensemble Kalman filter context and used in A01. Equation (3) depends only on the observing system and makes no assumptions about the prior joint state distribution or how it is represented.

b. An ensemble method for the filtering problem

In ensemble methods for solving (2), information about the prior distribution of the state variables, x^p, is available as a sample from N applications of a prediction model. An ensemble sample of the prior observation vector, y^p, can be created by applying the forward observation operator, h, to each ensemble sample of x^p.

Some Monte Carlo (ensemble) methods also have a weight, w, associated with each ensemble member. The possibility of weighted ensembles is not discussed in detail here, but the methods in this section are easily generalized to this case. Attempts to apply the most common types of weighting/resampling Monte Carlo algorithms in high-dimensional spaces have faced significant difficulties.

Observational error distributions of climate system observations are generally only poorly known and are often specified as Gaussian with zero mean (known instrument bias is usually corrected by removing the bias from the observation during a preprocessing step). Some observations have values restricted to certain ranges; for instance, precipitation must be positive. Redefining the observation variable as the log of the observation can lead to a Gaussian observational error distribution in this case (Tarantola 1987). Given Gaussian observational error distributions, observations can be decomposed into subsets where observational errors for observations in each subset are correlated but observational errors in different subsets are uncorrelated. In other words, 𝗥, the observational error covariance matrix, is block diagonal with each block being the size of the number of observations in the corresponding observation subset. Error distributions for the different subsets are independent, so the subsets can be assimilated sequentially in (2) in an arbitrary order.

For many commonly assimilated observations, each scalar observation has an error distribution that is independent of all others, allowing each scalar observation to be assimilated sequentially (Houtekamer and Mitchell 2001). If the observational covariance matrix is not strictly diagonal, a singular value decomposition (SVD; equivalent to an eigenvalue decomposition for a symmetric positive-definite matrix like 𝗥) can be performed on 𝗥. The prior joint state ensembles can be projected onto the singular vectors and the assimilation can proceed using this new basis, in which 𝗥′, the observational covariance matrix, is diagonal by definition. Upon completion of the assimilation computation, the updated state vectors can be projected back to the original state space. Given the application of this SVD, a mechanism for sequential assimilation of scalar observations implies no loss of generality for observations with arbitrary Gaussian error distributions. In everything that follows, results are presented only for assimilation of a single scalar observation so that m = 1, with joint state space size k = n + m = n + 1. Allowing arbitrary observational error distributions represented as a sum of Gaussians is a straightforward extension to the methods described below.

c. Two-step data assimilation procedure

Following A01, define the joint state space forward observation operator for a single observation as the order 1 × k linear operator 𝗛 = [0, 0, … , 0, 1]. The expected value of the observation can be calculated by applying 𝗛 to the joint state vector, z, which is equivalent to applying the possibly nonlinear operator h to x. The conversion of the possibly nonlinear h to the linear 𝗛 is a primary motivation for applying ensemble filters in the joint state space.

This suggests a partitioning of the assimilation of an observation into two parts. The first determines updated ensemble members for the observation variable y given the observation, y^o. To update the ensemble sample of y^p, an increment, Δy_i, is computed for each ensemble member, y^u_i = y^p_i + Δy_i, i = 1, … , N, where N is the ensemble size.

Given increments for the observation variable, the second step computes corresponding increments for each ensemble sample of each state variable, Δx_i,j (i indexes the ensemble member and j = 1, … , k indexes which joint state variable throughout this report). This requires assumptions about the prior relationship between the joint state variables. Although reasonable alternatives exist (Tarantola 1987), the assumption used here is that the prior distribution is Gaussian (or a sum of Gaussians that allows generality). This is equivalent to assuming that a least squares fit (local least squares fit) to the prior ensemble members summarizes the relationship between the joint state variables.

Figure 1 depicts the simplest example in which there is only a single state variable, x. The observation variable, y, is related to x by the operator h, which is nonlinear in the figure. Increments for each ensemble sample of y have been computed. The corresponding increments for x are then computed by a global least squares fit (linear regression) so that

View Expanded

The change in the ith ensemble sample of the state variable due to observation variable y is equal to the prior covariance of x with y, σ_x,y divided by the prior variance of y, σ_y,y times the change in the ith ensemble sample of the observation variable. This is just a statistical linearization and inversion of the observation operator h. This linearization can be done globally by computing the global sample covariance and using this for the regression for each ensemble member (Fig. 1).

The linearization can also be done locally (Fig. 2) by computing local estimates of covariance for each ensemble member. This can be done, for instance, by only using a set of nearest neighbors (in y, in x, or in some combined distance metric) to compute sample covariance. Figure 2 shows an idealized form of nearest neighbor linearization in which only a single closest ensemble member is used to compute the statistical linearization. Related methods for doing local Gaussian kernel approximations of this type can be found in Silverman (1986) and Bengtsson and Nychka (2001). When x is functionally related to y as in Fig. 2, local linearization methods like this can give significantly enhanced performance when h is strongly nonlinear over the prior ensemble range of y.

If h is nonlinear as in the figures, the statistical linearization is only valid locally. To minimize errors due to the linearization, whether global or local linearizations are applied, it is desirable that the observation variable increments, Δy_i, should be as small as possible. This is discussed further in section 4c.

This two-step method can be extended trivially to problems with arbitrary numbers of state variables. When a global linearization is applied using a least squares fit, a single Gaussian is assumed to approximate the prior relation of the variables. The increments, Δx_i,j, for each ensemble sample of each state variable in terms of Δy_i can be computed independently by regression:

View Expanded

Again, local linearizations could be performed using Gaussian (or extended Gaussian) kernel methods (Tarantola 1987) in which only some subset of local information is used to compute the covariance from the ensemble sample. In (6), all relevant information about the prior covariance of the model state variables, x, needed to compute increments is contained in the correlation of the individual scalar state variables with the observation variable y.

When the state variable being updated and the observation variable are not functionally related, the use of local linearizations can be more problematic. Figure 3 shows an example where state variable x₁ is being updated by an observation, y^o. The expected value of the observation is y = h (x₂), where x₂ is a second state variable, here moderately correlated with x₁. In this case, the linear regression for x₁ performs a statistical linearization in the presence of noise. Using large (global) regressions is useful to filter out this noise. On the other hand, using local linearizations can help to resolve more of the structure of h. Applying local regressions that are based on too few ensemble members can lead to disastrous overfitting behavior as demonstrated by the application of an idealized single nearest-neighbor linearization in Fig. 3. Appropriate trade-offs in choosing local versus global linearizations are an important part of tuning ensemble filters for improved performance.

a. Perturbed observation ensemble Kalman filter

In its traditional implementation, the perturbed observation ensemble Kalman filter (Houtekamer and Mitchell 1998) uses a random number generator to sample the observational error distribution (specified as part of the observing system) and adds these samples to the observation, y^o, to form an ensemble sample of the observation distribution, y^o_i, i = 1, … , N. In most implementations (Houtekamer and Mitchell 1998), the mean of the perturbations is adjusted to be 0 so that the perturbed observations have mean y^o; other clever methods for perturbing the observations can preserve other aspects of the distribution (Pham 2001) (the discussion below applies whether adjustment to the means or other types of perturbation algorithms are used or not). Here, Σ^p is computed using sample statistics from the prior joint state ensemble and (7) is computed once to find the value of Σ^u. Equation (8) is then applied N times with z^p replaced with z^p_i and y^o replaced by y^o_i in the ith application to compute N ensemble members for z^u. This method is described using more traditional Kalman filter terminology in Houtekamer and Mitchell (1998). As shown in Burgers et al. (1998), computing a random sample of the product as the product of random samples is a valid Monte Carlo approximation to the nonlinear filtering equation (2). Note that all ensemble members are assumed equally weighted in both the EnKF and EAKF so that (9) is not used; however, (9) may be relevant for other ensemble filtering methods (see section 4).

An equivalent two-step procedure for the EnKF begins by computing the update increments for the observation variable, y, a scalar problem independent of the other joint state variables (4). Perturbed observations are generated and (7) is used to compute an updated variance for the observation variable; all matrices here are order 1 × 1. Equation (8) is evaluated N times to compute y^u_i, with z^p and y^o replaced by y^p_i and y^o_i, where the subscript refers to the value of the ith ensemble member.

Equation (14) applies in this case, since the updated covariance in the full dimension EnKF is computed by (7), and can be used to compute the increments for all other state variables given the value of Δy_i = y^u_i − y^p_i (the kth component of the k-vector Δz_i). All components of Δz_i can be computed from (14) as

View Expanded

where the first subscript on Δz indexes the ensemble member and the second indexes the state variable. This is the regression formula (6) presented in section 2c derived from the assumption of a Gaussian relation between the prior state variables. Implementing the EnKF in this two-step fashion gives results identical (to computational roundoff) to previous implementations of the EnKF.

b. Ensemble adjustment Kalman filter–square root filter

Bishop et al. (2001), Whitaker and Hamill (2002), and Pham (2001) have all described other ensemble filtering methods similar to the EAKF in A01. Tippett et al. (2003), providing an analysis of the work of these different authors, point out that the methods are roughly equivalent and suggest that the name deterministic square root filter (Andrews 1968) may be more appropriate.

It can also be shown that the regression formula can be applied for the adjustment of the ensemble members around the mean in the EAKF. Equation (10) can be rewritten by taking the square root of the operator as

Σ^uΣ^pT(20)

where

View Expanded

If the deviation of the observation variable around the mean is updated as in (19), applying the linear regression (16) for state variable j gives

z_i,jσ_j,kσ_k,k⁻¹y_iσ_j,kσ_k,k⁻¹αy^p_i(23)

Then,

z^u_i,jz^p_i,jσ^p_j,ky^p_i(24)

where

View Expanded

Writing this in vector form for the updates of all joint state variables gives

z^u_iz^p_iz^p(26)

where

Σ^p_0k(27)

The matrix 𝗔 used in the regression update is identical to 𝗕 in (21) demonstrating that using regression in the two-step framework is identical to the implementation of the EAKF described in A01.

In summary, the EnKF and EAKF assume a Gaussian relation between the variables in the joint state space prior distribution. However, these methods do not use (7) and (8) directly to compute an updated distribution. Both methods can be recast in terms of the two-step assimilation context developed in section 2c. First, update increments are computed for each ensemble sample of the observation variable using scalar versions of the traditional algorithms. Once increments for ensemble samples of the observation variable have been computed, (16) is used to solve for the increments, Δz_j,i, for each state variable in turn in terms of Δy_i by linear regression. The appendix discusses this method in the case where the prior covariance matrix is degenerate.

There are a number of implications about the computational complexity of ensemble (Kalman) filtering that can be drawn from (16). First, there is no need to compute the prior covariance among the model state variables (only the prior cross covariance of each state variable with the observation variable is needed, along with the variance of the observation variable) or the complete updated covariance, Σ^u (only the updated variance of the observation variable is needed). Second, once the observational variables are updated, the increments for the state variables depend only on ratios of prior (co)variances. Any multiplication of the prior covariance matrix by some type of covariance inflation factor, as is done in many existing ensemble Kalman filter implementations (Anderson and Anderson 1999; Whitaker and Hamill 2002), does not impact the solution to (16). The impacts of covariance inflation are still felt in the first step in which the increments of the observational variable are computed. Note that many types of covariance inflation directly increase the spread of the joint prior distribution.

There are important caveats to this discussion of enhanced computational efficiency. First, as noted in section 2b, correlations between observational errors require a rotation of the problem to apply the sequential methods discussed here. In the limit where most of the observational errors are correlated, the cost of doing this rotation could end up offsetting the savings from avoiding matrix multiplies. At present, most operational centers do not assume that there are correlated observational errors, but this could become important at a later date. If only certain sets of observations have correlated errors, a block diagonal covariance structure results and rotation can still be done more efficiently than computing the full matrix products. Second, the sequential method outlined here may present challenges for implementation on highly parallel computer architectures; relative performance on such hardware will require further analysis.

a. A kernel filter

If the prior distribution of the observation variable may have significant non-Gaussian structure, a kernel method similar to the one employed in Anderson and Anderson (1999) may be useful for computing the update increments. One simple example of kernel methods is the Fukunaga method (Silverman 1986). In this algorithm, the prior distribution is represented as a sum of Gaussians with identical variance but different means. The means are the individual prior ensemble samples and the variance is the prior sample variance multiplied by a scaling factor, η. The prior distribution is then

View Expanded

where N(a, b) is a Gaussian with mean a and variance b.

The product of a prior expressed as a sum of Gaussians and a Gaussian observational distribution is equal to the sum of the products of the individual prior Gaussians and the observational Gaussian. The variance of all Gaussians summed in the product is identical in this case and can be computed by a single scalar application of (7). The means will all be different and can be computed by N scalar applications of (12). In the most naïve application of this method, an updated ensemble can be generated from this continuous representation by randomly sampling the sum of Gaussians as in Anderson and Anderson (1999).

This kernel method can be extended in a number of ways by allowing more general kernels. For instance, kernels with different means and different variances can be used following a variety of techniques like the class of nearest-neighbor methods (Silverman 1986; Bengtsson and Nychka 2001). In addition, kernels from the class of “generalized” Gaussians as described in Tarantola (1987) can lead to related kernel algorithms.

b. Quadrature product methods

Update methods that are based directly on “quadrature” solutions to (4) can also be used to find increments for observation variables. One implementation of such a method could begin by computing a continuous approximation to the prior distribution from the ensemble sample; again, kernel methods are an example. Quadrature methods can then be used to divide the real line into a set of intervals over which the product in the numerator of (4) is computed to approximate the updated distribution. An appropriate method can then be used to sample this updated distribution to generate new ensemble methods.

c. General requirements for an observation variable update

Several characteristics may be important for algorithms used to update the observation variables. First, low quality observations should have small impacts on the ensemble. For atmospheric and oceanic models, the prior distributions may be sampling model “attractors” that have a great deal of structure. Allowing low impact observations to change the ensembles has the potential to destroy valuable information. Pure resampling algorithms would be an example of an undesirable method. In this case, the prior ensemble would be converted to a continuous representation that would then be only subtly modified by a low information observation. This updated continuous distribution would then be resampled to generate an ensemble, leading to possibly large increments to ensemble members. The ensemble kernel filter as described above suffers from this deficiency and generally produces assimilations with larger ensemble mean error than do the EnKF and EAKF despite the fact that in many instances it produces more accurate samples of the updated observation variable distribution when the prior is significantly non-Gaussian. Modifications to the kernel filter that limit the impact of low information observations are required to make this method more generally useful.

For related reasons, it is desirable to limit the size of the increments for observation variables as noted in section 2. Since the regression used to update the state variables is a statistical linearization, it is likely to be an increasingly poor approximation as the increments increase. For instance, the updated mean and covariance of the observation variable for the EnKF would be the same if the pairing between the updated, z^u_k, and prior, z^p_k, observational variable ensemble members was changed before the computation of update increments, Δz_k.

In some applications, the performance of the EnKF can be dramatically improved by pairing the updated observational variable ensemble members with the prior members so as to reduce the value of the update increments. The most obvious way to do this is to sort the prior and updated observational variable ensemble members and to associate the nth sorted updated ensemble member with the nth sorted prior ensemble member. Doing this reduces much of the difference between the EnKF and EAKF reported in A01. Doing ordered pairing also significantly improves the performance of kernel filter algorithms (section 4a) by reducing the size of the increments.

As an idealized example of this sorting procedure, suppose that for a three-member ensemble, the prior observation variable sample z^p_k = {1, 5, 7} and the updated sample z^u_k = {4.9, 6.6, 1.2}. Unordered pairings like this can result when perturbed observations are used in the EnKF. Pairing these in this order gives increments Δz_k = {3.9, 1.6, −5.8}. If both sets are sorted, the statistics of each remains unchanged, but the increments are Δz_k = {0.2, −0.1, −0.4}. The errors resulting from the linear regression in the second step of the assimilation are clearly expected to be significantly less in the sorted case.

While many existing ensemble filters can be recast in the framework presented here, there are some that apparently cannot. For instance, the particle filter of Pham (2001) requires information about the joint distribution of observation and state variables when doing resampling. Methods like the singular second-order-exact EnKF of Pham (2001), however, can be expressed in the two-step framework.