a. Nonadaptive statistics

Let us consider the problem of estimating the evolution of a system described by the state vector x(t), between times t₀ and t_N+1, given a set of observation vectors y_k at times t_k, k = 1, … , N(t_k < t_k+1):

View Expanded

where x_k = x(t_k), 𝗛_k, and ϵ_k are the state vector, the observation operator, and the observational error at time t_k, respectively. We also assume that we have information on the initial condition x₀ = x(t₀) and optionally on dynamical laws governing the time evolution of x(t). In many situations, it is useful to solve the filtering problem, in which the estimation at time t is computed using only past information. This means that the information only needs to be propagated forward in time from t₀ to t_N+1, with discrete updates each time that an observation vector is available. In a probabilistic framework, this amounts to computing sequentially the following probability distributions:

View Expanded

where p₀(x₀) and p_N+1(x_N+1) are the initial and final probability distributions, and where p_k^f(x_k) and p_k^a(x_k) are the probability distributions at time t_k before and after that the observation vector y_k is taken into account (superscripts “f” and “a” stand for “forecast” and “analysis”). Since we solve a filtering problem, it is implicit that every probability distribution is conditioned on the past observations [i.e., y_k′ with k′ < k for p_k^f(x_k), and y_k′ with k′ ≤ k for p_k^a(x_k)].

In this study, it is assumed that every probability distribution of the sequence (2) is Gaussian:

View Expanded

where x_k^f and x_k^a are the expected forecast and analysis state vectors at time t_k, respectively, and where 𝗣_k^f and 𝗣_k^a are the corresponding forecast and analysis error covariance matrices, respectively. They are computed by repeating the following two steps in sequence from k = 1 to k = N. The forecast step computes x_k^f and 𝗣_k^f from by exploiting the prior knowledge about the time dependence of x_k and x_k−1 (as expressed, e.g., by approximate dynamical laws or by a time decorrelation model). The analysis step (or observational update) computes x_k^a and 𝗣_k^a from x_k^f and 𝗣_k^f by conditioning the prior distribution p_k^f(x_k) on the observation vector y_k using the Bayes’s theorem: p_k^a(x_k) ∼ p_k^f(x_k)p(y_k|x_k). It is well known that the observational update preserves Gaussianity as soon as p(y_k|x_k) is Gaussian: p(y_k|x_k) = (𝗛_kx_k, 𝗥_k), where 𝗥_k = 〈ϵ_kϵ_k^T〉 is the observation error covariance matrix, and that x_k^a and 𝗣_k^a can be computed by the classic linear observational update formulas:

View Expanded

View Expanded

where d_k is the innovation vector and 𝗞_k is the Kalman gain. It is useful to remark that the Gaussian parameters x_k^f and x_k^a, k = 1, … , N (and the innovations d_k as well) functionally depend on the sequence of observation vectors y_k, k = 1, … , N (even if, in practice, x_k^f and x_k^a are usually directly determined from the actual value of the observations: y_k = y_k^o). Conversely, the Gaussian parameters 𝗣_k^f and 𝗣_k^a do not depend on the observations y_k (but on the observation operator 𝗛_k and on the observation error covariance matrix 𝗥_k) and as long as all input covariance matrices are assumed to be error free, the resulting 𝗣_k^f and 𝗣_k^a are also known matrices, which do not need to be inferred from the observations. This is precisely what is going to be changed in section 2b.

If the forecast error covariance matrices 𝗣_k^f are available in square root form:

View Expanded

as in square root or ensemble Kalman filters, then it is possible to reformulate the observational update by linearly transforming the state and observation vectors x_k and y_k into the new vectors ξ_k and η_k:

View Expanded

where δ_k is the projection of the innovation vector d_k on the square root of the forecast error covariance matrix 𝗛_k𝗦_k^f (with metric ):

View Expanded

and 𝗨_k (unitary matrix) and Λ_k (diagonal matrix) are the matrices with the normalized eigenvectors and inverse eigenvalues of the matrix:

View Expanded

By the transformation in (7), the probability distributions p_k^f(x_k) and p(y_k|x_k) transform to

View Expanded

so that

View Expanded

from which it is easy to compute x_k^a and 𝗣_k^a by inverting the transformation in (7):

View Expanded

By the transformation 𝗨_k, the observational update of every component of the ξ_k vector are thus independent [all covariance matrices in (10) and (11) are diagonal].

The computational complexity of the observational update is also structurally modified by the transformation. In the linear observational update algorithm, the computational cost mainly results from the dependence between the vector components. In the presence of correlation, weighting optimally the forecast and observational information indeed requires the inversion of a full matrix (with computational complexity proportional to the cube of the size of the matrix). The main difference introduced by the transformation is thus that, in Eq. (5), the inversion is performed in the observation space, while in Eq. (9), it is performed in the state space [the cost of the inversion of 𝗥 is here assumed negligible, as shown by Brankart et al. (2009), for large classes of observation error correlation models]. Moreover, if 𝗣_k^f is rank deficient, the size of the matrix Γ_k in (9) is not the size of the state vector, but the rank of 𝗣_k^f, which is given by the number of independent columns in 𝗦_k^f (i.e., the error modes or the ensemble members). For such problems, the transformation in (7) is here introduced as a simple way of obtaining directly the reduced observational update formulas, that are otherwise deduced from (4) and (5) using the Sherman–Morrison–Woodbury formula (Pham et al. 1998). This transformed algorithm is particularly efficient for the reduced rank problems involving many observations, which are quite common in atmospheric and oceanic applications of square root or ensemble Kalman filters. The key property of the algorithm is indeed that its computational complexity is linear in the number y of observations, making it possible to deal efficiently with large size observation vectors [see Brankart et al. (2009), for a detailed comparison of the computational complexity of the transformed versus the original algorithm].

b. Adaptive statistics

In the filtering problem described in section 2a, optimal observational updates can only be obtained if the forecast and observation error covariance matrices 𝗣_k^f and 𝗥_k are accurate. However, in realistic atmospheric or oceanic applications, both matrices depend on inaccurate parameters. On the one hand, in the forecast step, modeling the time dependence of the errors usually requires questionable parameterizations (e.g., to account for errors in the dynamical laws governing the time evolution of the system). On the other hand, accurate observation error statistics are not always available. This is especially true for representation errors [difference between the truth of the problem and the real world, see Cohn (1997)], which can occur for instance if the state vector of the problem only contains a subrange of the scales that are present in the real system. In addition, the computation of the covariance matrices 𝗣_k^f and 𝗣_k^a never involves the observations y_k themselves, so that no feedback using differences with respect to the real world is possible; consequently, any inaccuracy in the parameterization of the observation or system noises can lead to instability of the error statistics produced by the filter. This is a well-known effect in Kalman filters, which is usually circumvented (in atmospheric and oceanic applications) by adjusting uncertain parameters in the forecast or observation error covariance matrices using statistics (variance or covariance) of the innovation sequence (adaptive Kalman filters). See for instance Dee (1995) for a more precise justification.

To describe the adaptive mechanism in a probabilistic framework, we introduce vectors of uncertain parameters α_k and β_k in the description of the forecast and observation error covariance matrices 𝗣_k^f(α_k) and 𝗥_k(β_k), with probability distributions p_k^f(α_k, β_k) and p_k^a(α_k, β_k) before and after their update using observations y_k. Thus, α_k and β_k are additional random vectors that must be estimated from the observational information: they are additional degrees of freedom that are introduced in the estimation problem to account for possible uncertainty in the Gaussian error covariances. In principle, assuming that the parameters α_k are uncertain transforms the probability distribution p_k^f(x_k) into

View Expanded

where p_k^f(x_k|α_k) is the Gaussian probability distribution of x_k that is assumed in section 2a if the vector of parameters α_k is known. In this equation, we use the updated parameter probability distribution p_k^a(α_k, β_k) for the parameters to express that all available information before and including time t_k is taken into account to estimate the uncertain parameters in the prior state probability distribution p_k^f(x_k), that is, the observational update of the parameter probability distribution is performed before the observational update of the state probability distribution. However, in (13), which explicitly takes into account uncertainties in the parameters, p_k^f(x_k) is no longer Gaussian in general, so that the filter equations described in section 2a do not apply anymore. An approximation is needed. To close the problem and be able to compute an explicit solution both for the state of the system and for the parameters, the central assumption is that the forecast probability distribution p_k^f(x_k) is still Gaussian (as in section 2a), but with covariances corresponding to the current best estimate α*_k of the parameters:

View Expanded

This approximation means that uncertainty in the parameters is not taken into account in the computation of the state estimates, which does not make a large difference as soon as the parameters remain sufficiently accurate. Equation (14) is indeed the zeroth-order term in the development of the integral in (13) if the parameter variance is used as the small quantity. By applying the same chain of arguments to the observation probability distribution, we directly obtain

View Expanded

Consequently, with the approximations in (14) and (15), the state filtering problem remains formally unchanged with respect to section 2a, and it remains only to show how best estimates of the parameters α*_k and β*_k can be obtained.

In this study, the successive values of α*_k and β*_k are computed by solving an additional filtering problem for the parameters, which amounts to sequentially computing the following probability distributions (which are not Gaussian in general):

View Expanded

The forecast step computes p_k^f(α_k, β_k) from p_k^a(α_k−1, β_k−1) by exploiting a prior knowledge p(α_k, β_k|α_k−1, β_k−1) about the time dependence of the parameter values (which must be specified, see section 3):

View Expanded

The analysis step (or observational update) computes p_k^a(α_k, β_k) from p_k^f(α_k, β_k) by conditioning the prior distribution p_k^f(α_k, β_k) on the innovation vector d_k using the Bayes theorem:

View Expanded

In writing this equation, it is assumed that innovation d_k contains information about the parameters that is independent from that contained in the previous innovations, which is already included in p_k^f(α_k, β_k). As in the state filtering problem, this can be justified by the hypothesis that model errors and observation errors (that parameters α_k and β_k are meant to model) are independent for times t_k and t_k′ (k ≠ k′). Furthermore, since p_k^f(x_k|α_k) and p_k(y_k|x_k, β_k) are the Gaussian distributions given in section 2a, the probability distribution p(d_k|α_k, β_k) of the innovation vector d_k is also a Gaussian distribution:

View Expanded

This fully defines the update of the (α_k, β_k) distribution defined by Eq. (18). It is then the purpose of section 3 to describe how the probability distributions sequence in (16) can be computed efficiently from (17) and (18) in practical applications and how to deduce from them the best estimates α*_k and β*_k to use for the state observational update at time t_k.

This joint state and parameter estimation problem can even be solved without the assumptions in (14) and (15) as soon as the state observational update is performed by a separate application of Eq. (4) for every member of the ensemble [as proposed by Evensen and van Leeuwen (1996), for the ensemble Kalman filter]. In that case, the prior non-Gaussian forecast probability distribution given by the integral in (13) can be simulated by randomly drawing a different parameter vector from p_k^a(α_k, β_k) to perform the update of each member. With that scheme, uncertainty in the parameter of the prior distribution can thus be explicitly taken into account, with asymptotic convergence of the prior distribution to the integral in (13) for large ensemble size. However, using (13) instead of (14) may not be appropriate if p_k^a(α_k, β_k) is not very accurate, for instance if the dispersion of the parameters (second-order moment) is not correctly simulated. Inaccuracy of the second-order moment is indeed the very reason why adaptivity is needed in the filter in (2) for the system state vector. Since this cannot be repeated for the parameter filter in (16), the use of the best parameter estimates α*_k and β*_k with Eqs. (14) and (15) can also be viewed as a closure that prevents inaccuracies in the second-order moments of p_k^a(α_k, β_k) from affecting directly the observational update of the state vector.

a. Constant parameters

If the parameters α_k and β_k are assumed constant in time, then p(α_k, β_k|α_k−1, β_k−1) = δ(α_k − α_k−1, β_k − β_k−1) so that the forecast step in (17) reduces to

View Expanded

which means that the previous knowledge of the parameters is not altered by the passing of time. In this simple case, we can suppress the index k to the parameter vectors, and all probability distributions of the sequence in (16) can be obtained by a recursive application of (18):

View Expanded

where L_k(α, β) is the likelihood function (as defined, e.g., in Von Mises 1964). From this expression, the best estimates of α*_k and β*_k of the parameters α and β at time t_k can be obtained as the mean (minimum variance estimator) or the mode (maximum probability estimator) of p_k^a(α, β). In this study, we only consider the mode of (21), which minimizes the cost function:

View Expanded

where the Gaussian shape of p(d_k′|α, β) in (19) is explicitly introduced, and Cst is an arbitrary constant.

b. Nonconstant parameters

If the parameters α_k and β_k are not assumed constant in time, it is necessary to introduce a prior knowledge p(α_k, β_k|α_k−1, β_k−1) about the time dependence of the parameters in order to perform the forecast step in (17). In this study, it is assumed that the effect of time is only to diffuse parameter probability densities from high probability regions to low probability regions, according to the simple model:

View Expanded

For a Gaussian distribution, the exponent f simply corresponds to multiplying the covariance by a factor 1/f, which means that the effect of time is simply to increase the error variance on the previous estimate by a factor 1/f. We call f the “forgetting exponent” because it corresponds to the exponential rate at which old information must be forgotten in the estimation of the parameters. The cost function in (22) indeed transforms to

View Expanded

Measured in number of assimilation cycles, the e-folding forgetting time scale is thus k^e = (ln1/f )⁻¹. Using a very small f exponent ( f → 0) means that only the last innovation is used to estimate the current parameters. This particular case exactly corresponds to the solution proposed by Dee (1995).

In addition, the solution of Dee (1995) is written without the first term in the cost function in (22) or (24), which corresponds to computing the maximum likelihood estimator of the parameters (as defined, e.g., in Von Mises 1964, chapter 10). The parameters α*_k and β*_k are then said to maximize the likelihood of the observed innovation sequence (i.e., the conditional probability of the innovation sequence for given parameters). This is useful in absence of reliable prior information on the parameters. With the parameterization in (23), this initial information is anyway progressively forgotten with time, as shown by the exponentially decreasing factor f^k in the cost function in (24).

c. Evaluation of the cost function

Minimizing the cost functions (22) or (24) with respect to the parameters requires the application of an iterative method, and thus the possibility of evaluating J_k for the successive iterates of the parameters α, β (for the sake of simplicity, subscript k is removed for the parameters; it is now implicit that they are computed for the current cycle k). The main difficulty with the expressions in (22) or (24) is that the evaluation of J_k requires the computation of the inverse and determinant of the covariance matrix 𝗖_k′(α, β) for the full sequence k′ ≤ k of previous innovation vectors, and this is needed for all successive iterates of α, β (let p be the number of iterates needed to reach the minimum with sufficient accuracy). Even if we truncate the innovation sequence to the K last innovations, this corresponds to a computational complexity proportional to pKy³ (leading behavior for large y), just to compute the optimal parameters α* and β* at time t_k [i.e., a factor pK with respect to the computational complexity of the observational update (4) and (5), or more precisely, with respect to the leading component (proportional to y³) of this computational complexity for large size observation vectors]. This large computational cost explains why using K = 1 (as in Dee 1995) is the only affordable solution (with this classic observational update algorithm) to compute optimal adaptive parameters in realistic atmospheric or oceanic assimilation systems. But, even in this special case, the computational complexity of the parameter estimation is still a factor p larger than the estimation of the state vector with Eqs. (4) and (5).

The objective of this section is to show how the cost function can be computed using the reduced innovation vector δ_k (size r) defined by Eq. (8), that is, by exploiting the transformation in (7) that transports the inversion problem from the observation space to the reduced dimension space defined by the square root or ensemble representation of the forecast error covariance matrix 𝗣_k^f(α). For that purpose, assume first that we know the reduced innovation vectors δ_k′(α, β) and the matrix Γ_k′(α, β), for k′ ≤ k, defined by Eqs. (8) and (9) as a function of the parameters α, β, together with the corresponding eigenvalues Λ_k′(α, β), and eigenvectors 𝗨_k′(α, β) as defined by Eq. (9). From this, we need to compute two kinds of terms in the cost function: . On the one hand, can be computed by inverting:

View Expanded

using the Sherman–Morrison–Woodbury formula:

View Expanded

which gives directly:

View Expanded

or

View Expanded

On the other hand, can be computed using the general formula:

View Expanded

where 𝗭 is any rectangular matrix and 𝗫 a regular square matrix. By applying this formula to the matrix in (25) with 𝗫 = 𝗥_k′(β) and , and using the definition (9) of Γ_k, we obtain the determinant:

View Expanded

In addition, using the definition (9) of Λ_k and since , we finally obtain the following:

View Expanded

Efficiently computing the determinant of the covariance matrix 𝗖_k is a classic difficulty in many problems involving the estimation of Gaussian parameters, which is here circumvented by introducing the diagonal covariance matrix Λ_k instead of 𝗖_k using the transformation in (7).

Yet, the transformed Eqs. (28) and (31) are still not a solution by themselves to efficiently compute the cost function as a function of the parameters, since the computation of δ_k′(α, β), Γ_k′(α, β), and then Λ_k′(α, β) and 𝗨_k′(α, β) is required for the full sequence of previous innovation vectors k′ ≤ k, and for all successive iterates of (α, β) (needed to minimize the cost function); that is, there is still a factor pK with respect to the computational complexity of the state observational update (8) and (9), or more precisely, with respect to the leading component (proportional to y) of this computational complexity for large size observation vectors. However, with the transformed equations in (28) and (31), there are classes of parameters α, β for which the additional computational complexity of the cost function minimization is either negligible with respect to the state observational update, or at least independent of the number y of observations. These classes of parameters are presented in sections 3d–f. Such simplifications are not possible with the original expressions in (22) or (24), because the matrix 𝗖_k(α, β) is the sum of two matrices: one depending on α and the other on β. The inverse and determinant must therefore be computed explicitly for every iteration of α and β.

d. Scaling of the forecast error covariance matrix

Let us assume first that a scaling factor α > 0 is introduced at each time t_k to rescale the forecast error covariance matrix 𝗣_k^f = α_k^f that is normally produced by the filter. [An efficient method to compute optimal estimates of this scaling factor is also proposed by Anderson (2007, 2009).] This can be done for instance to compensate a possible collapse of the ensemble forecast that can result from an insufficient system noise parameterization or the inadequacy of the Gaussian approximation. Then, if α is the only parameter to estimate, the cost functions in (28) and (31) reduce to

View Expanded

where δ̃_k′, Λ̃_k′, and _k′ are already available from the state observational update performed at time t_k′, so that the computational complexity of (32) is negligible (it is proportional to r, because the product is also already available). Even the gradient of the cost function can be computed for a similar numerical cost. Consequently, if the observational update is performed in the reduced space defined by the transformation in (7), an optimal adaptive estimate of the scaling factor α* can always be obtained without significant additional cost. Moreover, this can be directly generalized to a vector of parameters, scaling separately the eigenmodes of the forecast error covariance matrix (i.e., by independent scale factors applied to the eigenvalues λ_l,k′, l = 1, … , r). This can be done for instance by parameterizing a function α(λ) giving the scaling factor α as a function of the relative importance of each mode in the square root or ensemble representation of .

However, when estimating the inflation parameter α, it is important to keep in mind that nothing prevents the maximum likelihood estimator of α from being negative, which means that innovations are too small to explain the observation error covariance 𝗥_k′ alone, so that the adaptive scheme is attempting to reduce 𝗖_k′(α) by subtracting from 𝗥_k′. This usually results from an incorrect parameterization of the observation error covariance matrix, which underestimates the accuracy of the observations. This problem cannot occur with the maximum probability estimator since the prior probability distribution p₀(α) is equal to zero for α ≤ 0, so that optimization always provides a positive value for α*. In this case, the optimal value is, however, closely related to the behavior of p₀(α) near to zero and no longer to the statistics of the innovation sequence. This means that the adaptive scheme ascribes undeserved accuracy to the forecast (i.e., a small-scale factor α*), to compensate an incorrect parameterization of the observation errors. This phenomenon always results in further underestimating the weight of the observations in the assimilation system, and it can only be avoided either by improving the parameterization of the observation errors or by including observation error parameters β in the list of adaptive parameters (see sections 3e,f).

e. Scaling of the observation error covariance matrix

A second possibility is to introduce a scaling factor β at each time t_k to rescale the observation error covariance matrix 𝗥_k = β_k, where _k is the default prior estimate for this matrix. This can be useful to adjust inaccurate observation error statistics resulting in particular from the unknown amplitude of representation errors. Thus, if we also keep the parameter α (introduced in section 3d) in the control vector, Eqs. (28) and (31) reduce to

View Expanded

and

View Expanded

where y_k′ is the number of observations at time t_k′. Again, the computational complexity of (33) and (34) is negligible. The term can indeed be computed once for all for each time t_k′, with a computational complexity 2y_k′ (for a diagonal _k′) that is negligible with respect to that of the observational update. We can then conclude that, if the observational update is performed in the reduced space defined by the transformation in (7), optimal adaptive estimates of both scaling factors α*_k and β*_k can always be obtained without significant additional cost.

In addition, it is possible to generalize the simple problem of estimating one single scaling factor β for the whole observation error covariance matrix to the more general problem of estimating separate scaling factors for several segments of the observation vector. This may be useful for instance if the observation vector results from the merging of several datasets (i = 1, … , N), originating from different instruments, whose errors need to be scaled separately. However this generalized problem requires additional matrix operations that may significantly increase the computational complexity, because changing the structure of the matrix 𝗥_k modifies the eigenvalue problem defined by Eq. (9), which must then be solved again for every new iterate of the vector of parameters β. The first step is to express the matrix as the sum of the matrices (separately scaled with factor 1/β_i), which are nonzero only for observations belonging to the corresponding dataset:

View Expanded

and to separately store the corresponding vectors δ_k,i and matrices Γ_k,i given by Eqs. (8) and (9). This computation does not involve any additional operations since this can be done once for all, and since every scalar product only needs to be extended to the part of with nonzero values (which are complementary so that the count of operations remains unchanged). From these precomputed elements, the cost function can be evaluated by performing the following operations for every iterate of the vector β and for every k′ ≤ k: (i) compute the overall δ_k′(β) and Γ_k′(β) as

View Expanded

and (ii) compute the eigenvalue–eigenvector decomposition of Γ_k′(β) from which the second term of Eqs. (28) and (31) can be easily computed [with factor α as in (32) if required]. Concerning the first term of (28) or (31), it can be computed as the sum of the individual elements:

View Expanded

where y_k′,i is the number of observations in the segment number i. This computation is straightforward since the individual can also be computed once for all at time step k′. The dominant cost of these additional operations (for N ≪ r) comes from the necessity to recompute Λ_k′ and 𝗨_k′ (computational complexity proportional to r³) for every iterate of β and every k′ ≤ k, so that the overall additional complexity is proportional to pKr³, still independent of the number of observations, but no more linear in r.

Finally, it is important to keep in mind that, in order to attempt the joint control of observation and forecast error covariance scaling factors α and β, we must be certain that both parameters can be simultaneously identified through the innovation sequence. [See Li et al. (2009), who also propose a method to control these two parameters.] This is for instance clearly impossible if observation and forecast errors have identical covariance structures (in the observation space), because then parameters α and β play exactly the same role in the innovation covariance matrix in (25). In this ill-conditioned situation, parameters α and β are not jointly controllable whatever the number of observations or the length of the innovation sequence.

f. Observation error correlation length scale

All developments presented so far are valid whatever the shape of the observation error covariance matrix. However, performing the observational update in the reduced space using Eqs. (7)–(12) can only be efficient (i.e., linear in the number of observations) if the observation error covariance matrix 𝗥_k can be inverted at low cost (e.g., if it can be assumed diagonal). It is nevertheless possible to preserve the efficiency of the transformed observational update algorithm in presence of observation error correlations, as shown by Brankart et al. (2009). Their method consists in augmenting the observation vector y_k with new observations that are linear combinations of the original observations, and assuming a diagonal observation error covariance matrix in the augmented observation space. Since the computational complexity of the algorithm is linear in the number of observations, the size of the observation vector can indeed be increased without prohibitive consequence on the numerical cost. If the augmented observation vector y⁺, the augmented observation operator 𝗛⁺, and the associated diagonal observation error covariance matrix 𝗥⁺ are defined as

View Expanded

where 𝗧 is any linear transformation operator, and 𝗥₀, 𝗥₁ are diagonal matrices, then, the observational update that is performed using y⁺ and 𝗥⁺ is equivalent to an observational update performed using only observations y with a nondiagonal observation matrix 𝗥 given by

View Expanded

It can be shown for instance that adding gradient observations (𝗧 is then the discrete gradient operator) is equivalent to assuming observation error correlation decreasing exponentially with the distance, with a correlation length scale given by ℓ = σ₀/σ₁ (for homogeneous 𝗥₀ = σ₀²𝗜 and 𝗥₁ = σ₁²𝗜). The purpose of this section is to show how the optimal adaptive algorithm described above must be modified if this kind of parameterization of the observation errors correlations is applied.

First of all, it can be easily verified that the terms of the cost function in (27) are not affected by the observation vector transformation and can thus be identically computed using y and 𝗥 or y⁺ and 𝗥⁺. The vector δ_k and the matrix Γ_k are indeed left unchanged by this change of observation vector, if the observation error covariance matrices 𝗥 and 𝗥⁺ are related by Eq. (39). (It is the purpose of this transformation to keep the observational update unchanged.) This is also true for the term d^T𝗥⁻¹d in Eq. (27):

View Expanded

These terms can thus be computed efficiently using a diagonal observation error covariance matrix in the augmented observation space, using the elements δ_k′, Γ_k′, 𝗨_k′, and Λ_k′ that are already available from the observational update performed at time t_k′.

However, the adaptive algorithm cannot be applied blindly with the augmented observation vector defined by (38), because the determinant of the observation error covariance matrix, which is present in the terms of the cost function in (31), is modified by the transformation: |𝗥| ≠ |𝗥⁺|. The evaluation of the cost function thus requires an explicit computation of the determinant |𝗥| using Eq. (39). Fortunately, this computation can be simplified a great deal using the following transformation. First, the determinant of (39) can be rewritten as

View Expanded

Then using again Eq. (29) in the same way as it is used to deduce (30) and (31), we obtain

View Expanded

where μ_l are the eigenvalues of ^T. In general, computing the μ_l eigenvalues is a problem with computational complexity proportional to y³, but in many practical situations, a sufficient knowledge of this spectrum may not be very difficult to acquire if 𝗧 is kept simple enough [as is required for an efficient observational update, see Brankart et al. (2009)]. For instance, if 𝗧 is a discrete gradient operator and 𝗥₀ = σ₀²𝗜 and 𝗥₁ = σ₁²𝗜 are homogeneous, μ_l are simply the eigenvalues of a discrete Laplacian operator.

Let us assume now that the observation error covariance matrix depends on two uncertain parameters β₀ and β₁, which must be controlled:

View Expanded

where ₀ and ₁ are the default prior estimate of the diagonal matrices 𝗥₀ and 𝗥₁ defined by (39). Here β₀ and β₁ are thus scaling factors for these two matrices: 𝗥₀ = β₀₀ and 𝗥₁ = β₁₁. If 𝗧 is a combination of derivative operators of successive orders (and 𝗥₀, 𝗥₁ are spatially homogeneous: 𝗥₀ = σ₀²𝗜, 𝗥₁ = σ₁²𝗜), then the scaling factor for the correlation length scale ℓ is a function of . If 𝗧 is a discrete gradient, this reduces to (see Brankart et al. 2009). With the parameterization in (43) together with the scaling parameter α, Eqs. (28) and (31) for the components of the cost function reduce to

View Expanded

and

View Expanded

because δ_k′(α, β₀, β₁) and Γ_k′(α, β₀, β₁) defined in (36) can be written as

View Expanded

Equations (44)–(46) show that the eigenvalue/eigenvector decomposition in (9) needs only to be recomputed when iterating on the ratio β₀/β₁, which scales the observation error correlation length. The computational complexity of the minimization is thus proportional to pKr³ (leading behavior for large y and r), where p is here only the number of iterations requiring the update of the correlation length scale. In summary, by minimizing the cost function in (22) or (24) with components (44) and (45), it is possible to adapt simultaneously scaling factors for the forecast error covariance matrix, for the observation error covariance matrix, and for the observation error correlation length scale (given by α, β₀ and β₀/β₁, respectively).

As a particular case, the problem with α = 0 corresponds to estimating parameters β₀ and β₁ of the covariance 𝗥 of a zero mean random vector from a finite sample of independent events d_k′, k′ = 1, … , k. In this case, the last term disappears in (44) and (45) and we can obtain the likelihood function as

View Expanded

where a constant size y of the observation vector, a constant operator 𝗧, and constant diagonal matrices ₀ and ₁ are assumed. If a prior probability distribution p₀(β₀, β₁) is available, the posterior distribution is then p_k^a(β₀, β₁) ∼ p₀(β₀, β₁) L(β₀, β₁). Parameters β₀ and β₁ jointly govern the observation error variance σ² and the observation error correlation length scale ℓ (while the shape of the correlation function is governed by the operator 𝗧, which is assumed to be known). The likelihood function in (47) is thus also a likelihood function for σ² and ℓ as soon as their relation to β₀ and β₁ is known. This relation can be deduced from the functions f (ℓ) = σ₀²/σ₁² and g(ℓ) = σ₀²/σ² that can be obtained from (43) with ₀ = σ₀²𝗜 and ₁ = σ₁²𝗜 (see Brankart et al. 2009). Using these functions, the likelihood function in (47) can be rewritten as a function of σ² and ℓ:

View Expanded

where μ_l are here directly the eigenvalues of 𝗧𝗧^T (and not of ^T as before). If 𝗧 is the gradient operator, the μ_l are simply the eigenvalues of the Laplacian operator, f (ℓ) = ℓ and g(ℓ) is a decreasing function of ℓ, behaving proportionally to ℓⁿ (in n dimensions) if ℓ is large with respect to the distance between observations. If the correlation length is known, we retrieve the classic likelihood function for the variance σ² of a Gaussian random vector. Conversely, if the variance is known, Eq. (48) is simply the likelihood function L(ℓ) for the correlation length of a random vector with known variance and known correlation shape. The equation shows that it can be computed explicitly at low cost as soon as 𝗧, μ_l, f (ℓ), and g(ℓ) are known (see section 4d).

a. Description of the experiments

The reference mesoscale signal is simulated using a primitive equation model of an idealized square and 5000-m-deep flat bottom ocean at midlatitudes (between 25° and 45°N). In this square basin a double-gyre circulation is created by a constant zonal wind forcing blowing westward in the northern and southern parts of the basin and eastward in the middle part of the basin (with sinusoidal latitude dependence: τ = −τ₀ cos[(λ − λ_min)/(λ_max − λ_min)] and τ₀ = 0.1 N m⁻²). The western intensification of these two gyres produces western boundary currents that feed an eastward jet in the middle of the square basin (see the resulting mean dynamic height in Fig. 1). This jet is unstable (Le Provost and Verron 1987) so that the flow is dominated by chaotic mesoscale dynamics, with largest eddies that are ∼100 km wide, and to which correspond velocities of ∼1 m s⁻¹ and dynamic height differences of ∼1 m (see the resulting dynamic height standard deviation in Fig. 1). All this is very similar in shape and magnitude to what is observed in the Gulf Stream (North Atlantic) or in the Kuroshio (North Pacific).

The time evolution of this chaotic system is computed using the Nucleus for European Modelling of the Ocean (NEMO) numerical ocean model, with a horizontal resolution of ¼° × ¼° cosλ and 11 levels in the vertical (see Cosme et al. 2010 for more detail about the model configuration). The main three physical parameters governing the dominant characteristics of the flow are the stratification, the bottom friction, and the horizontal viscosity. The model is started from rest with uniform stratification and can be considered to reach equilibrium statistics after 20 yr of simulation. In this paper, we thus concentrate on the estimation of the 100-yr signal from years 21 to 120. Moreover, we focus our study to a limited subdomain in the middle of the jet (about 650 × 650 km, as shown by the black square in Fig. 1) with intense and quite homogeneous mesoscale activity. It is also assumed that the reference simulation is known with a time resolution of 1 snapshot every 10 days. Figure 2 shows for instance the simulated dynamic height in October of year 21, with a time resolution of 10 days, which is clearly sufficient to observe the slow westward (upstream) motion of the main eddies.

To estimate the time evolution of this mesoscale flow, we assume that the ocean altimetry is observed every 10 days at model resolution. Without a dynamical model to constrain the estimation problem, it is indeed important to have observations with a sufficient horizontal coverage. However, in order to generate the synthetic observations, an artificial observational noise is added to the reference simulation. This noise is meant to simulate measurement and representation errors that always exist in a real observation dataset. In our system, the representation error mainly includes the subgrid-scale eddies that are not resolved by the model discretization. This component of the observation error is thus often dominant and poorly known, which justifies adjusting its main statistics (variance and correlation length scale) using the available observations. In this study, three kinds of correlation model are used to simulate the observational noise: (A) uncorrelated errors, (B) exponential decorrelation function: ρ(r) = exp(−r/ℓ), and (C) a smooth noise correlation model: ρ(r) = (r/l)K₁(r/l), where K₁ is the second kind modified Bessel function of order 1. The two last models have the property that they can be efficiently parameterized using a diagonal observation error covariance matrix in an augmented observation space [as in Eq. (38)]. Model B just requires including gradient observations (with error standard deviation σ₁ = σ₀/ℓ), while model C requires including both gradient and curvature (with error standard deviations σ₁ = σ₀/ℓ2 and σ₂ = σ₀/ℓ²). See Brankart et al. (2009) for more detail about this correspondence. Figure 3 shows an example of the simulated observation noise corresponding to the three correlation models. They are randomly sampled from the Gaussian distribution (0, 𝗥), where the observation error covariance matrix 𝗥 is characterized by a uniform standard deviation σ = 0.2 m and one of the correlation models A, B, or C, with a correlation length scale ℓ equal to 5 grid points (for the last 2 models).

b. Control of the forecast error covariance scaling

The first set of experiments is dedicated to the estimation of a single global scaling factor α for the forecast error covariance matrix, by applying the method described in section 3d. In these first experiments, observation errors are uncorrelated, with standard deviation σ = 0.2 m, and the corresponding observation error covariance matrix is assumed perfectly known: 𝗥 = σ𝗜. Problems with a constant and nonconstant parameter α are successively considered.

c. Control of the observation error covariance scaling

The second set of experiments is dedicated to the estimation of a single global scaling factor β for the observation error covariance matrix, by applying the method described in section 3e. In these experiments, observation errors are still uncorrelated, with a standard deviation of σ = 0.2 m, but the scaling of the observation error covariance matrix is assumed unknown: 𝗥 = β, with = σ𝗜. We first consider the same example as in section 4b(2), but with known forecast error covariance matrix 𝗣_k^f = 𝗤. The only parameter to control is thus the scaling factor β. This is done by solving the joint filtering problem for the state of the system and for the parameter β [as in section 4b(2) for parameter α, but this time using the expression of the cost function given in section 3e], using again the prior probability distribution p₀(β) = exp(−β), but forgetting the exponent f = 1, since β can be assumed constant.

Figure 7 (top panel, thick lines) shows the resulting estimate β*(t_k) that is obtained as the mode of the posterior probability distribution at time t_k, together with percentiles 0.1 and 0.9. As expected, the optimal estimate β*(t_k) quickly converges toward the correct value β = 1, with an associated error decreasing to zero as more observations become available. Old observations are indeed never forgotten since the parameter is assumed constant. The figure also illustrates the corresponding error on the state estimate (as in Fig. 6), showing that we obtain the same result as in Fig. 6 as soon as parameter β becomes sufficiently accurate. This result is compared with another simulation that is performed without the adaptive mechanism (thin lines) using the inaccurate value β = 0.5. We observe again that, without adaptive statistics, the error estimate remains inconsistent, so that the resulting nonoptimal scheme can only produce larger errors, together with badly estimated error variance (underestimated in this example).

On the other hand, it is also interesting to investigate what happens if we control the scaling factor β alone, in presence of inaccurate forecast error covariance scaling. For that purpose, we just redo the same experiment with 𝗣_k^f = α𝗤, with constant α = ¼. The resulting estimate β*(t_k) is also shown in Fig. 7 (top panel, thin line). As can be observed, β*(t_k) quickly diverges from the correct value β = 1, because the adaptive mechanism is trying to compensate for the underestimation of the forecast error variance by overestimating the observation error, in order to account for the total innovation signal. It is thus very important to stress again the assumption that all statistical parameters that are not included in the control vector are accurately known. Inconsistency in the prior statistical parameterization can easily lead to grossly incorrect adaptive parameters.

d. Control of the correlation length scale

The third set of experiments is dedicated to the estimation of the correlation length scale from a random sample of a Gaussian distribution, by applying the method described in section 3f. In these experiments, it is assumed that this Gaussian distribution is characterized by a zero mean and a covariance 𝗥. This application thus corresponds to the problem described in section 3f with the simplification that α = 0 (i.e., 𝗥 is the only remaining term in the covariance 𝗖 of the random vector), so that we can directly apply Eqs. (47) and (48) to estimate the signal variance and/or correlation length scale.

As an illustration, let us consider the problem of estimating the correlation length scale of the observational noise described in section 4a (and illustrated in Fig. 3) from samples of various sizes. In this experiment, it is assumed that the noise standard deviation σ = 0.2 m and the correlation structure in model B [ρ(r) = exp(−r/ℓ)] or model C [ρ(r) = (r/l)K₁(r/l)] are known, so that the correlation length scale ℓ is the only parameter that must be estimated. Moreover, we have already said that these two correlation structures can be consistently parameterized using the parameterization in (39) with a transformation operator 𝗧 that includes the gradient for model B or the gradient and curvature for model C. From this operator, it is easy to evaluate the eigenvalues μ_l of 𝗧𝗧^T, and the function f (ℓ) and g(ℓ) using Eq. (43). Figure 8 illustrates the resulting likelihood function L(ℓ) for the correlation length scale ℓ, as computed using Eq. (48), for several sizes k of the sample. (The figure is scaled so that the maximum is always equal to 1.) The figure shows that, for the correlation model B, the likelihood function narrows close to the correct value ℓ = 2 grid points (left panel) or, ℓ = 5 grid points (middle panel), as more observations are available, which indicates that the adaptive mechanism presented in this paper is able to control the correlation length scale of a random signal, as soon as a correct assumption about the correlation structure can be formulated. Concerning the correlation model C, also illustrated in Fig. 8, there is a discrepancy between the true correlation length scale (ℓ = 5 grid points) and the estimated value (ℓ ∼ 3.2 grid points). This problem results from an approximate evaluation of the function g(ℓ) in Eq. (48), for which we assumed an infinite domain (whereas our real domain is not very large with respect to the correlation length scale). This difficulty does not arise if the adaptive parameters are β₀ and β₁ (instead of σ² and ℓ), but this points out the sensitivity of the estimation to inaccuracies in the representation of the correlation structure.

e. Joint control of forecast and observation error parameterizations

Last, in a fourth set of experiments, we solve a more general assimilation problem in which two adaptive parameters considered in the previous sections are controlled together: a parameter α to adjust the scaling of the forecast error covariance matrix, and a parameter β to adjust the scaling of the observation error covariance matrix. For that purpose, we use observations with uncorrelated or correlated errors (σ = 0.2 m and ℓ = 5 grid points) using correlation model A, B, or C, as illustrated in Fig. 3. In addition, in the experiments, the structure of the observation error covariance matrix is assumed to be known: it is given by Eq. (39) with an operator 𝗧 consistent with the real correlation model (A, B, or C), and with diagonal matrices 𝗥₀ = σ₀𝗜 and 𝗥₁ = σ₁𝗜 parameterized in such a way that the true value of the global scaling parameter is β = 1. As prior probability distribution for the parameters, we use p₀(α, β) = exp[−(α + β)], that is, independent prior distributions with exponential probability density for each parameter. As in section 4b(2), the parameters are not assumed constant in time, and the same forgetting factor f = 0.9 is used. With these assumptions, we can solve the joint filtering problem for the state of the system and for the parameters α and β. In particular, the optimal parameter estimates are obtained at each time t_k by minimizing the cost function whose components are given by Eqs. (33) and (34) as a function of the parameters α and β (with y_k′ equal to the number of real observations, i.e., excluding the fictious gradient or curvature observations if any).

Figure 9 (top two panels, thick lines) shows the resulting estimates α*(t_k) and β*(t_k) that are obtained as the mode of the posterior probability distribution at time t_k. As expected, for all correlation models A, B, or C, the optimal forecast error covariance scaling α*(t_k) converges toward the same solution as in section 4b(2), and the optimal observation error covariance scaling β*₁(t_k) converges approximately toward the correct values β = 1. Figure 9 (bottom panel) also illustrates the corresponding error on altimetry. For correlation model A, the result is similar to the solution shown in Fig. 6 (obtained with known β), whereas for the two other correlation models B or C, the error is larger, as a result of the lower quality of the observations that are assimilated (correlated instead of uncorrelated observation error). But in any case, the error estimate produced by the adaptive filter (dotted thick line) is always a consistent estimate of the real error standard deviation. Without adaptivity (not shown in the figure), the error estimate can become grossly inconsistent (as explicitly shown in the examples of sections 4b and 4c) with the same negative consequences on filter optimality.

Finally, in order to characterize more completely the knowledge that is acquired about parameters α and β, Fig. 10 represents their joint likelihood function (based on the first two innovation vectors), as obtained in the three experiments (i.e., with correlation model A, B, or C for the observation error). The first thing that can be observed is that β is diagnosed with a better accuracy than α, as a consequence of the larger number of degrees of freedom in the observation error as compared to the forecast error (see Figs. 2 and 3). Second, in these experiments, the slope of the first principal axis of the sensitivity ellipse happens to be only slightly negative, which indicates that, in this case study, the adaptive scheme is able to make a clear distinction between forecast and observation errors. This absence of an unstable direction with large inaccuracy (which would correspond to a correct representation of the total covariance whatever α and β along that line) explains why parameters α and β can be jointly identified. Again, this is linked to the very distinct structures of the forecast and observation errors (cf. Figs. 2 and 3) so that their respective variances can be easily diagnosed as soon as their respective covariance structures are known.