• Anderson, J. L. , 2003: A local least squares framework for ensemble filtering. Mon. Wea. Rev., 131 , 634642.

  • Anderson, J. L. , 2007: An adaptive covariance inflation error correction algorithm for ensemble filters. Tellus, 59A , 210224.

  • Bishop, C. H. , and D. Hodyss , 2009: Ensemble covariances adaptively localized with ECO-RAP. Part 2: A strategy for the atmosphere. Tellus, 61A , 97111.

    • Search Google Scholar
    • Export Citation
  • Bishop, C. H. , B. J. Etherton , and S. J. Majumdar , 2001: Adaptive sampling with the ensemble transform Kalman filter. Part I: Theoretical aspects. Mon. Wea. Rev., 129 , 420436.

    • Search Google Scholar
    • Export Citation
  • Brankart, J-M. , C-E. Testut , P. Brasseur , and J. Verron , 2003: Implementation of a multivariate data assimilation scheme for isopycnic coordinate ocean models: Application to a 1993–96 hindcast of the North Atlantic Ocean circulation. J. Geophys. Res., 108 , 3074. doi:10.1029/2001JC001198.

    • Search Google Scholar
    • Export Citation
  • Brankart, J-M. , C. Ubelmann , C-E. Testut , E. Cosme , P. Brasseur , and J. Verron , 2009: Efficient parameterization of the observation error covariance matrix for square root or ensemble Kalman filters: Application to ocean altimetry. Mon. Wea. Rev., 137 , 19081927.

    • Search Google Scholar
    • Export Citation
  • Brankart, J-M. , E. Cosme , C-E. Testut , P. Brasseur , and J. Verron , 2010: Efficient adaptive error parameterization for square root or ensemble Kalman filters: Application to the control of ocean mesoscale signals. Mon. Wea. Rev., 138 , 932950.

    • Search Google Scholar
    • Export Citation
  • Brusdal, K. , J. Brankart , G. Halberstadt , G. Evensen , P. Brasseur , P. van Leeuwen , E. Dombrowsky , and J. Verron , 2003: A demonstration of ensemble-based assimilation methods with a layered OGCM from the perspective of operational ocean forecasting systems. J. Mar. Syst., 40–41 , 253289.

    • Search Google Scholar
    • Export Citation
  • Buehner, M. , 2005: Ensemble-derived stationary and flow-dependent background-error covariances: Evaluation in a quasi-operational NWP setting. Quart. J. Roy. Meteor. Soc., 131 , 10131043.

    • Search Google Scholar
    • Export Citation
  • Burgers, G. , P. J. van Leeuwen , and G. Evensen , 1998: Analysis scheme in the ensemble Kalman filter. Mon. Wea. Rev., 126 , 17191724.

    • Search Google Scholar
    • Export Citation
  • Corazza, M. , E. Kalnay , and S. Yang , 2007: An implementation of the Local Ensemble Kalman filter in a quasi-geostrophic model and comparison with 3D-Var. Nonlinear Processes Geophys., 14 , 89101.

    • Search Google Scholar
    • Export Citation
  • Cosme, E. , J-M. Brankart , J. Verron , P. Brasseur , and M. Krysta , 2010: Implementation of a reduced-rank, square-root smoother for high-resolution ocean data assimilation. Ocean Modell., 33 , 87100.

    • Search Google Scholar
    • Export Citation
  • Dee, D. , 1995: Online estimation of error covariance parameters for atmospheric data assimilation. Mon. Wea. Rev., 123 , 11281145.

    • Search Google Scholar
    • Export Citation
  • Drévillon, M. , and Coauthors , 2008: The GODAE/Mercator-Ocean global ocean forecasting system: Results, applications and prospects. J. Oper. Oceanogr., 1 , 5157.

    • Search Google Scholar
    • Export Citation
  • Evensen, G. , 1994: Sequential data assimilation with a non linear quasigeostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res., 99 , (C5). 1014310162.

    • Search Google Scholar
    • Export Citation
  • Evensen, G. , and P. J. van Leeuwen , 1996: Assimilation of Geosat altimeter data for the Agulhas Current using the ensemble Kalman filter with a quasi-geostrophic model. Mon. Wea. Rev., 124 , 8596.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M. , J. S. Whitaker , and C. Snyder , 2001: Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter. Mon. Wea. Rev., 129 , 27762790.

    • Search Google Scholar
    • Export Citation
  • Heemink, A. W. , M. Verlaan , and A. J. Segers , 2001: Variance reduced ensemble Kalman filtering. Mon. Wea. Rev., 129 , 17181728.

  • Houtekamer, P. L. , and H. L. Mitchell , 1998: Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev., 126 , 796811.

    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L. , and H. L. Mitchell , 2001: A sequential ensemble Kalman filter for atmospheric data assimilation. Mon. Wea. Rev., 129 , 123137.

    • Search Google Scholar
    • Export Citation
  • Hunt, B. , E. Kostelich , and I. Szunyogh , 2007: Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filter. Physica D, 230 , 112126.

    • Search Google Scholar
    • Export Citation
  • Lermusiaux, P. F. J. , and A. R. Robinson , 1999: Data assimilation via error subspace statistical estimation. Part I: Theory and schemes. Mon. Wea. Rev., 127 , 13851407.

    • Search Google Scholar
    • Export Citation
  • Lorenc, A. C. , 2003: The potential of the ensemble Kalman filter for NWP—A comparison with 4D-VAR. Quart. J. Roy. Meteor. Soc., 129 , 31833203.

    • Search Google Scholar
    • Export Citation
  • Ott, E. , and Coauthors , 2004: A local ensemble Kalman filter for atmospheric data assimilation. Tellus, 56A , 415428.

  • Pham, D. T. , J. Verron , and M. C. Roubaud , 1998: Singular evolutive extended Kalman filter with EOF initialization for data assimilation in oceanography. J. Mar. Syst., 16 , 323340.

    • Search Google Scholar
    • Export Citation
  • Szunyogh, I. , E. Kostelich , G. Gyarmati , D. Patil , B. Hunt , E. Kalnay , E. Ott , and J. Yorke , 2005: Assessing a local ensemble Kalman filter: Perfect model experiments with the National Centers for Environmental Prediction global model. Tellus, 57A , 528545.

    • Search Google Scholar
    • Export Citation
  • Testut, C. , P. Brasseur , J. Brankart , and J. Verron , 2003: Assimilation of sea-surface temperature and altimetric observations during 1992–1993 into an eddy permitting primitive equation model of the North Atlantic Ocean. J. Mar. Syst., 40–41 , 291316.

    • Search Google Scholar
    • Export Citation
  • Tippett, M. K. , J. L. Anderson , C. H. Bishop , T. M. Hamill , and J. S. Whitaker , 2003: Ensemble square root filters. Mon. Wea. Rev., 131 , 14851490.

    • Search Google Scholar
    • Export Citation
  • Verlaan, M. , and A. W. Heemink , 1997: Tidal flow forecasting using reduced rank square root filters. Stochastic Hydrol. Hydraul., 11 (5) 349368.

    • Search Google Scholar
    • Export Citation
  • Whitaker, J. S. , and T. M. Hamill , 2002: Ensemble data assimilation without perturbed observations. Mon. Wea. Rev., 130 , 19131924.

    • Search Google Scholar
    • Export Citation
  • Whitaker, J. S. , T. M. Hamill , X. Wei , Y. Song , and Z. Toth , 2008: Ensemble data assimilation with the NCEP global forecast system. Mon. Wea. Rev., 136 , 463482.

    • Search Google Scholar
    • Export Citation
  • Yang, S-C. , E. Kalnay , B. Hunt , and N. E. Bowler , 2009: Weight interpolation for efficient data assimilation with the Local Ensemble Transform Kalman filter. Quart. J. Roy. Meteor. Soc., 135 , 251262.

    • Search Google Scholar
    • Export Citation
  • View in gallery
    Fig. 1.

    Comparison of the global (solid line) and local ensemble (dashed line) observational updates to the exact solution (dotted line): (top left) forecast covariance with respect to the central location, (top right) updated mean, (bottom left) updated standard deviation, and (bottom right) updated covariance with respect to the central location.

  • View in gallery
    Fig. 2.

    (left) Eigenvalue spectrum of the local ensemble forecast covariance, as obtained with an identity metric (solid line) and with a metric giving a significant importance to the central location only (dotted line). (right) Percentage of variance that is resolved as a function of the position in the subdomain by keeping 99% of the total variance (with each metric).

  • View in gallery
    Fig. 3.

    Difference with respect to the reference local parameterization using explicit square root decomposition (solid line) or bulk parameterization (dotted line): (top left) forecast ensemble covariance with respect to the central location, (top right) updated ensemble mean, (bottom left) updated ensemble standard deviation, and (bottom right) updated ensemble covariance with respect to the central location. For comparison, the figure also shows the difference between the classic local parameterization and the exact solution (dotted line).

  • View in gallery
    Fig. 4.

    (left) Model-simulated dynamic height anomaly and (right) random perturbation simulating observation error.

  • View in gallery
    Fig. 5.

    Sensitivity of correlation estimates to (left) sample size and (right) rank reduction. The figure represents dynamic height correlation as function of latitude (along the 15°E meridional section, with respect to a point located at the center of the jet at 34.5°N, 15°E). (right) With 10, 20, and 80 principal components, the percentage of resolved variance is 54%, 68%, and 93%, respectively.

  • View in gallery
    Fig. 6.

    Observational update resulting from one perfect observation located at 34.37°N, 25.18°E: (top left) global ensemble parameterization (with m = 5000 and r = 5000), (top middle) global ensemble parameterization (with m = 200 and r = 200), (top right) global reduced-rank parameterization (with m = 5000 and r = 20), (bottom left) local ensemble parameterization (with m = 200, r = 200 and constant 𝗪S), (bottom middle) local ensemble parameterization (with m = 200, r = 200 and decreasing 𝗪S), and (bottom right) local reduced-rank parameterization (with m = 5000, r = 20 and decreasing 𝗪S).

  • View in gallery
    Fig. 7.

    Root-mean-square residual error after the observational update: (left) global ensemble parameterization (with m = 5000 and r = 5000), (middle) global reduced-rank parameterization (with m = 5000 and r = 20), and (right) local reduced-rank parameterization (with m = 5000, r = 20 and decreasing 𝗪S).

  • View in gallery
    Fig. 8.

    Updated error standard deviation: (left) global ensemble parameterization (with m = 5000 and r = 5000), (middle) global reduced-rank parameterization (with m = 5000 and r = 20), and (right) local reduced-rank parameterization (with m = 5000, r = 20 and decreasing 𝗪S).

  • View in gallery
    Fig. 9.

    Maps of estimated (top) α and (bottom) β using (from left to right) 1, 3, or 5 observation vectors.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 77 29 3
PDF Downloads 27 8 0

Efficient Local Error Parameterizations for Square Root or Ensemble Kalman Filters: Application to a Basin-Scale Ocean Turbulent Flow

Jean-Michel BrankartLEGI/CNRS-Université de Grenoble, Grenoble, France

Search for other papers by Jean-Michel Brankart in
Current site
Google Scholar
PubMed
Close
,
Emmanuel CosmeLEGI/CNRS-Université de Grenoble, Grenoble, France

Search for other papers by Emmanuel Cosme in
Current site
Google Scholar
PubMed
Close
,
Charles-Emmanuel TestutLEGI/CNRS-Université de Grenoble, Grenoble, France

Search for other papers by Charles-Emmanuel Testut in
Current site
Google Scholar
PubMed
Close
,
Pierre BrasseurLEGI/CNRS-Université de Grenoble, Grenoble, France

Search for other papers by Pierre Brasseur in
Current site
Google Scholar
PubMed
Close
, and
Jacques VerronLEGI/CNRS-Université de Grenoble, Grenoble, France

Search for other papers by Jacques Verron in
Current site
Google Scholar
PubMed
Close
Full access

Abstract

In large-sized atmospheric or oceanic applications of square root or ensemble Kalman filters, it is often necessary to introduce the prior assumption that long-range correlations are negligible and force them to zero using a local parameterization, supplementing the ensemble or reduced-rank representation of the covariance. One classic algorithm to perform this operation consists of taking the Schur product of the ensemble covariance matrix by a local support correlation matrix. However, with this parameterization, the square root of the forecast error covariance matrix is no more directly available, so that any observational update algorithm requiring this square root must include an additional step to compute local square roots from the Schur product. This computation generates an additional numerical cost or produces high-rank square roots, which may deprive the observational update from its original efficiency. In this paper, it is shown how efficient local square root parameterizations can be obtained, for use with a specific square root formulation (i.e., eigenbasis algorithm) of the observational update. Comparisons with the classic algorithm are provided, mainly in terms of consistency, accuracy, and computational complexity. As an application, the resulting parameterization is used to estimate maps of dynamic topography characterizing a basin-scale ocean turbulent flow. Even with this moderate-sized system (a 2200-km-wide square basin with 100-km-wide mesoscale eddies), it is observed that more than 1000 ensemble members are necessary to faithfully represent the global correlation patterns, and that a local parameterization is needed to produce correct covariances with moderate-sized ensembles. Comparisons with the exact solution show that the use of local square roots is able to improve the accuracy of the updated ensemble mean and the consistency of the updated ensemble variance. With the eigenbasis algorithm, optimal adaptive estimates of scaling factors for the forecast and observation error covariance matrix can also be obtained locally at negligible additional numerical cost. Finally, a comparison of the overall computational cost illustrates the decisive advantage that efficient local square root parameterizations may have to deal simultaneously with a larger number of observations and avoid data thinning as much as possible.

* Current affiliation: Mercator-Ocean, Toulouse, France

Corresponding author address: Jean-Michel Brankart, LEGI/CNRS, BP53X, 38041 Grenoble CEDEX, France. Email: jean-michel.brankart@hmg.inpg.fr

Abstract

In large-sized atmospheric or oceanic applications of square root or ensemble Kalman filters, it is often necessary to introduce the prior assumption that long-range correlations are negligible and force them to zero using a local parameterization, supplementing the ensemble or reduced-rank representation of the covariance. One classic algorithm to perform this operation consists of taking the Schur product of the ensemble covariance matrix by a local support correlation matrix. However, with this parameterization, the square root of the forecast error covariance matrix is no more directly available, so that any observational update algorithm requiring this square root must include an additional step to compute local square roots from the Schur product. This computation generates an additional numerical cost or produces high-rank square roots, which may deprive the observational update from its original efficiency. In this paper, it is shown how efficient local square root parameterizations can be obtained, for use with a specific square root formulation (i.e., eigenbasis algorithm) of the observational update. Comparisons with the classic algorithm are provided, mainly in terms of consistency, accuracy, and computational complexity. As an application, the resulting parameterization is used to estimate maps of dynamic topography characterizing a basin-scale ocean turbulent flow. Even with this moderate-sized system (a 2200-km-wide square basin with 100-km-wide mesoscale eddies), it is observed that more than 1000 ensemble members are necessary to faithfully represent the global correlation patterns, and that a local parameterization is needed to produce correct covariances with moderate-sized ensembles. Comparisons with the exact solution show that the use of local square roots is able to improve the accuracy of the updated ensemble mean and the consistency of the updated ensemble variance. With the eigenbasis algorithm, optimal adaptive estimates of scaling factors for the forecast and observation error covariance matrix can also be obtained locally at negligible additional numerical cost. Finally, a comparison of the overall computational cost illustrates the decisive advantage that efficient local square root parameterizations may have to deal simultaneously with a larger number of observations and avoid data thinning as much as possible.

* Current affiliation: Mercator-Ocean, Toulouse, France

Corresponding author address: Jean-Michel Brankart, LEGI/CNRS, BP53X, 38041 Grenoble CEDEX, France. Email: jean-michel.brankart@hmg.inpg.fr

1. Introduction

A common difficulty with ensemble Kalman filters is to obtain the required ensemble size for representing weak correlations with sufficient accuracy. Small-sized ensembles systematically overestimate the information that observations contain about the system, and give undue confidence to suboptimal posterior estimates (ensemble collapse). In realistic atmospheric or oceanic applications of the ensemble Kalman filter, the horizontal dimensions of the system are generally much larger than the correlation length scales, so that most elements of the correlation matrix (coupling distant state variables) are close to zero, and the required rank for the error covariance matrices can be significantly larger than any affordable ensemble size. To circumvent this difficulty, the usual solution (Houtekamer and Mitchell 1998, 2001; Anderson 2003; Brusdal et al. 2003; Brankart et al. 2003; Testut et al. 2003; Ott et al. 2004; Szunyogh et al. 2005; Corazza et al. 2007; Hunt et al. 2007; Bishop and Hodyss 2009; Yang et al. 2009) is to add the prior information that long range correlations are negligible and force them to zero in the forecast ensemble covariance matrix. In that way, the rank of the matrix is artificially increased, and any spurious long-range influence of the observations is removed. To produce such local parameterizations of the error covariances, the accepted reference algorithm (proposed by Houtekamer and Mitchell 2001) modifies the ensemble covariance matrix by a Schur product with a local support correlation matrix (this method is commonly referred to as localization of the ensemble covariance). It can be shown indeed that this preserves the positive definiteness of the original matrix, and that it is appropriate for a valid and well-conditioned observational update (see Houtekamer and Mitchell 2001; Hamill et al. 2001, for more details).

However, with this parameterization, the square root of the forecast error covariance matrix is no more directly available, so that any observational update algorithm requiring this square root must include an additional step to compute local square roots from the Schur product. This computation generates an additional numerical cost or produces high-rank square roots, which may deprive the observational update from its remarkable efficiency. Algorithms taking benefit from the direct availability of a low-rank square root have been initially developed for square root Kalman filters (Verlaan and Heemink 1997; Pham et al. 1998; Lermusiaux and Robinson 1999), and have also been introduced in ensemble Kalman filters (Bishop et al. 2001; Heemink et al. 2001; Tippett et al. 2003, see also section 2b). The chief benefit that can be obtained from these algorithms is that the computational complexity of the observational update can be made linear in the number of observations. This is very useful to reduce the need for data thinning and allow the optimal analysis of a larger number of observations (see e.g., Brankart et al. 2009, who also show how this advantage can be preserved in presence of observation error correlations). In addition, in a recent study, Brankart et al. (2010) have shown that a variant of these algorithms (the eigenbasis observational update) can produce an optimal adaptive estimate (optimal in the sense of Dee 1995) of some important statistical parameters at negligible numerical cost. With this variant, it is also directly possible to avoid adding random perturbations to the observations (a virtue that is shared by the serial processing algorithm of Whitaker et al. 2008). There is thus a significant benefit to expect if these properties of the eigenbasis observational update could be preserved with localized error covariance matrices. Proposing possible answers to this question is the main objective of the present paper.

One possible way of computing the required local square roots is the direct factorization of the Schur product (as recently proposed by Bishop and Hodyss 2009). In this way, the computation of the local square roots is very efficient, but high-rank square roots are produced, which may be detrimental to the efficiency of the eigenbasis algorithm (especially with adaptive statistics). On the other hand, local implementations of algorithms working with local square roots have already been developed [for the singular evolutive extended Kalman filter in Brankart et al. (2003); Testut et al. (2003), or for the ensemble transform Kalman filter in Ott et al. (2004); Hunt et al. (2007); Yang et al. (2009)], but they do not rely on the Schur product by a correlation matrix to localize the ensemble covariances. By introducing low-rank local square roots producing the same effect as the Schur product, a subsidiary result of our paper (see section 3b) is also to suggest a possible link between these two approaches.

In section 2, we start by describing how the eigenbasis observational update algorithm (linear in the number of observations, with costless optimal adaptive parameter estimates) can be used to perform ensemble observational update (as an alternative to the original algorithm of the ensemble Kalman filter), and we propose a comparison of their respective merits, mainly in terms of computational complexity. In section 3, we present compatible local square root parameterizations of the forecast error covariance matrix, as an approximation to the parameterization of Houtekamer and Mitchell (2001). An academic example is used to illustrate the effect of the various schemes, which are also compared in terms of computational complexity. And finally, in section 4, we provide a more realistic application of the scheme by analyzing synthetic altimetric observations resulting from a long-term (100 yr) model simulation of a basin-scale ocean turbulent flow. We study in particular why local parameterizations are needed (in terms of accuracy and efficiency), how accurately the exact solution can be approached, and how local adaptive parameter estimates can be obtained.

2. Efficient ensemble observational update

a. Classic ensemble observational update

In the ensemble Kalman filter (Evensen 1994; Evensen and van Leeuwen 1996), the time propagation of the probability distribution for the state of the system is approximated by an ensemble model forecast, which is meant to provide a random sample of the system probability distribution at any future time. Each time that a new observation vector y is available, this probability distribution must be updated to take into account the new observational information. For that purpose, Gaussianity is assumed and the ensemble forecast is updated so that its mean and covariance are consistent with the result given by the standard Kalman filter observational update formula. Thus, if the ensemble forecast is noted xif, i = 1, … , m (where m is the size of the ensemble) with mean and covariance:
i1520-0493-139-2-474-e1
then the updated ensemble xia, i = 1, … , m must be characterized by a mean and a covariance given by the formulas:
i1520-0493-139-2-474-e2
i1520-0493-139-2-474-e3
or, equivalently
i1520-0493-139-2-474-e4
where 𝗛 is the observation operator, 𝗥 is the observation error covariance matrix, and 𝗞 is the Kalman gain:
i1520-0493-139-2-474-e5
In the classic formulation of the ensemble observational update (Houtekamer and Mitchell 1998; Burgers et al. 1998), this objective is achieved by defining an ensemble of observation vectors:
i1520-0493-139-2-474-e6
where ϵi is a random Gaussian noise with zero mean and covariance 𝗥, and by updating each ensemble member using the classic Kalman formula:
i1520-0493-139-2-474-e7
It can be easily verified (see Burgers et al. 1998, for more detail) that this elegant solution provides an ensemble with the right mean in (2) and covariance in (3), as soon as the finite random sample ϵi, i = 1, … , m can be assumed to have a zero mean and a covariance 𝗥:
i1520-0493-139-2-474-e8
This condition is asymptotically realized for m → ∞, and the misfit is decreasing with m as 1/.

b. Eigenbasis ensemble observational update

In the ensemble Kalman filter, the forecast error covariance matrix defined by (1) can be directly written in square root form:
i1520-0493-139-2-474-e9
by defining each column1 S(i)f of 𝗦f as S(i)f = δxif/, so that the observational update can be reformulated using the eigenbasis square root algorithm, as described in Brankart et al. (2010). This requires first to compute (i) the eigenbasis decomposition of the matrix Γ (m × m) defined as
i1520-0493-139-2-474-e10
where 𝗨 (unitary matrix) and Λ (diagonal matrix) are the matrices containing the eigenvectors and the inverse eigenvalues of Γ, and (ii) the projection (with metric 𝗥−1) of the ensemble of innovation vectors on each column of 𝗦f, which defines the vectors δi(m):
i1520-0493-139-2-474-e11
Then, we can define the transformed state and observation vectors ξi and ηi (with dimension m) using the linear transformations:
i1520-0493-139-2-474-e12
Finding the updated member ξia (as a correction to ξif = 0) amounts to solving the observational update problem in a new control space, defined by the linear transformation in (12). [For more details refer to Brankart et al. (2010), who describe the same transformation for a square root Kalman filter.] The transformation is useful because, in this new basis, the transformed forecast state is ξif = 0 with error covariance 𝗜 (identity matrix) and the transformed observation is ηi with error covariance Λ (diagonal matrix). The update of the m components of ξi thus become independent, and the updated ensemble can be obtained as
i1520-0493-139-2-474-e13
from which it is easy to compute the ensemble in the original space using transformation (12):
i1520-0493-139-2-474-e14
This updated ensemble computed using (10)(14) is mathematically exactly equal to the solution computed using the classic equations in (5) and (7). The decision to use one of these two algorithms thus only depends on their relative computational complexity (see section 2d). In addition, transforming the state and observation vectors using (12) is equivalent to what is done in the ensemble transform Kalman filter (Bishop et al. 2001) to perform the observational update, so that all methods discussed in the present paper are also directly applicable in that framework.

c. Modified eigenbasis algorithm

However, if we explore a little further, we observe that the updated ensemble (14) can be rewritten as the sum of the mean:
i1520-0493-139-2-474-e15
and an ensemble of anomalies δxia = xiaxa with respect to the mean:
i1520-0493-139-2-474-e16
which can be rewritten as
i1520-0493-139-2-474-e17
i1520-0493-139-2-474-e18
The mean of the ensemble corresponds to (2) as soon as ϵ → 0 and the anomalies δxia can be interpreted as originating from the forecast anomalies δxif = 𝗦if, which are first transformed to the eigenbasis (by the factor 𝗨), then scaled down by the factor (𝗜 + Λ)−1Λ, and finally transformed back to the original basis (by the factor 𝗨T). However, this scaling down of the anomalies, corresponding exactly to the first term of (3), is too strong to correctly simulate the error reduction resulting from the observations. This is why the observation noise ϵi is added, in order to simulate the additional ensemble dispersion that is needed for a consistent representation of the second term of (3).
Now, we can see that, provided that we apply the eigenbasis transformation in (12), it is possible to avoid adding ϵi by directly writing the updated anomalies as
i1520-0493-139-2-474-e19
i1520-0493-139-2-474-e20
The square root scaling in (20) can be verified (see e.g., Brankart et al. 2010) to be exactly what is needed to provide the updated covariance matrix given by (4), and the easiest way of obtaining (19) is to come back from (20), by using the same path that was followed (in the reverse sense) to obtain (18) from (17). Consequently, we obtain a (slightly) modified eigenbasis algorithm for the ensemble observational update that fits our initial requirements (right mean and covariance), that does not require adding a random noise to the observations, and that is structurally very similar to the classic solution (18). From an algorithmic point of view, the modification reduces to applying (10), (11), (13), and (14) to the mean only (by replacing yi and xif by y and xf), and use (20) to compute the updated ensemble of anomalies. [This modified eigenbasis algorithm can be viewed as a particular application of the general property of square root algorithms to allow ensemble data assimilation without perturbed observations, as already demonstrated by Whitaker and Hamill (2002).]

In this framework, it is also easy to evaluate the relative weight of the two terms in (18), and thus in (3), in the dispersion of the updated ensemble. Along any principal direction of Γ, with eigenvalue λ, the fraction of the updated variance that is due to the first term of (18) can be computed as the ratio between this first term and the total updated variance given by (20); it is equal to f (λ) = λ1/2/(1 + λ)1/2, and the remaining part of the work is performed by the second term (i.e., the random noise perturbating the observations). Consequently, for a very large ratio between observation and forecast error variances [λ → ∞, f (λ) → 1], most of the updated variance is due to the first term in (18), and the random noise ϵi plays a negligible role. Conversely, for a very small ratio (λ → 0), the weight of the first term f (λ) → 0, so that most of the dispersion results from the perturbations ϵi of the observations. The two terms play equal roles for λ = ⅓, since f (⅓) = ½. Thus, for λ ≪ 1, it is of primary importance for the validity of ensembles (7) or (14) that condition (8) be accurately verified, since the final dispersion largely depends on these perturbations. In this case, it is certainly safer to directly compute a consistent ensemble of anomalies using (20).

d. Computational complexity

The computational complexity of the observational update algorithms described in sections 2a and 2b depends on the dimensions of the problem that can be described by the size x of the state vector, the size y of the observation vector, and the size m of the ensemble. In this discussion, we only consider the leading behavior of the computational complexity for large size problems (i.e., large x, y, and m), so that the complexity of the classic algorithm (section 2a) only results from the application of (5) and (7):
i1520-0493-139-2-474-e21
The first term corresponds to the inversion of the symmetric matrix 𝗛𝗣f𝗛T + 𝗥, the second term includes the computation of the matrix 𝗛𝗣f𝗛T from the ensemble and the application of the inverse matrix to the ensemble of innovations, and the third term corresponds to the left multiplication by (𝗛𝗣f)T.
In practical applications of the ensemble Kalman filter involving a large number of observations, the numerical cost C0 associated to the classic algorithm is often strongly reduced by a serial processing of the observations: the ensemble forecast is updated sequentially by assimilating the observations one by one. In this way, the computational complexity of the algorithm becomes linear in the number of observations:
i1520-0493-139-2-474-e22
This complexity formula corresponds to the algorithm described in Whitaker et al. (2008), who also include a square root formalism avoiding the perturbation of the observations. In this algorithm, the dominant numerical cost corresponds to the update of all ensemble members (size mx) for all observations (which gives a first mxy term), and to the computation of the background covariance from the ensemble (which gives another mxy term). The main drawback of this algorithm is that the observation error covariance matrix 𝗥 must be assumed diagonal, but, as in the eigenbasis algorithm, this drawback can be circumvented using the augmented observation vector approach proposed by Brankart et al. (2009): with a numerical cost C0serial linear in the number y of observations, the serial processing algorithm can also afford increasing y to simulate a nondiagonal 𝗥.
To describe the computational complexity of the eigenbasis algorithm (section 2b), we must introduce the rank r of the ensemble covariance matrix (which may be smaller than the ensemble size: rm), since only the nonzero eigenvalues need to be considered in the computation of the observational update. The leading behavior of the computational complexity, which corresponds to (10), (11), and (14), can then be written as
i1520-0493-139-2-474-e23
The first term corresponds to the computation of Γ and δi from the ensemble (with a diagonal 𝗥 matrix); the second term, to the computation of the eigenvalues Λ−1 and eigenvectors 𝗨 of Γ; and the third term, to the left multiplication by 𝗦f to obtain the ensemble of corrections. [The factor α depends on the algorithm that is used in practice; it is for instance equal to 12 for the singular value decomposition (SVD) algorithm.] The computational complexity C1 in (23) stands for the eigenbasis algorithm as described in section 2b, but a similar scaling of the numerical cost can be obtained with other formulations of the algorithm like the ensemble transform Kalman filter (initially introduced by Bishop et al. 2001). The key advantage of these transformed algorithms with respect to the classic algorithm is that the computational complexity C1 is linear in the number of observations. It is thus particularly useful if the number y of observations is much larger than the ensemble size m, which is the most common situation in atmospheric or oceanic applications of the ensemble Kalman filter. Moreover, as shown in Brankart et al. (2009), this numerical efficiency (linearity in y) can be preserved in presence of observation error correlations, by augmenting the observation vector with new observations that are linear combinations of the original observations (e.g., as gradients). In that way, large classes of correlation structures can be efficiently parameterized.
Concerning the modified eigenbasis algorithm (section 2c), its computational complexity is even smaller than C1:
i1520-0493-139-2-474-e24
because (11) must only be computed for the mean innovation y − 𝗛xf, and not for the ensemble of innovations yi − 𝗛xif (which is only done for the sake of applying the Kalman gain to the ensemble of observational perturbations ϵi).

e. Properties of the eigenbasis algorithm

To conclude this section, we now summarize the list of reasons why, in many situations, the eigenbasis square root algorithm may be a useful solution to perform the observational update.

  • The first reason is that it is very efficient for large-sized observation vectors, which are quickly prohibited in the classic algorithm by the cubic term (in y) in the complexity in (21). This efficiency can be preserved with a nondiagonal observation error covariance matrix using the method presented in Brankart et al. (2009).

  • Second, with the eigenbasis algorithm, it becomes possible to compute, at negligible additional cost, optimal adaptive estimates of several important statistical parameters, including scaling factors for the forecast error covariance matrix, the observation error covariance matrix, or for the observation error correlation length scale (see Brankart et al. 2010 for more details). This computation can only be very expensive with the classic formulation of the ensemble observational update.

  • Third, the use of the linear transformation in (12) transporting the observational update in a new space, in which forecast and observation error covariance matrices become diagonal, makes the algorithm very intuitive and easy to understand. Equation (13) is then simply the Gauss formula applied to the independent components (with a zero forecast and observations ηi, resulting from the projection of the original observations onto every eigenvector). With (20), transporting the anomalies in the eigenspace, then scaling them down, before transporting them back in the original space, we can be sure that the structure of the forecast ensemble is as carefully preserved as it can be during the observational update. This property is lost as soon as a random noise is added to the observations, since this noise can play the major role (as soon as λ ≪ 1) in the dispersion of the updated ensemble (see example in section 3b). In addition, the algorithm also computes the spectrum of the ensemble Λ (with metric 𝗥−1), which gives the opportunity of diagnosing a possible alignment of the ensemble along a smaller dimension manifold (ensemble degeneracy).

  • The fourth reason is that the eigenbasis algorithm is as easy to implement as the classic algorithm. Equations (10), (11), (14), and (20) are all straightforward linear algebra operations, with easily interpretable intermediate results, which leaves little space for coding errors.

  • And finally, the eigenbasis algorithm is interesting because it can be applied, unchanged, to a wider range of Kalman filters, including not only ensemble Kalman filters but also the square root Kalman filters [using (20) without factor to update the square root of the forecast error covariance matrix]. The only condition is that the forecast error covariance matrix is available in the square root form of (9).

Nevertheless, there is at least one important thing that it is not straightforward to do with the eigenbasis algorithm: to apply the classic method for a local parameterization of the error covariances (i.e., the Schur product by a correlation matrix). But the potential benefits listed above are too significant to be abandoned, and it is the purpose of the next section to propose a local square root error parameterization, which preserves the above properties of the eigenbasis algorithm.

3. Reduced-rank local error parameterizations

a. Classic local parameterization

The accepted reference method to obtain a local parameterization of the error covariances (proposed by Houtekamer and Mitchell 2001) consists in modifying the ensemble forecast error covariance matrix 𝗣f by taking the Schur product with a local support correlation matrix 𝗖:
i1520-0493-139-2-474-e25
where is the resulting local parameterization of the forecast error covariance matrix. The basic reason for choosing a parameterization like (25) is that it guarantees that the matrix remains positive definite, which is a prerequisite for a valid and well-conditioned observational update (see Houtekamer and Mitchell 2001; Hamill et al. 2001, for more detail about this).

In practice, the correlation matrix 𝗖 is usually designed in such a way that correlation only depends on the horizontal distance between state variables, decreasing from 1 (for variables at the same horizontal location) to 0 (for a horizontal distance larger than a cutting length ℓ), according to a correlation function γ(r). With such parameterization of , the observational update of any state variable can only depend on the observations that are closer than the distance ℓ, and can be performed separately for every node i = 1, … , n of the model horizontal grid using the local submatrix (including only the variables within a distance ℓ from node i). Since the computational complexity in (21) of this algorithm is cubic in the number of observations, it is indeed cheaper to perform a large number of observational update with a small subset of the observations, than only one observational update with the full observation vector.

To analyze the behavior of this local parameterization, it is useful to give an academic example, which is further used in the next section to show the effect of our reduced rank approximation. We consider a one-dimensional random signal on the interval [0, 1], with zero mean, unitary variance and a homogeneous and isotropic correlation function: γs(r) = exp(−r2/ℓs2), with ℓs = 0.01. This signal is discretized on the grid xi = iδx, i = 1, … , n, with δx = 0.001 and n = 1000. The signal covariance is approximated by a 200-member ensemble, randomly sampled from a Gaussian distribution with the above-mentioned mean and covariance. Figure 1 (top-left panel) shows the ensemble covariance with respect to the reference location xref = 0.5, as compared to the exact covariance given by γs(xxref). The figure shows that the ensemble correctly represents the short-range covariances, while the long-range covariances do not approach zero as in the exact function. The difficulty to accurately represent these long-range correlations with a limited size ensemble, and the important cumulated impact that they can have on the observational update are the two reasons why a local error parameterization is so important in many ensemble Kalman filter applications. The figure also shows the local parameterization in (25), using the correlation function γ(r) = exp(−r2/ℓ2), with ℓ = 3ℓs. This parameterization provides a much better approximation of the exact covariance function than the original ensemble covariance, since the spurious long-range correlations are removed.

The observational update of this ensemble is then performed using observations yi, i = 1, … , y at locations xi = [i/(y + 1)]Δx, with Δx = 20δx and y = 50. These observations are sampled from a synthetic true signal (also drawn from the above mentioned probability distribution) and perturbed by white noise (with a standard deviation of 0.5) simulating observation errors. Figure 1 shows the resulting updated ensemble, as described by the ensemble mean (top-right panel), the ensemble standard deviation (bottom-left panel), and the ensemble covariance with respect to the reference location xref = 0.5 (bottom-right panel). Two solutions are represented: one is computed with the original ensemble with covariance 𝗣f, without local parameterization, and the other is computed using the local parameterization given by (25), with the correlation function γ(r) given above. Both are compared to the exact solution, which is computed using the exact global covariance of the random signal given by γs(r). This result shows that, in this case study, and with that kind of ensemble size (m = 200), a local parameterization like (25) is needed to obtain a correct approximation of the exact solution. With the original ensemble covariance matrix 𝗣f, the spurious influence of the numerous distant observations (each one with a representer like in the top-left panel of Fig. 1, solid line) is sufficient to produce a solution that is largely incorrect. Another way of understanding the same thing is to observe that the exact covariance matrix (defined by γs) is characterized by a very wide spectrum, with very few negligible eigenvalues, which is difficult to approximate accurately by a limited size ensemble. This is why the rank of 𝗣f must be artificially increased by a Schur product with a local support 𝗖 matrix. This is justified here by exploiting the additional prior information that the long-range correlations are truly negligible. Fortunately, the same assumption is very reasonable in many atmospheric and oceanic applications.

b. Square root local parameterizations

The problem with the parameterization in (25) is that the submatrices , i = 0, … , n corresponding to the global and local problems, become full rank matrices, which are not directly available in square root form, so that the eigenbasis algorithm (described in section 2b) could not be applied together with the classic local parameterization (described in section 3a) without recomputing local square roots:
i1520-0493-139-2-474-e26
where are the submatrices of 𝗣f, 𝗦f, and 𝗖 corresponding to each local problem and are the local square roots that we are looking for.

Moreover, remembering that the numerical cost of the eigenbasis algorithm quickly increases with the rank of these square roots (see discussion in sections 2d and 3c), we are also interested in obtaining approximate low-rank square roots , especially if they produce the same observational update as (25) at the central location of each subdomain. This is indeed the only part of each local observational update that is used to build the global observational update. Considering that we start with an ensemble square root parameterization of the forecast error covariance matrix 𝗣f, with a maximum rank equal to the ensemble size m, there must exist a method to obtain a consistent update at the central location of each subdomain without increasing the rank of the local error covariance matrices.

1) with explicit square root decomposition

To obtain an exact square root of , the direct solution is to explicitly compute the eigenvalue/eigenvector decomposition of this full-rank matrix. Low-rank approximations are then also immediately available by keeping the first few eigenvalues representing a given fraction of the total variance. Moreover, since we are only interested in representing the variance at the central location, we can do this operation with a metric giving a negligible importance to any other location, so that a few eigenvalues are usually sufficient for representing accurately the total variance. It is this property that the total variance at the central location can be explained by a limited number of eigenmodes (see example below) that justifies the use of low rank approximate square roots.

(i) Rank reduction accuracy

For instance, if we consider the example given in section 3a (in Fig. 1), and focus on the observational update for the location xref = 0.5 (i.e., i = 500), we can study the rank of the local square roots that is needed to obtain a given accuracy for the total variance at this location (which is directly related to the accuracy of the observational update at this point). For this location, the submatrix only covers the subdomain between i = 481 and i = 519 (i.e., within a distance ℓ = 20 from the central location i = 500). Figure 2 (left panel) compares the eigenvalue spectra of that are obtained (i) with an identity metric (giving the same importance to every location between i = 481 and i = 519) and (ii) with a metric giving a significant importance to the location i = 500 only. From this result we can observe that we need up to 200 eigenvalues to represent 99% of the total variance over the full subdomain, but that only 1 eigenvalue is sufficient to represent the full variance at the central location (because there is only 1 state variable there, in this simple example). Figure 2 (right panel) shows, in both cases, the percentage of variance that is resolved as a function of the position in the subdomain. In the second case, we see that we miss an important part of the total signal variance in the most external regions of the subdomain. This unresolved variance must be interpreted as an additional representativity error corresponding to the part of the total signal that has been truncated from the local forecast error covariance matrix (because it only projects on the eigenmodes with a zero signature at the central location of the subdomain). Thus, to perform a consistent local observational update using the truncated local square root of (with a rank reduced to r = 1), the covariance of this additional representativity error must be included in the local observation error covariance matrices. The resulting solution is presented in Fig. 3 showing that the difference with respect to the classic local parameterization (solid line) is smooth on the horizontal and very small with respect to the intrinsic error inherent to the classic local parameterization (dotted line).

(ii) Horizontal coherence of the solution

Since the observational updates at every horizontal locations are computed separately, it is important to check that the algorithm preserves the horizontal coherence of every ensemble members. With perturbation of the observations (algorithm of sections 2a and 2b), this coherence is obvious since every member result from a separate observational update, with a gain 𝗞 that results from statistics that are consistent over the global domain. This is thus also true for the updated ensemble anomalies in (18), which correspond to the same solution. Without perturbation of the observations [i.e., using (20) to update the anomalies], the coherence results from the particular structure of the eigenbasis algorithm, which updates all ensemble anomalies δxif by a single operator 𝗔1 that is structurally very similar to operator 𝗔0 in (17). It is important to notice that this property requires the forward and backward transformations in (20). For instance, the last factor 𝗨T in (20), interpreted in section 2b as transforming the updated anomalies back into the original basis, can in principle be omitted for a global observational update of ensemble anomalies; this would make a perfectly valid alternative algorithm since the unitary transformation does not change the updated covariance. However, it cannot be omitted for a local observational update since it is necessary to write the updated anomalies in the form of (19) (i.e., a single operator 𝗔1 applied to all ensemble anomalies). Going back to the original basis is thus the key element that ensures the horizontal consistency of the adjacent local observational updates [as in (17) or (18), which are equivalent to the classic algorithm].

(iii) Non-Gaussian and large-scale effects

With random perturbations of the observations, the local observational update adds a random Gaussian signal (with local support covariance) to every member of the ensemble. This is the term 𝗞(ϵiϵ) in (18). As soon as conditions in (8) are verified, this contribution is exactly what is required to produce an updated ensemble with the right mean and covariance, but the detailed structure of the forecast ensemble can be deeply modified. This makes no difference if the forecast ensemble is truly Gaussian and if large-scale patterns are really meaningless, but, in many situations, it can be highly beneficial to preserve as much as possible the asymmetries or large-scale structures of the forecast ensemble. It is thus an interesting property of (19) to have a unique linear operator 𝗔1 applied to all anomalies, and giving directly the right updated covariance, without this additional local Gaussian signal. Equivalently, with (20), the algorithm transforms the forecast anomalies into the eigenbasis of the observational update (with unitary transformation 𝗨), so that each component can be directly scaled with the right square root factor (1 + λ)1/2λ1/2. Even if the relative weight of every component can be modified by this operation, transforming them back into the original basis (with transformation 𝗨T) can restore as much as possible of the global structure of the ensemble forecast.

2) With factorization of the Schur product

In a recent paper, Bishop and Hodyss (2009) propose computing the square root of the local submatrices by factorizing the Schur product using a mathematical property, already used by Lorenc (2003) and Buehner (2005) in a variational analysis system: if 𝗭i is a square root of the correlation matrix: , then (25) may be rewritten as
i1520-0493-139-2-474-e27
where each column of is computed as the Schur product of a column of with a column of Zi [see Bishop and Hodyss (2009), for more detail]. The number of columns r of the square root is thus equal to the ensemble size m (number of columns of ) times the rank rC of 𝗖i (number of columns of 𝗭i): r = mrC.

This algorithm is interesting because it provides a direct way of computing an exact square root of the local forecast error covariance matrices at a very low numerical cost. Moreover, as pointed out by Bishop and Hodyss (2009), the columns of the resulting square root may be interpreted as a modulation of the ensemble anomalies by the columns of 𝗭i, which provides a rational framework for truncating 𝗭i to its most significant components. However, even if the number of columns in 𝗭i is kept moderate (it is for instance equal to rC = 126 in the work of Bishop and Hodyss 2009), it comes as a factor of the ensemble size in the number of columns of . This may thus be a serious limitation to the efficiency of the eigenbasis algorithm, even if the cost of the factorization method can be significantly reduced (without approximation) by aggregating the update of several model grid points in each local update (see the discussion in section 3c).

3) With bulk parameterization

In the above algorithms, the explicit computation of the square roots generates an additional numerical cost or produces high-rank square roots, which may deprive the eigenbasis algorithm from its original efficiency (see section 3c) so that it is useful to look for more direct (but cruder) local parameterizations. For that purpose, we assume that the decrease of the useful variance as a function of the distance with respect to the central location (i.e., observed in the above-described reduced-rank local parameterization, and illustrated in Fig. 2, right panel) can be simulated by an equivalent effect on every member of the forecast ensemble, so that we can directly write:
i1520-0493-139-2-474-e28
where 𝗪S,i is a diagonal matrix with the fraction of the variance that must be represented at every grid point of the local subdomain (i.e., decreasing from 1 at the central location to 0 at the boundaries). On the other hand, to be consistent, the covariance that we miss with the parameterization in (28) corresponds to an additional representativity error (resulting from the truncation of the local covariance matrices) and must be added to the observation error covariance matrix:
i1520-0493-139-2-474-e29
It is also possible to keep the 𝗥 matrix diagonal using the following approximation, which preserves the total variance: diag(𝗛𝗣f𝗛T + 𝗥):
i1520-0493-139-2-474-e30
Rescaling the observation error variances as a function of the distance to the central location is also the technique that is used by Brankart et al. (2003) or Hunt et al. (2007) to avoid discontinuities in the local observational update. We suggest here that, providing that two scaling factors 𝗪S,i (for the forecast error covariance) and 𝗪R,i (for the observation error covariance) are introduced and consistently defined, this rescaling of the variances can be viewed as a bulk parameterization, producing the same effect as the localization of the ensemble error covariance. On the other hand, this bulk parameterization also corresponds to the factorization method of Bishop and Hodyss (2009) when the correlation matrix 𝗖 is truncated to one single eigenmode. Again, if this rank reduction of 𝗖 is performed with a metric giving a significant importance to the central location only, one eigenmode is always sufficient for the to represent the full ensemble covariance at the central location, and the corresponding first eigenmode of 𝗖 provides the single modulation of the ensemble anomalies that is appropriate to simulate the localization of the ensemble covariance by the correlation matrix 𝗖. In this case, the observational update at the central location is indeed equivalent to that obtained with (25) providing that the observation error covariance matrix is consistently modified according to (29), and only approximate if the diagonal approximation of (30) is applied (as in the following examples).
(i) Accuracy of the statistics

This bulk parameterization is now applied to our academic example (illustrated in Figs. 1 and 2). As an approximation for 𝗪S,i, we use here the variance reduction factor that is shown in Fig. 2 (right panel). This corresponds to rescaling the local forecast error covariance matrix with the same effect on the variance as the explicit subtraction of the covariance of the part of the signal that is independent from what happens at the central location of the subdomain (as done in the previous subsection). Figure 3 (dashed line) shows the relative error with respect to parameterization (25) for the statistics that are shown in Fig. 1 [i.e., the forecast error covariance with respect to the reference location (top-left panel), the mean (top-right panel) and standard deviation (bottom-left panel) of the updated ensemble, and the updated covariance with respect to the reference location (bottom-right panel)]. This error is compared to the error that is made with the explicit square root decomposition of section 1. The results show that the bulk parameterization in (28) produces a significant difference with respect to the parameterization in (25). However, this difference has the same order of magnitude as the error associated to the classic local parameterization (dotted line), so that the real accuracy of the solution is not altered by the approximation.

(ii) Discontinuity problem

To avoid problems, it is important to realize that further simplifying the algorithm using identity 𝗪S,i matrices is not a valid and well-conditioned solution. Indeed, this exactly corresponds to leaving the global ensemble error covariance unchanged for every local analysis, but keeping only the observations that are closer than the cutting length scale ℓ. It is thus also equivalent to a parameterization like (25) with a matrix 𝗖 equal to 1 for variables closer than ℓ and equal to 0 otherwise (which is thus not a valid correlation matrix anymore since it is not positive definite). This solution is equivalent to performing a global observational update with a modified forecast error “covariance” matrix , which is no more positive definite, with the dangerous consequence that the eigenmodes with negative eigenvalues can be artificially amplified for the purpose of fitting the discrete observations [see Houtekamer and Mitchell (2001) for more details]. The result of this phenomenon is usually the presence of discontinuities in the solution: grid-scale structures are triggered although they are never assumed to represent a significant part of the forecast error variance.

On the other hand, the solution obtained with bulk parameterization in (28) with appropriate 𝗪S,i is free of this discontinuity problem and very close to the reference solution. But it is important to notice that this bulk parameterization does not correspond anymore to any global least squares problem: it is impossible to identify global forecast and observation error covariance matrices that are mathematically equivalent to using parameterizations (28) and (30) for the local problems. The global problem in (25) is here replaced by a series of local problems, in which the effect of the Schur product by a correlation matrix is simulated by the simplified bulk equations in (28) and (30). The discussion and the example show that the resulting algorithmic shortcut produces accurate estimates for the state of the system and the associated error covariance (Fig. 3), and that it is sufficient to avoid any spurious unstable behavior.

c. Computational complexity

The computational complexity of applying the reference local parameterization in (25) with the classic observational update algorithm (described in section 2a) can often be made smaller than the complexity C0 in (21) of the global observational update by solving the problem separately for every horizontal location of the model domain and keeping only the observations that are closer that the cutting length scale ℓ. For large-sized problems, the leading behavior of the complexity can be written as
i1520-0493-139-2-474-e31
where z is the number of local problems, y is the number of observations that are used in the corresponding local problems, x is the number of state variables to update, and y2, y3, xy are averages of these quantities over the set of local problems. With respect to C0, the third term is kept global since it is computationally more efficient to compute the elements of the matrix 𝗛𝗣f𝗛T + 𝗥 only once for every pair of observations.
With localization of the forecast error covariance matrix, the serial processing algorithm becomes somewhat approximate (since the forecast error covariance matrix is no more equal to the ensemble covariance 𝗣f), but, for large y, its numerical cost is still much smaller than the classic algorithm:
i1520-0493-139-2-474-e32
where X is the average number of state variables that are influenced by each observation. It is still linear in the number of observations, with a coefficient reduced from 2mx to 2mX, and there is no significant additional cost associated to the localization of the covariance.
To use the parameterization in (25) with the eigenbasis observational update algorithm (described in section 2b), one possible option is to perform an explicit square root decomposition in (26) of all submatrices (as explained in section 1). The size X of each of these submatrices typically includes all state variables that are closer than the cutting length scale. With these explicit square root decompositions, the computational complexity of the local eigenbasis algorithm [without perturbation of the observations, as in (24)] can be written as
i1520-0493-139-2-474-e33
where rm is the reduced rank of the local ensemble covariance matrices and yr2, r3, X3 and xr are still averages over the set of local problems. It is interesting to point out that the factor β can be as small as ⅙ if a Cholesky decomposition is performed to obtain the square root decomposition in (26), but then it is impossible to seek a rank reduction for these local submatrices so that we must set r = X in (33). In any case, even with explicit square root decomposition, the computational complexity of the eigenbasis algorithm remains linear in the number of observations, so that a large number of observations can be explicitly taken into account without prohibitive consequence on the numerical cost.
If the local square roots are computed by factorization of the Schur product [as proposed by Bishop and Hodyss (2009)], the costs associated to the computation of the local submatrices from the ensemble (zmX2) and to their explicit square root decomposition (X3) disappear, to be replaced by the much smaller cost of the factorization (zmXrC):
i1520-0493-139-2-474-e34
where the number of columns r of the square root is here a multiple of the ensemble size: r = mrC. Since this number can usually be made much smaller than the size of the local state vector (rXl), this factorization method generally represents an important computational benefit with respect to explicit square root decompositions. Moreover, the numerical cost of the method can be further reduced by a factor k by aggregating the update of k model grid points in each local update [as suggested by Bishop and Hodyss (2009)].
Nevertheless, there exist practical applications for which the computation of exact local square roots remains penalizing, either because their explicit computation is too expensive (X3 in C1) or because the number of columns of the resulting square roots is too large (r in C1ℓ,fact). Then, the local error parameterization may be the right place to introduce an approximation, because much computational effort can be gained from approximate low-rank square roots, without a significant effect on the results (see example above). For instance, with the bulk parameterization in (28) of the local square root matrices, the computational complexity of the algorithm is reduced to
i1520-0493-139-2-474-e35
(The terms are simplified because, with this parameterization, r = m and zmxr = zm2x = m2x.) If the size becomes a problem, there is indeed no choice but simplifying the problem, either by approximating the local statistics [with the parameterization in (26) to reduce the complexity to C1ℓ,bulk], or by approximating the observation vector (data thinning) until C0 or C0ℓ,serial becomes affordable. The bulk parameterization in (28) can thus also be viewed as an alternative approximation to avoid suboptimal data thinning as much as possible, or conversely as a way of preserving approximately the “optimal data thinning” given by the transformation in (11), which defines the reduced observation vector δi as the projection (with metric 𝗥−1) of the full innovation vector on the ensemble anomalies.

To better illustrate why an approximation may sometimes be useful, it is interesting to compare the computational behavior of the various algorithms described above in a practical case study. The target application for which the methods described in this paper have been developed is the Mercator-Ocean operational system (Drévillon et al. 2008), which produces weekly operational forecasts of the global ocean at a ¼° horizontal resolution (a system is already running in preoperational mode). The number of model grid points is z ∼ 9 × 105 for the ¼° model or z ∼ 8.1 × 106 for the model. The number of control variables at each model grid points is x = 209 (i.e., 4 three-dimensional variables with 50 vertical levels, and 9 two-dimensional variables). With a cutting length scales ℓ of about 400 km, the size of the local state vector is X ∼ 1.6 × 105 for the ¼° model or X ∼ 2.5 × 106 for the model. The forecast error covariance matrix is constructed as the covariance of an ensemble of m = 250 anomalies (which are here obtained from one single model simulation, but this makes no difference regarding the computational complexity of the observational update).

With these settings and the bulk parameterization described above, we can afford including up to y ∼ 12 000 observations in each local observational update, for a numerical cost of about C1ℓ,bulk ∼ (1/2)zym2 operations. In practice, this cost is usually further reduced (by a factor of about 10 in the Mercator system) by aggregating the update of several model grid points in each local analysis [as already suggested by Bishop and Hodyss (2009), for the factorization method], but this reduction is not taken into account in the comparison below. The other algorithms presented above could also be applied to the Mercator system, but they would require other kinds of approximation to keep the same numerical cost. For instance, a reduction in the size of the observation vector by a factor of 12 (because y3 ∼ 1.5y) would be needed for the standard algorithm (with complexity C0), and by a factor of 3 (because Xyxy) for the serial processing algorithm (with complexity C0ℓ,serial). In the Mercator application, the large ensemble size penalizes the factorization method (with complexity C1ℓ,fact), which would require significant rank reductions, whereas explicit square root decompositions (with complexity C1) could not be afforded without decreasing the cutting length scale ℓ until X becomes much smaller. In conclusion, even if the relative cost of these algorithms may strongly depend on the size of every particular application, reducing the dependence of the computational complexity to the number of observations is certainly valuable for any practical application involving large observation datasets.

4. Demonstration experiments

To study the need and effects of the local parameterizations using a more realistic example, we now investigate the problem of updating a prior probability distribution for a basin-scale ocean turbulent flow using synthetic observations of the ocean dynamic topography.

a. Description of the experiments

Our reference basin-scale flow is a double-gyre circulation that is created in an idealized square (2200 × 2200 km2) and 3000-m-deep flat-bottom ocean by a constant zonal wind blowing westward in the southern and northern parts of the basin and eastward in the middle part of the basin. The anticylonic gyre (in the southern half of the basin) and the cyclonic gyre (in the northern half of the basin) are separated by an eastward jet in the middle of the basin that is fed by the two western boundary currents that result from the western intensification of the two gyres. This jet is unstable so that the flow is dominated by a chaotic mesoscale dynamics, with the largest eddies that are ∼100 km wide, and to which correspond velocities of ∼1 m s−1 and dynamic height differences of ∼1 m. This eddy field is responsible for most of the system variability, which is illustrated in Fig. 4 (left panel) by a snapshot of the dynamic height anomaly (with respect to the mean). All this is very similar in shape and magnitude to what is observed in the Gulf Stream (North Atlantic) or in the Kuroshio (North Pacific). This ideal system is simulated using the Nucleus for European Modelling of the Ocean (NEMO) numerical ocean model, with a horizontal resolution of ¼° × ¼° cosλ, and 11 levels on the vertical [see Cosme et al. (2009) for more details about the model configuration]. As in Brankart et al. (2010), we use the model time series from years 21 to 120 (sampled every 10 days), after statistical equilibrium is reached.

As synthetic observations, we use the model simulated dynamic height (Fig. 4, left panel) with a random perturbation (Fig. 4, right panel) to simulate observation errors. This observational noise is assumed Gaussian with zero mean homogeneous standard deviation σ = 0.05 m and exponential horizontal correlation function ρ(r) = exp(−r/ℓ). The horizontal correlation length ℓ is set to ℓ = 5 grid points, so that the error patterns (illustrated in Fig. 4, right panel) are similar in size to the real signal (Fig. 4, left panel). This exponential correlation model has the property that it can be parameterized using a diagonal observation error covariance matrix 𝗥 in an augmented observation space including the gradient of the original observations [see Brankart et al. (2009) for more details]. With this correlation model, the efficiency of the eigenbasis algorithm (in section 2b) is thus preserved without approximation.

As prior probability distribution, we use the equilibrium distribution for the state of the system. Assuming ergodicity, this distribution is approximated by an ensemble of model states sampled from a long-term phase trajectory (given here by the 100-yr model simulation mentioned above). Such simple definition of the prior distribution amounts to testing the observational update algorithm in absence of any knowledge of the past evolution of the system, but at least we are sure that the horizontal covariance structure is realistic and thus appropriate for checking the local error parameterizations described in this paper. Assuming Gaussianity, the mean and covariance of this ensemble are indeed the only statistical information that is exploited, and the justification for a local parameterization only depends on the difficulty to represent accurately this covariance structure with small size ensembles.

b. Sensitivity of correlations to rank deficiency

It is indeed a well-known property of weak correlations to require a large sample to be adequately identified. This is illustrated in Fig. 5 (left panel), showing dynamic height correlations as function of latitude (along the 15°E meridional section, with respect to a point located at the center of the jet at 34.5°N, 15°E), as estimated using samples of various sizes. The decrease of the sample size (from m = 5000 to 1000, 200 and 40) here corresponds to an increase of the time interval between the model snapshots that are used to estimate the correlations (from 5 days to 25, 125, and 625 days). In view of the time decorrelation of the mesoscale dynamic topography (about 10 days), it is only in the largest sample that the 5000 elements of the sample cannot be considered perfectly independent. Independence is an important condition for computing confidence intervals for the estimated correlations. These intervals (which narrow with increasing sample size) are known to be much smaller for large correlations coefficients. This behavior can also be observed in the figure, where the large correlation coefficients (which also happen to be the short-distance correlations) are correctly estimated whatever the size of the sample, while small correlation coefficients may remain inaccurate even with a 200-member ensemble (for which 0 correlation have the confidence interval [−0.14, 0.14,] at the 95% level). This is the well-known property justifying the application of local parameterizations in ensemble Kalman filters.

Less documented is the sensitivity of correlations to rank reduction by a principal component analysis of the available sample. This is illustrated in Fig. 5 (right panel), showing dynamic height correlation (the same as in the left panel) as estimated using the largest sample (5000 members), but keeping only a few principal components (r = 10, 20, or 80) of the covariance matrix. This result shows that obtaining a given accuracy for the correlation estimates requires less principal components than ensemble members. And, at least in this energetic region of the model domain, 20 principal components and 200 members produce the same level of accuracy. However, the main behavior is identical: the small long-range correlations are the first to be penalized by rank reduction and a local parameterization is needed as long as a high-rank global parameterization cannot be afforded.

c. Effect of local parameterizations on observation influence

The most direct way of analyzing the effect of local parameterizations is by computing the observational update resulting from one perfect observation at some location of the model domain. Extrapolation is indeed the situation for which spurious long-distance effects are most penalizing. Figure 6 (top-left panel) shows the result corresponding to one dynamic height observation located at 34.37°N, 25.18°E (in a region with relatively small signal variance), as computed with the largest sample size (m = 5000) and without rank reduction (r = 5000). Most of the observation effect is local, with a clear negative lobe west of the positive maximum, and three other moderate negative lobes in the eastern, southern, and northern directions, respectively. And except for secondary lobes farther to the west, the observations only receives a weak influence in the rest of the domain.

If the size of the sample is reduced to m = 200 (Fig. 6, top-middle panel) or the rank is reduced to r = 20 (Fig. 6, top-right panel, pay attention to the different color bar), the immediate consequence is the appearance of a very important spurious long-range influence of the observation. This spurious effect is mainly due to the misrepresentation of the small long-range correlations, even if it is here strongly amplified by the very large ratio between the signal variance in the western part of the basin (along the main jet) and at the location of the observation. (The observational update is indeed equal to the correlation times this ratio.) With the rank reduction to the r = 20 first principal components (top-right panel), these distant effects are even further amplified (by a factor of ∼4) as a result of the nonuniform underestimation of the model variance over the model domain. Principal components are indeed computed to keep most of the total signal variance, which means prioritizing the signal in the most energetic regions to the prejudice of the least energetic regions. This phenomenon leads to an artificial increase of the variance ratio (which is absent in the ensemble approximation, with m = 200), which amplifies the long-range influence of the observations.

To remove this spurious long-range influence, we apply a local parameterization as described in section 3 with the cutting length scale ℓ equal to 30 grid points. Figure 6 (bottom-left panel) shows the result obtained using the bulk parameterization of section 3 with WS,i = I(∀i). The result is exactly identical to the global solution (Fig. 6, top-middle panel), except that the correction is set to zero at a distance r ≥ ℓ from the observation. As sufficiently explained in section 3, this is not a possible solution because it does not approximate any least squares problem, and thus introduces nonphysical scales (discontinuities) in the solution. As a more sensible choice, we then use a modified WS, decreasing as a function of r as
i1520-0493-139-2-474-e36
with ℓs = 10 grid points, together with the corresponding 𝗪R, given by (30). The resulting observational update (for m = 200 and r = 20) are also represented in Fig. 6 (bottom-middle and right panels), showing that the remote influence is still forced to zero (for r > ℓ), but this time, connects smoothly with the local effect. With r = 20 principal components (bottom-left panel), the parameters ℓ = 30 grid points and ℓs = 10 grid points are well adjusted to get the most of this quite poor reduced-rank description of the covariance, and restore a global shape of the observational update that is similar to the maximum rank (r = 5000) solution (top-left panel): even the southern and eastern negative lobes and the secondary positive lobe to the west are partly captured by the local parameterization. On the contrary, with m = 200 ensemble members (bottom-middle panel), the same parameters are quite far from optimal, since the original ensemble description of the covariance (top-middle panel) is more accurate and only the far west influence of the observation needed to be removed. Larger ℓ and ℓs parameters could have produced a far better parameterization, but this cannot be easily anticipated and, in real situations, it is usually safer to rely on local observations, which always dominate if the observation network is dense enough.

d. Comparison between global and local observational updates

After analyzing the effect of one single perfect observation, we now turn to another idealized observation system made of a full coverage of imperfect observations, with the aim of testing the ability of the local parameterization to correctly filter the observational noise. For this purpose, we use one model snapshot (e.g., Fig. 4, left panel) as reference dynamic height (i.e., the synthetic truth of the ideal estimation problem), and generate the observations by adding a random perturbation (e.g., Fig. 4, right panel) to simulate observation errors. This reference model snapshot is taken from the model simulation more than 10 yr after the m = 5000 sample that is used to estimate the background error covariance matrix, so that we can be sure that no direct information about the truth (except the observation and the background) is spuriously introduced in the experiments. On the other hand, the observation error covariance matrix is parameterized consistently with the real observation error with a standard deviation of σ = 0.05 m and an exponential correlation function of ρ(r) = exp(−r/ℓ) with ℓ = 5 grid points. This is done by adding gradient observations to the observation vector, and using a diagonal error covariance matrix in the augmented observation space, as explained in Brankart et al. (2009). The standard deviation on the dynamic height and on the gradient must be σ0 = 0.34 m and σ1 = 0.069 meters per grid point, respectively.

An ensemble of 360 such experiments has been performed using a set of independent true states and observation errors. Figure 7 shows the resulting root-mean-square error after the observational update, as computed explicitly as the difference with respect to the truth. This result can be directly compared to the standard deviation of the updated ensemble presented in Fig. 8. The reference result (obtained with m = 5000 and r = 5000, left panel) is characterized by the smallest error (it is the optimal solution), with consistent error estimates. This ideal solution is however very expensive to compute, since it requires a large amount of ensemble members. Moreover, since it is not small with respect to the number of observations, the eigenbasis algorithm is not more efficient than the classic algorithm: C1/C0 ∼ 1. On the other hand, if the background error covariance matrix is approximated with a reduced rank r = 20 (Fig. 7, middle panel), a much better efficiency of the eigenbasis algorithm is obtained (C1/C0 ∼ 10−5), at the expense of a larger residual difference with respect to the truth, together with inconsistent error estimates. With overestimated long-range correlations, the error estimates are systematically underestimated (Fig. 8, middle panel), thus becoming grossly inconsistent with the real error, which is larger (Fig. 7, middle panel). This systematic overestimation of weak correlations by small-sized ensembles or low-rank covariance matrices is one of the primary reason for ensemble collapse in ensemble or square root Kalman filter applications. Moreover, if the covariance approximation is too crude to represent faithfully the long-range correlations (as shown in section 4c), it is also unable to grasp the full complexity of the target signal: a large part of the relevant signal contained in the observations is out of the background error subspace and is simply filtered off from the data. This is why an artificial increase of the rank of the covariance matrix by a local parameterization is needed. Figure 7 (right panel) shows how the solution can be improved using the same local parameterization as in the previous section (ℓ = 30 and ℓs = 10 grid points). The relative efficiency of the eigenbasis algorithm is still very good (C1/C0 ∼ 10−4), and the local parameterization produces a smaller error (even if still suboptimal) with a better consistency of the associated error estimates (Fig. 8, right panel).

e. Local adaptive parameters

On the other hand, in many Kalman filter applications, the forecast error covariance can often become inconsistent with the real error as a result of inappropriate assumptions about model error or observation error statistics. In such a situation, the usual solution is to turn to adaptive filtering techniques, which consist in estimating some unknown statistical parameters in addition to the state of the system (Dee 1995). The most common adaptive parameter is typically a scaling factor for the forecast error covariance matrix (as in Anderson 2007), in order to ensure that the background error variances have at least the right magnitude. The parameter is then simply a supplementary degree of freedom in the estimation problem, and it is updated using the newly available observations, before performing the state observational update. In general, with the classic formulation of the ensemble observational update (as presented in section 2a), the optimal update of these adaptive parameters is prohibitively expensive for large-sized applications (much more expensive than the observational update of the state of the system). Yet, in a recent study, Brankart et al. (2010) have shown that, with the eigenbasis algorithm (as presented in section 2b), the optimal update of a few kinds of statistical parameters can be obtained at negligible computational cost. The parameters for which this level of efficiency can be achieved are scaling factors for the forecast or observation error covariance matrix or parameters modifying the shape of the observation error correlation function. The purpose of this section is to show that this adaptive algorithm can be used together with the local square root error parameterization proposed in section 3 (which are needed to apply the eigenbasis algorithm), in such a way that a map of local scaling factors are produced instead of a single scaling of the covariance matrix [as in Brankart et al. (2010)].

For that purpose, we repeated the same experiment as in section 4d (but only for the case with m = 200 and r = 200), with two additional unknowns: a scaling factor α for the forecast error covariance matrix and a scaling factor β for the observation error covariance matrix. These unknown parameters are included in the control vector (and not set to α = 1 and β = 1 as in section 4d) to be estimated from the observations, by applying locally the optimal algorithm described in Brankart et al. (2010). Figure 9 (left panels) shows the resulting maps for the parameters α and β, as estimated from the current innovation vector only. In this idealized experiment, we know the true value of the parameters: α = 1 and β = 1 (because the covariance matrices used in the experiment of section 4d were correct), and we can see from the figure that, if these parameters are unknown, they cannot be accurately estimated from one single innovation vector: estimation can be up to a factor of 2 away from the true value. However, as more and more observations are made available, the adaptive scheme progressively gets a better learning of the parameters, and a better accuracy can progressively be obtained. Figure 9 (middle and right panels) shows for instance that with three and then five observation vectors, both parameters are improved everywhere in the model domain.

5. Conclusions

Square root algorithms provide a useful solution for performing ensemble observational updates because of their distinctive computational complexity and because a consistent result can be obtained without adding perturbations to the observations. In this paper (section 2), we briefly described a specific formulation (eigenbasis observational update) transforming the problem into a set of independent scalar observational updates. With this particular scheme, the updated ensemble (mathematically equal to that obtained with the classic algorithm) can be computed at a numerical cost becoming linear in the number of observations, and options exist to avoid the perturbations of the observations or to obtain costless optimal adaptive parameters. Moreover, we have shown (section 3) that this eigenbasis algorithm can be made compatible with a local error parameterization. Starting from the classic localization of the ensemble covariance (by a Schur product with a local support correlation matrix), we performed an explicit square root decomposition of the full-rank local covariance matrices, which shows that low-rank square roots are always sufficient to represent the full variance at the central location of the local domain (i.e., where the local observational update is required). In terms of factorization of the Schur product, this means that 1 single modulation of the ensemble anomalies (decreasing from 1 at the central location to 0 at the boundaries of the subdomains) is always sufficient to produce a consistent observational update at the central location, providing that the missing covariance (transported in observation space) is added to the observation error covariance as an additional representativity error. Together with the approximation that this additional observation error covariance can be simulated by a simple rescaling of the original observation error covariance matrix (with an additional variance increasing from 0 at the central location to the forecast error variance at the boundaries of the subdomains), these low-rank local square roots define the local error parameterization that we are proposing for use with the eigenbasis algorithm. This bulk formulation is numerically very efficient and usually sufficient to obtain an accurate and well-conditioned approximation of the exact solution.

As an application (section 4), we considered the problem of estimating maps of dynamic topography characterizing a basin-scale ocean turbulent flow. Since the correlation scale is much smaller than the size of the ocean basin, this is the typical example where local parameterizations are needed to obtain a correct representation of the global covariance using a moderate-sized ensemble. We observed indeed that a minimum of 1000 ensemble members are necessary to represent faithfully the global correlation patterns, while the application of the local observational update (with bulk parameterization, without perturbations of the observations) can produce a correct solution with only 200 members or 20 principal components. Comparisons with the exact solution (approximated using a 5000 ensemble size) show that, with moderate-sized ensembles or reduced-rank covariance matrices, this local parameterization improve the accuracy of the updated ensemble mean and the consistency of the updated ensemble variance. With the same scheme, it was also possible to add two degrees of freedom in the local estimation problems: accurate estimates of scaling factors for the forecast and observation error covariance matrices have been computed at a numerical cost remaining negligible with respect to that of the state observational update. Finally, a comparison of computational costs illustrated the decisive advantage that efficient local square root parameterizations may have to deal simultaneously with a large number of observations. Even if slightly suboptimal, they must be considered as a way of avoiding data thinning as much as possible, and of keeping more observational information inside the estimation system.

Acknowledgments

This work was conducted as a contribution to the MyOcean Project funded by the E.U. (Grant FP7-SPACE-2007-1-CT-218812-MYOCEAN), with additional support from CNES. We also wish to thank the anonymous reviewers for their useful comments and suggestions. The calculations were performed using HPC resources from GENCI-IDRIS (Grant 2009-011279).

REFERENCES

  • Anderson, J. L. , 2003: A local least squares framework for ensemble filtering. Mon. Wea. Rev., 131 , 634642.

  • Anderson, J. L. , 2007: An adaptive covariance inflation error correction algorithm for ensemble filters. Tellus, 59A , 210224.

  • Bishop, C. H. , and D. Hodyss , 2009: Ensemble covariances adaptively localized with ECO-RAP. Part 2: A strategy for the atmosphere. Tellus, 61A , 97111.

    • Search Google Scholar
    • Export Citation
  • Bishop, C. H. , B. J. Etherton , and S. J. Majumdar , 2001: Adaptive sampling with the ensemble transform Kalman filter. Part I: Theoretical aspects. Mon. Wea. Rev., 129 , 420436.

    • Search Google Scholar
    • Export Citation
  • Brankart, J-M. , C-E. Testut , P. Brasseur , and J. Verron , 2003: Implementation of a multivariate data assimilation scheme for isopycnic coordinate ocean models: Application to a 1993–96 hindcast of the North Atlantic Ocean circulation. J. Geophys. Res., 108 , 3074. doi:10.1029/2001JC001198.

    • Search Google Scholar
    • Export Citation
  • Brankart, J-M. , C. Ubelmann , C-E. Testut , E. Cosme , P. Brasseur , and J. Verron , 2009: Efficient parameterization of the observation error covariance matrix for square root or ensemble Kalman filters: Application to ocean altimetry. Mon. Wea. Rev., 137 , 19081927.

    • Search Google Scholar
    • Export Citation
  • Brankart, J-M. , E. Cosme , C-E. Testut , P. Brasseur , and J. Verron , 2010: Efficient adaptive error parameterization for square root or ensemble Kalman filters: Application to the control of ocean mesoscale signals. Mon. Wea. Rev., 138 , 932950.

    • Search Google Scholar
    • Export Citation
  • Brusdal, K. , J. Brankart , G. Halberstadt , G. Evensen , P. Brasseur , P. van Leeuwen , E. Dombrowsky , and J. Verron , 2003: A demonstration of ensemble-based assimilation methods with a layered OGCM from the perspective of operational ocean forecasting systems. J. Mar. Syst., 40–41 , 253289.

    • Search Google Scholar
    • Export Citation
  • Buehner, M. , 2005: Ensemble-derived stationary and flow-dependent background-error covariances: Evaluation in a quasi-operational NWP setting. Quart. J. Roy. Meteor. Soc., 131 , 10131043.

    • Search Google Scholar
    • Export Citation
  • Burgers, G. , P. J. van Leeuwen , and G. Evensen , 1998: Analysis scheme in the ensemble Kalman filter. Mon. Wea. Rev., 126 , 17191724.

    • Search Google Scholar
    • Export Citation
  • Corazza, M. , E. Kalnay , and S. Yang , 2007: An implementation of the Local Ensemble Kalman filter in a quasi-geostrophic model and comparison with 3D-Var. Nonlinear Processes Geophys., 14 , 89101.

    • Search Google Scholar
    • Export Citation
  • Cosme, E. , J-M. Brankart , J. Verron , P. Brasseur , and M. Krysta , 2010: Implementation of a reduced-rank, square-root smoother for high-resolution ocean data assimilation. Ocean Modell., 33 , 87100.

    • Search Google Scholar
    • Export Citation
  • Dee, D. , 1995: Online estimation of error covariance parameters for atmospheric data assimilation. Mon. Wea. Rev., 123 , 11281145.

    • Search Google Scholar
    • Export Citation
  • Drévillon, M. , and Coauthors , 2008: The GODAE/Mercator-Ocean global ocean forecasting system: Results, applications and prospects. J. Oper. Oceanogr., 1 , 5157.

    • Search Google Scholar
    • Export Citation
  • Evensen, G. , 1994: Sequential data assimilation with a non linear quasigeostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res., 99 , (C5). 1014310162.

    • Search Google Scholar
    • Export Citation
  • Evensen, G. , and P. J. van Leeuwen , 1996: Assimilation of Geosat altimeter data for the Agulhas Current using the ensemble Kalman filter with a quasi-geostrophic model. Mon. Wea. Rev., 124 , 8596.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M. , J. S. Whitaker , and C. Snyder , 2001: Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter. Mon. Wea. Rev., 129 , 27762790.

    • Search Google Scholar
    • Export Citation
  • Heemink, A. W. , M. Verlaan , and A. J. Segers , 2001: Variance reduced ensemble Kalman filtering. Mon. Wea. Rev., 129 , 17181728.

  • Houtekamer, P. L. , and H. L. Mitchell , 1998: Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev., 126 , 796811.

    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L. , and H. L. Mitchell , 2001: A sequential ensemble Kalman filter for atmospheric data assimilation. Mon. Wea. Rev., 129 , 123137.

    • Search Google Scholar
    • Export Citation
  • Hunt, B. , E. Kostelich , and I. Szunyogh , 2007: Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filter. Physica D, 230 , 112126.

    • Search Google Scholar
    • Export Citation
  • Lermusiaux, P. F. J. , and A. R. Robinson , 1999: Data assimilation via error subspace statistical estimation. Part I: Theory and schemes. Mon. Wea. Rev., 127 , 13851407.

    • Search Google Scholar
    • Export Citation
  • Lorenc, A. C. , 2003: The potential of the ensemble Kalman filter for NWP—A comparison with 4D-VAR. Quart. J. Roy. Meteor. Soc., 129 , 31833203.

    • Search Google Scholar
    • Export Citation
  • Ott, E. , and Coauthors , 2004: A local ensemble Kalman filter for atmospheric data assimilation. Tellus, 56A , 415428.

  • Pham, D. T. , J. Verron , and M. C. Roubaud , 1998: Singular evolutive extended Kalman filter with EOF initialization for data assimilation in oceanography. J. Mar. Syst., 16 , 323340.

    • Search Google Scholar
    • Export Citation
  • Szunyogh, I. , E. Kostelich , G. Gyarmati , D. Patil , B. Hunt , E. Kalnay , E. Ott , and J. Yorke , 2005: Assessing a local ensemble Kalman filter: Perfect model experiments with the National Centers for Environmental Prediction global model. Tellus, 57A , 528545.

    • Search Google Scholar
    • Export Citation
  • Testut, C. , P. Brasseur , J. Brankart , and J. Verron , 2003: Assimilation of sea-surface temperature and altimetric observations during 1992–1993 into an eddy permitting primitive equation model of the North Atlantic Ocean. J. Mar. Syst., 40–41 , 291316.

    • Search Google Scholar
    • Export Citation
  • Tippett, M. K. , J. L. Anderson , C. H. Bishop , T. M. Hamill , and J. S. Whitaker , 2003: Ensemble square root filters. Mon. Wea. Rev., 131 , 14851490.

    • Search Google Scholar
    • Export Citation
  • Verlaan, M. , and A. W. Heemink , 1997: Tidal flow forecasting using reduced rank square root filters. Stochastic Hydrol. Hydraul., 11 (5) 349368.

    • Search Google Scholar
    • Export Citation
  • Whitaker, J. S. , and T. M. Hamill , 2002: Ensemble data assimilation without perturbed observations. Mon. Wea. Rev., 130 , 19131924.

    • Search Google Scholar
    • Export Citation
  • Whitaker, J. S. , T. M. Hamill , X. Wei , Y. Song , and Z. Toth , 2008: Ensemble data assimilation with the NCEP global forecast system. Mon. Wea. Rev., 136 , 463482.

    • Search Google Scholar
    • Export Citation
  • Yang, S-C. , E. Kalnay , B. Hunt , and N. E. Bowler , 2009: Weight interpolation for efficient data assimilation with the Local Ensemble Transform Kalman filter. Quart. J. Roy. Meteor. Soc., 135 , 251262.

    • Search Google Scholar
    • Export Citation

Fig. 1.
Fig. 1.

Comparison of the global (solid line) and local ensemble (dashed line) observational updates to the exact solution (dotted line): (top left) forecast covariance with respect to the central location, (top right) updated mean, (bottom left) updated standard deviation, and (bottom right) updated covariance with respect to the central location.

Citation: Monthly Weather Review 139, 2; 10.1175/2010MWR3310.1

Fig. 2.
Fig. 2.

(left) Eigenvalue spectrum of the local ensemble forecast covariance, as obtained with an identity metric (solid line) and with a metric giving a significant importance to the central location only (dotted line). (right) Percentage of variance that is resolved as a function of the position in the subdomain by keeping 99% of the total variance (with each metric).

Citation: Monthly Weather Review 139, 2; 10.1175/2010MWR3310.1

Fig. 3.
Fig. 3.

Difference with respect to the reference local parameterization using explicit square root decomposition (solid line) or bulk parameterization (dotted line): (top left) forecast ensemble covariance with respect to the central location, (top right) updated ensemble mean, (bottom left) updated ensemble standard deviation, and (bottom right) updated ensemble covariance with respect to the central location. For comparison, the figure also shows the difference between the classic local parameterization and the exact solution (dotted line).

Citation: Monthly Weather Review 139, 2; 10.1175/2010MWR3310.1

Fig. 4.
Fig. 4.

(left) Model-simulated dynamic height anomaly and (right) random perturbation simulating observation error.

Citation: Monthly Weather Review 139, 2; 10.1175/2010MWR3310.1

Fig. 5.
Fig. 5.

Sensitivity of correlation estimates to (left) sample size and (right) rank reduction. The figure represents dynamic height correlation as function of latitude (along the 15°E meridional section, with respect to a point located at the center of the jet at 34.5°N, 15°E). (right) With 10, 20, and 80 principal components, the percentage of resolved variance is 54%, 68%, and 93%, respectively.

Citation: Monthly Weather Review 139, 2; 10.1175/2010MWR3310.1

Fig. 6.
Fig. 6.

Observational update resulting from one perfect observation located at 34.37°N, 25.18°E: (top left) global ensemble parameterization (with m = 5000 and r = 5000), (top middle) global ensemble parameterization (with m = 200 and r = 200), (top right) global reduced-rank parameterization (with m = 5000 and r = 20), (bottom left) local ensemble parameterization (with m = 200, r = 200 and constant 𝗪S), (bottom middle) local ensemble parameterization (with m = 200, r = 200 and decreasing 𝗪S), and (bottom right) local reduced-rank parameterization (with m = 5000, r = 20 and decreasing 𝗪S).

Citation: Monthly Weather Review 139, 2; 10.1175/2010MWR3310.1

Fig. 7.
Fig. 7.

Root-mean-square residual error after the observational update: (left) global ensemble parameterization (with m = 5000 and r = 5000), (middle) global reduced-rank parameterization (with m = 5000 and r = 20), and (right) local reduced-rank parameterization (with m = 5000, r = 20 and decreasing 𝗪S).

Citation: Monthly Weather Review 139, 2; 10.1175/2010MWR3310.1

Fig. 8.
Fig. 8.

Updated error standard deviation: (left) global ensemble parameterization (with m = 5000 and r = 5000), (middle) global reduced-rank parameterization (with m = 5000 and r = 20), and (right) local reduced-rank parameterization (with m = 5000, r = 20 and decreasing 𝗪S).

Citation: Monthly Weather Review 139, 2; 10.1175/2010MWR3310.1

Fig. 9.
Fig. 9.

Maps of estimated (top) α and (bottom) β using (from left to right) 1, 3, or 5 observation vectors.

Citation: Monthly Weather Review 139, 2; 10.1175/2010MWR3310.1

1

Subscript (i) means the ith column of the matrix; this notation is also used in (18) and (20).

Save