## Abstract

An intercomparison of the Environment Canada variational and ensemble Kalman filter (EnKF) data assimilation systems is presented in the context of global deterministic NWP. In an EnKF experiment having the same spatial resolution as the inner loop in the four-dimensional variational data assimilation system (4D-Var), the mean of each analysis ensemble is used to initialize the higher-resolution deterministic forecasts. Five different variational data assimilation experiments are also conducted. These include both 4D-Var and 3D-Var (with first guess at appropriate time) experiments using either (i) prescribed background-error covariances similar to those used operationally, which are static in time and include horizontally homogeneous and isotropic correlations; or (ii) flow-dependent covariances computed from the EnKF background ensembles with spatial covariance localization applied. The fifth variational data assimilation experiment is a new approach called the Ensemble-4D-Var (En-4D-Var). This approach uses *4D* flow-dependent background-error covariances estimated from EnKF ensembles to produce a 4D analysis without the need for tangent-linear or adjoint versions of the forecast model. In this first part of a two-part paper, results from a series of idealized assimilation experiments are presented. In these experiments, only a single observation or vertical profile of observations is assimilated to explore the impact of various fundamental differences among the EnKF and the various variational data assimilation approaches considered. In particular, differences in the application of covariance localization in the EnKF and variational approaches are shown to have a significant impact on the assimilation of satellite radiance observations. The results also demonstrate that 4D-Var and the EnKF can both produce similar 4D background-error covariances within a 6-h assimilation window. In the second part, results from medium-range deterministic forecasts for the study period of February 2007 are presented for the EnKF and the five variational data assimilation approaches considered.

## 1. Introduction

Variational data assimilation approaches are used at many numerical weather prediction (NWP) centers for operationally assimilating meteorological observations to provide a single “best” estimate of the current atmospheric state (e.g., Parrish and Derber, 1992; Rabier et al. 2000; Gauthier et al. 2007; Rawlins et al. 2007). The resulting analysis is used to initialize deterministic forecast models to produce short- and medium-range forecasts. Observations are assimilated to correct a single short-term forecast where the uncertainty associated with this background state must be specified, typically in terms of a background-error covariance matrix. Four-dimensional variational data assimilation (4D-Var) uses the tangent-linear and adjoint versions (of a simplified version) of the forecast model to estimate the 4D atmospheric state that best fits the assimilated observations distributed over a specified time window.

On the other hand, ensemble approaches strive to explicitly incorporate the unavoidable uncertainty in estimates of the atmospheric state that arise primarily from errors in the observations and the forecast model. The goal of these approaches is to estimate the probability distribution functions (PDFs) of the analyses and forecasts using a relatively small ensemble of random realizations from the distributions. Ensemble data assimilation approaches use the ensemble estimate of the short-term forecast PDF when assimilating the observations. Recent versions of the ensemble Kalman filter (EnKF) approach use ensemble estimates of the error covariances of the *4D* atmospheric state (Hunt et al. 2004; Houtekamer and Mitchell, 2005). This feature was introduced into the Canadian operational EnKF system on 10 July 2007. Consequently, similar to 4D-Var, this allows the EnKF to estimate the 4D atmospheric state that best fits the assimilated observations distributed over time.

Since 2005, both the 4D-Var (Gauthier et al. 2007) and EnKF (Houtekamer and Mitchell, 2005) data assimilation approaches have been part of the operational suite at the Canadian Meteorological Centre (CMC). As is common at many NWP centers, both deterministic and ensemble forecasts are produced operationally at CMC with the ensemble forecasts being produced at a lower spatial resolution to compensate for the added computational cost of performing an ensemble of forecasts. The deterministic analysis produced by the 4D-Var is used to initialize the deterministic forecast. The EnKF is used to supply the initial conditions for the ensemble prediction system to produce medium- to extended-range ensemble forecasts.

The operational 4D-Var and EnKF data assimilation systems both use the Global Environmental Multiscale (GEM) model (Côté et al. 1998), but with different configurations. Both systems assimilate a similar set of observations using nearly identical observation operators and observation-error statistics, though the 4D-Var currently assimilates several additional types of remotely sensed data. Because of the extent of the similarities between the two systems, it was thought that a rigorous intercomparison of the two approaches would be feasible. Nonetheless, for the purpose of the present study, numerous minor modifications were made to both systems to eliminate as many of the differences as possible.

In addition to comparing the 4D-Var and EnKF using configurations similar to the operational systems, several additional configurations of the variational data assimilation approach are considered. These additional configurations attempt to bridge the gap between the two systems by using background-error covariances in the variational system that are estimated from the ensemble of background states produced by the EnKF. The ensemble covariances are used to specify the 3D background-error covariances in both 4D-Var and 3D-Var with first guess at the appropriate time (3D-FGAT) experiments. This approach has previously been examined by Buehner (2005) in the context of 3D-Var using covariances derived from a preoperational version of the Canadian EnKF. Similar approaches were considered earlier by Hamill and Snyder (2000) and Lorenc (2003b). However, this is the first time the approach has been applied to 4D-Var in a near-operational context and rigorously compared with the EnKF approach.

A relatively new approach is also employed in which the 4D background-error covariances are estimated from the EnKF ensemble members to produce a 4D analysis with the variational data assimilation approach, but without the need for the tangent-linear or adjoint versions of the forecast model. This approach, which uses the ensemble covariances in a very similar way as in the Canadian EnKF system, has been called the Ensemble-4D-Var (En-4D-Var) approach and several variations of this approach have been recently examined in idealized settings (Tian et al. 2008; Liu et al. 2008; Liu et al. 2009).

The relative advantages and disadvantages of variational and EnKF approaches to data assimilation have been considered in several previous studies. The EnKF was evaluated by Lorenc (2003b) as a possible alternative to 4D-Var for operational NWP. Several challenges associated with the EnKF were highlighted, including difficulties using nonlinear observation operators, the impact on dynamical balance from covariance localization, and the inability to fit small-scale observations. An approach for incorporating spatially localized ensemble background-error covariances in a variational data assimilation system was also presented. Kalnay et al. (2007) compared EnKF and 4D-Var approaches using idealized models and discussed results from other studies that employed realistic atmospheric models and real observations. One of these recent comparisons between the EnKF and variational data assimilation for operational NWP was presented by Whitaker et al. (2008). Using a lower resolution version of the operational forecast model and the operational observations, except without satellite radiances, the operational 3D-Var system at the National Centers for Environmental Prediction was compared against an EnKF. Medium-range forecasts initialized from the EnKF ensemble-mean analysis and the 3D-Var analysis (both at the same spatial resolution) showed that using the EnKF analyses led to large improvements in the Southern Hemisphere and more modest improvements in the Northern Hemisphere. In a similar experimental context, Szunyogh et al. (2008) compared the local ensemble transform Kalman filter (LETKF) to 3D-Var and found results similar to Whitaker et al. (2008). The EnKF has also been compared with 3D-Var and 4D-Var in the context of an artificially degraded observational network (only surface pressure) and realistic forecast models (Whitaker et al. 2009). In that study, comparable results were obtained with the EnKF and 4D-Var experiments and both produced much better forecasts than when using 3D-Var. Yang et al. (2009) compared the LETKF with 3D-Var and 4D-Var in idealized experiments with a quasigeostrophic model and also obtained similar quality results from the 4D-Var and LETKF, which were both better than those obtained with 3D-Var.

The goal of this two-part study is to compare the variational and EnKF approaches within the context of global deterministic NWP. In the next section details regarding the configurations of the EnKF and variational data assimilation systems used in this study are given. Section 3 highlights differences in the two approaches related to the use of a single “deterministic” analysis versus an ensemble of analyses. In section 4 differences in the solution algorithm used for each are considered. Differences due to the application of spatial localization to ensemble background-error covariances are discussed in section 5. In section 6 differences in the various approaches with respect to the temporal evolution of the background-error covariances are examined. The results from single-observation experiments are given in section 7 to illustrate two of the important differences between the assimilation approaches considered: the spatial localization and temporal evolution of background-error covariances. Finally some conclusions are given in section 8. Results from realistic 1-month data assimilation experiments using real observations and producing 6-day deterministic forecasts are the focus of the second part (Buehner et al. 2010, hereafter Part II) of this two-part paper.

## 2. Configurations of the variational and EnKF data assimilation systems

Both the operational 4D-Var and EnKF systems were modified on 28 May 2008 to expand the set of assimilated observations. The resulting configurations (hereafter referred to as the “2008 operational” configurations) were operational until subsequent modifications were made on 12 March 2009 for 4D-Var and 22 June 2009 for the EnKF. Nevertheless, because these configurations were operational when this study began, they were used as the basis for the configurations described below. For the purpose of this study, numerous changes were made to both the 4D-Var and EnKF systems relative to the 2008 operational configurations. The goal was to eliminate as many of the differences between the two systems as possible so that only the differences that are fundamental to each of the approaches remain. Each of the changes was tested against the corresponding 2008 operational configuration with the result that no degradation in the quality of the analyses and subsequent forecasts was incurred, except as a result of the elimination of several types of observations from the 4D-Var because they are not used in the EnKF system. A 3D-FGAT experiment was performed that assimilates the same observations as the 4D-Var and EnKF experiments and uses the same background-error covariances as the 4D-Var experiment. In addition to the standard configurations of the variational and EnKF data assimilation systems, experiments with both 4D-Var and 3D-FGAT were performed in which the background-error covariances were replaced by the 3D flow-dependent ensemble covariances from the EnKF. The final experiment uses the En-4D-Var approach in which the 4D ensemble covariances are used in the variational data assimilation system. The five variational data assimilation experiments are summarized in Table 1.

### a. Spatial and temporal resolution

In the 2008 operational configuration of the 4D-Var data assimilation cycle, the background state and medium-range forecasts are both produced using the GEM model with a global uniform latitude–longitude 800 × 600 grid (∼35-km grid spacing) and 58 terrain-following vertical levels with the top level at 10 hPa (Bélair et al. 2009). The incremental approach to 4D-Var (Courtier et al. 1994) is used in which the analysis increment is computed at a lower resolution using linearized versions of the forecast model and observation operators and their adjoints. The linearization of the forecast model is applied to a version with simplified physical parameterizations (Zadra et al. 2004). The cost function is first reduced by allowing the quasi-Newton minimization algorithm to execute 30 iterations. Then, the high-resolution nonlinear forecast model is again integrated over the assimilation window. The forecast model and observation operators are linearized about this updated 4D state estimate and an additional 25 iterations of the minimization algorithm are performed. Each set of iterations of the minimization algorithm, which involves the tangent linear and adjoint models, is referred to as the “inner loop” and each time the forecast model and observation operators are linearized about an integration of the nonlinear model this is referred to as an iteration of the “outer loop.” The analysis increment is computed within the inner loop using a coarser horizontal grid, as compared with the resolution of the background state, consisting of only 240 × 120 grid points with Gaussian latitudes. The analysis is produced by interpolating this lower-resolution analysis increment to the same 800 × 600 grid as the background state before adding the two.

In the 2008 operational EnKF (Houtekamer et al. 2009), ensembles of 96 members are cycled sequentially in time through analysis and forecast steps. Each member uses a global uniform latitude–longitude 400 × 200 grid and 28 terrain-following vertical levels with the top level at 10 hPa for both the analysis and forecast steps. As in all implementations of the EnKF, only the full nonlinear versions of the forecast model and observation operators are used.

For the purpose of the present study, the horizontal and vertical resolution of the 4D-Var inner loop and the EnKF ensemble members were both modified to make them equal. Hence, the horizontal grid of the 4D-Var inner loop was modified from a 240 × 120 Gaussian grid to a 400 × 200 Gaussian grid. Similarly, the number of vertical levels used by the EnKF for the analysis and forecast steps was increased from 28 to 58. This set of vertical levels is the same as those used in both the 4D-Var and the high-resolution deterministic forecasts. A minor difference in the grids still exists since the 4D-Var inner loop uses a Gaussian grid and the EnKF uses a uniform latitude–longitude grid. This causes small differences in gridpoint locations in the meridional direction. It should also be noted that the other variational data assimilation approaches considered (3D-FGAT and En-4D-Var) also compute analysis increments on a 400 × 200 Gaussian grid; however, they differ from 4D-Var in that there is no outer loop and instead a single set of 55 iterations is performed.

By modifying the spatial resolution of the EnKF and the inner loop of the 4D-Var, the analysis increments in the configurations of both systems used in this study are produced at the same spatial resolution (400 × 200 × 58 L). However, since the variational data assimilation system uses the incremental approach, its analysis is at a higher horizontal resolution (800 × 600) than the 96 members of the EnKF analysis ensemble (400 × 200). The spatial resolution of the analysis produced by the variational data assimilation system is the same as the forecast model used for the medium-range forecasts. To enable the same model configuration to be used to produce deterministic medium-range forecasts with an analysis from the EnKF, the mean of the EnKF analysis ensemble is first computed and then interpolated to the (800 × 600 × 58 L) model grid. Consequently in this study, the early portion of the medium-range forecasts obtained using the ensemble-mean EnKF analysis may be negatively affected because of the adjustment of the model fields to the higher-resolution grid and surface topography field. In all experiments a digital filter finalization procedure (Fillion et al. 1995) is applied over the first 6 h of both the medium-range forecasts and the short-term forecasts used to produce the background state.

Both the EnKF and variational data assimilation systems assimilate observations distributed over 6-h time windows, centered at 0000, 0600, 1200, and 1800 UTC. In 4D-Var, the background state and increment at each iteration of the minimization procedure are both interpolated to the time of the observations by simply selecting the model state with the closest time from a set of 9 model states separated by 45 min. To reduce disk storage requirements, the ensemble of model states used by the EnKF are only stored every 90 min, however these model states are linearly interpolated to the time of the observations. The amplitude of the error due to the approximation in this interpolation procedure used in the EnKF was determined by Houtekamer and Mitchell (2005) to be well below that of the initialization increment and more than an order of magnitude smaller than the analysis increment. The En-4D-Var experiment uses a combination of the time interpolation approaches just described: the background state is interpolated by choosing the time step closest to the observation time among 9 model states separated by 45 min and the 4D increment to the background state is linearly interpolated in time from 5 states separated by 90 min.

### b. Background-error covariances

The background-error covariances used in the operational 4D-Var were computed using the differences between a series of 48- and 24-h forecasts valid at the same time (the so-called NMC method; Parrish and Derber 1992) following the procedure described by Gauthier et al. (1999). These covariances are nearly constant in time (computed for each month by interpolating between a set of summer and winter covariances). The background-error correlations are horizontally homogeneous and isotropic for the set of independent analysis variables associated with the control vector and represented as a block-diagonal matrix in spectral space up to the horizontal total wavenumber 108. A local geostrophic balance operator is used to model the covariances between the mass and wind fields and another balance operator is used to model the covariances between the rotational and divergent wind components near the surface. All other cross covariances between analysis variables, except for the covariances between the unbalanced components of surface pressure and temperature, are explicitly set to zero. In 4D-Var these error covariances are used to specify the background cost function that directly constrains the analysis increment at the beginning of the assimilation time window. To take full advantage of the increase in horizontal resolution (from 240 × 120 to 400 × 200) in the 4D-Var inner loop, the background-error covariances were modified to include correlations up to total wavenumber 180, instead of only up to 108 as in the operational system.

During the EnKF analysis step, each of the 96 ensemble members is updated by assimilating the observations after they have been independently perturbed for each member. These perturbations simulate the effect of random observation error and are computed such that the ensemble mean of the perturbations for each observation is zero. During the forecast step, perturbation fields (with an ensemble mean equal to zero at each grid point) are added to the forecast initial conditions and different configurations of the model physical parameterizations are used to simulate the effect of model error (Houtekamer et al. 2009). These simulated sources of uncertainty have the effect of producing and maintaining a spread in the ensemble of background states that is representative of the error in the ensemble mean. The background-error covariances used when assimilating observations to update each 24-member subensemble are obtained from the other 72 ensemble members. This is referred to as the four-subensemble configuration in Mitchell and Houtekamer (2009). The background-error covariances include the temporal cross-covariances between five time steps, separated by 90 min, over the 6-h assimilation window. Unlike the 4D-Var covariances based on the NMC method, very few modifications are made to the raw sample covariances computed from the 72 ensemble members. The only modification is the application of spatial localization to both the horizontal (Houtekamer and Mitchell 2001; Hamill et al. 2001) and vertical (Houtekamer et al. 2005) covariances. This is applied by computing the Schur product between the ensemble covariances and a compactly supported function (the fifth-order piecewise rational function of Gaspari and Cohn 1999) that becomes 0 at a distance of 2800 km in the horizontal and 2 scale heights (i.e., the distance over which the natural logarithm of pressure changes by 2 units) in the vertical.

Additional 4D-Var and 3D-FGAT experiments that use covariances computed from the background ensembles produced by the EnKF experiment are included in this study. By using the same flow-dependent covariances in the variational data assimilation system as in the EnKF, one of the major differences between the approaches is removed. The covariances are modeled in a very similar way as in the EnKF; however, one key difference is that, instead of the 72 members used for updating each member in the EnKF, all 96 ensemble members are used. This does not give rise to inbreeding since the background state is independent of the 96 ensemble members. None of the same assumptions used for the prescribed operational 4D-Var covariances, such as homogeneity, isotropy, or the form of the cross covariances between different analysis variables, are applied when using the EnKF covariances in the variational system. The approach for applying spatial localization to the EnKF covariances within the variational system is described by Buehner (2005). The same compactly supported function was used for horizontal and vertical covariance localization as in the EnKF analysis step. However, unlike in the EnKF algorithm, covariance localization in the variational data assimilation system is applied directly to the ensemble background-error covariances, independent of the observations being assimilated. To obtain a practical implementation of the EnKF for realistic NWP applications, covariance localization in that system is applied to the ensemble estimates of the matrices 𝗕𝗛^{T} and 𝗛𝗕𝗛^{T} (Houtekamer and Mitchell 2001), where 𝗕 is the background-error covariance matrix and 𝗛 is the observation operator that relates the analysis variables to the observations. This difference between the two systems is discussed in section 5.

The En-4D-Var experiment uses the EnKF covariances within the variational data assimilation system in a manner very similar to that used in the EnKF system itself by employing the ensemble members at five time levels within the assimilation window. This allows the estimation of the 4D background-error covariances and the estimation of a 4D analysis increment without the need for the tangent linear or adjoint version of the forecast model. The same spatial covariance localization is applied to the covariances at each time level and to all of the between-time-level cross covariances, as is also done in the EnKF system. The approach is similar to that described by Liu et al. (2008), except that covariance localization is employed (following the approach of Buehner 2005). Also, instead of only applying the nonlinear observation operators to the ensemble members, as in the EnKF, the tangent linear version of the observation operators is applied to the 4D analysis increment as in 4D-Var.

### c. Assimilated observations

The types of observations assimilated in the 2008 operational 4D-Var system are wind, temperature, and humidity from radiosondes; wind and temperature from aircraft; wind (only over water), temperature, pressure, and humidity from in situ surface observations; winds from profilers over the United States; cloud-tracked winds from geostationary and polar-orbiting satellites; surface winds over water from the Quick Scatterometer (QuikSCAT); and radiances from the Advanced Microwave Sounding Unit A and B (AMSU-A/B), Atmospheric Infrared Sounder (AIRS), Special Sensor Microwave Imager (SSM/I), and geostationary satellites. The 2008 operational EnKF assimilates the same observations except for the radiances from AIRS, SSM/I, and geostationary satellites. Several modifications were made to procedures related to the assimilated observations to eliminate as many differences between the two systems as possible. A detailed description of the procedures used in both the operational and experimental configurations considered in this study is given in Part II.

Table 2 summarizes the differences between the configurations used for the variational data assimilation experiments and the EnKF experiment.

## 3. Deterministic versus ensemble data assimilation

All configurations of the variational data assimilation approach considered in this study make use of a single deterministic background state and also produce a single analysis at each analysis time. In contrast, the EnKF assimilates observations to update each member of a 96-member background ensemble to produce a corresponding ensemble of analyses. In this study, the ensemble mean analysis is used to initialize medium-range forecasts and therefore it is reasonable to consider how the ensemble mean analysis relates to the analysis produced by the variational data assimilation system. The En-4D-Var approach is compared with the EnKF in this section since it is the approach most similar to the EnKF.

If all observations are only linearly related to the analysis variables, then the ensemble mean of the innovation vector (observation-minus-background state projected into observation space) should equal the innovation vector from the variational data assimilation system given a background state that equals the EnKF ensemble mean background state. This is ensured because the random perturbations added to the observations have a zero ensemble mean. However, assuming no covariance localization and with all observations being assimilated simultaneously, a difference in the analysis increment will still occur because of the use of 72-member subsets of the complete background ensemble when computing the background-error covariances in the EnKF. Because of the matrix inversion in the analysis equation, the ensemble mean of the Kalman gain matrices, 𝗞* _{i}*, does not equal the equivalent gain matrix for the analysis with the En-4D-Var approach. This can be seen by considering the EnKF analysis equation (assuming a linear observation operator):

where Δ**x*** _{k}^{a}* is the analysis increment, 𝗕

*is the background-error covariance matrix,*

_{i}**y**

*is the vector of observations, 𝗛 is the observation operator, 𝗥 is the observation-error covariance matrix, and*

_{k}**x**

*is the background state. The index*

_{k}^{b}*i*refers to the 72-member ensemble subset, which does not include the member being updated, for computing the background-error covariances (

*i*= 1, … , 4) and

*k*is the index of the ensemble member being updated (

*k*= 1, … , 96). Averaging (1) over all 96 ensemble members does not produce the same result as replacing 𝗕

*with a covariance matrix estimated from the entire background ensemble, which is equivalent to what is done in the En-4D-Var approach.*

_{i}The assimilation of observations that are nonlinearly related to the analysis variables introduces an additional difference between the two systems. Nonlinear observation operators affect the calculation of the innovation vector, for which the ensemble mean will no longer equal the innovation vector computed with the ensemble mean observations and background state. Also, the ensemble estimates for the matrix products 𝗕𝗛^{T} and 𝗛𝗕𝗛^{T} in (1) are computed in the EnKF using the nonlinear observation operators applied to the background ensemble. This differs from the equivalent calculation in the variational system, which uses the observation operator linearized with respect to the background state (or a partially updated background state during the second iteration of the outer loop in 4D-Var). It is not clear if this effect is significant since the relationships of the assimilated observations with the analysis variables are often only weakly nonlinear.

## 4. Differences in solution algorithm

All observations within a 6-h window are assimilated simultaneously in all configurations of the variational data assimilation system. This is accomplished by computing the value of the control vector ** ξ** that minimizes the cost function

where **y**(*t*) is the vector of observations at time *t*, [·] is the original observation operator, 𝗛 is the linearized version of [·], 𝗥 is the observation-error covariance matrix, and **x*** ^{b}*(

*t*) is the background state at time

*t*. The relationship between the control vector

**and the increment at time**

*ξ**t*is defined as either

for 3D-FGAT,

for 4D-Var, or

for En-4D-Var, where 𝗠* _{t}* is the tangent linear model that maps perturbations from the beginning of the assimilation window to time

*t*and 𝗕

^{1/2}is the square root of the background-error covariance matrix valid either at the beginning (4D-Var) or middle (3D-FGAT) of the assimilation time window or the 4D ensemble covariances valid for 5 time levels over the entire (0 ≤

*t*≤

*T*) window (En-4D-Var). Note that for 4D-Var, (2) is valid only for the first outer-loop iteration. To facilitate the minimization process, the gradient of (2) is obtained and used together with an optimization algorithm based on the quasi-Newton approach (Gauthier et al. 2007).

The data assimilation step in the EnKF is performed sequentially by assimilating batches of several hundred observations to update each ensemble member (Houtekamer and Mitchell 2001). This is accomplished by explicitly computing the Kalman gain matrix using the ensemble of background states to specify the background-error covariance matrix. Once a particular batch of observations has been used to update all members of the background ensemble, the resulting updated ensemble covariances are used for recomputing the Kalman gain matrix for assimilating the next batch of observations. The explicit solution of the Kalman filter analysis equation is computationally feasible only if the size of the batches is sufficiently small. For the original Kalman filter this sequential approach is equivalent to assimilating all observations simultaneously only if observations with correlated error are included in the same batch (Gelb 1974, p. 304). For an EnKF with covariance localization and a diagonal observation-error covariance matrix (𝗥), Houtekamer and Mitchell (2001) found a sensitivity of the ensemble mean analysis error with respect to the size of the observation batches, especially when the number of ensemble members was small.

## 5. Differences in spatial covariance localization

As mentioned previously, spatial covariance localization is applied in a somewhat different manner in the variational and EnKF data assimilation systems. This difference has the largest impact when assimilating observations that are each related to the analysis variables over many grid points or vertical levels (Campbell et al. 2010). The main instances of this type of observation are the radiances measured by space-based instruments. These observations are typically related to temperature and/or humidity over many vertical levels as modeled by a radiative transfer model.

Without covariance localization and assuming the observation is linearly related to the analysis vector, the analysis increment from using the EnKF to assimilate a single observation is proportional to 𝗕_{ens}**h**^{T}, where 𝗕_{ens} is the sample covariance matrix computed from the EnKF ensemble and **h** is a row vector corresponding to the observation operator for the assimilated observation. When using the EnKF covariances with covariance localization in the variational data assimilation system, the analysis increment is proportional to , where *ρ*_{var} is the localization function and the symbol represents the Schur product. In contrast, the analysis increment produced by the EnKF system with covariance localization applied is proportional to . Consequently, to define the appropriate localization function for the EnKF, a unique location must be associated with the assimilated observation, even when the observation is related to the analysis variables over many grid points or vertical levels. As a result of how covariance localization is applied in the EnKF system, the analysis increment that results from assimilating a single observation is itself forced to be local, which may not always be appropriate. Results from several single-observation experiments that demonstrate this difference between the two systems are presented in section 7a.

## 6. Differences in the temporal evolution of error covariances

Both the EnKF and 4D-Var data assimilation systems make use of background-error covariances that are evolved through time over the assimilation time window. While the covariances are being evolved, they are modified in a way that depends on the current meteorological situation. However, the way in which the covariances are evolved in the two systems is quite different. These differences are described in this section and illustrated with results from single-observation experiments in section 7b.

A key difference regarding the evolution of the background-error covariances in the EnKF and 4D-Var systems is due to the nonlinearity of the forecast model. The 4D-Var approach requires that the background-error covariances be specified at the beginning of the assimilation time window. Within the inner loop of the incremental implementation of 4D-Var, these covariances are then implicitly evolved to the times of the observations using the tangent linear and adjoint versions of the forecast model (Lorenc 2003a). This can be seen by considering the equation used to compute the increment at time *t* which, for the configuration of 4D-Var that uses the EnKF background ensemble, is defined as

where **x*** _{k}^{b}*(0) is the

*k*th background ensemble member and

**x**

^{b}(0) denotes the ensemble mean background state with both being valid at the beginning of the assimilation window,

*ξ*is the control vector element associated with the

_{k}*k*th ensemble member, and

*K*is the ensemble size. The recalculation of the forecast model linearization with respect to the partially updated background state during an outer-loop iteration modifies the implicit evolution of the covariances to the extent that nonlinearities in the forecast model are significant. For the EnKF approach, the full nonlinear model is used to evolve each ensemble member throughout the assimilation window and the covariances are then estimated from the time-evolved ensemble members. This can be seen by considering the equation for computing the increment to the background state at time

*t*in the En-4D-Var approach, which employs evolved ensemble covariances in the same way as in the EnKF:

where represents integration of the nonlinear forecast model from the beginning of the assimilation window to time *t* and **x**^{b}(*t*) is defined by

Note that, for clarity, spatial covariance localization is not included in (6) and (7). The quantity in the numerator of (7) is the deviation of the forecast initialized with the *k*th background ensemble member at the beginning of the assimilation window from the ensemble mean of such forecasts. The difference between (6) and (7) vanishes when the following equality is true:

which holds only when the difference between ensemble members is small and the meteorological situation is such that the influence of highly nonlinear processes (such as convection) is negligible. If (8) does not hold, then the evolution of the spread in the ensemble by the nonlinear model cannot be well approximated by the tangent linear version of the model. The use of the En-4D-Var approach eliminates this difference between the variational and EnKF data assimilation approaches.

Because the prescribed covariances at the beginning of the assimilation window are evolved implicitly in the 4D-Var approach, it is computationally feasible to employ a covariance matrix of high rank that may even span the entire phase space of the analysis vector. This allows spatial covariance localization to be applied to the 3D covariances before they are evolved in time. The inclusion of spatial localization results in a substantial increase to both the rank of the covariance matrix and the dimension of the control vector (the latter is equal to the product of the number of ensemble members and the rank of the matrix used for localization), however, it does not increase the number of multiplications by the tangent linear model 𝗠* _{t}* in (6). In contrast, the explicit time evolution of the ensemble covariances in the EnKF approach implies that the covariances being evolved are defined in the low-dimensional subspace spanned by the ensemble members. Consequently, this precludes the possibility of applying covariance localization before time evolution. Spatial localization is therefore applied to the evolved 4D covariances at each time level and also to the cross covariances of background error between different time levels. In the current EnKF system (and En-4D-Var approach), the same spatial localization is applied to all temporal cross covariances as for the covariances of the background error at each individual time level. This may not be appropriate in situations where the background error may have a maximum temporal cross covariance at distant locations, due to rapid advection or wave propagation over the assimilation time window. An alternative localization technique that addresses this problem is proposed by Bishop and Hodyss (2009).

## 7. Results of single-observation analysis experiments

In this section results from single-observation experiments are used to demonstrate the impact of some of the differences discussed earlier between the data assimilation approaches considered in this study. The first set of experiments shows the impact of differences in the application of spatial covariance localization on the analysis increments when assimilating satellite radiances or radiosonde temperatures. The second set of experiments demonstrates the impact of differences in the temporal evolution of the background-error covariances.

### a. Impact of differences in spatial covariance localization

Figure 1 shows the analysis increments from assimilating a single radiance observation of AMSU-A located over the ocean between Australia and New Zealand at 0000 UTC 3 February 2007. The observed radiance is from channel 9, which is sensitive to temperature within a vertical layer centered near 70 hPa. The analysis increments resulting from using the EnKF and variational data assimilation systems are computed with each using the same EnKF ensemble covariances. The result from the variational data assimilation system is produced using the En-4D-Var approach that employs the ensemble covariances in a very similar manner as the EnKF system. The time of the observation is exactly at the center of the assimilation window, the same time for which the analysis increments are computed and therefore the time correlations in the ensemble-based covariances do not influence the result. Figure 1a shows the analysis increments of temperature when using the same amount of vertical localization as in the EnKF (i.e., the localization function reaches zero at a vertical distance of two scale heights). In this case the resulting analysis increments of temperature from both the EnKF and variational data assimilation systems have a maximum value near 70 hPa. However, above this level, the analysis increment from the EnKF gradually approaches zero whereas the analysis increment from the variational system remains near half the maximum value. The results change somewhat by reducing the amount of vertical localization so that the localization function reaches zero at a distance of four scale heights, as shown in Fig. 1b. Above 70 hPa the analysis increments are now more similar than previously shown due to a slight increase in the EnKF increments and a slight decrease in the increments from the variational system. Below 70 hPa the analysis increments from the two systems remain much more similar. Figure 1c shows the result when the vertical localization is reduced so that it has almost no effect (with the localization function reaching 0 only at a distance of 100 scale heights). Now the analysis increment resulting from assimilating the single radiance observation with either the EnKF or variational data assimilation system is almost identical.

It is interesting to note that, when using the variational data assimilation system, the application of more vertical localization results in an apparently less local analysis increment above 70 hPa. This is most evident for the highest level (10 hPa) where the analysis increment decreases from a value of more than half the value at 70 hPa, when using the most vertical localization, to near 0 when using effectively no vertical localization. Such an effect likely results from the negative temperature correlations that are often observed near the top of the model in the vertical background-error correlations. This can be understood by first considering that, with no vertical correlations, the analysis increment for temperature would be proportional to the corresponding weights in the linearized radiance observation operator, as computed by the Radiative Transfer for (A)TOVS (RTTOV) model, multiplied by the temperature background-error variances [see (1)]. Since the observation operator weights are positive over a broad layer, the temperature analysis increment would therefore have the same sign over this layer as the scalar innovation. However in the case of large negative background-error correlations near the top of the model, such an increment would result in large values for the background term of the cost function. Consequently, the only way to reasonably satisfy both the background and observation constraints in the cost function is to reduce the increments where the background-error correlations are negative. With increasingly severe vertical covariance localization, however, the negative correlations are reduced and the increment near the top of the model approaches the larger values obtained in the case of no vertical correlations.

Figure 2 shows the results from a similar single observation experiment as shown in the previous figure, but using an observation of AMSU-A channel 10, which is sensitive to the temperature within a vertical layer centered near 30 hPa. The analysis increments shown in Figs. 2a–c are again computed using a vertical localization function that forces the covariances to zero at a distance of 2, 4, or 100 scale heights, respectively. Like in the previous example, the differences between the analysis increments of temperature when using either the EnKF or variational data assimilation system is largest when the most severe vertical localization is used (Fig. 2a). When effectively no vertical localization is used, the analysis increments are again almost identical (Fig. 2c). However, unlike the previous example, the largest analysis increment does not consistently occur at the level where the sensitivity of the radiance observation to temperature is maximum (i.e., at 30 hPa). The maximum temperature increment only occurs at 30 hPa when using the most vertical localization with the EnKF. For the other cases the maximum analysis increment is located at levels higher than 30 hPa and most often at 10 hPa. Again, as in the previous example, negative background-error correlations reduce the analysis increment at 10 hPa produced by the variational system. Therefore, the analysis increment at 10 hPa tends to be larger when applying more severe vertical covariance localization. In contrast, the analysis increment produced by the EnKF system is forced by the localization procedure to be almost zero at 10 hPa, regardless of the sensitivity of the radiance observation to temperature at this level.

Figure 3a shows the effect of assimilating the radiance observations from the full set of AMSU-A channels (4–10) at the same location and time used for the single-observation experiments and with the vertical localization function that reaches zero at a distance of two scale heights. Over most of the vertical levels the analysis increments of temperature produced by the EnKF and variational data assimilation systems are very similar, except for the levels above about 70 hPa where large differences occur. These differences appear to be related to those seen in the single observation experiments with only channel 9 or 10. Therefore, the largest differences due to how covariance localization is applied in the two systems when assimilating AMSU-A radiance observations are confined to the uppermost analysis levels. This is consistent with the results shown in Fig. 15 of Part II. Analysis increments of temperature obtained by assimilating temperature observations from a radiosonde profile located near the radiance observations used previously are shown in Fig. 3b. In contrast to the analysis increments of temperature obtained from assimilating the AMSU-A radiances, the increments obtained from assimilating in situ temperatures with the EnKF and variational data assimilation systems are nearly identical throughout the entire vertical domain. This is consistent with the fact that the difference in how covariance localization is applied in the two systems only causes differences in the analysis increments when the assimilated observations are each related to many grid points or vertical levels. The background-error standard deviation of temperature at the location and time of the assimilated observations is shown in Fig. 3c. Throughout the troposphere, the standard deviation is approximately 0.5 K. Above 100 hPa it increases to a maximum value of almost 1.5 K at 10 hPa.

Recently, Campbell et al. (2010) have argued that localization performed directly on the background-error covariances in model space should lead to better results. In practice, however, because of unrealistic model behavior near the top of the model, the approach of spatially localizing the influence of the radiance observations in the vertical, as done in the EnKF, appears to be beneficial in our system.

### b. Impact from differences in the temporal evolution of error covariances

A series of single observation experiments were performed to illustrate the impact of the differences between the EnKF and variational data assimilation systems with respect to the temporal evolution of background-error covariances. In all cases a single radiosonde temperature observation located over China at 500 hPa is assimilated. The time of the observation is manipulated so that it coincides with the beginning, middle, or end of the 6-h assimilation window centered on 0000 UTC 3 February 2007. The same 6-h segment of a model forecast, coinciding with the 6-h assimilation window, is used as the background state for all experiments. The value of the temperature observation is artificially set to 1 K below the background value at the same location and time as the observation. The observation is located to the west of a low pressure system in a region of strong northwesterly winds. Note that for all 4D-Var experiments described in this section, the linearizations of the forecast model and observation operators are only performed with respect to the background state and are therefore not refreshed as part of an outer-loop iteration. This ensures that the 4D-Var response is simply proportional to the artificially specified innovation. Also, since the 4D-Var analysis increments are always computed at the beginning of the data assimilation window, the 4D-Var increments shown as being valid at the middle of the window are, in fact, computed from a 3-h forecast using the nonlinear model.

In the first experiment a single 500-hPa temperature observation at the beginning of the assimilation time window is assimilated. The resulting analysis increment of temperature and horizontal wind at the middle of the assimilation time window is shown in Fig. 4 for the model level closest to 500 hPa. The analysis increments obtained from using the EnKF (the ensemble mean analysis increment) and three different configurations of the variational data assimilation system are shown. The first configuration of the variational data assimilation system is 4D-Var-Bnmc (Fig. 4a). The second configuration is 4D-Var-Benkf (Fig. 4b), which differs from the first in that the background-error covariances specified at the beginning of the assimilation window are derived from the EnKF background ensemble. The third configuration is the En-4D-Var approach (Fig. 4c), which is most similar to the EnKF (Fig. 4d) in how the background-error covariances are evolved through time. That is, the En-4D-Var approach uses 4D error covariances computed from the 96 background ensemble members output from the nonlinear model at 5 time levels throughout the assimilation time window. For all results the location of the maximum temperature increment does not exactly coincide with the observation location (denoted by the white circle), but is displaced toward the south or southeast. This can be explained because the analysis increment shown is valid 3 h after the observation time and the background wind field is directed toward the southeast. The temperature increment from the 4D-Var-Bnmc configuration has a broader horizontal structure, but with some small-scale structure to the northeast of the observation location, and the wind increment has a much smaller amplitude than the other three results. The other approaches produce generally similar temperature and wind increments. The results from using the En-4D-Var and EnKF approaches are very nearly identical for both temperature and wind increments. Both produce analysis increments for wind that have a slightly larger amplitude than with the 4D-Var-Benkf configuration. However, all three produce a similar spatial structure for the wind increment consisting of a cell of cyclonic circulation centered just to the southwest of the maximum temperature increment. The temperature increments for these three results are also elongated toward the northwest and northeast, aligned with the isolines of the background state geopotential height field. The 4D-Var-Benkf result appears to share some of the same small-scale structure to the northeast of the observation location as the analysis increment produced by the 4D-Var-Bnmc configuration. This suggests that these features are created by an aspect unique to the 4D-Var system and not related to the specified background-error covariances.

The second experiment is identical to the first, except that the time of the observation coincides with the middle of the assimilation time window, instead of the beginning. The same four data assimilation configurations are used and the results shown in Fig. 5. The location of the maximum temperature increment is now more closely aligned with the location of the observation for all four results. Again, the temperature increment for the 4D-Var-Bnmc approach (Fig. 5a) is much broader than the other three, but the wind increment has a slightly higher amplitude than in the first experiment with a spatial structure that is similar, but smoother, than the other three results. The small-scale features to the northeast are still present in the temperature increments of the two 4D-Var configurations (Figs. 5a,b). The En-4D-Var (Fig. 5c) and EnKF (Fig. 5d) approaches again produce nearly identical temperature and wind increments. The wind increment produced by all three approaches that employ the EnKF ensemble covariances form a cell of cyclonic circulation that extends much farther toward the northwest than any other direction. This elongated structure appears to be associated with the closely spaced isolines of the background state geopotential height field (and therefore strong geostrophic winds). Consequently the wind increment will have the effect of displacing toward the west the jet of strong northwesterly winds in the background state.

For the third experiment (Fig. 6), the time of the temperature observation coincides with the end of the assimilation time window. In all four results, the location of the maximum temperature increment is now upstream, that is to the northwest, of the observation location. The wind increment again forms a cell of cyclonic circulation centered slightly to the west of the maximum temperature increment. This feature of the wind field is still much weaker and broader in the 4D-Var-Bnmc result (Fig. 6a) than for the other three. Apparently, because the analysis increment is farther upstream and away from the location where the geostrophic winds change direction toward the east, the temperature increment is even more elongated along the isolines of background geopotential height toward the northwest and is much more compact in the orthogonal direction than for the previous experiment. This effect can also be seen in the 4D-Var-Bnmc result, but to a lesser extent.

As a point of comparison, the analysis increment from assimilating the single temperature observation was also computed using the two 3D-FGAT configurations. Because of the absence of covariance evolution in the 3D-FGAT approach and the fact that the observed temperature is artificially set to ensure the same observation-minus-background value for each observation time, the analysis increments produced by the 3D-FGAT experiments are independent of observation time. Consequently, only the analysis increments for which the time of the observation coincides with the middle of the assimilation window are shown (Fig. 7). Unlike the previously shown results, it is clear that the resulting analysis increments of temperature and wind from the 3D-FGAT-Bnmc configuration (Fig. 7a) are completely independent of the local meteorological conditions. This is due to the background-error covariances being based on temporally and spatially averaged statistics and the assumptions of homogeneity and isotropy imposed on the horizontal correlations of the set of independent analysis variables. In contrast, the 3D-FGAT-Benkf configuration produces an analysis increment (Fig. 7b) exactly equal to that produced by the En-4D-Var approach when the observation time coincides with the middle of the assimilation window (Fig. 5c).

## 8. Conclusions

The EnKF and five configurations of a variational data assimilation system were described and compared to highlight both similarities and inherent differences between them. Numerous modifications were made to the 2008 operational configurations of the EnKF and variational data assimilation systems to eliminate many unnecessary differences between the systems. However, the data assimilation systems considered still differ in several important ways, including the use of a deterministic versus an ensemble data assimilation approach, the use of a variational versus a sequential solution algorithm, the approach for temporally evolving the error covariances, the use of static versus flow-dependent background-error covariances, and the approach for applying spatial covariance localization. By comparing the results from the different approaches, it can be demonstrated how these differences affect the resulting analysis and forecast quality. A set of single-observation experiments helped to illustrate the impact of some of these differences. In the second part of this two-part paper, results from medium-range deterministic forecasts for the study period of February 2007 are presented for the EnKF and the variational data assimilation approaches considered.

Several fundamental differences still require additional detailed investigation, including the impact of differences in how nonlinearities (in both the observation operators and the forecast model) are treated in the various approaches considered. The difference in solution algorithm (variational versus sequential) is another fundamental difference that may be significant; however, from the set of experiments considered in this study it is not possible to isolate this difference from other important differences. Finally, it should be noted that the difference in the type of grid used to compute the analysis increment in the variational and EnKF data assimilation systems has recently been eliminated by modifying the EnKF code to employ a Gaussian grid. In the future, this will allow for cleaner comparisons of the systems and simplify the use of EnKF covariances in the variational system by eliminating the interpolation of ensemble members onto the 4D-Var analysis grid.

## Acknowledgments

The authors thank Luc Fillion and other members of the “WWRP/THORPEX Workshop on 4D-Var and Ensemble Kalman Filter Inter-comparisons” Program Committee for providing the initial impetus for this work and Gilbert Brunet and Godelieve Deblonde for their continued support of this project. The authors also thank Craig Bishop, Eugenia Kalnay, and Jeff Whitaker for their helpful official reviews.

## REFERENCES

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

**,**

## Footnotes

*Corresponding author address:* Mark Buehner, Meteorological Research Division, Environment Canada, 2121 TransCanada Hwy., Dorval, QC H9P 1J3, Canada. Email: mark.buehner@ec.gc.ca

This article included in the Intercomparisons of 4D-Variational Assimilation and the Ensemble Kalman Filter and the Targeted Observations, Data Assimilation, and Tropical Cyclone Predictability special collections.