1. Introduction

For many hydrological applications, it is desirable to update an entire spatial field through the assimilation of sparse soil moisture observations. Information propagation over a soil volume can be accomplished by full three-dimensional (3D) Richards equation–based models. However, for computational efficiency, coarse-scale land surface models (LSM) often treat soil moisture profiles as independent, individual vertical columns, ignoring horizontal water flows. In a synthetic data assimilation study, Reichle and Koster (2003) demonstrated that taking horizontal model error correlations into account in the a priori error covariance matrix for a 3D ensemble Kalman filter (EnKF) improved the estimation accuracy for soil moisture, even in a regional-scale application. Other existing approaches to enforce lateral information flow in an assimilation scheme with a 1D (column) model are a priori interpolation of the observations to locations where they are desired to affect the model state, as in analysis nudging (Stauffer and Seaman 1990; Houser et al. 1998), and combined spatial interpolation and temporal propagation as explored in the space–time Kalman filter (Huang and Cressie 1996). Generally, the choice is between a better or more complex LSM with a simple assimilation scheme and a simple LSM with a more complex assimilation scheme. Exploring the possibility of accounting for horizontal information flow within the assimilation scheme is worthwhile, because “it is unreasonable to rely on models that relate each pixel of a large grid to every other pixel” (McLaughlin 2002). It should also be recognized that for very coarse-scale modeling, it is not needed to horizontally connect all the state variables within the model, because of the limited correlation length of earth state variables.

For several decades it has been discussed that, for a given system model and observation network (Daley 1992b), the efficient use of the information brought by the observations is largely determined by the second-order a priori error statistics: the a priori (or background) error covariance matrix 𝗣_i⁻, used in the Kalman gain (or any blending matrix used during assimilation) at each time step i, distributes information in space and between state variables and is the key to optimal data assimilation. Furthermore, the magnitude of the blending matrix elements essentially only depends on the signal-to-noise ratio 𝗥_i/𝗣_i⁻, with 𝗥_i as the observation error covariance matrix. Different blending gains mean different filter bandwidth and a distinct sensitivity to noise: the larger the a priori state error (or the smaller the observation error), the larger the filter bandwidths and the more variation (resulting from incorporating observations) can be expected in the assimilation analyses.

In the sequential statistical interpolation method (Daley 1991) and in most variational assimilation schemes, 𝗣_i⁻ is generally set empirically, after splitting 𝗣_i⁻ into a time-invariant correlation matrix 𝗣_C and a possibly time-variant diagonal matrix 𝗣_Si whose elements are the error standard deviations for the individual variables in the state, that is, 𝗣_i⁻ = (𝗣_Si𝗣_C𝗣_Si)⁻. The specification of the correlation structure is a critical issue, because it selects the area from which observations are allowed to influence the model variables. Mostly, stationary, homogeneous, horizontal/vertical separable, and isotropic error covariance functions as functions of distance have been proposed for which only a few parameters were to be estimated, for example, by the traditional innovation or observational method (Hollingsworth and Lönnberg 1986; Daley 1991; Houser et al. 1998). With the implementation of the Kalman filter, the focus was moved to the definition of the model error covariance (Daley 1992a), because 𝗣_i⁻ can easily be calculated by the explicit Lyapunov propagation equation in linear stationary systems as the sum of the predictability and model error covariances. For optimization, several adaptive filters have been developed to estimate the model error covariances, mostly involving some whitening of the innovation (observation minus forecast) sequence (Kailath 1968; Mehra 1970; Maybeck 1982) or matching (elements of) the zero lag innovation covariance matrix to the theoretically optimal values (Jazwinski 1969; Myers and Tapley 1976; Maybeck 1982; Mehra 1972; Dee 1995; Dee and da Silva 1999). The majority of the techniques focus more on the estimation of the error variances than on the error correlation. In nonlinear systems, as are most land surface models, ensembles allow a flow-dependent statistical approximation of 𝗣_i⁻ for the EnKF (Evensen (1994)). However, the EnKF only quantifies the uncertainty in the space spanned by the ensembles, and it is particularly difficult to reflect the “true” uncertainty in the initial or analysis state and model error into an optimal perturbation magnitude and structure for the forcings, model structure, state, and parameters. It is often useful to tune the filter by adapting the error variance—for example, by inflation, to avoid model divergence in case of an underestimated a priori error covariance. An adaptive inflation factor in time was proposed by Anderson (2007), who adjusted the variance of each state component, whereas the correlation between pairs of components remained unchanged. Mitchell and Houtekamer (2000) applied the maximum-likelihood method to estimate model error variance parameters in an EnKF framework, where initially only the uncertainty in the initial state or analysis was considered.

In hydrology, the statistical nature of the a priori (forecast) errors has often been prescribed in synthetic studies or arbitrarily chosen in real studies. Sometimes, the filter performance is evaluated afterward by providing innovation statistics (Reichle et al. 2002a; Crow 2003; De Lannoy et al. 2007b). In synthetic studies, filters can be calibrated by contrasting the filter output to some truth, as in Reichle et al. (2002b) and Reichle and Koster (2003), to allow for a comparison of several Kalman filter setups for soil moisture estimation, each in optimal conditions. If no truth is available, the innovation statistics could be used to optimize the filter. Van Geer et al. (1991) and Zhang et al. (2004) applied the covariance matching technique (Mehra 1972) to optimize the uncertainty estimation in groundwater and soil moisture simulations, respectively. In the covariance matching method, the innovation covariance matrix, obtained as a result from the filter, is iteratively contrasted to its theoretical value, which is a function of the tunable model error covariance matrix. The innovation covariance matrix is typically calculated as the outer product of the centralized innovation vectors over some time (proxy of ensemble). Drécourt et al. (2006) extended and changed this approach to estimate both the system and bias model error covariance in a synthetic groundwater simulation study. Crow and van Loon (2006) applied the maximum-likelihood estimation (Dee 1995) to find a covariance scaling parameter to adjust the error variances only, while assuming a fixed, predefined (spatial) correlation structure. This approach involves the inner product (or, the trace of the outer product) of the innovation vectors—that is, all the variables’ uncertainties are lumped into one value. They illustrated how tuning the wrong error source to produce better innovation statistics led to a reduced accuracy of the filter results, when only assimilating surface soil moisture. Additionally, there was a risk to find the “globally” best innovation statistics through the optimization of the wrong error type, if different sources of error were taken into account. Reichle et al. (2008) used the same maximum-likelihood approach to identify model and observation error variances for soil moisture assimilation in a synthetic experiment. The multiscale Kalman filter (Parada and Liang 2004; Kumar 1999) efficiently considers the spatial dependence and scaling properties of soil moisture. Parada and Liang (2004) applied a multiscale Kalman filter in conjunction with an expectation maximization algorithm to estimate temporally evolving statistical observation and model error parameters inherent to the Kalman filter to assimilate surface soil moisture in an LSM. Besides interest in the 𝗣_i⁻, observation uncertainty (error variance) is also an important factor in filtering. In several observation system simulation experiments (OSSEs), for example by Walker and Houser (2004), researchers have tried to quantify the maximal acceptable (to be useful in data assimilation) observation uncertainty to guide instrumentation design. It is reasonable to assume that, similar to forecast error, observation error variance is also state dependent and hence an adaptive filtering approach could be beneficial (Romanowicz et al. 2006).

Here, in situ soil moisture profiles in a small agricultural field [Optimizing Production Inputs for Economic and Environmental Enhancement (OPE³)] are simulated by individual, unconnected soil columns in the Community Land Model, version 2.0 (CLM2.0). Ensembles are used to quantify the variables’ uncertainty in each individual profile. Then, the ensemble a priori error variance is adapted, and the error correlation between the profiles is sought for in an adaptive filtering scheme to introduce lateral flow of information through the assimilation scheme. The adaptive information is extracted from as many field observations as available. In a successive time period, this information is used to update the complete field while assimilating only a few observations. This study differs from the EnKF approach of Reichle and Koster (2003), who imposed and calibrated (against some truth) the model error field in a synthetic study to impose lateral flow of information. Although data assimilation updates the observable prognostic state variables, it also adjusts the unobserved diagnostic flow variables. This study only focuses on the possibility of propagating soil moisture profiles horizontally and only validates observable prognostic soil moisture. An earlier study (De Lannoy et al. 2007b) highlighted the effect of soil moisture assimilation on diagnostic evapotranspiration, subsurface drainage, and runoff.

2. Model and observations

Soil moisture profiles are estimated with the CLM2.0 (Dai et al. 2003) at 36 locations (Fig. 1) in a 21-ha corn field in Beltsville, Maryland, where the OPE³ experiment is conducted (Gish et al. 2002; De Lannoy et al. 2006b). The model simulates land surface processes by calculating hourly vertical water and heat fluxes and states for each profile without interaction between profiles—that is, the linearized system matrix is block diagonal. To limit the computational costs, only the 10-layer soil moisture prognostic variables are included in the assimilation scheme—that is, the state vector x̂_i has a dimension of 360. The depths of the soil nodes are set to 2.5, 5, 10, 20, 30, 50, 80, 120, 150, and 180 cm. The nonlinear land surface model f_i,i−1 propagates the state vector from time step i − 1 to i:

View Expanded

where u_i is the meteorological forcings, which are hourly observed data and assumed spatially homogeneous over all the profiles in this study. The initial condition for the land surface model is the a posteriori state estimate (or analysis) . This a posteriori state estimate is obtained from updating the a priori state estimate at time i − 1 (discussed below).

During our observation period from 1 May 2001 through 30 April 2002, hourly measurements were obtained from 36 capacitance probes (EnviroSCAN of SENTEK Pty Ltd., South Australia), measuring soil moisture at three to seven layers (H probes: 10, 30, and 80 cm; M probes: 10, 30, 50, 120, 150, and 180 cm; L probes: 10, 30, 50, 80, 120, 150, and 180 cm depth; H, M, and L probes were installed at locations with expected different infiltration regimes, i.e., high, medium, or low clay content.). A layout of the observation probes in the OPE³ field is shown in Fig. 1. The observations y_i (vector dimension varies with the specific assimilation experiment) are linearly related to the state by the observation (or forward) operator 𝗛_i, that is,

View Expanded

where v_i is zero-mean assumed random error. Here, 𝗛_i contains only values of 1 and 0, because the observations are direct measurements of the state variables.

The CLM2.0 was calibrated and initialized for each individual profile through weak constraint variational assimilation of observed data from 1 May through 1 October 2001 (De Lannoy et al. 2006a), with a focus on parameter estimation in the period from 2 September through 2 October 2001. All state updating experiments in this paper start after this calibration period. A limited subset of observations will be assimilated for state and uncertainty estimation, while withholding other soil moisture profile observations for validation of the filtering performance. This study only focuses on assimilation of “complete” profiles. Depending on the type of probe and the availability of data, this means simultaneous assimilation of observations at three up to seven soil depths.

3. EnKF implementation

The Kalman filter is a sequential assimilation method used to update forecasted model states with observational information. The analysis or a posteriori state estimate is a combination of the forecast and observations, each weighted as a function of their uncertainty, which is captured in the a priori and observation error covariance matrices, respectively. The uncertainty in the forecasts is approximated by N = 64 ensemble (Monte Carlo) member predictions :

View Expanded

where w_j,i is a realization of the (zero-mean assumed) model error that represents the complete effect of perturbations to forcings, parameters, and state variables, quantified as in De Lannoy et al. (2006a). The a priori state (background) error covariance 𝗣_i⁻ is initially obtained from the ensemble sample covariance . Because of the model structure (independent profiles) and our choice of uncorrelated forcing perturbations, a priori state errors at different locations were effectively uncorrelated. The spurious elements (without actual statistical significance) ending up outside of the diagonal blocks corresponding to the variables in one profile in were set to 0. This type of localization would be well suited for global- or regional-scale studies of variables that are only correlated over a small spatial domain and for which the error correlations over larger distances can be assumed to be very small.

When observations y_i are available, each ensemble member j is updated individually to obtain the a posteriori state estimate:

View Expanded

The actual analysis state x̂_i⁺ is the ensemble mean. The term equals the innovation vector r_i⁻. The observations are perturbed to ensure sufficient spread—that is, y_j,i = y_i + v_j,i, where v_j,i is an imposed zero-mean Gaussian perturbation with an error covariance of 𝗥 = (0.022 m³ m⁻³)² 𝗜, based on the sensor calibration information and assuming zero cross correlation between the observation errors (De Lannoy et al. 2007b). The Kalman gain 𝗞_i is identical for all ensemble members and given by

View Expanded

Basically, the term 𝗣_i⁻𝗛_i^T maps the analysis increments on the model grid. If no observations are available, then x̂_i⁺ in Eq. (1) simply equals x̂_i⁻.

4. Adaptive EnKF (ADEnKF)

a. Method

The correct characterization of 𝗣_i⁻ largely determines the success of the Kalman filter. A first approximation of 𝗣_i⁻ is the ensemble-based . Within the blocks of , we can assume that ensemble error correlations are well estimated: an assimilated observation in a single layer would be efficiently propagated to other layers within the same profile (Fig. 2). However, the magnitude of the error variances could possibly be optimized, because it is defined by a subjectively chosen “realistic” perturbation magnitude. Tuning the variances is typically the goal of most adaptive filtering schemes. Therefore, we will allow a variance correction within the adaptive Kalman scheme ADEnKF, while keeping the error correlations in the blocks as retrieved from the ensembles. Because the ensemble approach does not allow the estimation of the error structure outside the diagonal blocks in 𝗣_i⁻, the adaptive scheme is also explored to retrieve this “missing” between-profiles correlation information and hence allow horizontal information propagation (Fig. 2).

Most adaptive schemes calculate the model error covariance matrix 𝗤_i as part of 𝗣_i⁻ = 𝗣_i^p,− + 𝗤_i, assuming that predictability error, which is the error in the forecasts resulting from errors in the previous analysis or initial conditions (with covariance 𝗣_i^p,−), and model error (with covariance 𝗤_i) are independent (Daley 1992a). One of the schemes that allows for the estimation of all individual elements in 𝗤_i is the method of Myers and Tapley (1976). It introduces the analysis (a posteriori estimate) as a proxy for the true state and an approximation of model noise w*_i,

View Expanded

which in fact equals the analysis increment 𝗞_i[y_i − 𝗛_ix̂_i⁻]. For an optimal Kalman filter, the covariance of this quantity is given by

View Expanded

View Expanded

View Expanded

View Expanded

where E[w*_i, w*_i] = E[(w*_i−)(w*_i−)^T] and is the theoretical ensemble mean. The matrix 𝗣_i⁺ is the a posteriori (analysis) error covariance matrix and the predictability error covariance matrix 𝗣_i^p,− can be calculated by propagating the previous a posteriori (or analysis) error covariance at i − 1 by the model to time i (for a linearized model version . The practical expression to obtain matrix 𝗤_i from Eq. (10) introduces a time averaging over a window of λ time steps:

View Expanded

An analysis increment time series (window of λ elements) is stored and centralized by its time mean 〈w*〉, to eliminate the time mean bias. The time mean approach is basically to obtain a statistically viable estimate of the analysis increment covariance matrix as an approximation of the ensemble mean analysis increment covariance matrix. The advantage of this method, as compared to the computationally more attractive approach of Dee (1995), is its ability to tune the variance at each state variable individually, whereas simpler methods typically come up with an inflation factor to homogeneously scale the a priori covariance matrix by a constant value. The latter approach could yield adverse results, as suggested by the findings of Crow and van Loon (2006).

Whereas in the above deduction 𝗤_i is the unknown part of 𝗣_i⁻ = 𝗣_i^p,− + 𝗤_i, we now have different missing information in 𝗣_i⁻. With the EnKF, the objective is to determine 𝗣_i⁻ based on information in the ensembles: a good approach of the error covariance would result from the propagation of the ensemble through the model dynamics. However, here the model does not simulate lateral flow, so only a part of the actual 𝗣_i⁻ is captured by the ensemble information, that is , which is block diagonal and contains some predictability and model error (i.e., part of 𝗣_i^p,− and 𝗤_i, respectively) in the diagonal blocks by perturbing the initial conditions, parameters, and forcings for each profile individually. The ensembles do not provide any information on the covariance between errors in variables of different profiles (off block diagonal) because of a model structural deficiency (no lateral flow). This is now the missing information in 𝗣_i⁻. A new matrix, , will be calculated in the adaptive scheme to adapt the ensemble-based block-diagonal , to obtain the total 𝗣_i⁻:

View Expanded

The method of Myers and Tapley (1976), as in Eq. (11), is slightly changed to obtain as follows:

View Expanded

assuming that for i = λ, instead of 𝗣_λ⁻ = 𝗣_λ^p,− + 𝗤_λ. The practical implementation is as follows: at each time step, (i) calculate the matrix [w*_i − 〈w*〉] [w*_i − 〈w*〉]^T and (ii) derive and 𝗣_i⁺ from the ensemble state members, the latter after having filtered observations with . Then, the time mean at the right-hand side of Eq. (13) is calculated. In our example, the estimate of is based on increment information in the λ = 40 previous assimilation time steps for a filtering frequency of once every two days, from 21 December 2001 through 19 March 2002. For λ ≤ 20, we found that the retrieved matrix information could not be statistically consistent, because it was too sensitive to information brought in at additional time steps.

Once is found in a first period, it can be used to optimize 𝗣_i⁻ for the filtering in a next assimilation period. The variances on the diagonal of are used to correct the ensemble variances on the diagonal of at each assimilation step i, that is:

View Expanded

If and the resulting [𝗣_i⁻]_kk would result in a negative value, then [𝗣_i⁻]_kk is not corrected by Eq. (14), but we choose to set [𝗣_i⁻]_kk/2. Next, the covariances within the blocks are updated for the corrected variances [𝗣_i⁻]_kk (the ensemble correlations are unaltered, but the variances are updated) as follows:

View Expanded

if k and l belong to the same profile. Outside the blocks, the covariances are calculated with the time-invariant correlations found in and the corrected time-variant variances from Eq. (14):

View Expanded

if k and l belong to different profiles. Here, is the ensemble-based a priori error variance for the kth state variable, averaged over the λ assimilation time steps. The off-block-diagonal elements in contain covariances containing variances of (with zero off-block-diagonal elements), plus (the additional correction). The final 𝗣_i⁻ is not imposed on the ensemble state members along the model run but only used for the calculation of the Kalman gain. In summary, we apply a hybrid method to fully determine 𝗣_i⁻: ensemble-based state-dependent variances are corrected by the adaptive information and are used together with (i) ensemble-based state-dependent correlations in the blocks and (ii) time-invariant correlations outside the blocks. Unlike many EnKF studies here, no covariance localization (Hamill et al. 2002; Reichle and Koster 2003) is applied after filling the off-block-diagonal elements in 𝗣_i⁻.

b. Innovations versus analysis increments

The above proposed method uses the information in the analysis increments (𝗞_i[y_i − 𝗛_ix̂_i⁻]) rather than the innovations (y_i − 𝗛_ix̂_i⁻), which are used in most other adaptive filters. A direct consequence is that all involved matrices as well as the resulting are determined in full state space, while other adaptive filters mostly reduce the computations to the observation space. The traditional innovation method (Hollingsworth and Lönnberg 1986; Daley 1991; Houser et al. 1998) implicitly provides a way to find the a priori state (background) error correlations at observed locations and to use them to cover the larger part of state space by fitting a distance-dependent correlation function. This technique builds on the following second-order innovation statistics for each pair (k, l) of observation locations:

View Expanded

View Expanded

where E[r_i⁻,r_i⁻] is the ensemble covariance matrix at one time step i. In practice, time series of (ensemble mean) innovations r_k⁻, calculated during the EnKF application at individual locations k, can be used as a proxy to estimate the ensemble innovation covariance. The centralized second-order statistic is calculated to remove the time mean discrepancy between forecasts and observations. Because in our study, a reasonable time-invariant estimate of the diagonal 𝗥 matrix is available—that is, [𝗥]_kk = (0.022 m³ m⁻³)² and [𝗥]_kl = 0 (m³ m⁻³)²—the a priori error correlations ρ_kl can be estimated by

View Expanded

where E[[r⁻]_k, [r⁻]_l] = E[([r⁻]_k − 〈[r⁻]_k〉) ([r⁻]_l − 〈[r⁻]_l〉)^T] is calculated over time and 〈[r⁻]_k〉 is the time mean innovation for the kth element of vector r⁻. For each pair of elements, this correlation can then be plotted as a function of the distance and an empirical homogeneous (in space) function can then be fitted.

In this study, we could easily change this procedure, or any other innovation-based adaptive method, to find the off-block-diagonal correlation values of in the observation space to fill in the individual entries in 𝗣_i⁻ in the state space as well as to find the variance correction, like in our adapted Myers and Tapley (1976) method. However, that would only be possible in the case of this study, where 𝗛_i only contains 0 and 1, and the transfer of information from the observation to the state space would be well determined (leaving unobserved entries in 𝗣_i⁻ with some default or zero correlation). For more general and complex observation operators 𝗛_i, the method of Myers and Tapley (1976) is more attractive than other adaptive filters to find in the state space.

5. Results

b. Adaptive filter: Spatial effects

For the main experiment, the combined state estimation (updating) and uncertainty estimation was performed during the period from 2 October through 21 December 2001 with the adapted Myers and Tapley (1976) method. All 36 complete observed soil moisture profiles were assimilated every two days (40 time steps) to calculate missing off-block-diagonal information and to obtain a correction to the variances in the a priori error covariance. Thereafter, from 23 December 2001 to 19 March 2002, only a single profile was assimilated using the novel error covariance information to enhance the soil moisture estimates at all 36 simulated soil moisture profiles in the field. During this period the ensemble-based was updated with time-invariant information in as described in Eqs. (14)–(16).

Figure 5 shows the a priori state error correlation matrix on 23 December 2001, prior to and after the adaptive correction. It illustrates how the ensemble-based block-diagonal (i.e., within a profile) correlations are kept unaltered and how the off-block-diagonal (i.e., between profiles) error correlation elements are partially filled in. For a random zoom example, where error correlations between state variables corresponding to profile AL2 and BM4 are shown, only six rows (state variables corresponding to the six observation layers in M-probes) and seven columns (seven observation layers in L-probes) are filled in. The more observational layers available for a pair of profiles, the more error correlation information can be retrieved. Because the model layers at 2.5, 5, and 20 cm are never observed by the OPE³ probes, there will never be information on error statistics at these layers. Again, no spatial structure could be found in the retrieved error correlations at any soil layer (not shown). Similar to the above findings, the determined error correlation was higher in the top soil layer but much smaller than what was found through the innovation method.

Figure 6 shows for two individual soil depths at 10 and 80 cm, the temporal evolution of the spatial root-mean-square error (RMSE) between observations and ensemble mean forecasts without assimilation as well as analyses with assimilation of complete observed soil moisture profile data. During the first period (from 2 October to 21 December 2001) all available profile information is assimilated, causing significant drops in RMSE. However, the observational information is not taken up completely by the model at all profiles, mainly as a result of forecast bias (sometimes the model and observation climatologies are different; De Lannoy et al. 2007b), causing a rapid increase of RMSE after each assimilation event. In the second period, only one selected profile (here, DL1) is assimilated, either without or with inclusion of the estimated off-block-diagonal a priori error covariance elements. The further reduced RMSE, when including the off-block-diagonal information, shows that there is a benefit to spreading information in the horizontal space.

As an illustration, the spatial soil moisture fields in Fig. 7 are generated for 13 March 2002 as a weighted average of two spatial interpolations of the individual point analysis results (only for visualization purposes) by (i) a multivariate regression between the soil moisture and the terrain characteristics (texture, elevation, and topographic index) at the observing locations and (ii) kriging with an exponential variogram with parameters as observed in De Lannoy et al. (2006b). The observations and simulations show a discrepancy in the spatial soil moisture field: a part is a result of model bias, another part results from random error. When assimilating the complete observed profile at DL1 with the regular EnKF, no other profile is updated and the spatial view is dominated by the ensemble mean integration without assimilation. When the estimated off-block-diagonal a priori error covariance elements are included, ADEnKF of DL1 observations updates all profiles in the field. The full circles in Fig. 7 show regions where the ADEnKF obviously brings the simulated field close to the observed one, while only assimilating data at one location in the field and using time-invariant correlation information off the diagonal blocks in 𝗣_i⁻. The dashed circle shows a region where a major dry forecast bias cannot be dealt with by a state filter and where the adaptive filter only has a minor effect, because it is mainly designed to treat random error and probably also because of the time-invariant correlation structure. There is an attempt to increase the soil moisture by the ADEnKF, but it is mainly effective only at one location (CL2).

Figure 8 summarizes the effect on the spatial soil moisture estimation, when assimilating observations from each individual profile only, during a validation period from 23 December 2001 through 24 March 2002. The time-averaged spatial RMSE is shown for the case without and with inclusion of information. The RMSE is generally smaller with inclusion of off-block-diagonal a priori error covariance elements, even though the off-block-diagonal structure was determined during the period preceding the validation period and kept time-invariant (and hence suboptimal) during the validation period. A maximum decrease in spatial RMSE by 22% was achieved at 180 cm by assimilating either DM2 or DM3 profile data, a maximum RMSE decrease of 13% was found at 80 cm for the assimilation of AM1 data, and at 10 cm a 9% RMSE decrease resulted from assimilation of BM2 data. At the other observing layers, the maximum RMSE decrease was 11% (30 cm), 13% (50 and 80 cm), 17% (120 cm), and 7% (150 cm) by complete profile assimilation. For this study, the maximum decrease in spatial RMSE, averaged over all field profile layers, was 11% when assimilating probe AM1. It turns out that AM1 is the most time stable profile in the OPE³ field (De Lannoy et al. 2007a; i.e., its measured soil moisture time series best resembles the spatial mean soil moisture behavior in time at most soil depths). It should also be noted that validation of the performance in a single layer sometimes shows further reduced spatial RMSE values, when only assimilating a single observation in that single layer (not shown). This is because when updating the soil moisture in many layers simultaneously, the analysis profile may not reflect the preferred model balance and seemingly contradictory (to the model) updates for the different layers may be cancelled out quickly over time after the analysis. The update in a single layer often persists longer in time, when neighboring layers are adapting to the update in that single layer, rather than possibly competing against this update with updates at their own respective depths. It is evident that, when evaluating the total field profile performance (as opposed to the previous discussion on the effect in a single layer), assimilation with probes having more observational depths (six for M-probes and seven for L-probes) generally results in the best improvements in RMSE. However, it is important to notice how H-probes with only three observation layers are able to enhance the spatial RMSE at unobserved (e.g., 180 cm) layers.

In some cases (here, it is for 80-cm depth), the RMSE is increased, as a result of either a bad propagation of information over the vertical layers or an inaccurate determination of the between-profiles error correlation. In an exceptional case, the assimilation of a single profile did not improve the soil moisture field estimate in any layer: through assimilation of probe BL2, the spatial RMSE was increased outside the range of the plot for almost all soil layers when including the spatial error correlation information. The problem here was a collapsed ensemble spread (and the time-invariant adaptive correction was inappropriate to solve this problem) at some layers, where a model-defined minimal soil moisture was reached, causing adverse effects in other layers at distant profiles.

The spatial RMSE covers both the discrepancy in the spatial mean soil moisture field and the spatial structure of the soil moisture field. The last three columns in Table 1 summarize the time root-mean-square difference between the observed and modeled spatial mean soil moisture field (during the validation period 23 December 2001–24 March 2002) for the cases (i) without assimilation, (ii) with regular EnKF assimilation of a single profile (here, DL1), and (iii) with assimilation of the same single profile by the proposed adaptive EnKF. When only state updating is performed (EnKF), the RMSE is slightly reduced, only because of the improving effect on the DL1 soil moisture time series within the spatial mean. Clearly, when including adaptive information, the error in the spatial mean soil moisture is largely reduced by assimilating a single profile. Only at 150 cm (for this case), a minor adverse effect can be observed, caused by a bad vertical interaction between layers. Maybe this could be solved by using the adaptive filter to correct the ensemble-based error correlations within the profile. The first three columns in Table 1 give the time mean of the root-mean-square difference in semivariances γ between the observed (γ_o) and modeled (γ_m) spatial soil moisture fields over the same period of length T,

View Expanded

The spatial lag in the calculation of γ_o and γ_m was set at 20 m (to allow some averaging over pairs of observations in each spatial lag-interval) for a maximum interdistance of 420 m (beyond half of the field length, semivariance values would be calculated for too few sample pairs). The mean square difference in semivariances was thus calculated over a maximum number of 21 spatial lags, that is, L = 21. Table 1 shows that, through simple EnKF, the spatial pattern was slightly improved, again because of the enhanced representation of soil moisture at the assimilation location only. With the inclusion of adaptive information, the differences in semivariances were further reduced. Consequently, assimilation of a single profile (here, DL1) with an adaptive EnKF is able to partially rectify both incorrectly simulated spatial soil moisture mean and variance.

6. Discussion and perspectives

In this paper, the missing error correlations and an error covariance correction are retrieved only over a single time window and are to be used in a period succeeding the period where it was actually calculated. A more frequent batch processing could be suggested to obtain more time-variant adaptive information. Definitely for large-scale applications, where the variability in the precipitation errors may alter the error correlations over time, a frequent update of the missing error correlations would be beneficial. However, there is a trade-off between an increased time variability and the statistical consistency of the error covariance estimates, as found in Eq. (13): some time averaging is needed to obtain viable covariance estimates. Future research could investigate how the training period can be limited by exploring the ensemble information in the innovations at each time step, instead of studying the ensemble mean information in time.

The analysis results can be further improved by adding a bias filter to the state estimation (De Lannoy et al. 2007b). However, since the bias error covariance is often linked to the (adapted) a priori error covariance, more research is needed to assure both proper interactions between these error covariance and the optimality of the filter. Note that the Myers and Tapley (1976) method removes some bias from the increments to find in Eq. (13), but the state estimates remain biased. It should also be recognized that the a priori error correlations should only represent information about the random error structure, as long-range a priori state error correlations derived from innovations (or analysis increments) may sometimes originate from spatial bias fields in either the observations or forecasts.

This approach to determine the between-profiles error covariance can be of practical importance to propagate satellite observations from observed locations to temporarily masked or unobserved regions (e.g., clouds in visual products and hydrometeors for high-frequency microwave products), or propagation of in situ profile information to locations where sensors are temporarily out of service. In satellite data assimilation schemes, the modeled grid cells are typically covered by the observations (after reprojecting the data). This allows an update of all the a priori error correlations at each time observations are available and use them, whenever part of the observations falls out and information needs to be propagated to the nonobserved pixels. Furthermore, training of the correlations during one satellite pass (covering a limited banded area) allows observations in the overlapping area of the next pass to be propagated all over the area covered by previous (shifted) pass, if some time-invariance in error correlations could be assumed. However, for large-scale applications, we generally assume that the a priori error correlation would be limited (e.g., different dominant forcing errors) and not worth the increased numerical expenses to deal with a more complex 𝗣_i⁻. As a very simple approximation, the actual (mostly observed) state correlation length is often used to limit the a priori state error correlations at large distances, even though we should realize that the actual state and its estimation errors (and hence their respective correlations lengths) are in fact independent.

A problem is that with a finer model resolution, more observations are needed to retrieve the correct error covariances. If a soil moisture profile is available every 20 m (assume it representative for a 10 × 10 m² area) and the model simulates at a 10-m resolution, then the correlation between the forecast errors of two neighboring grid cells at 10 m apart will not be retrieved by the proposed technique, whereas the correlation between forecast errors in two observed grid cells at 20 m apart can be found. One limitation of the proposed method is the need to determine the adaptive information based on as many observations as available, before it can be used with a restricted dataset.

In general, the proposed technique is expected to advance the optimal use of satellite and other data to fill in unobserved areas and to overcome the shortcomings of completely time invariant homogeneous (in space) correlation matrices, as in optimal interpolation techniques. Typical empirical distance-dependent correlation functions may give a false impression of smooth correlation structures in the analyses. It could be advised to review assumptions on empirical correlation functions for hydrological variables at different scales by dynamical error correlation insights obtained from adaptive filters.

7. Summary

This study explores the potential of updating the full 3D system state when assimilating only single soil moisture profile data in the independent column-based CLM2.0 with an adaptive ensemble Kalman filter (ADEnKF). Assimilation of individual in situ soil moisture profiles from the small-scale OPE³ field with an optimized knowledge of a priori error covariances resulted in a partial rectification of the inaccurately simulated spatial mean and variability in soil moisture fields at different soil depths.

Because no simple spatial error correlation structure could be found based on the traditional innovation method, it was necessary to find all elements of the a priori error covariance matrix individually. Ensembles were used to obtain (i) a first guess of each state variable uncertainty, that is, the magnitude of error variance, and (ii) an estimate of the error correlation between state variables within each individual profile. Then, during a first period, a time-invariant variance correction and the between-profiles error correlations were sought based on an adaptive method of Myers and Tapley (1976). This allowed for the adjustment of the time-variable ensemble-based state error covariance matrix before each filtering time step in the successive period.

Even though only one time-invariant estimate was used to correct the a priori error covariances, the results show improvements up to between 7% and 22% reduction in spatial RMSE in a single soil layers, when assimilating only a single complete profile [higher percentage reductions can be achieved by assimilating only in the single validation layer, which avoids contradictory updates over the different soil layers (not shown)]. Even the assimilation of a single profile with only a few observed layers resulted in enhanced soil moisture field estimates at most depths. With knowledge of the off-block-diagonal a priori error covariances (ADEnKF), any observation in a single layer will affect both the total profile where it is assimilated in and all layers of surrounding profiles, while a regular EnKF with the CLM2.0 only allows for updating of a single profile.

This study shows an example of retrieving the missing error correlations and an error covariance correction during a first time window and using it in a period succeeding the period where it was actually calculated. A more frequent batch processing would result in more time-variant adaptive information and the information in the ensembles may be further explored to limit the training periods, while keeping the estimates statistically consistent.

The determination of between-profiles error covariance can be important to propagate satellite observations from observed locations to temporarily unobserved or masked regions or propagation of in situ profile information to locations where sensors are temporarily out of service. The limitation of this method is the need to determine the adaptive information based on as many observations as available, before it can be used with a restricted dataset.