## 1. Introduction

For many hydrological applications, it is desirable to update an entire spatial field through the assimilation of sparse soil moisture observations. Information propagation over a soil volume can be accomplished by full three-dimensional (3D) Richards equation–based models. However, for computational efficiency, coarse-scale land surface models (LSM) often treat soil moisture profiles as independent, individual vertical columns, ignoring horizontal water flows. In a synthetic data assimilation study, Reichle and Koster (2003) demonstrated that taking horizontal model error correlations into account in the a priori error covariance matrix for a 3D ensemble Kalman filter (EnKF) improved the estimation accuracy for soil moisture, even in a regional-scale application. Other existing approaches to enforce lateral information flow in an assimilation scheme with a 1D (column) model are a priori interpolation of the observations to locations where they are desired to affect the model state, as in analysis nudging (Stauffer and Seaman 1990; Houser et al. 1998), and combined spatial interpolation and temporal propagation as explored in the space–time Kalman filter (Huang and Cressie 1996). Generally, the choice is between a better or more complex LSM with a simple assimilation scheme and a simple LSM with a more complex assimilation scheme. Exploring the possibility of accounting for horizontal information flow within the assimilation scheme is worthwhile, because “it is unreasonable to rely on models that relate each pixel of a large grid to every other pixel” (McLaughlin 2002). It should also be recognized that for very coarse-scale modeling, it is not needed to horizontally connect all the state variables within the model, because of the limited correlation length of earth state variables.

For several decades it has been discussed that, for a given system model and observation network (Daley 1992b), the efficient use of the information brought by the observations is largely determined by the second-order a priori error statistics: the a priori (or background) error covariance matrix 𝗣_{i}^{−}, used in the Kalman gain (or any blending matrix used during assimilation) at each time step *i*, distributes information in space and between state variables and is the key to optimal data assimilation. Furthermore, the magnitude of the blending matrix elements essentially only depends on the signal-to-noise ratio 𝗥_{i}/𝗣_{i}^{−}, with 𝗥* _{i}* as the observation error covariance matrix. Different blending gains mean different filter bandwidth and a distinct sensitivity to noise: the larger the a priori state error (or the smaller the observation error), the larger the filter bandwidths and the more variation (resulting from incorporating observations) can be expected in the assimilation analyses.

In the sequential statistical interpolation method (Daley 1991) and in most variational assimilation schemes, 𝗣_{i}^{−} is generally set empirically, after splitting 𝗣_{i}^{−} into a time-invariant correlation matrix 𝗣* _{C}* and a possibly time-variant diagonal matrix 𝗣

*whose elements are the error standard deviations for the individual variables in the state, that is, 𝗣*

_{Si}_{i}

^{−}= (𝗣

*𝗣*

_{Si}*𝗣*

_{C}*)*

_{Si}^{−}. The specification of the correlation structure is a critical issue, because it selects the area from which observations are allowed to influence the model variables. Mostly, stationary, homogeneous, horizontal/vertical separable, and isotropic error covariance functions as functions of distance have been proposed for which only a few parameters were to be estimated, for example, by the traditional innovation or observational method (Hollingsworth and Lönnberg 1986; Daley 1991; Houser et al. 1998). With the implementation of the Kalman filter, the focus was moved to the definition of the model error covariance (Daley 1992a), because 𝗣

_{i}

^{−}can easily be calculated by the explicit Lyapunov propagation equation in linear stationary systems as the sum of the predictability and model error covariances. For optimization, several adaptive filters have been developed to estimate the model error covariances, mostly involving some whitening of the innovation (observation minus forecast) sequence (Kailath 1968; Mehra 1970; Maybeck 1982) or matching (elements of) the zero lag innovation covariance matrix to the theoretically optimal values (Jazwinski 1969; Myers and Tapley 1976; Maybeck 1982; Mehra 1972; Dee 1995; Dee and da Silva 1999). The majority of the techniques focus more on the estimation of the error variances than on the error correlation. In nonlinear systems, as are most land surface models, ensembles allow a flow-dependent statistical approximation of 𝗣

_{i}

^{−}for the EnKF (Evensen (1994)). However, the EnKF only quantifies the uncertainty in the space spanned by the ensembles, and it is particularly difficult to reflect the “true” uncertainty in the initial or analysis state and model error into an optimal perturbation magnitude and structure for the forcings, model structure, state, and parameters. It is often useful to tune the filter by adapting the error variance—for example, by inflation, to avoid model divergence in case of an underestimated a priori error covariance. An adaptive inflation factor in time was proposed by Anderson (2007), who adjusted the variance of each state component, whereas the correlation between pairs of components remained unchanged. Mitchell and Houtekamer (2000) applied the maximum-likelihood method to estimate model error variance parameters in an EnKF framework, where initially only the uncertainty in the initial state or analysis was considered.

In hydrology, the statistical nature of the a priori (forecast) errors has often been prescribed in synthetic studies or arbitrarily chosen in real studies. Sometimes, the filter performance is evaluated afterward by providing innovation statistics (Reichle et al. 2002a; Crow 2003; De Lannoy et al. 2007b). In synthetic studies, filters can be calibrated by contrasting the filter output to some truth, as in Reichle et al. (2002b) and Reichle and Koster (2003), to allow for a comparison of several Kalman filter setups for soil moisture estimation, each in optimal conditions. If no truth is available, the innovation statistics could be used to optimize the filter. Van Geer et al. (1991) and Zhang et al. (2004) applied the covariance matching technique (Mehra 1972) to optimize the uncertainty estimation in groundwater and soil moisture simulations, respectively. In the covariance matching method, the innovation covariance matrix, obtained as a result from the filter, is iteratively contrasted to its theoretical value, which is a function of the tunable model error covariance matrix. The innovation covariance matrix is typically calculated as the outer product of the centralized innovation vectors over some time (proxy of ensemble). Drécourt et al. (2006) extended and changed this approach to estimate both the system and bias model error covariance in a synthetic groundwater simulation study. Crow and van Loon (2006) applied the maximum-likelihood estimation (Dee 1995) to find a covariance scaling parameter to adjust the error variances only, while assuming a fixed, predefined (spatial) correlation structure. This approach involves the inner product (or, the trace of the outer product) of the innovation vectors—that is, all the variables’ uncertainties are lumped into one value. They illustrated how tuning the wrong error source to produce better innovation statistics led to a reduced accuracy of the filter results, when only assimilating surface soil moisture. Additionally, there was a risk to find the “globally” best innovation statistics through the optimization of the wrong error type, if different sources of error were taken into account. Reichle et al. (2008) used the same maximum-likelihood approach to identify model and observation error variances for soil moisture assimilation in a synthetic experiment. The multiscale Kalman filter (Parada and Liang 2004; Kumar 1999) efficiently considers the spatial dependence and scaling properties of soil moisture. Parada and Liang (2004) applied a multiscale Kalman filter in conjunction with an expectation maximization algorithm to estimate temporally evolving statistical observation and model error parameters inherent to the Kalman filter to assimilate surface soil moisture in an LSM. Besides interest in the 𝗣_{i}^{−}, observation uncertainty (error variance) is also an important factor in filtering. In several observation system simulation experiments (OSSEs), for example by Walker and Houser (2004), researchers have tried to quantify the maximal acceptable (to be useful in data assimilation) observation uncertainty to guide instrumentation design. It is reasonable to assume that, similar to forecast error, observation error variance is also state dependent and hence an adaptive filtering approach could be beneficial (Romanowicz et al. 2006).

Here, in situ soil moisture profiles in a small agricultural field [Optimizing Production Inputs for Economic and Environmental Enhancement (OPE^{3})] are simulated by individual, unconnected soil columns in the Community Land Model, version 2.0 (CLM2.0). Ensembles are used to quantify the variables’ uncertainty in each individual profile. Then, the ensemble a priori error variance is adapted, and the error correlation between the profiles is sought for in an adaptive filtering scheme to introduce lateral flow of information through the assimilation scheme. The adaptive information is extracted from as many field observations as available. In a successive time period, this information is used to update the complete field while assimilating only a few observations. This study differs from the EnKF approach of Reichle and Koster (2003), who imposed and calibrated (against some truth) the model error field in a synthetic study to impose lateral flow of information. Although data assimilation updates the observable prognostic state variables, it also adjusts the unobserved diagnostic flow variables. This study only focuses on the possibility of propagating soil moisture profiles horizontally and only validates observable prognostic soil moisture. An earlier study (De Lannoy et al. 2007b) highlighted the effect of soil moisture assimilation on diagnostic evapotranspiration, subsurface drainage, and runoff.

## 2. Model and observations

^{3}experiment is conducted (Gish et al. 2002; De Lannoy et al. 2006b). The model simulates land surface processes by calculating hourly vertical water and heat fluxes and states for each profile without interaction between profiles—that is, the linearized system matrix is block diagonal. To limit the computational costs, only the 10-layer soil moisture prognostic variables are included in the assimilation scheme—that is, the state vector

**x̂**

_{i}has a dimension of 360. The depths of the soil nodes are set to 2.5, 5, 10, 20, 30, 50, 80, 120, 150, and 180 cm. The nonlinear land surface model

**f**

_{i,i−1}propagates the state vector from time step

*i*− 1 to

*i*: where

**u**

*is the meteorological forcings, which are hourly observed data and assumed spatially homogeneous over all the profiles in this study. The initial condition for the land surface model is the a posteriori state estimate (or analysis)*

_{i}*i*− 1 (discussed below).

^{3}field is shown in Fig. 1. The observations

**y**

*(vector dimension varies with the specific assimilation experiment) are linearly related to the state by the observation (or forward) operator 𝗛*

_{i}*, that is, where*

_{i}**v**

*is zero-mean assumed random error. Here, 𝗛*

_{i}*contains only values of 1 and 0, because the observations are direct measurements of the state variables.*

_{i}The CLM2.0 was calibrated and initialized for each individual profile through weak constraint variational assimilation of observed data from 1 May through 1 October 2001 (De Lannoy et al. 2006a), with a focus on parameter estimation in the period from 2 September through 2 October 2001. All state updating experiments in this paper start after this calibration period. A limited subset of observations will be assimilated for state and uncertainty estimation, while withholding other soil moisture profile observations for validation of the filtering performance. This study only focuses on assimilation of “complete” profiles. Depending on the type of probe and the availability of data, this means simultaneous assimilation of observations at three up to seven soil depths.

## 3. EnKF implementation

*N*= 64 ensemble (Monte Carlo) member predictions

**w**

_{j,i}is a realization of the (zero-mean assumed) model error that represents the complete effect of perturbations to forcings, parameters, and state variables, quantified as in De Lannoy et al. (2006a). The a priori state (background) error covariance 𝗣

_{i}

^{−}is initially obtained from the ensemble sample covariance

**y**

*are available, each ensemble member*

_{i}*j*is updated individually to obtain the a posteriori state estimate:

**x̂**

_{i}

^{+}is the ensemble mean. The term

**r**

_{i}

^{−}. The observations are perturbed to ensure sufficient spread—that is,

**y**

_{j,i}=

**y**

*+*

_{i}**v**

_{j,i}, where

**v**

_{j,i}is an imposed zero-mean Gaussian perturbation with an error covariance of 𝗥 = (0.022 m

^{3}m

^{−3})

^{2}𝗜, based on the sensor calibration information and assuming zero cross correlation between the observation errors (De Lannoy et al. 2007b). The Kalman gain 𝗞

*is identical for all ensemble members and given by*

_{i}Basically, the term 𝗣_{i}^{−}𝗛_{i}^{T} maps the analysis increments on the model grid. If no observations are available, then **x̂**_{i}^{+} in Eq. (1) simply equals **x̂**_{i}^{−}.

## 4. Adaptive EnKF (ADEnKF)

### a. Method

The correct characterization of 𝗣_{i}^{−} largely determines the success of the Kalman filter. A first approximation of 𝗣_{i}^{−} is the ensemble-based _{i}^{−}, the adaptive scheme is also explored to retrieve this “missing” between-profiles correlation information and hence allow horizontal information propagation (Fig. 2).

*as part of 𝗣*

_{i}_{i}

^{−}= 𝗣

_{i}

^{p,−}+ 𝗤

*, assuming that predictability error, which is the error in the forecasts resulting from errors in the previous analysis or initial conditions (with covariance 𝗣*

_{i}_{i}

^{p,−}), and model error (with covariance 𝗤

*) are independent (Daley 1992a). One of the schemes that allows for the estimation of all individual elements in 𝗤*

_{i}*is the method of Myers and Tapley (1976). It introduces the analysis (a posteriori estimate) as a proxy for the true state and an approximation of model noise*

_{i}**w***

*, which in fact equals the analysis increment 𝗞*

_{i}*[*

_{i}**y**

_{i}− 𝗛

_{i}**x̂**

_{i}

^{−}]. For an optimal Kalman filter, the covariance of this quantity is given by where

*E*[

**w***

*,*

_{i}**w***

*] =*

_{i}*E*[(

**w***

*−*

_{i}**w***

*−*

_{i}^{T}] and

_{i}

^{+}is the a posteriori (analysis) error covariance matrix and the predictability error covariance matrix 𝗣

_{i}

^{p,−}can be calculated by propagating the previous a posteriori (or analysis) error covariance at

*i*− 1 by the model to time

*i*(for a linearized model version

*from Eq. (10) introduces a time averaging over a window of*

_{i}*λ*time steps:

An analysis increment time series (window of *λ* elements) is stored and centralized by its time mean 〈**w***〉, to eliminate the time mean bias. The time mean approach is basically to obtain a statistically viable estimate of the analysis increment covariance matrix as an approximation of the ensemble mean analysis increment covariance matrix. The advantage of this method, as compared to the computationally more attractive approach of Dee (1995), is its ability to tune the variance at each state variable individually, whereas simpler methods typically come up with an inflation factor to homogeneously scale the a priori covariance matrix by a constant value. The latter approach could yield adverse results, as suggested by the findings of Crow and van Loon (2006).

*is the unknown part of 𝗣*

_{i}_{i}

^{−}= 𝗣

_{i}

^{p,−}+ 𝗤

*, we now have different missing information in 𝗣*

_{i}_{i}

^{−}. With the EnKF, the objective is to determine 𝗣

_{i}

^{−}based on information in the ensembles: a good approach of the error covariance would result from the propagation of the ensemble through the model dynamics. However, here the model does not simulate lateral flow, so only a part of the actual 𝗣

_{i}

^{−}is captured by the ensemble information, that is

_{i}

^{p,−}and 𝗤

*, respectively) in the diagonal blocks by perturbing the initial conditions, parameters, and forcings for each profile individually. The ensembles do not provide any information on the covariance between errors in variables of different profiles (off block diagonal) because of a model structural deficiency (no lateral flow). This is now the missing information in 𝗣*

_{i}_{i}

^{−}. A new matrix,

_{i}

^{−}: The method of Myers and Tapley (1976), as in Eq. (11), is slightly changed to obtain

*i*=

*λ*,

_{λ}

^{−}= 𝗣

_{λ}

^{p,−}+ 𝗤

*. The practical implementation is as follows: at each time step, (i) calculate the matrix [*

_{λ}**w***

*− 〈*

_{i}**w***〉] [

**w***

*− 〈*

_{i}**w***〉]

^{T}and (ii) derive

_{i}

^{+}from the ensemble state members, the latter after having filtered observations with

*λ*= 40 previous assimilation time steps for a filtering frequency of once every two days, from 21 December 2001 through 19 March 2002. For

*λ*≤ 20, we found that the retrieved matrix information could not be statistically consistent, because it was too sensitive to information brought in at additional time steps.

_{i}

^{−}for the filtering in a next assimilation period. The variances on the diagonal of

*i*, that is:

_{i}

^{−}]

*would result in a negative value, then [𝗣*

_{kk}_{i}

^{−}]

*is not corrected by Eq. (14), but we choose to set [𝗣*

_{kk}_{i}

^{−}]

*/2. Next, the covariances within the blocks are updated for the corrected variances [𝗣*

_{kk}_{i}

^{−}]

*(the ensemble correlations are unaltered, but the variances are updated) as follows: if*

_{kk}*k*and

*l*belong to the same profile. Outside the blocks, the covariances are calculated with the time-invariant correlations found in

*k*and

*l*belong to different profiles. Here,

*k*th state variable, averaged over the

*λ*assimilation time steps. The off-block-diagonal elements in

_{i}

^{−}is not imposed on the ensemble state members along the model run but only used for the calculation of the Kalman gain. In summary, we apply a hybrid method to fully determine 𝗣

_{i}

^{−}: ensemble-based state-dependent variances are corrected by the adaptive information and are used together with (i) ensemble-based state-dependent correlations in the blocks and (ii) time-invariant correlations outside the blocks. Unlike many EnKF studies here, no covariance localization (Hamill et al. 2002; Reichle and Koster 2003) is applied after filling the off-block-diagonal elements in 𝗣

_{i}

^{−}.

### b. Innovations versus analysis increments

_{i}[

**y**

_{i}− 𝗛

_{i}

**x̂**

_{i}

^{−}]) rather than the innovations (

**y**

_{i}− 𝗛

_{i}

**x̂**

_{i}

^{−}), which are used in most other adaptive filters. A direct consequence is that all involved matrices as well as the resulting

*k*,

*l*) of observation locations: where

*E*[

**r**

_{i}^{−},

**r**

_{i}^{−}] is the ensemble covariance matrix at one time step

*i*. In practice, time series of (ensemble mean) innovations

**r**

_{k}^{−}, calculated during the EnKF application at individual locations

*k*, can be used as a proxy to estimate the ensemble innovation covariance. The centralized second-order statistic is calculated to remove the time mean discrepancy between forecasts and observations. Because in our study, a reasonable time-invariant estimate of the diagonal 𝗥 matrix is available—that is, [𝗥]

*= (0.022 m*

_{kk}^{3}m

^{−3})

^{2}and [𝗥]

*= 0 (m*

_{kl}^{3}m

^{−3})

^{2}—the a priori error correlations

*ρ*can be estimated by where

_{kl}*E*[[

**r**

^{−}]

*, [*

_{k}**r**

^{−}]

*] =*

_{l}*E*[([

**r**

^{−}]

*− 〈[*

_{k}**r**

^{−}]

*〉) ([*

_{k}**r**

^{−}]

*− 〈[*

_{l}**r**

^{−}]

*〉)*

_{l}^{T}] is calculated over time and 〈[

**r**

^{−}]

*〉 is the time mean innovation for the*

_{k}*k*th element of vector

**r**

^{−}. For each pair of elements, this correlation can then be plotted as a function of the distance and an empirical homogeneous (in space) function can then be fitted.

In this study, we could easily change this procedure, or any other innovation-based adaptive method, to find the off-block-diagonal correlation values of _{i}^{−} in the state space as well as to find the variance correction, like in our adapted Myers and Tapley (1976) method. However, that would only be possible in the case of this study, where 𝗛* _{i}* only contains 0 and 1, and the transfer of information from the observation to the state space would be well determined (leaving unobserved entries in 𝗣

_{i}

^{−}with some default or zero correlation). For more general and complex observation operators 𝗛

*, the method of Myers and Tapley (1976) is more attractive than other adaptive filters to find*

_{i}## 5. Results

### a. Innovation analysis: Correlation structure

Before implementing the adaptive method based on Myers and Tapley (1976) described above, it was tested if a time-invariant horizontal error correlation structure could be defined as a function of separation distance between profiles, based on the information in the innovation covariances, as in the traditional innovation method.

For a two-weekly assimilation of complete profiles from 2 October 2001 through 19 March 2002, the resulting 13-element time series of (ensemble mean) innovations were used to approximate a time-invariant ensemble innovation covariance. The deduced background error correlations in Fig. 3 show that it is impossible to fit an empirical homogeneous isotropic correlation function to the calculated forecast error correlations at any observation depth. Similar results were found for scenarios with more frequent assimilation (and hence more elements in the innovation time series).

The CLM does not simulate horizontal water flows and is applied for each profile individually, while studies in the OPE^{3} field have revealed pathways for active preferential horizontal flow (Gish et al. 2002). It could be expected that profiles situated along these pathways may, therefore, show correlated model structural errors and hence correlated innovations. However, there were too few reported actively connected probes with available observations (and hence innovations) at corresponding soil depths to quantitatively analyze this hypothesis. Furthermore, the innovation correlation as a result of correlation in model structural errors may have been reduced by error compensation during the individual profile calibration procedure.

The lack of spatial error structure in Fig. 3 is probably because (i) the error correlation structure changes with direction (i.e., anisotropy), (ii) the different profiles were calibrated independently, and (iii) this is a small field-scale analysis, where the local highly variable soil conditions (and their potential parameterization errors) largely determine a close to random spatial soil moisture structure (and hence the derived innovation statistics). Note that at the regional scale, we might find spatial correlation structures controlled by forcing error patterns.

Figure 3 also shows that error correlations are higher and less variable in shallow layers than in deeper profile layers. Similarly, the analysis of the spatial structure in the soil moisture observations showed that the correlation (and correlation length) was higher in shallow layers (De Lannoy et al. 2006b). This suggests that for highly correlated time series of observed soil moisture, highly correlated innovation time series are also found. This is reasonable, because the calibrated model deficiencies will be similar for the similarly observed soil moisture time series. Basically, the innovation time sequences for each point will contain sequences of short time-scale biases that might be cross-correlated. When a sequential bias estimation (De Lannoy et al. 2007b) was included and the bias was removed from the innovations at each assimilation step, before the innovation time series was centralized by its time mean (longer time-scale “bias”), the correlations in the top layers indeed dropped down (results not shown).

To study the correlation stationarity in time, daily assimilation of complete profiles was performed over the same period (2 October 2001 through 19 March 2002, now resulting in 167-element time series). Through batch processing, the state error correlation was calculated from the innovations over 5 (and 10) windows of 33 (and 16) hourly time steps. A random selection of soil moisture state variable pairs at identical depths (10 cm in Fig. 4) shows that the retrieved error correlation values are nonstationary at the high-resolution time scale (e.g., 10 windows) but change relatively little when smoothed over fewer windows.

### b. Adaptive filter: Spatial effects

For the main experiment, the combined state estimation (updating) and uncertainty estimation was performed during the period from 2 October through 21 December 2001 with the adapted Myers and Tapley (1976) method. All 36 complete observed soil moisture profiles were assimilated every two days (40 time steps) to calculate missing off-block-diagonal information and to obtain a correction to the variances in the a priori error covariance. Thereafter, from 23 December 2001 to 19 March 2002, only a single profile was assimilated using the novel error covariance information to enhance the soil moisture estimates at all 36 simulated soil moisture profiles in the field. During this period the ensemble-based

Figure 5 shows the a priori state error correlation matrix on 23 December 2001, prior to and after the adaptive correction. It illustrates how the ensemble-based block-diagonal (i.e., within a profile) correlations are kept unaltered and how the off-block-diagonal (i.e., between profiles) error correlation elements are partially filled in. For a random zoom example, where error correlations between state variables corresponding to profile AL2 and BM4 are shown, only six rows (state variables corresponding to the six observation layers in M-probes) and seven columns (seven observation layers in L-probes) are filled in. The more observational layers available for a pair of profiles, the more error correlation information can be retrieved. Because the model layers at 2.5, 5, and 20 cm are never observed by the OPE^{3} probes, there will never be information on error statistics at these layers. Again, no spatial structure could be found in the retrieved error correlations at any soil layer (not shown). Similar to the above findings, the determined error correlation was higher in the top soil layer but much smaller than what was found through the innovation method.

Figure 6 shows for two individual soil depths at 10 and 80 cm, the temporal evolution of the spatial root-mean-square error (RMSE) between observations and ensemble mean forecasts without assimilation as well as analyses with assimilation of complete observed soil moisture profile data. During the first period (from 2 October to 21 December 2001) all available profile information is assimilated, causing significant drops in RMSE. However, the observational information is not taken up completely by the model at all profiles, mainly as a result of forecast bias (sometimes the model and observation climatologies are different; De Lannoy et al. 2007b), causing a rapid increase of RMSE after each assimilation event. In the second period, only one selected profile (here, DL1) is assimilated, either without or with inclusion of the estimated off-block-diagonal a priori error covariance elements. The further reduced RMSE, when including the off-block-diagonal information, shows that there is a benefit to spreading information in the horizontal space.

As an illustration, the spatial soil moisture fields in Fig. 7 are generated for 13 March 2002 as a weighted average of two spatial interpolations of the individual point analysis results (only for visualization purposes) by (i) a multivariate regression between the soil moisture and the terrain characteristics (texture, elevation, and topographic index) at the observing locations and (ii) kriging with an exponential variogram with parameters as observed in De Lannoy et al. (2006b). The observations and simulations show a discrepancy in the spatial soil moisture field: a part is a result of model bias, another part results from random error. When assimilating the complete observed profile at DL1 with the regular EnKF, no other profile is updated and the spatial view is dominated by the ensemble mean integration without assimilation. When the estimated off-block-diagonal a priori error covariance elements are included, ADEnKF of DL1 observations updates all profiles in the field. The full circles in Fig. 7 show regions where the ADEnKF obviously brings the simulated field close to the observed one, while only assimilating data at one location in the field and using time-invariant correlation information off the diagonal blocks in 𝗣_{i}^{−}. The dashed circle shows a region where a major dry forecast bias cannot be dealt with by a state filter and where the adaptive filter only has a minor effect, because it is mainly designed to treat random error and probably also because of the time-invariant correlation structure. There is an attempt to increase the soil moisture by the ADEnKF, but it is mainly effective only at one location (CL2).

Figure 8 summarizes the effect on the spatial soil moisture estimation, when assimilating observations from each individual profile only, during a validation period from 23 December 2001 through 24 March 2002. The time-averaged spatial RMSE is shown for the case without and with inclusion of ^{3} field (De Lannoy et al. 2007a; i.e., its measured soil moisture time series best resembles the spatial mean soil moisture behavior in time at most soil depths). It should also be noted that validation of the performance in a single layer sometimes shows further reduced spatial RMSE values, when only assimilating a single observation in that single layer (not shown). This is because when updating the soil moisture in many layers simultaneously, the analysis profile may not reflect the preferred model balance and seemingly contradictory (to the model) updates for the different layers may be cancelled out quickly over time after the analysis. The update in a single layer often persists longer in time, when neighboring layers are adapting to the update in that single layer, rather than possibly competing against this update with updates at their own respective depths. It is evident that, when evaluating the total field profile performance (as opposed to the previous discussion on the effect in a single layer), assimilation with probes having more observational depths (six for M-probes and seven for L-probes) generally results in the best improvements in RMSE. However, it is important to notice how H-probes with only three observation layers are able to enhance the spatial RMSE at unobserved (e.g., 180 cm) layers.

In some cases (here, it is for 80-cm depth), the RMSE is increased, as a result of either a bad propagation of information over the vertical layers or an inaccurate determination of the between-profiles error correlation. In an exceptional case, the assimilation of a single profile did not improve the soil moisture field estimate in any layer: through assimilation of probe BL2, the spatial RMSE was increased outside the range of the plot for almost all soil layers when including the spatial error correlation information. The problem here was a collapsed ensemble spread (and the time-invariant adaptive correction was inappropriate to solve this problem) at some layers, where a model-defined minimal soil moisture was reached, causing adverse effects in other layers at distant profiles.

*γ*between the observed (

*γ*) and modeled (

_{o}*γ*) spatial soil moisture fields over the same period of length

_{m}*T*, The spatial lag in the calculation of

*γ*and

_{o}*γ*was set at 20 m (to allow some averaging over pairs of observations in each spatial lag-interval) for a maximum interdistance of 420 m (beyond half of the field length, semivariance values would be calculated for too few sample pairs). The mean square difference in semivariances was thus calculated over a maximum number of 21 spatial lags, that is,

_{m}*L*= 21. Table 1 shows that, through simple EnKF, the spatial pattern was slightly improved, again because of the enhanced representation of soil moisture at the assimilation location only. With the inclusion of adaptive information, the differences in semivariances were further reduced. Consequently, assimilation of a single profile (here, DL1) with an adaptive EnKF is able to partially rectify both incorrectly simulated spatial soil moisture mean and variance.

## 6. Discussion and perspectives

In this paper, the missing error correlations and an error covariance correction are retrieved only over a single time window and are to be used in a period succeeding the period where it was actually calculated. A more frequent batch processing could be suggested to obtain more time-variant adaptive information. Definitely for large-scale applications, where the variability in the precipitation errors may alter the error correlations over time, a frequent update of the missing error correlations would be beneficial. However, there is a trade-off between an increased time variability and the statistical consistency of the error covariance estimates, as found in Eq. (13): some time averaging is needed to obtain viable covariance estimates. Future research could investigate how the training period can be limited by exploring the ensemble information in the innovations at each time step, instead of studying the ensemble mean information in time.

The analysis results can be further improved by adding a bias filter to the state estimation (De Lannoy et al. 2007b). However, since the bias error covariance is often linked to the (adapted) a priori error covariance, more research is needed to assure both proper interactions between these error covariance and the optimality of the filter. Note that the Myers and Tapley (1976) method removes some bias from the increments to find

This approach to determine the between-profiles error covariance can be of practical importance to propagate satellite observations from observed locations to temporarily masked or unobserved regions (e.g., clouds in visual products and hydrometeors for high-frequency microwave products), or propagation of in situ profile information to locations where sensors are temporarily out of service. In satellite data assimilation schemes, the modeled grid cells are typically covered by the observations (after reprojecting the data). This allows an update of all the a priori error correlations at each time observations are available and use them, whenever part of the observations falls out and information needs to be propagated to the nonobserved pixels. Furthermore, training of the correlations during one satellite pass (covering a limited banded area) allows observations in the overlapping area of the next pass to be propagated all over the area covered by previous (shifted) pass, if some time-invariance in error correlations could be assumed. However, for large-scale applications, we generally assume that the a priori error correlation would be limited (e.g., different dominant forcing errors) and not worth the increased numerical expenses to deal with a more complex 𝗣_{i}^{−}. As a very simple approximation, the actual (mostly observed) state correlation length is often used to limit the a priori state error correlations at large distances, even though we should realize that the actual state and its estimation errors (and hence their respective correlations lengths) are in fact independent.

A problem is that with a finer model resolution, more observations are needed to retrieve the correct error covariances. If a soil moisture profile is available every 20 m (assume it representative for a 10 × 10 m^{2} area) and the model simulates at a 10-m resolution, then the correlation between the forecast errors of two neighboring grid cells at 10 m apart will not be retrieved by the proposed technique, whereas the correlation between forecast errors in two observed grid cells at 20 m apart can be found. One limitation of the proposed method is the need to determine the adaptive information based on as many observations as available, before it can be used with a restricted dataset.

In general, the proposed technique is expected to advance the optimal use of satellite and other data to fill in unobserved areas and to overcome the shortcomings of completely time invariant homogeneous (in space) correlation matrices, as in optimal interpolation techniques. Typical empirical distance-dependent correlation functions may give a false impression of smooth correlation structures in the analyses. It could be advised to review assumptions on empirical correlation functions for hydrological variables at different scales by dynamical error correlation insights obtained from adaptive filters.

## 7. Summary

This study explores the potential of updating the full 3D system state when assimilating only single soil moisture profile data in the independent column-based CLM2.0 with an adaptive ensemble Kalman filter (ADEnKF). Assimilation of individual in situ soil moisture profiles from the small-scale OPE^{3} field with an optimized knowledge of a priori error covariances resulted in a partial rectification of the inaccurately simulated spatial mean and variability in soil moisture fields at different soil depths.

Because no simple spatial error correlation structure could be found based on the traditional innovation method, it was necessary to find all elements of the a priori error covariance matrix individually. Ensembles were used to obtain (i) a first guess of each state variable uncertainty, that is, the magnitude of error variance, and (ii) an estimate of the error correlation between state variables within each individual profile. Then, during a first period, a time-invariant variance correction and the between-profiles error correlations were sought based on an adaptive method of Myers and Tapley (1976). This allowed for the adjustment of the time-variable ensemble-based state error covariance matrix before each filtering time step in the successive period.

Even though only one time-invariant estimate was used to correct the a priori error covariances, the results show improvements up to between 7% and 22% reduction in spatial RMSE in a single soil layers, when assimilating only a single complete profile [higher percentage reductions can be achieved by assimilating only in the single validation layer, which avoids contradictory updates over the different soil layers (not shown)]. Even the assimilation of a single profile with only a few observed layers resulted in enhanced soil moisture field estimates at most depths. With knowledge of the off-block-diagonal a priori error covariances (ADEnKF), any observation in a single layer will affect both the total profile where it is assimilated in and all layers of surrounding profiles, while a regular EnKF with the CLM2.0 only allows for updating of a single profile.

This study shows an example of retrieving the missing error correlations and an error covariance correction during a first time window and using it in a period succeeding the period where it was actually calculated. A more frequent batch processing would result in more time-variant adaptive information and the information in the ensembles may be further explored to limit the training periods, while keeping the estimates statistically consistent.

The determination of between-profiles error covariance can be important to propagate satellite observations from observed locations to temporarily unobserved or masked regions or propagation of in situ profile information to locations where sensors are temporarily out of service. The limitation of this method is the need to determine the adaptive information based on as many observations as available, before it can be used with a restricted dataset.

## Acknowledgments

The first author is a postdoctoral research fellow of the Research Foundation Flanders (FWO). We thank the USDA Beltsville Agricultural Research Center/Agricultural Research Service for their data. The reviewers are thanked for their constructive comments.

## REFERENCES

Anderson, J. L., 2007: An adaptive covariance inflation error correction algorithm for ensemble filters.

,*Tellus***59A****,**210–224.Crow, W. T., 2003: Correcting land surface model predictions for the impact of temporally sparce rainfall rate measurements using an ensemble Kalman filter and surface brightness temperature observations.

,*J. Hydrometeor.***4****,**960–973.Crow, W. T., , and van Loon E. , 2006: Impact of incorrect model error assumptions on the sequential assimilation of remotely sensed surface soil moisture.

,*J. Hydrometeor.***7****,**421–432.Dai, Y., and Coauthors, 2003: The Common Land Model.

,*Bull. Amer. Meteor. Soc.***84****,**1013–1023.Daley, R., 1991:

*Atmospheric Data Analysis*. Cambridge University Press, 455 pp.Daley, R., 1992a: Estimating model-error covariances for application to atmospheric data assimilation.

,*Mon. Wea. Rev.***120****,**1735–1746.Daley, R., 1992b: Forecast-error statistics for homogeneous and inhomogeneous observation networks.

,*Mon. Wea. Rev.***120****,**627–643.Dee, D. P., 1995: On-line estimation of error covariance parameters for atmospheric data assimilation.

,*Mon. Wea. Rev.***123****,**1128–1145.Dee, D. P., , and da Silva A. M. , 1999: Maximum-likelihood estimation of forecast and observation error covariance parameters. Part I: Methodology.

,*Mon. Wea. Rev.***127****,**1822–1834.De Lannoy, G. J. M., , Houser P. R. , , Pauwels V. R. N. , , and Verhoest N. E. C. , 2006a: Assessment of model uncertainty for soil moisture through ensemble verification.

,*J. Geophys. Res.***111****,**D10101. doi:10.1029/2005JD006367.De Lannoy, G. J. M., , Verhoest N. E. C. , , Houser P. R. , , Gish T. , , and Van Meirvenne M. , 2006b: Spatial and temporal characteristics of soil moisture in an intensively monitored agricultural field (OPE3).

,*J. Hydrol.***331****,**719–730.De Lannoy, G. J. M., , Houser P. R. , , Verhoest N. E. C. , , Pauwels V. R. N. , , and Gish T. , 2007a: Upscaling of point soil moisture observations to field averages at the OPE3 site.

,*J. Hydrol.***343****,**1–11.De Lannoy, G. J. M., , Reichle R. H. , , Houser P. , , Pauwels V. R. N. , , and Verhoest N. E. C. , 2007b: Correcting for forecast bias in soil moisture assimilation with the ensemble Kalman filter.

,*Water Resour. Res.***43****,**W09410. doi:10.1029/2006WR005449.Drécourt, J-P., , Madsen H. , , and Rosbjerg D. , 2006: Joint calibration and uncertainty analysis of a groundwater model coupled with a Kalman filter.

,*Adv. Water Resour.***29****,**719–734.Evensen, G., 1994: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics.

,*J. Geophys. Res.***99****,**(C5). 10143–10162.Gish, T., , Dulaney W. , , Kung K-J. S. , , Daughtry C. , , Doolittle J. , , and Miller P. , 2002: Evaluating use of ground-penetrating radar for identifying subsurface flow pathways.

,*Soil Sci. Soc. Amer. J.***66****,**1620–1629.Hamill, T., , Snyder C. , , and Morss R. , 2002: Analysis-error statistics of a quasigeostrophic model using three-dimensional variational assimilation.

,*Mon. Wea. Rev.***130****,**2777–2790.Hollingsworth, A., , and Lönnberg P. , 1986: The statistical structure of short-range forecast errors as determined from radiosonde data. Part I: The wind field.

,*Tellus***38A****,**111–136.Houser, P. R., , Shuttleworth W. J. , , Famiglietti J. S. , , Gupta H. V. , , Syed K. H. , , and Goodrich D. C. , 1998: Integration of soil moisture remote sensing and hydrologic modeling using data assimilation.

,*Water Resour. Res.***34****,**3405–3420.Huang, H-C., , and Cressie N. , 1996: Spatio-temporal prediction of snow water equivalent using the Kalman filter.

,*Comput. Stat. Data Anal.***22****,**159–175.Jazwinski, A., 1969: Adaptive filtering.

,*Automatica***5****,**475–485.Kailath, T., 1968: An innovation approach to least-squares estimation. Part I: Linear filtering in additive white noise.

,*IEEE Trans. Autom. Control***AC-13****,**646–655.Kumar, P., 1999: A multiple scale state-space model of characterizing subgrid scale variability of near-surface soil moisture.

,*IEEE Trans. Geosci. Remote Sens.***37****,**182–197.Maybeck, P. S., 1982:

*Stochastic Models, Estimation, and Control*. Vol. 2. Academic Press, 423 pp.McLaughlin, D., 2002: An integrated approach to hydrologic data assimilation: Interpolation, smoothing, and filtering.

,*Adv. Water Resour.***25****,**1275–1286.Mehra, R., 1970: On the identification of variances and adaptive Kalman filtering.

,*IEEE Trans. Autom. Control***15****,**175–184.Mehra, R., 1972: Approaches to adaptive filtering.

,*IEEE Trans. Autom. Control***17****,**693–698.Mitchell, H. L., , and Houtekamer P. L. , 2000: An adaptive ensemble Kalman filter.

,*Mon. Wea. Rev.***128****,**416–433.Myers, K., , and Tapley B. , 1976: Adaptive sequential estimation with unknown noise statistics.

,*IEEE Trans. Autom. Control***21****,**520–523.Parada, L. M., , and Liang X. , 2004: Optimal multiscale Kalman filter assimilation for near-surface soil moisture into land surface models.

,*J. Geophys. Res.***109****,**D24109. doi:10.1029/2004JD004745.Reichle, R. H., , and Koster R. , 2003: Assessing the impact of horizontal error correlations in background fields on soil moisture estimation.

,*J. Hydrometeor.***4****,**1229–1242.Reichle, R. H., , McLaughlin D. B. , , and Entekhabi D. , 2002a: Hydrologic data assimilation with the ensemble Kalman filter.

,*Mon. Wea. Rev.***130****,**103–114.Reichle, R. H., , Walker J. P. , , Houser P. R. , , and Koster R. D. , 2002b: Extended versus ensemble Kalman filtering for land data assimilation.

,*J. Hydrometeor.***3****,**728–740.Reichle, R. H., , Crow W. , , and Keppenne C. , 2008: An adaptive ensemble Kalman filter for soil moisture data assimilation.

,*Water Resour. Res.***44****,**W03423. doi:10.1029/2007WR006357.Romanowicz, R. J., , Young P. C. , , and Beven K. J. , 2006: Data assimilation and adaptive forecasting of water levels in the River Severn catchment, United Kingdom.

,*Water Resour. Res.***42****,**W06407. doi:10.1029/2005WR004373.Stauffer, D. R., , and Seaman N. L. , 1990: Use of four-dimensional data assimilation in a limited-area mesoscale model. Part I: Experiments with synoptic-scale data.

,*Mon. Wea. Rev.***118****,**1250–1277.van Geer, F. C., , Te Stroet C. B. , , and Yangxiao Z. , 1991: Using Kalman filtering to improve and quantify the uncertainty of numerical groundwater simulations. 1. The role of system noise and its calibration.

,*Water Resour. Res.***27****,**1987–1994.Walker, J. P., , and Houser P. R. , 2004: Requirements of a global near-surface soil moisture satellite mission: Accuracy, repeat time, and spatial resolution.

,*Adv. Water Resour.***27****,**785–801.Zhang, S-W., , Qiu C-J. , , and Xu A. , 2004: Estimating soil water contents from soil temperature measurements by using an adaptive Kalman filter.

,*J. Appl. Meteor.***43****,**379–389.

Time-averaged RMS difference between observed and modeled soil moisture spatial semivariance (DSsv, vol%^{2}) and spatial mean (DSm, vol%), over a validation period from 23 Dec 2001 though 24 Mar 2002. Ensemble mean integration without any assimilation is noted as Ens. EnKF and ADEnKF are for 2-daily ensemble Kalman filtering of the single DL1 profile during the period 23 Dec 2001 through 19 Mar 2002, without and with inclusion of adaptive information, respectively.