• Anderson, J. L., 2007: An adaptive covariance inflation error correction algorithm for ensemble filters. Tellus, 59A , 210224.

  • Crow, W. T., 2003: Correcting land surface model predictions for the impact of temporally sparce rainfall rate measurements using an ensemble Kalman filter and surface brightness temperature observations. J. Hydrometeor., 4 , 960973.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Crow, W. T., and van Loon E. , 2006: Impact of incorrect model error assumptions on the sequential assimilation of remotely sensed surface soil moisture. J. Hydrometeor., 7 , 421432.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Dai, Y., and Coauthors, 2003: The Common Land Model. Bull. Amer. Meteor. Soc., 84 , 10131023.

  • Daley, R., 1991: Atmospheric Data Analysis. Cambridge University Press, 455 pp.

  • Daley, R., 1992a: Estimating model-error covariances for application to atmospheric data assimilation. Mon. Wea. Rev., 120 , 17351746.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Daley, R., 1992b: Forecast-error statistics for homogeneous and inhomogeneous observation networks. Mon. Wea. Rev., 120 , 627643.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Dee, D. P., 1995: On-line estimation of error covariance parameters for atmospheric data assimilation. Mon. Wea. Rev., 123 , 11281145.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Dee, D. P., and da Silva A. M. , 1999: Maximum-likelihood estimation of forecast and observation error covariance parameters. Part I: Methodology. Mon. Wea. Rev., 127 , 18221834.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • De Lannoy, G. J. M., Houser P. R. , Pauwels V. R. N. , and Verhoest N. E. C. , 2006a: Assessment of model uncertainty for soil moisture through ensemble verification. J. Geophys. Res., 111 , D10101. doi:10.1029/2005JD006367.

    • Search Google Scholar
    • Export Citation
  • De Lannoy, G. J. M., Verhoest N. E. C. , Houser P. R. , Gish T. , and Van Meirvenne M. , 2006b: Spatial and temporal characteristics of soil moisture in an intensively monitored agricultural field (OPE3). J. Hydrol., 331 , 719730.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • De Lannoy, G. J. M., Houser P. R. , Verhoest N. E. C. , Pauwels V. R. N. , and Gish T. , 2007a: Upscaling of point soil moisture observations to field averages at the OPE3 site. J. Hydrol., 343 , 111.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • De Lannoy, G. J. M., Reichle R. H. , Houser P. , Pauwels V. R. N. , and Verhoest N. E. C. , 2007b: Correcting for forecast bias in soil moisture assimilation with the ensemble Kalman filter. Water Resour. Res., 43 , W09410. doi:10.1029/2006WR005449.

    • Search Google Scholar
    • Export Citation
  • Drécourt, J-P., Madsen H. , and Rosbjerg D. , 2006: Joint calibration and uncertainty analysis of a groundwater model coupled with a Kalman filter. Adv. Water Resour., 29 , 719734.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Evensen, G., 1994: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res., 99 , (C5). 1014310162.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gish, T., Dulaney W. , Kung K-J. S. , Daughtry C. , Doolittle J. , and Miller P. , 2002: Evaluating use of ground-penetrating radar for identifying subsurface flow pathways. Soil Sci. Soc. Amer. J., 66 , 16201629.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hamill, T., Snyder C. , and Morss R. , 2002: Analysis-error statistics of a quasigeostrophic model using three-dimensional variational assimilation. Mon. Wea. Rev., 130 , 27772790.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hollingsworth, A., and Lönnberg P. , 1986: The statistical structure of short-range forecast errors as determined from radiosonde data. Part I: The wind field. Tellus, 38A , 111136.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Houser, P. R., Shuttleworth W. J. , Famiglietti J. S. , Gupta H. V. , Syed K. H. , and Goodrich D. C. , 1998: Integration of soil moisture remote sensing and hydrologic modeling using data assimilation. Water Resour. Res., 34 , 34053420.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Huang, H-C., and Cressie N. , 1996: Spatio-temporal prediction of snow water equivalent using the Kalman filter. Comput. Stat. Data Anal., 22 , 159175.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jazwinski, A., 1969: Adaptive filtering. Automatica, 5 , 475485.

  • Kailath, T., 1968: An innovation approach to least-squares estimation. Part I: Linear filtering in additive white noise. IEEE Trans. Autom. Control, AC-13 , 646655.

    • Search Google Scholar
    • Export Citation
  • Kumar, P., 1999: A multiple scale state-space model of characterizing subgrid scale variability of near-surface soil moisture. IEEE Trans. Geosci. Remote Sens., 37 , 182197.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Maybeck, P. S., 1982: Stochastic Models, Estimation, and Control. Vol. 2. Academic Press, 423 pp.

  • McLaughlin, D., 2002: An integrated approach to hydrologic data assimilation: Interpolation, smoothing, and filtering. Adv. Water Resour., 25 , 12751286.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mehra, R., 1970: On the identification of variances and adaptive Kalman filtering. IEEE Trans. Autom. Control, 15 , 175184.

  • Mehra, R., 1972: Approaches to adaptive filtering. IEEE Trans. Autom. Control, 17 , 693698.

  • Mitchell, H. L., and Houtekamer P. L. , 2000: An adaptive ensemble Kalman filter. Mon. Wea. Rev., 128 , 416433.

  • Myers, K., and Tapley B. , 1976: Adaptive sequential estimation with unknown noise statistics. IEEE Trans. Autom. Control, 21 , 520523.

  • Parada, L. M., and Liang X. , 2004: Optimal multiscale Kalman filter assimilation for near-surface soil moisture into land surface models. J. Geophys. Res., 109 , D24109. doi:10.1029/2004JD004745.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Reichle, R. H., and Koster R. , 2003: Assessing the impact of horizontal error correlations in background fields on soil moisture estimation. J. Hydrometeor., 4 , 12291242.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Reichle, R. H., McLaughlin D. B. , and Entekhabi D. , 2002a: Hydrologic data assimilation with the ensemble Kalman filter. Mon. Wea. Rev., 130 , 103114.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Reichle, R. H., Walker J. P. , Houser P. R. , and Koster R. D. , 2002b: Extended versus ensemble Kalman filtering for land data assimilation. J. Hydrometeor., 3 , 728740.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Reichle, R. H., Crow W. , and Keppenne C. , 2008: An adaptive ensemble Kalman filter for soil moisture data assimilation. Water Resour. Res., 44 , W03423. doi:10.1029/2007WR006357.

    • Search Google Scholar
    • Export Citation
  • Romanowicz, R. J., Young P. C. , and Beven K. J. , 2006: Data assimilation and adaptive forecasting of water levels in the River Severn catchment, United Kingdom. Water Resour. Res., 42 , W06407. doi:10.1029/2005WR004373.

    • Search Google Scholar
    • Export Citation
  • Stauffer, D. R., and Seaman N. L. , 1990: Use of four-dimensional data assimilation in a limited-area mesoscale model. Part I: Experiments with synoptic-scale data. Mon. Wea. Rev., 118 , 12501277.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • van Geer, F. C., Te Stroet C. B. , and Yangxiao Z. , 1991: Using Kalman filtering to improve and quantify the uncertainty of numerical groundwater simulations. 1. The role of system noise and its calibration. Water Resour. Res., 27 , 19871994.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Walker, J. P., and Houser P. R. , 2004: Requirements of a global near-surface soil moisture satellite mission: Accuracy, repeat time, and spatial resolution. Adv. Water Resour., 27 , 785801.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zhang, S-W., Qiu C-J. , and Xu A. , 2004: Estimating soil water contents from soil temperature measurements by using an adaptive Kalman filter. J. Appl. Meteor., 43 , 379389.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • View in gallery

    Fig 1. Digital elevation model (dashed contour lines, 0.5-m interval) with indication of the boundary coordinates and location of soil moisture probes (three digits) in the OPE3 field with four subwatersheds, A–D. Defective probes during the study period are crossed out. The bottom sketch shows how the system is modeled as 36 individual, unconnected soil profiles with multiple soil layers.

  • View in gallery

    Fig 2. (top) Schematic of EnKF and ADEnKF functionality with the CLM, which only simulates vertical fluxes (black dashed arrows). The white arrows represent the information transfer from the observation in one particular layer to model states in other soil layers of the same profile (EnKF and ADEnKF) and neighboring profiles (ADEnKF only) by the a priori state error covariance statistics 𝗣i in the filter. (bottom) Matrices represent 𝗣i for two profiles and with multiple state variables per profile. For the EnKF, 𝗣i is block diagonal. For the ADEnKF, it is tried to (i) optimize the a priori error variances on the diagonal of 𝗣i and (ii) retrieve the off-block-diagonal error correlations between the different profiles, schematically indicated by crosshatched matrix parts.

  • View in gallery

    Fig 3. A priori horizontal error correlation as function of separation distance, calculated by the traditional innovation method at 10- and 80-cm depth, with 13 innovations obtained after assimilation of complete profiles every two weeks from 2 Oct 2001 through 19 Mar 2002.

  • View in gallery

    Fig 4. Calculated error correlation at 10-cm depth for some randomly selected pairs: + (DH1, A21), * (AM1, BL1), ▵ (AH2, DH3), ⋄ (BH2, BL2), □ (BL3, CL1), × (BM4, DM2). Small gray symbols are calculated over 10 windows of 16 hourly time steps and thick black symbols over 5 windows of 33 time steps. The symbols on the right are calculated over the total time series of 167 hourly time steps.

  • View in gallery

    Fig 5. A priori error correlation matrix (top) prior to and (bottom) after adaptive correction at 23 December 2001. Within each profile 10 soil moisture state variables (2.5, 5, 10, 20, 30, 50, 80, 120, 150, and 180 cm) are present. The absolute correlation values are shown.

  • View in gallery

    Fig 6. Spatial RMSE in soil moisture over all available sensors at two different depths. The arrows indicate the assimilation events: (top) from 2 Oct through 21 Dec 2001 all profile information is assimilated, (bottom) thereafter through 19 Mar 2002 only profile data from probe DL1 are assimilated. The gray zone shows the model calibration period. The dashed line is for the ensemble mean integration without any assimilation, the gray line and the full black line are assimilation runs without and with inclusion of off-block-diagonal information extracted from the innovations, respectively. The gap is caused by missing observations.

  • View in gallery

    Fig 7. Spatial soil moisture fields at 50-cm depth on 13 Mar 2002, obtained by spatial interpolation of the (left) observed values, (middle) ensemble mean forecasts with EnKF assimilation of the complete profile information at DL1 and (right) ensemble mean analyses obtained by ADEnKF assimilation of the complete profile information at DL1, with propagation of the update over the whole field. The arrows indicate the assimilation location, the full ellipses show regions where the ADEnKF has a clear improving effect, the dashed circle shows a region where the EnKF largely underestimates soil moisture, and the ADEnKF only has a very limited effect. The black dots are the actual observed and simulated locations at 50 cm depth.

  • View in gallery

    Fig 8. Spatial RMSE in soil moisture over all available sensors at three different depths, averaged over a validation period of 23 Dec 2001 through 24 Mar 2002, covering the period where only a single profile (x axis) is assimilated. The symbols □ and + are without and with inclusion of off-block-diagonal information extracted from the analysis increments, respectively.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 75 70 7
PDF Downloads 42 41 4

Adaptive Soil Moisture Profile Filtering for Horizontal Information Propagation in the Independent Column-Based CLM2.0

View More View Less
  • 1 Laboratory of Hydrology and Water Management, Ghent University, Ghent, Belgium, and Center for Research on Environment and Water, George Mason University, Calverton, Maryland
  • | 2 Center for Research on Environment and Water, George Mason University, Calverton, Maryland
  • | 3 Laboratory of Hydrology and Water Management, Ghent University, Ghent, Belgium
© Get Permissions
Full access

Abstract

Data assimilation aims to provide an optimal estimate of the overall system state, not only for an observed state variable or location. However, large-scale land surface models are typically column-based and purely random ensemble perturbation of states will lead to block-diagonal a priori (or background) error covariance. This facilitates the filtering calculations but compromises the potential of data assimilation to influence (unobserved) vertical and horizontal neighboring state variables. Here, a combination of an ensemble Kalman filter and an adaptive covariance correction method is explored to optimize the variances and retrieve the off-block-diagonal correlations in the a priori error covariance matrix. In a first time period, all available soil moisture profile observations in a small agricultural field are assimilated into the Community Land Model, version 2.0 (CLM2.0) to find the adaptive second-order a priori error information. After that period, only observations from single individual soil profiles are assimilated with inclusion of this adaptive information. It is shown that assimilation of a single profile can partially rectify the incorrectly simulated soil moisture spatial mean and variability. The largest reduction in the root-mean-square error in the soil moisture field varies between 7% and 22%, depending on the soil depth, when assimilating a single complete profile every two days during three months with a single time-invariant covariance correction.

Corresponding author address: Gabriëlle J. M. De Lannoy, Laboratory of Hydrology and Water Management, Ghent University, Coupure links 653, B-9000 Ghent, Belgium. Email: gabrielle.delannoy@ugent.be

This article included in the Catchment-scale Hydrological Modelling & Data Assimilation (CAHMDA) III special collection.

Abstract

Data assimilation aims to provide an optimal estimate of the overall system state, not only for an observed state variable or location. However, large-scale land surface models are typically column-based and purely random ensemble perturbation of states will lead to block-diagonal a priori (or background) error covariance. This facilitates the filtering calculations but compromises the potential of data assimilation to influence (unobserved) vertical and horizontal neighboring state variables. Here, a combination of an ensemble Kalman filter and an adaptive covariance correction method is explored to optimize the variances and retrieve the off-block-diagonal correlations in the a priori error covariance matrix. In a first time period, all available soil moisture profile observations in a small agricultural field are assimilated into the Community Land Model, version 2.0 (CLM2.0) to find the adaptive second-order a priori error information. After that period, only observations from single individual soil profiles are assimilated with inclusion of this adaptive information. It is shown that assimilation of a single profile can partially rectify the incorrectly simulated soil moisture spatial mean and variability. The largest reduction in the root-mean-square error in the soil moisture field varies between 7% and 22%, depending on the soil depth, when assimilating a single complete profile every two days during three months with a single time-invariant covariance correction.

Corresponding author address: Gabriëlle J. M. De Lannoy, Laboratory of Hydrology and Water Management, Ghent University, Coupure links 653, B-9000 Ghent, Belgium. Email: gabrielle.delannoy@ugent.be

This article included in the Catchment-scale Hydrological Modelling & Data Assimilation (CAHMDA) III special collection.

1. Introduction

For many hydrological applications, it is desirable to update an entire spatial field through the assimilation of sparse soil moisture observations. Information propagation over a soil volume can be accomplished by full three-dimensional (3D) Richards equation–based models. However, for computational efficiency, coarse-scale land surface models (LSM) often treat soil moisture profiles as independent, individual vertical columns, ignoring horizontal water flows. In a synthetic data assimilation study, Reichle and Koster (2003) demonstrated that taking horizontal model error correlations into account in the a priori error covariance matrix for a 3D ensemble Kalman filter (EnKF) improved the estimation accuracy for soil moisture, even in a regional-scale application. Other existing approaches to enforce lateral information flow in an assimilation scheme with a 1D (column) model are a priori interpolation of the observations to locations where they are desired to affect the model state, as in analysis nudging (Stauffer and Seaman 1990; Houser et al. 1998), and combined spatial interpolation and temporal propagation as explored in the space–time Kalman filter (Huang and Cressie 1996). Generally, the choice is between a better or more complex LSM with a simple assimilation scheme and a simple LSM with a more complex assimilation scheme. Exploring the possibility of accounting for horizontal information flow within the assimilation scheme is worthwhile, because “it is unreasonable to rely on models that relate each pixel of a large grid to every other pixel” (McLaughlin 2002). It should also be recognized that for very coarse-scale modeling, it is not needed to horizontally connect all the state variables within the model, because of the limited correlation length of earth state variables.

For several decades it has been discussed that, for a given system model and observation network (Daley 1992b), the efficient use of the information brought by the observations is largely determined by the second-order a priori error statistics: the a priori (or background) error covariance matrix 𝗣i, used in the Kalman gain (or any blending matrix used during assimilation) at each time step i, distributes information in space and between state variables and is the key to optimal data assimilation. Furthermore, the magnitude of the blending matrix elements essentially only depends on the signal-to-noise ratio 𝗥i/𝗣i, with 𝗥i as the observation error covariance matrix. Different blending gains mean different filter bandwidth and a distinct sensitivity to noise: the larger the a priori state error (or the smaller the observation error), the larger the filter bandwidths and the more variation (resulting from incorporating observations) can be expected in the assimilation analyses.

In the sequential statistical interpolation method (Daley 1991) and in most variational assimilation schemes, 𝗣i is generally set empirically, after splitting 𝗣i into a time-invariant correlation matrix 𝗣C and a possibly time-variant diagonal matrix 𝗣Si whose elements are the error standard deviations for the individual variables in the state, that is, 𝗣i = (𝗣Si𝗣C𝗣Si). The specification of the correlation structure is a critical issue, because it selects the area from which observations are allowed to influence the model variables. Mostly, stationary, homogeneous, horizontal/vertical separable, and isotropic error covariance functions as functions of distance have been proposed for which only a few parameters were to be estimated, for example, by the traditional innovation or observational method (Hollingsworth and Lönnberg 1986; Daley 1991; Houser et al. 1998). With the implementation of the Kalman filter, the focus was moved to the definition of the model error covariance (Daley 1992a), because 𝗣i can easily be calculated by the explicit Lyapunov propagation equation in linear stationary systems as the sum of the predictability and model error covariances. For optimization, several adaptive filters have been developed to estimate the model error covariances, mostly involving some whitening of the innovation (observation minus forecast) sequence (Kailath 1968; Mehra 1970; Maybeck 1982) or matching (elements of) the zero lag innovation covariance matrix to the theoretically optimal values (Jazwinski 1969; Myers and Tapley 1976; Maybeck 1982; Mehra 1972; Dee 1995; Dee and da Silva 1999). The majority of the techniques focus more on the estimation of the error variances than on the error correlation. In nonlinear systems, as are most land surface models, ensembles allow a flow-dependent statistical approximation of 𝗣i for the EnKF (Evensen (1994)). However, the EnKF only quantifies the uncertainty in the space spanned by the ensembles, and it is particularly difficult to reflect the “true” uncertainty in the initial or analysis state and model error into an optimal perturbation magnitude and structure for the forcings, model structure, state, and parameters. It is often useful to tune the filter by adapting the error variance—for example, by inflation, to avoid model divergence in case of an underestimated a priori error covariance. An adaptive inflation factor in time was proposed by Anderson (2007), who adjusted the variance of each state component, whereas the correlation between pairs of components remained unchanged. Mitchell and Houtekamer (2000) applied the maximum-likelihood method to estimate model error variance parameters in an EnKF framework, where initially only the uncertainty in the initial state or analysis was considered.

In hydrology, the statistical nature of the a priori (forecast) errors has often been prescribed in synthetic studies or arbitrarily chosen in real studies. Sometimes, the filter performance is evaluated afterward by providing innovation statistics (Reichle et al. 2002a; Crow 2003; De Lannoy et al. 2007b). In synthetic studies, filters can be calibrated by contrasting the filter output to some truth, as in Reichle et al. (2002b) and Reichle and Koster (2003), to allow for a comparison of several Kalman filter setups for soil moisture estimation, each in optimal conditions. If no truth is available, the innovation statistics could be used to optimize the filter. Van Geer et al. (1991) and Zhang et al. (2004) applied the covariance matching technique (Mehra 1972) to optimize the uncertainty estimation in groundwater and soil moisture simulations, respectively. In the covariance matching method, the innovation covariance matrix, obtained as a result from the filter, is iteratively contrasted to its theoretical value, which is a function of the tunable model error covariance matrix. The innovation covariance matrix is typically calculated as the outer product of the centralized innovation vectors over some time (proxy of ensemble). Drécourt et al. (2006) extended and changed this approach to estimate both the system and bias model error covariance in a synthetic groundwater simulation study. Crow and van Loon (2006) applied the maximum-likelihood estimation (Dee 1995) to find a covariance scaling parameter to adjust the error variances only, while assuming a fixed, predefined (spatial) correlation structure. This approach involves the inner product (or, the trace of the outer product) of the innovation vectors—that is, all the variables’ uncertainties are lumped into one value. They illustrated how tuning the wrong error source to produce better innovation statistics led to a reduced accuracy of the filter results, when only assimilating surface soil moisture. Additionally, there was a risk to find the “globally” best innovation statistics through the optimization of the wrong error type, if different sources of error were taken into account. Reichle et al. (2008) used the same maximum-likelihood approach to identify model and observation error variances for soil moisture assimilation in a synthetic experiment. The multiscale Kalman filter (Parada and Liang 2004; Kumar 1999) efficiently considers the spatial dependence and scaling properties of soil moisture. Parada and Liang (2004) applied a multiscale Kalman filter in conjunction with an expectation maximization algorithm to estimate temporally evolving statistical observation and model error parameters inherent to the Kalman filter to assimilate surface soil moisture in an LSM. Besides interest in the 𝗣i, observation uncertainty (error variance) is also an important factor in filtering. In several observation system simulation experiments (OSSEs), for example by Walker and Houser (2004), researchers have tried to quantify the maximal acceptable (to be useful in data assimilation) observation uncertainty to guide instrumentation design. It is reasonable to assume that, similar to forecast error, observation error variance is also state dependent and hence an adaptive filtering approach could be beneficial (Romanowicz et al. 2006).

Here, in situ soil moisture profiles in a small agricultural field [Optimizing Production Inputs for Economic and Environmental Enhancement (OPE3)] are simulated by individual, unconnected soil columns in the Community Land Model, version 2.0 (CLM2.0). Ensembles are used to quantify the variables’ uncertainty in each individual profile. Then, the ensemble a priori error variance is adapted, and the error correlation between the profiles is sought for in an adaptive filtering scheme to introduce lateral flow of information through the assimilation scheme. The adaptive information is extracted from as many field observations as available. In a successive time period, this information is used to update the complete field while assimilating only a few observations. This study differs from the EnKF approach of Reichle and Koster (2003), who imposed and calibrated (against some truth) the model error field in a synthetic study to impose lateral flow of information. Although data assimilation updates the observable prognostic state variables, it also adjusts the unobserved diagnostic flow variables. This study only focuses on the possibility of propagating soil moisture profiles horizontally and only validates observable prognostic soil moisture. An earlier study (De Lannoy et al. 2007b) highlighted the effect of soil moisture assimilation on diagnostic evapotranspiration, subsurface drainage, and runoff.

2. Model and observations

Soil moisture profiles are estimated with the CLM2.0 (Dai et al. 2003) at 36 locations (Fig. 1) in a 21-ha corn field in Beltsville, Maryland, where the OPE3 experiment is conducted (Gish et al. 2002; De Lannoy et al. 2006b). The model simulates land surface processes by calculating hourly vertical water and heat fluxes and states for each profile without interaction between profiles—that is, the linearized system matrix is block diagonal. To limit the computational costs, only the 10-layer soil moisture prognostic variables are included in the assimilation scheme—that is, the state vector i has a dimension of 360. The depths of the soil nodes are set to 2.5, 5, 10, 20, 30, 50, 80, 120, 150, and 180 cm. The nonlinear land surface model fi,i−1 propagates the state vector from time step i − 1 to i:
i1525-7541-10-3-766-e1
where ui is the meteorological forcings, which are hourly observed data and assumed spatially homogeneous over all the profiles in this study. The initial condition for the land surface model is the a posteriori state estimate (or analysis) . This a posteriori state estimate is obtained from updating the a priori state estimate at time i − 1 (discussed below).
During our observation period from 1 May 2001 through 30 April 2002, hourly measurements were obtained from 36 capacitance probes (EnviroSCAN of SENTEK Pty Ltd., South Australia), measuring soil moisture at three to seven layers (H probes: 10, 30, and 80 cm; M probes: 10, 30, 50, 120, 150, and 180 cm; L probes: 10, 30, 50, 80, 120, 150, and 180 cm depth; H, M, and L probes were installed at locations with expected different infiltration regimes, i.e., high, medium, or low clay content.). A layout of the observation probes in the OPE3 field is shown in Fig. 1. The observations yi (vector dimension varies with the specific assimilation experiment) are linearly related to the state by the observation (or forward) operator 𝗛i, that is,
i1525-7541-10-3-766-e2
where vi is zero-mean assumed random error. Here, 𝗛i contains only values of 1 and 0, because the observations are direct measurements of the state variables.

The CLM2.0 was calibrated and initialized for each individual profile through weak constraint variational assimilation of observed data from 1 May through 1 October 2001 (De Lannoy et al. 2006a), with a focus on parameter estimation in the period from 2 September through 2 October 2001. All state updating experiments in this paper start after this calibration period. A limited subset of observations will be assimilated for state and uncertainty estimation, while withholding other soil moisture profile observations for validation of the filtering performance. This study only focuses on assimilation of “complete” profiles. Depending on the type of probe and the availability of data, this means simultaneous assimilation of observations at three up to seven soil depths.

3. EnKF implementation

The Kalman filter is a sequential assimilation method used to update forecasted model states with observational information. The analysis or a posteriori state estimate is a combination of the forecast and observations, each weighted as a function of their uncertainty, which is captured in the a priori and observation error covariance matrices, respectively. The uncertainty in the forecasts is approximated by N = 64 ensemble (Monte Carlo) member predictions :
i1525-7541-10-3-766-e3
where wj,i is a realization of the (zero-mean assumed) model error that represents the complete effect of perturbations to forcings, parameters, and state variables, quantified as in De Lannoy et al. (2006a). The a priori state (background) error covariance 𝗣i is initially obtained from the ensemble sample covariance . Because of the model structure (independent profiles) and our choice of uncorrelated forcing perturbations, a priori state errors at different locations were effectively uncorrelated. The spurious elements (without actual statistical significance) ending up outside of the diagonal blocks corresponding to the variables in one profile in were set to 0. This type of localization would be well suited for global- or regional-scale studies of variables that are only correlated over a small spatial domain and for which the error correlations over larger distances can be assumed to be very small.
When observations yi are available, each ensemble member j is updated individually to obtain the a posteriori state estimate:
i1525-7541-10-3-766-e4
The actual analysis state i+ is the ensemble mean. The term equals the innovation vector ri. The observations are perturbed to ensure sufficient spread—that is, yj,i = yi + vj,i, where vj,i is an imposed zero-mean Gaussian perturbation with an error covariance of 𝗥 = (0.022 m3 m−3)2 𝗜, based on the sensor calibration information and assuming zero cross correlation between the observation errors (De Lannoy et al. 2007b). The Kalman gain 𝗞i is identical for all ensemble members and given by
i1525-7541-10-3-766-e5

Basically, the term 𝗣i𝗛iT maps the analysis increments on the model grid. If no observations are available, then i+ in Eq. (1) simply equals i.

4. Adaptive EnKF (ADEnKF)

a. Method

The correct characterization of 𝗣i largely determines the success of the Kalman filter. A first approximation of 𝗣i is the ensemble-based . Within the blocks of , we can assume that ensemble error correlations are well estimated: an assimilated observation in a single layer would be efficiently propagated to other layers within the same profile (Fig. 2). However, the magnitude of the error variances could possibly be optimized, because it is defined by a subjectively chosen “realistic” perturbation magnitude. Tuning the variances is typically the goal of most adaptive filtering schemes. Therefore, we will allow a variance correction within the adaptive Kalman scheme ADEnKF, while keeping the error correlations in the blocks as retrieved from the ensembles. Because the ensemble approach does not allow the estimation of the error structure outside the diagonal blocks in 𝗣i, the adaptive scheme is also explored to retrieve this “missing” between-profiles correlation information and hence allow horizontal information propagation (Fig. 2).

Most adaptive schemes calculate the model error covariance matrix 𝗤i as part of 𝗣i = 𝗣ip,− + 𝗤i, assuming that predictability error, which is the error in the forecasts resulting from errors in the previous analysis or initial conditions (with covariance 𝗣ip,−), and model error (with covariance 𝗤i) are independent (Daley 1992a). One of the schemes that allows for the estimation of all individual elements in 𝗤i is the method of Myers and Tapley (1976). It introduces the analysis (a posteriori estimate) as a proxy for the true state and an approximation of model noise w*i,
i1525-7541-10-3-766-e6
which in fact equals the analysis increment 𝗞i[yi − 𝗛ii]. For an optimal Kalman filter, the covariance of this quantity is given by
i1525-7541-10-3-766-e7
i1525-7541-10-3-766-e8
i1525-7541-10-3-766-e9
i1525-7541-10-3-766-e10
where E[w*i, w*i] = E[(w*i)(w*i)T] and is the theoretical ensemble mean. The matrix 𝗣i+ is the a posteriori (analysis) error covariance matrix and the predictability error covariance matrix 𝗣ip,− can be calculated by propagating the previous a posteriori (or analysis) error covariance at i − 1 by the model to time i (for a linearized model version . The practical expression to obtain matrix 𝗤i from Eq. (10) introduces a time averaging over a window of λ time steps:
i1525-7541-10-3-766-e11

An analysis increment time series (window of λ elements) is stored and centralized by its time mean 〈w*〉, to eliminate the time mean bias. The time mean approach is basically to obtain a statistically viable estimate of the analysis increment covariance matrix as an approximation of the ensemble mean analysis increment covariance matrix. The advantage of this method, as compared to the computationally more attractive approach of Dee (1995), is its ability to tune the variance at each state variable individually, whereas simpler methods typically come up with an inflation factor to homogeneously scale the a priori covariance matrix by a constant value. The latter approach could yield adverse results, as suggested by the findings of Crow and van Loon (2006).

Whereas in the above deduction 𝗤i is the unknown part of 𝗣i = 𝗣ip,− + 𝗤i, we now have different missing information in 𝗣i. With the EnKF, the objective is to determine 𝗣i based on information in the ensembles: a good approach of the error covariance would result from the propagation of the ensemble through the model dynamics. However, here the model does not simulate lateral flow, so only a part of the actual 𝗣i is captured by the ensemble information, that is , which is block diagonal and contains some predictability and model error (i.e., part of 𝗣ip,− and 𝗤i, respectively) in the diagonal blocks by perturbing the initial conditions, parameters, and forcings for each profile individually. The ensembles do not provide any information on the covariance between errors in variables of different profiles (off block diagonal) because of a model structural deficiency (no lateral flow). This is now the missing information in 𝗣i. A new matrix, , will be calculated in the adaptive scheme to adapt the ensemble-based block-diagonal , to obtain the total 𝗣i:
i1525-7541-10-3-766-e12
The method of Myers and Tapley (1976), as in Eq. (11), is slightly changed to obtain as follows:
i1525-7541-10-3-766-e13
assuming that for i = λ, instead of 𝗣λ = 𝗣λp,− + 𝗤λ. The practical implementation is as follows: at each time step, (i) calculate the matrix [w*i − 〈w*〉] [w*i − 〈w*〉]T and (ii) derive and 𝗣i+ from the ensemble state members, the latter after having filtered observations with . Then, the time mean at the right-hand side of Eq. (13) is calculated. In our example, the estimate of is based on increment information in the λ = 40 previous assimilation time steps for a filtering frequency of once every two days, from 21 December 2001 through 19 March 2002. For λ ≤ 20, we found that the retrieved matrix information could not be statistically consistent, because it was too sensitive to information brought in at additional time steps.
Once is found in a first period, it can be used to optimize 𝗣i for the filtering in a next assimilation period. The variances on the diagonal of are used to correct the ensemble variances on the diagonal of at each assimilation step i, that is:
i1525-7541-10-3-766-e14
If and the resulting [𝗣i]kk would result in a negative value, then [𝗣i]kk is not corrected by Eq. (14), but we choose to set [𝗣i]kk/2. Next, the covariances within the blocks are updated for the corrected variances [𝗣i]kk (the ensemble correlations are unaltered, but the variances are updated) as follows:
i1525-7541-10-3-766-e15
if k and l belong to the same profile. Outside the blocks, the covariances are calculated with the time-invariant correlations found in and the corrected time-variant variances from Eq. (14):
i1525-7541-10-3-766-e16
if k and l belong to different profiles. Here, is the ensemble-based a priori error variance for the kth state variable, averaged over the λ assimilation time steps. The off-block-diagonal elements in contain covariances containing variances of (with zero off-block-diagonal elements), plus (the additional correction). The final 𝗣i is not imposed on the ensemble state members along the model run but only used for the calculation of the Kalman gain. In summary, we apply a hybrid method to fully determine 𝗣i: ensemble-based state-dependent variances are corrected by the adaptive information and are used together with (i) ensemble-based state-dependent correlations in the blocks and (ii) time-invariant correlations outside the blocks. Unlike many EnKF studies here, no covariance localization (Hamill et al. 2002; Reichle and Koster 2003) is applied after filling the off-block-diagonal elements in 𝗣i.

b. Innovations versus analysis increments

The above proposed method uses the information in the analysis increments (𝗞i[yi − 𝗛ii]) rather than the innovations (yi − 𝗛ii), which are used in most other adaptive filters. A direct consequence is that all involved matrices as well as the resulting are determined in full state space, while other adaptive filters mostly reduce the computations to the observation space. The traditional innovation method (Hollingsworth and Lönnberg 1986; Daley 1991; Houser et al. 1998) implicitly provides a way to find the a priori state (background) error correlations at observed locations and to use them to cover the larger part of state space by fitting a distance-dependent correlation function. This technique builds on the following second-order innovation statistics for each pair (k, l) of observation locations:
i1525-7541-10-3-766-e17
i1525-7541-10-3-766-e18
where E[ri,ri] is the ensemble covariance matrix at one time step i. In practice, time series of (ensemble mean) innovations rk, calculated during the EnKF application at individual locations k, can be used as a proxy to estimate the ensemble innovation covariance. The centralized second-order statistic is calculated to remove the time mean discrepancy between forecasts and observations. Because in our study, a reasonable time-invariant estimate of the diagonal 𝗥 matrix is available—that is, [𝗥]kk = (0.022 m3 m−3)2 and [𝗥]kl = 0 (m3 m−3)2—the a priori error correlations ρkl can be estimated by
i1525-7541-10-3-766-e19
where E[[r]k, [r]l] = E[([r]k − 〈[r]k〉) ([r]l − 〈[r]l〉)T] is calculated over time and 〈[r]k〉 is the time mean innovation for the kth element of vector r. For each pair of elements, this correlation can then be plotted as a function of the distance and an empirical homogeneous (in space) function can then be fitted.

In this study, we could easily change this procedure, or any other innovation-based adaptive method, to find the off-block-diagonal correlation values of in the observation space to fill in the individual entries in 𝗣i in the state space as well as to find the variance correction, like in our adapted Myers and Tapley (1976) method. However, that would only be possible in the case of this study, where 𝗛i only contains 0 and 1, and the transfer of information from the observation to the state space would be well determined (leaving unobserved entries in 𝗣i with some default or zero correlation). For more general and complex observation operators 𝗛i, the method of Myers and Tapley (1976) is more attractive than other adaptive filters to find in the state space.

5. Results

a. Innovation analysis: Correlation structure

Before implementing the adaptive method based on Myers and Tapley (1976) described above, it was tested if a time-invariant horizontal error correlation structure could be defined as a function of separation distance between profiles, based on the information in the innovation covariances, as in the traditional innovation method.

For a two-weekly assimilation of complete profiles from 2 October 2001 through 19 March 2002, the resulting 13-element time series of (ensemble mean) innovations were used to approximate a time-invariant ensemble innovation covariance. The deduced background error correlations in Fig. 3 show that it is impossible to fit an empirical homogeneous isotropic correlation function to the calculated forecast error correlations at any observation depth. Similar results were found for scenarios with more frequent assimilation (and hence more elements in the innovation time series).

The CLM does not simulate horizontal water flows and is applied for each profile individually, while studies in the OPE3 field have revealed pathways for active preferential horizontal flow (Gish et al. 2002). It could be expected that profiles situated along these pathways may, therefore, show correlated model structural errors and hence correlated innovations. However, there were too few reported actively connected probes with available observations (and hence innovations) at corresponding soil depths to quantitatively analyze this hypothesis. Furthermore, the innovation correlation as a result of correlation in model structural errors may have been reduced by error compensation during the individual profile calibration procedure.

The lack of spatial error structure in Fig. 3 is probably because (i) the error correlation structure changes with direction (i.e., anisotropy), (ii) the different profiles were calibrated independently, and (iii) this is a small field-scale analysis, where the local highly variable soil conditions (and their potential parameterization errors) largely determine a close to random spatial soil moisture structure (and hence the derived innovation statistics). Note that at the regional scale, we might find spatial correlation structures controlled by forcing error patterns.

Figure 3 also shows that error correlations are higher and less variable in shallow layers than in deeper profile layers. Similarly, the analysis of the spatial structure in the soil moisture observations showed that the correlation (and correlation length) was higher in shallow layers (De Lannoy et al. 2006b). This suggests that for highly correlated time series of observed soil moisture, highly correlated innovation time series are also found. This is reasonable, because the calibrated model deficiencies will be similar for the similarly observed soil moisture time series. Basically, the innovation time sequences for each point will contain sequences of short time-scale biases that might be cross-correlated. When a sequential bias estimation (De Lannoy et al. 2007b) was included and the bias was removed from the innovations at each assimilation step, before the innovation time series was centralized by its time mean (longer time-scale “bias”), the correlations in the top layers indeed dropped down (results not shown).

To study the correlation stationarity in time, daily assimilation of complete profiles was performed over the same period (2 October 2001 through 19 March 2002, now resulting in 167-element time series). Through batch processing, the state error correlation was calculated from the innovations over 5 (and 10) windows of 33 (and 16) hourly time steps. A random selection of soil moisture state variable pairs at identical depths (10 cm in Fig. 4) shows that the retrieved error correlation values are nonstationary at the high-resolution time scale (e.g., 10 windows) but change relatively little when smoothed over fewer windows.

b. Adaptive filter: Spatial effects

For the main experiment, the combined state estimation (updating) and uncertainty estimation was performed during the period from 2 October through 21 December 2001 with the adapted Myers and Tapley (1976) method. All 36 complete observed soil moisture profiles were assimilated every two days (40 time steps) to calculate missing off-block-diagonal information and to obtain a correction to the variances in the a priori error covariance. Thereafter, from 23 December 2001 to 19 March 2002, only a single profile was assimilated using the novel error covariance information to enhance the soil moisture estimates at all 36 simulated soil moisture profiles in the field. During this period the ensemble-based was updated with time-invariant information in as described in Eqs. (14)(16).

Figure 5 shows the a priori state error correlation matrix on 23 December 2001, prior to and after the adaptive correction. It illustrates how the ensemble-based block-diagonal (i.e., within a profile) correlations are kept unaltered and how the off-block-diagonal (i.e., between profiles) error correlation elements are partially filled in. For a random zoom example, where error correlations between state variables corresponding to profile AL2 and BM4 are shown, only six rows (state variables corresponding to the six observation layers in M-probes) and seven columns (seven observation layers in L-probes) are filled in. The more observational layers available for a pair of profiles, the more error correlation information can be retrieved. Because the model layers at 2.5, 5, and 20 cm are never observed by the OPE3 probes, there will never be information on error statistics at these layers. Again, no spatial structure could be found in the retrieved error correlations at any soil layer (not shown). Similar to the above findings, the determined error correlation was higher in the top soil layer but much smaller than what was found through the innovation method.

Figure 6 shows for two individual soil depths at 10 and 80 cm, the temporal evolution of the spatial root-mean-square error (RMSE) between observations and ensemble mean forecasts without assimilation as well as analyses with assimilation of complete observed soil moisture profile data. During the first period (from 2 October to 21 December 2001) all available profile information is assimilated, causing significant drops in RMSE. However, the observational information is not taken up completely by the model at all profiles, mainly as a result of forecast bias (sometimes the model and observation climatologies are different; De Lannoy et al. 2007b), causing a rapid increase of RMSE after each assimilation event. In the second period, only one selected profile (here, DL1) is assimilated, either without or with inclusion of the estimated off-block-diagonal a priori error covariance elements. The further reduced RMSE, when including the off-block-diagonal information, shows that there is a benefit to spreading information in the horizontal space.

As an illustration, the spatial soil moisture fields in Fig. 7 are generated for 13 March 2002 as a weighted average of two spatial interpolations of the individual point analysis results (only for visualization purposes) by (i) a multivariate regression between the soil moisture and the terrain characteristics (texture, elevation, and topographic index) at the observing locations and (ii) kriging with an exponential variogram with parameters as observed in De Lannoy et al. (2006b). The observations and simulations show a discrepancy in the spatial soil moisture field: a part is a result of model bias, another part results from random error. When assimilating the complete observed profile at DL1 with the regular EnKF, no other profile is updated and the spatial view is dominated by the ensemble mean integration without assimilation. When the estimated off-block-diagonal a priori error covariance elements are included, ADEnKF of DL1 observations updates all profiles in the field. The full circles in Fig. 7 show regions where the ADEnKF obviously brings the simulated field close to the observed one, while only assimilating data at one location in the field and using time-invariant correlation information off the diagonal blocks in 𝗣i. The dashed circle shows a region where a major dry forecast bias cannot be dealt with by a state filter and where the adaptive filter only has a minor effect, because it is mainly designed to treat random error and probably also because of the time-invariant correlation structure. There is an attempt to increase the soil moisture by the ADEnKF, but it is mainly effective only at one location (CL2).

Figure 8 summarizes the effect on the spatial soil moisture estimation, when assimilating observations from each individual profile only, during a validation period from 23 December 2001 through 24 March 2002. The time-averaged spatial RMSE is shown for the case without and with inclusion of information. The RMSE is generally smaller with inclusion of off-block-diagonal a priori error covariance elements, even though the off-block-diagonal structure was determined during the period preceding the validation period and kept time-invariant (and hence suboptimal) during the validation period. A maximum decrease in spatial RMSE by 22% was achieved at 180 cm by assimilating either DM2 or DM3 profile data, a maximum RMSE decrease of 13% was found at 80 cm for the assimilation of AM1 data, and at 10 cm a 9% RMSE decrease resulted from assimilation of BM2 data. At the other observing layers, the maximum RMSE decrease was 11% (30 cm), 13% (50 and 80 cm), 17% (120 cm), and 7% (150 cm) by complete profile assimilation. For this study, the maximum decrease in spatial RMSE, averaged over all field profile layers, was 11% when assimilating probe AM1. It turns out that AM1 is the most time stable profile in the OPE3 field (De Lannoy et al. 2007a; i.e., its measured soil moisture time series best resembles the spatial mean soil moisture behavior in time at most soil depths). It should also be noted that validation of the performance in a single layer sometimes shows further reduced spatial RMSE values, when only assimilating a single observation in that single layer (not shown). This is because when updating the soil moisture in many layers simultaneously, the analysis profile may not reflect the preferred model balance and seemingly contradictory (to the model) updates for the different layers may be cancelled out quickly over time after the analysis. The update in a single layer often persists longer in time, when neighboring layers are adapting to the update in that single layer, rather than possibly competing against this update with updates at their own respective depths. It is evident that, when evaluating the total field profile performance (as opposed to the previous discussion on the effect in a single layer), assimilation with probes having more observational depths (six for M-probes and seven for L-probes) generally results in the best improvements in RMSE. However, it is important to notice how H-probes with only three observation layers are able to enhance the spatial RMSE at unobserved (e.g., 180 cm) layers.

In some cases (here, it is for 80-cm depth), the RMSE is increased, as a result of either a bad propagation of information over the vertical layers or an inaccurate determination of the between-profiles error correlation. In an exceptional case, the assimilation of a single profile did not improve the soil moisture field estimate in any layer: through assimilation of probe BL2, the spatial RMSE was increased outside the range of the plot for almost all soil layers when including the spatial error correlation information. The problem here was a collapsed ensemble spread (and the time-invariant adaptive correction was inappropriate to solve this problem) at some layers, where a model-defined minimal soil moisture was reached, causing adverse effects in other layers at distant profiles.

The spatial RMSE covers both the discrepancy in the spatial mean soil moisture field and the spatial structure of the soil moisture field. The last three columns in Table 1 summarize the time root-mean-square difference between the observed and modeled spatial mean soil moisture field (during the validation period 23 December 2001–24 March 2002) for the cases (i) without assimilation, (ii) with regular EnKF assimilation of a single profile (here, DL1), and (iii) with assimilation of the same single profile by the proposed adaptive EnKF. When only state updating is performed (EnKF), the RMSE is slightly reduced, only because of the improving effect on the DL1 soil moisture time series within the spatial mean. Clearly, when including adaptive information, the error in the spatial mean soil moisture is largely reduced by assimilating a single profile. Only at 150 cm (for this case), a minor adverse effect can be observed, caused by a bad vertical interaction between layers. Maybe this could be solved by using the adaptive filter to correct the ensemble-based error correlations within the profile. The first three columns in Table 1 give the time mean of the root-mean-square difference in semivariances γ between the observed (γo) and modeled (γm) spatial soil moisture fields over the same period of length T,
i1525-7541-10-3-766-e20
The spatial lag in the calculation of γo and γm was set at 20 m (to allow some averaging over pairs of observations in each spatial lag-interval) for a maximum interdistance of 420 m (beyond half of the field length, semivariance values would be calculated for too few sample pairs). The mean square difference in semivariances was thus calculated over a maximum number of 21 spatial lags, that is, L = 21. Table 1 shows that, through simple EnKF, the spatial pattern was slightly improved, again because of the enhanced representation of soil moisture at the assimilation location only. With the inclusion of adaptive information, the differences in semivariances were further reduced. Consequently, assimilation of a single profile (here, DL1) with an adaptive EnKF is able to partially rectify both incorrectly simulated spatial soil moisture mean and variance.

6. Discussion and perspectives

In this paper, the missing error correlations and an error covariance correction are retrieved only over a single time window and are to be used in a period succeeding the period where it was actually calculated. A more frequent batch processing could be suggested to obtain more time-variant adaptive information. Definitely for large-scale applications, where the variability in the precipitation errors may alter the error correlations over time, a frequent update of the missing error correlations would be beneficial. However, there is a trade-off between an increased time variability and the statistical consistency of the error covariance estimates, as found in Eq. (13): some time averaging is needed to obtain viable covariance estimates. Future research could investigate how the training period can be limited by exploring the ensemble information in the innovations at each time step, instead of studying the ensemble mean information in time.

The analysis results can be further improved by adding a bias filter to the state estimation (De Lannoy et al. 2007b). However, since the bias error covariance is often linked to the (adapted) a priori error covariance, more research is needed to assure both proper interactions between these error covariance and the optimality of the filter. Note that the Myers and Tapley (1976) method removes some bias from the increments to find in Eq. (13), but the state estimates remain biased. It should also be recognized that the a priori error correlations should only represent information about the random error structure, as long-range a priori state error correlations derived from innovations (or analysis increments) may sometimes originate from spatial bias fields in either the observations or forecasts.

This approach to determine the between-profiles error covariance can be of practical importance to propagate satellite observations from observed locations to temporarily masked or unobserved regions (e.g., clouds in visual products and hydrometeors for high-frequency microwave products), or propagation of in situ profile information to locations where sensors are temporarily out of service. In satellite data assimilation schemes, the modeled grid cells are typically covered by the observations (after reprojecting the data). This allows an update of all the a priori error correlations at each time observations are available and use them, whenever part of the observations falls out and information needs to be propagated to the nonobserved pixels. Furthermore, training of the correlations during one satellite pass (covering a limited banded area) allows observations in the overlapping area of the next pass to be propagated all over the area covered by previous (shifted) pass, if some time-invariance in error correlations could be assumed. However, for large-scale applications, we generally assume that the a priori error correlation would be limited (e.g., different dominant forcing errors) and not worth the increased numerical expenses to deal with a more complex 𝗣i. As a very simple approximation, the actual (mostly observed) state correlation length is often used to limit the a priori state error correlations at large distances, even though we should realize that the actual state and its estimation errors (and hence their respective correlations lengths) are in fact independent.

A problem is that with a finer model resolution, more observations are needed to retrieve the correct error covariances. If a soil moisture profile is available every 20 m (assume it representative for a 10 × 10 m2 area) and the model simulates at a 10-m resolution, then the correlation between the forecast errors of two neighboring grid cells at 10 m apart will not be retrieved by the proposed technique, whereas the correlation between forecast errors in two observed grid cells at 20 m apart can be found. One limitation of the proposed method is the need to determine the adaptive information based on as many observations as available, before it can be used with a restricted dataset.

In general, the proposed technique is expected to advance the optimal use of satellite and other data to fill in unobserved areas and to overcome the shortcomings of completely time invariant homogeneous (in space) correlation matrices, as in optimal interpolation techniques. Typical empirical distance-dependent correlation functions may give a false impression of smooth correlation structures in the analyses. It could be advised to review assumptions on empirical correlation functions for hydrological variables at different scales by dynamical error correlation insights obtained from adaptive filters.

7. Summary

This study explores the potential of updating the full 3D system state when assimilating only single soil moisture profile data in the independent column-based CLM2.0 with an adaptive ensemble Kalman filter (ADEnKF). Assimilation of individual in situ soil moisture profiles from the small-scale OPE3 field with an optimized knowledge of a priori error covariances resulted in a partial rectification of the inaccurately simulated spatial mean and variability in soil moisture fields at different soil depths.

Because no simple spatial error correlation structure could be found based on the traditional innovation method, it was necessary to find all elements of the a priori error covariance matrix individually. Ensembles were used to obtain (i) a first guess of each state variable uncertainty, that is, the magnitude of error variance, and (ii) an estimate of the error correlation between state variables within each individual profile. Then, during a first period, a time-invariant variance correction and the between-profiles error correlations were sought based on an adaptive method of Myers and Tapley (1976). This allowed for the adjustment of the time-variable ensemble-based state error covariance matrix before each filtering time step in the successive period.

Even though only one time-invariant estimate was used to correct the a priori error covariances, the results show improvements up to between 7% and 22% reduction in spatial RMSE in a single soil layers, when assimilating only a single complete profile [higher percentage reductions can be achieved by assimilating only in the single validation layer, which avoids contradictory updates over the different soil layers (not shown)]. Even the assimilation of a single profile with only a few observed layers resulted in enhanced soil moisture field estimates at most depths. With knowledge of the off-block-diagonal a priori error covariances (ADEnKF), any observation in a single layer will affect both the total profile where it is assimilated in and all layers of surrounding profiles, while a regular EnKF with the CLM2.0 only allows for updating of a single profile.

This study shows an example of retrieving the missing error correlations and an error covariance correction during a first time window and using it in a period succeeding the period where it was actually calculated. A more frequent batch processing would result in more time-variant adaptive information and the information in the ensembles may be further explored to limit the training periods, while keeping the estimates statistically consistent.

The determination of between-profiles error covariance can be important to propagate satellite observations from observed locations to temporarily unobserved or masked regions or propagation of in situ profile information to locations where sensors are temporarily out of service. The limitation of this method is the need to determine the adaptive information based on as many observations as available, before it can be used with a restricted dataset.

Acknowledgments

The first author is a postdoctoral research fellow of the Research Foundation Flanders (FWO). We thank the USDA Beltsville Agricultural Research Center/Agricultural Research Service for their data. The reviewers are thanked for their constructive comments.

REFERENCES

  • Anderson, J. L., 2007: An adaptive covariance inflation error correction algorithm for ensemble filters. Tellus, 59A , 210224.

  • Crow, W. T., 2003: Correcting land surface model predictions for the impact of temporally sparce rainfall rate measurements using an ensemble Kalman filter and surface brightness temperature observations. J. Hydrometeor., 4 , 960973.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Crow, W. T., and van Loon E. , 2006: Impact of incorrect model error assumptions on the sequential assimilation of remotely sensed surface soil moisture. J. Hydrometeor., 7 , 421432.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Dai, Y., and Coauthors, 2003: The Common Land Model. Bull. Amer. Meteor. Soc., 84 , 10131023.

  • Daley, R., 1991: Atmospheric Data Analysis. Cambridge University Press, 455 pp.

  • Daley, R., 1992a: Estimating model-error covariances for application to atmospheric data assimilation. Mon. Wea. Rev., 120 , 17351746.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Daley, R., 1992b: Forecast-error statistics for homogeneous and inhomogeneous observation networks. Mon. Wea. Rev., 120 , 627643.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Dee, D. P., 1995: On-line estimation of error covariance parameters for atmospheric data assimilation. Mon. Wea. Rev., 123 , 11281145.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Dee, D. P., and da Silva A. M. , 1999: Maximum-likelihood estimation of forecast and observation error covariance parameters. Part I: Methodology. Mon. Wea. Rev., 127 , 18221834.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • De Lannoy, G. J. M., Houser P. R. , Pauwels V. R. N. , and Verhoest N. E. C. , 2006a: Assessment of model uncertainty for soil moisture through ensemble verification. J. Geophys. Res., 111 , D10101. doi:10.1029/2005JD006367.

    • Search Google Scholar
    • Export Citation
  • De Lannoy, G. J. M., Verhoest N. E. C. , Houser P. R. , Gish T. , and Van Meirvenne M. , 2006b: Spatial and temporal characteristics of soil moisture in an intensively monitored agricultural field (OPE3). J. Hydrol., 331 , 719730.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • De Lannoy, G. J. M., Houser P. R. , Verhoest N. E. C. , Pauwels V. R. N. , and Gish T. , 2007a: Upscaling of point soil moisture observations to field averages at the OPE3 site. J. Hydrol., 343 , 111.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • De Lannoy, G. J. M., Reichle R. H. , Houser P. , Pauwels V. R. N. , and Verhoest N. E. C. , 2007b: Correcting for forecast bias in soil moisture assimilation with the ensemble Kalman filter. Water Resour. Res., 43 , W09410. doi:10.1029/2006WR005449.

    • Search Google Scholar
    • Export Citation
  • Drécourt, J-P., Madsen H. , and Rosbjerg D. , 2006: Joint calibration and uncertainty analysis of a groundwater model coupled with a Kalman filter. Adv. Water Resour., 29 , 719734.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Evensen, G., 1994: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res., 99 , (C5). 1014310162.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gish, T., Dulaney W. , Kung K-J. S. , Daughtry C. , Doolittle J. , and Miller P. , 2002: Evaluating use of ground-penetrating radar for identifying subsurface flow pathways. Soil Sci. Soc. Amer. J., 66 , 16201629.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hamill, T., Snyder C. , and Morss R. , 2002: Analysis-error statistics of a quasigeostrophic model using three-dimensional variational assimilation. Mon. Wea. Rev., 130 , 27772790.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hollingsworth, A., and Lönnberg P. , 1986: The statistical structure of short-range forecast errors as determined from radiosonde data. Part I: The wind field. Tellus, 38A , 111136.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Houser, P. R., Shuttleworth W. J. , Famiglietti J. S. , Gupta H. V. , Syed K. H. , and Goodrich D. C. , 1998: Integration of soil moisture remote sensing and hydrologic modeling using data assimilation. Water Resour. Res., 34 , 34053420.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Huang, H-C., and Cressie N. , 1996: Spatio-temporal prediction of snow water equivalent using the Kalman filter. Comput. Stat. Data Anal., 22 , 159175.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jazwinski, A., 1969: Adaptive filtering. Automatica, 5 , 475485.

  • Kailath, T., 1968: An innovation approach to least-squares estimation. Part I: Linear filtering in additive white noise. IEEE Trans. Autom. Control, AC-13 , 646655.

    • Search Google Scholar
    • Export Citation
  • Kumar, P., 1999: A multiple scale state-space model of characterizing subgrid scale variability of near-surface soil moisture. IEEE Trans. Geosci. Remote Sens., 37 , 182197.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Maybeck, P. S., 1982: Stochastic Models, Estimation, and Control. Vol. 2. Academic Press, 423 pp.

  • McLaughlin, D., 2002: An integrated approach to hydrologic data assimilation: Interpolation, smoothing, and filtering. Adv. Water Resour., 25 , 12751286.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mehra, R., 1970: On the identification of variances and adaptive Kalman filtering. IEEE Trans. Autom. Control, 15 , 175184.

  • Mehra, R., 1972: Approaches to adaptive filtering. IEEE Trans. Autom. Control, 17 , 693698.

  • Mitchell, H. L., and Houtekamer P. L. , 2000: An adaptive ensemble Kalman filter. Mon. Wea. Rev., 128 , 416433.

  • Myers, K., and Tapley B. , 1976: Adaptive sequential estimation with unknown noise statistics. IEEE Trans. Autom. Control, 21 , 520523.

  • Parada, L. M., and Liang X. , 2004: Optimal multiscale Kalman filter assimilation for near-surface soil moisture into land surface models. J. Geophys. Res., 109 , D24109. doi:10.1029/2004JD004745.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Reichle, R. H., and Koster R. , 2003: Assessing the impact of horizontal error correlations in background fields on soil moisture estimation. J. Hydrometeor., 4 , 12291242.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Reichle, R. H., McLaughlin D. B. , and Entekhabi D. , 2002a: Hydrologic data assimilation with the ensemble Kalman filter. Mon. Wea. Rev., 130 , 103114.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Reichle, R. H., Walker J. P. , Houser P. R. , and Koster R. D. , 2002b: Extended versus ensemble Kalman filtering for land data assimilation. J. Hydrometeor., 3 , 728740.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Reichle, R. H., Crow W. , and Keppenne C. , 2008: An adaptive ensemble Kalman filter for soil moisture data assimilation. Water Resour. Res., 44 , W03423. doi:10.1029/2007WR006357.

    • Search Google Scholar
    • Export Citation
  • Romanowicz, R. J., Young P. C. , and Beven K. J. , 2006: Data assimilation and adaptive forecasting of water levels in the River Severn catchment, United Kingdom. Water Resour. Res., 42 , W06407. doi:10.1029/2005WR004373.

    • Search Google Scholar
    • Export Citation
  • Stauffer, D. R., and Seaman N. L. , 1990: Use of four-dimensional data assimilation in a limited-area mesoscale model. Part I: Experiments with synoptic-scale data. Mon. Wea. Rev., 118 , 12501277.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • van Geer, F. C., Te Stroet C. B. , and Yangxiao Z. , 1991: Using Kalman filtering to improve and quantify the uncertainty of numerical groundwater simulations. 1. The role of system noise and its calibration. Water Resour. Res., 27 , 19871994.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Walker, J. P., and Houser P. R. , 2004: Requirements of a global near-surface soil moisture satellite mission: Accuracy, repeat time, and spatial resolution. Adv. Water Resour., 27 , 785801.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zhang, S-W., Qiu C-J. , and Xu A. , 2004: Estimating soil water contents from soil temperature measurements by using an adaptive Kalman filter. J. Appl. Meteor., 43 , 379389.

    • Crossref
    • Search Google Scholar
    • Export Citation

i1525-7541-10-3-766-f01

Fig 1. Digital elevation model (dashed contour lines, 0.5-m interval) with indication of the boundary coordinates and location of soil moisture probes (three digits) in the OPE3 field with four subwatersheds, A–D. Defective probes during the study period are crossed out. The bottom sketch shows how the system is modeled as 36 individual, unconnected soil profiles with multiple soil layers.

Citation: Journal of Hydrometeorology 10, 3; 10.1175/2008JHM1037.1

i1525-7541-10-3-766-f02

Fig 2. (top) Schematic of EnKF and ADEnKF functionality with the CLM, which only simulates vertical fluxes (black dashed arrows). The white arrows represent the information transfer from the observation in one particular layer to model states in other soil layers of the same profile (EnKF and ADEnKF) and neighboring profiles (ADEnKF only) by the a priori state error covariance statistics 𝗣i in the filter. (bottom) Matrices represent 𝗣i for two profiles and with multiple state variables per profile. For the EnKF, 𝗣i is block diagonal. For the ADEnKF, it is tried to (i) optimize the a priori error variances on the diagonal of 𝗣i and (ii) retrieve the off-block-diagonal error correlations between the different profiles, schematically indicated by crosshatched matrix parts.

Citation: Journal of Hydrometeorology 10, 3; 10.1175/2008JHM1037.1

i1525-7541-10-3-766-f03

Fig 3. A priori horizontal error correlation as function of separation distance, calculated by the traditional innovation method at 10- and 80-cm depth, with 13 innovations obtained after assimilation of complete profiles every two weeks from 2 Oct 2001 through 19 Mar 2002.

Citation: Journal of Hydrometeorology 10, 3; 10.1175/2008JHM1037.1

i1525-7541-10-3-766-f04

Fig 4. Calculated error correlation at 10-cm depth for some randomly selected pairs: + (DH1, A21), * (AM1, BL1), ▵ (AH2, DH3), ⋄ (BH2, BL2), □ (BL3, CL1), × (BM4, DM2). Small gray symbols are calculated over 10 windows of 16 hourly time steps and thick black symbols over 5 windows of 33 time steps. The symbols on the right are calculated over the total time series of 167 hourly time steps.

Citation: Journal of Hydrometeorology 10, 3; 10.1175/2008JHM1037.1

i1525-7541-10-3-766-f05

Fig 5. A priori error correlation matrix (top) prior to and (bottom) after adaptive correction at 23 December 2001. Within each profile 10 soil moisture state variables (2.5, 5, 10, 20, 30, 50, 80, 120, 150, and 180 cm) are present. The absolute correlation values are shown.

Citation: Journal of Hydrometeorology 10, 3; 10.1175/2008JHM1037.1

i1525-7541-10-3-766-f06

Fig 6. Spatial RMSE in soil moisture over all available sensors at two different depths. The arrows indicate the assimilation events: (top) from 2 Oct through 21 Dec 2001 all profile information is assimilated, (bottom) thereafter through 19 Mar 2002 only profile data from probe DL1 are assimilated. The gray zone shows the model calibration period. The dashed line is for the ensemble mean integration without any assimilation, the gray line and the full black line are assimilation runs without and with inclusion of off-block-diagonal information extracted from the innovations, respectively. The gap is caused by missing observations.

Citation: Journal of Hydrometeorology 10, 3; 10.1175/2008JHM1037.1

i1525-7541-10-3-766-f07

Fig 7. Spatial soil moisture fields at 50-cm depth on 13 Mar 2002, obtained by spatial interpolation of the (left) observed values, (middle) ensemble mean forecasts with EnKF assimilation of the complete profile information at DL1 and (right) ensemble mean analyses obtained by ADEnKF assimilation of the complete profile information at DL1, with propagation of the update over the whole field. The arrows indicate the assimilation location, the full ellipses show regions where the ADEnKF has a clear improving effect, the dashed circle shows a region where the EnKF largely underestimates soil moisture, and the ADEnKF only has a very limited effect. The black dots are the actual observed and simulated locations at 50 cm depth.

Citation: Journal of Hydrometeorology 10, 3; 10.1175/2008JHM1037.1

i1525-7541-10-3-766-f08

Fig 8. Spatial RMSE in soil moisture over all available sensors at three different depths, averaged over a validation period of 23 Dec 2001 through 24 Mar 2002, covering the period where only a single profile (x axis) is assimilated. The symbols □ and + are without and with inclusion of off-block-diagonal information extracted from the analysis increments, respectively.

Citation: Journal of Hydrometeorology 10, 3; 10.1175/2008JHM1037.1

Table 1.

Time-averaged RMS difference between observed and modeled soil moisture spatial semivariance (DSsv, vol%2) and spatial mean (DSm, vol%), over a validation period from 23 Dec 2001 though 24 Mar 2002. Ensemble mean integration without any assimilation is noted as Ens. EnKF and ADEnKF are for 2-daily ensemble Kalman filtering of the single DL1 profile during the period 23 Dec 2001 through 19 Mar 2002, without and with inclusion of adaptive information, respectively.

Table 1.
Save