1. Introduction
Atmospheric and other similar models (e.g., oceanic, hydrological, ecological) still have large but unknown deficiencies. This is one of the major obstacles for further research progress, since it can seriously undermine research results by a large unknown error. Even though it is currently not possible to fully eliminate model deficiencies, they can be appropriately taken into account using advanced data assimilation methods. These methods are capable of extracting significant information about the model error from the observations. Data assimilation methods, based on classical Kalman filtering (Kalman 1960) and the weak-constraint variational approach (Sasaki 1970), are now reaching enough maturity to successfully estimate and correct model errors in realistic applications. For example, a bias correction approach was successfully applied within Kalman filtering by Dee (1995), Dee and da Silva (1998), Martin et al. (1999), Kalnay (2002), and Nichols (2003). The weak-constraint four-dimensional variational data assimilation (4DVAR) method was employed in Derber (1989), DeMaria and Jones (1993), Bennett et al. (1993, 1996), Zupanski (1993), Zupanski (1997), Uboldi and Kamachi (2000), Vidard et al. (2000), D’Andrea and Vautard (2000), Sasaki (2003), etc. The state augmentation method (e.g., Jazwinski 1970) for estimating a systematic model error has been used successfully in both Kalman filter and 4DVAR data assimilation schemes (Dee and da Silva 1998; Griffith and Nichols 1996, 2001; Martin et al. 2002). Recent studies indicated substantial benefits of model error adjustment in cases of assimilation of precipitation observations (e.g., Hou et al. 2000, 2001; D. Zupanski et al. 2002; M. Zupanski et al. 2002; Lee and Lee 2003). The bias correction technique of Derber (1989) is also being successfully applied to radiative transfer models (e.g., Derber and Wu 1998).
Ensemble data assimilation (EnsDA) methods are rapidly emerging as new-generation data assimilation techniques. The EnsDA methodologies are also referred to as ensemble Kalman filter (EnKF)-like approaches (e.g., Hansen 2002). Starting from the pioneering work of Evensen (1994), many techniques have been proposed as different variants of EnsDA (e.g., Houtekamer and Mitchell 1998; Hamill and Snyder 2000; Keppenne 2000; Mitchell and Houtekamer 2000; Anderson 2001; Bishop et al. 2001; van Leeuwen 2001; Reichle et al. 2002a, b; Whitaker and Hamill 2002; Tippett et al. 2003; Zhang et al. 2004; Ott et al. 2004; Zupanski 2005). More discussion about the differences in the current approaches can be found in Evensen (2003), Anderson (2003), and Tippett et al. (2003). Independently of the specific variant of the EnsDA, or the EnKF-like approach, the state estimate (or analysis) is obtained in an ensemble-spanned subspace, defined by a limited number of forecast model realizations (ensembles). In addition, ensemble-based data assimilation techniques provide a consistent mathematical formalism for updating the analysis and the forecast error covariance matrices, employing information from the observations. Because of the ease of the ensemble framework, a prediction model (e.g., atmospheric, oceanic, carbon, and other similar models) of any complexity can be used in EnsDA, employing essentially the same algorithm.
EnsDA methodologies, however, often assume that forecast models are perfect (i.e., model error is neglected). There are problems with this assumption, even in applications to relatively simple models, using simulated observations, as will be shown in this study. One possible way to relax the perfect model assumption is to simulate model error as a random or a serially correlated noise with prescribed error covariances (e.g., Heemink et al. 2001). A similar approach is used in Mitchell et al. (2002), where a balanced random noise, generated from a multivariate probability distribution function, is included into the model. Hansen (2002) applied an EnKF and a probabilistic 4DVAR to account for model error in a chaotic model. In Reichle et al. (2002a) a serially correlated model error was estimated employing an EnKF approach and the state augmentation method using a prescribed time-correlated model error correlation matrix.
Parameter estimation approaches are simplifications of the more complex model error estimation methods. The parameter estimation methods have previously been examined mostly within the frameworks of variational or Kalman filtering methods (e.g., Bennett and McIntosh 1982; Cohn and Dee 1988; Le Dimet and Navon 1988; Panchang and O’Brien 1989; Zou et al. 1992; Muccino and Bennett 2002; also see the review by Navon 1997 for more information). The EnKF-like approaches are also emerging as new parameter estimation techniques (Mitchell and Houtekamer 2000; Anderson 2001; Kivman 2003; Jones et al. 2004). These studies indicated that the parameter estimation is an effective approach to model error estimation when the amount of observed information and the ensemble size are not sufficient to constrain a more complex model error (e.g., serially correlated error). There is a theoretical framework, examined mostly within variational and Kalman filter approaches, that can be used for better understanding how and why the parameter estimation method should work. The framework is based on determining observability, that is, the amount of observed information about a parameter being estimated, and indentifiability, that is, the uniqueness of the parameter being estimated (e.g., Cohn and Dee 1988; Navon 1997). Only in cases when the estimated parameters are observable and identifiable is the parameter estimation method expected to work properly. When dealing with EnsDA approaches the observability and the indentifiability criteria are dependent on the ensemble size, thus requiring deeper understanding of the links between the ensemble size and the number of degrees of freedom (DOF) in the control variable and the observations.
The studies mentioned above have been just initial steps toward resolving the problem of model error estimation. This problem remains fundamental for future progress in variational, Kalman filter, and EnsDA methods, in ensemble forecasting (Buizza et al. 1999), and in predictability (Orrell 2003).
The methodology proposed here incorporates three approaches: 1) EnsDA, 2) state augmentation, and 3) parameter and model bias estimation. While there are more studies dealing with each of the three approaches separately, experience available so far in dealing with all three components as parts of the same methodology is much more limited. Here we list some of the pioneering studies incorporating these three components. For example, in Mitchell and Houtekamer (2000), Anderson (2001), Kivman (2003), and Jones et al. (2004), parameter estimation approaches were examined. In Reichle et al. (2002a), a serially correlated model error of a soil hydrology model was estimated.
The EnsDA approach used in this study is the maximum likelihood ensemble filter (MLEF) proposed by Zupanski (2005). This article can be considered as a continuation of the previous work by Zupanski (2005), since the same ensemble data assimilation approach (MLEF) and the same model, Korteweg–de Vries–Burgers (KdVB), are used. Although the MLEF approach is selected for this study, its use is not of essential importance. This is because the minimization, introduced by the MLEF, converges in a single iteration, since the observation operator is linear. Therefore, the proposed model error estimation is applicable to other EnKF-like approaches as well.
The paper is organized as follows. Serially correlated model error is described in section 2. The state augmentation method and its application within the framework of the EnsDA are explained in section 3. Experimental design is presented in section 4, and the results are discussed in section 5. The conclusions and future work are given in section 6.
2. Serially correlated model error (bias)
We assume that the model error is a serially correlated first-order Markov process variable (e.g., Daley 1992). Different variants of a serially correlated model error have been successfully used in variational (Derber 1989; Bennett et al. 1993, 1996; DeMaria and Jones 1993; Zupanski 1993; Griffith and Nichols 2001; Zupanski 1997) and Kalman filter (e.g., Dee 1995; Dee and da Silva 1998; Martin et al. 1999; Nichols 2003) approaches.
Assuming that Φ0 is known, and that b is estimated, (1) and (2) describe time evolution of both the model state and the model error. Thus, the estimation of the bias Φn reduces to the estimation of b, a vector assumed constant over a period of time (data assimilation interval). In this study we assume that both Φ0 and the guess of b (i.e., the value of b before optimization) are zero in the first data assimilation cycle, since no better guess is available. In any other cycle these values are determined from the previous data assimilation cycle (explained in more detail in appendix A).
The model bias parameter b can be defined in the same space as the model state variable x. This allows for possible corrections of model state variables in each grid point and for each component (e.g., temperature, pressure, wind, humidity). This is of special importance for limited-area models, where assumption (2) can be used to estimate and correct the error in the horizontal boundary conditions (e.g., Zupanski 1997). On the other hand, defining the serially correlated model error in the same space as the model state vector could be impractical for complex atmospheric models, since the number of DOF in the control variable would become very large (of the order of 108–1010). In such cases, an appropriate choice of the operator G may be needed to reduce the number of DOF of the bias estimation problem. These problems will be examined in this study.
3. State augmentation method
Equation (6) indicates that the observation operator H is rank deficient, with nonzero values corresponding to the x component only. This is because observations for the model error components of z (components b and v) are not available. A consequence is that the model error is only influenced by the observations through the cross covariance, describing correlations between the error of the model state estimate and the error of the model bias estimate, that is, 𝗣 fxb part of the full covariance 𝗣f . Therefore, the actual estimation of model bias is only possible if these cross correlations are nonzero. Note that 𝗣 fxb is not neglected in this study. Neglecting 𝗣 fxb has a consequence of being able to only simulate the impact of model error through a prescribed model error covariance matrix (often denoted 𝗤).
As mentioned before, the model error estimation procedure, described above, should be applicable not only to the MLEF, but to also to other EnKF-like approaches. These include the Monte Carlo EnKF (e.g., Evensen 1994; Houtekamer and Mitchell 1998), ensemble transform Kalman filter (ETKF) of Bishop et al. (2001), ensemble adjustment Kalman filter (EAKF) of Anderson (2001), the local ensemble Kalman filter (LEKF) approach of Ott et al. (2004), etc. The differences between various EnKF-like approaches do not impact the model error estimation methodology, since the equivalence between the Kalman gain and inverse Hessian/gradient calculation (as shown in Zupanski 2005) is still valid in the augmented state variable approach.
4. Experimental design
a. Model
b. Observations
The observations are defined by perturbing the true state (12), where perturbations are drawn from a white-noise Gaussian distribution. The observation error covariance 𝗥 is diagonal (e.g., variance), with prescribed elements equal to a constant ε2obs = 0.052. The observation number varies in the experiments. In most experiments only 10 irregularly spaced observations are assimilated, targeted to sample the crests of the two waves as in Zupanski (2005). A full set of observations (101 per data assimilation cycle) is also used in some experiments to examine the impact of additional information given by the observations. The observations are always defined at the end of the data assimilation interval, corresponding to 200 model time steps (the integration time in each cycle is 2 nondimensional units). In all experiments presented 100 data assimilation cycles are performed; however, when appropriate, the results from later data assimilation cycles (up to 200) are discussed. In Fig. 1 we illustrate the true model solution and observations used in the experiments with the reduced number of observations. The experiments with a full set of observations use the same true model solution, but observations at each grid point are used (figure not shown).
The value of diffusion coefficient in Fig. 1 is ν = 0.07. As Fig. 1 indicates, the amplitude of the model state solution is decreasing with time, due to diffusion, which might contribute to stabilizing the filter performance by reducing the root-mean-square (rms) errors of the estimated solution, in all experiments. Relative comparisons between the experiments (if performed under the same conditions) are still valid.
The observation operator chosen for this study is a linear (identity) transformation operator, defined as H(u) = u. The choice of observation operator does not significantly impact the model error estimation results; for the cases of nonlinear observation operators, more minimization iterations are needed (2–3 instead of 1). The impact of different observation operators on the minimization performance is examined in Zupanski (2005).
c. Experiments
1) Cold start procedure
Control variable (initial conditions, model bias parameter, and empirical parameters) and ensemble perturbations have to be initialized in the first data assimilation cycle, that is, a cold start procedure has to be performed. This initialization is not needed in subsequent cycles, since the data assimilation scheme updates both the control variable and the ensemble perturbations. The cold start procedure used here is similar to the procedures commonly used in other EnKF-like approaches. Because of the specific, less common, extension of the EnsDA methodology to include model bias and parameter estimation, we will explain the cold start procedure in more detail.
Regarding the initial conditions component of the control variable, two possibilities are used: perfect initial conditions and imperfect initial conditions. Perfect initial conditions are used to initialize the true KdVB model. The perfect initial conditions are defined as a two-soliton analytic solution of the Korteweg–de Vries (KdV) equation (Vvedensky 1993). For data assimilation purposes, we assume that the knowledge about the truth is not available and use imperfect initial conditions. The imperfect initial conditions are created using a slightly different KdV analytical solution. Namely, the difference between the perfect and the imperfect initial conditions is roughly 10% in the amplitudes of the soliton waves and there is a lag in time corresponding to 100 time steps of the numerical KdVB model (half of the data assimilation interval).
The model bias component of the control variable is initialized with zero (the actual bias is, however, different from zero in the bias estimation experiments). The third component of the control variable, the diffusion parameter, is initialized with values 0.07 or 0.20, depending on the particular experiment, as explained later.
Initial ensemble perturbations are imposed upon the initial values of the control variable. These perturbations are defined as white-noise Gaussian random variables for all components of the control variable. The standard deviation of the perturbations is set to 10%–20% of the typical initial magnitude of the corresponding control variable. For the model bias, the standard deviation of the ensemble perturbations could not be set to a fraction of the typical value of the control variable, since its initial magnitude is zero. In this case, the standard deviation is defined as a fraction (1%) of the standard deviation used for generating perturbations for the initial conditions. The initial ensemble perturbations are only defined for the control variable components needed in the particular experiment. For example, in the data assimilation experiments with the perfect model (referred to as correct_model experiments) and in the experiments with neglecting model error (referred to as neglect_err experiments), only the initial conditions are perturbed. Likewise, in the experiments with model bias estimation the initial conditions and the model bias are perturbed; in the experiments with parameter estimation the initial conditions and the diffusion parameter are perturbed. Finally, the initial forecast error covariance is defined by integrating the forecast model (5) from a set of the initial ensemble perturbations two cycles prior to the first observation time. In this way the dynamical constraint of the forecast model is added on the initial random perturbations.
Two sets of data assimilation experiments are performed to examine the impact of (i) parameter estimation, as one of the simplest, and the impact of (ii) model bias estimation, as one of the most complex, approaches to model error estimation.
2) Parameter estimation
Parameter estimation experiments are designed to assess the impact of incorrect diffusion in the KdVB model. In the correct model, which is used to define observations, as well as in one of the control experiments explained below, the diffusion coefficient is ν = 0.20. In the erroneous model the diffusion coefficient is ν = 0.07. The following experiments are performed: (i) correct_model, where the model with correct diffusion is used (defines the upper limit of success of the model error estimation), (ii) neglect_err, where erroneous model is used, but the error was neglected (used as the lower limit of success of the model error estimation method), and (iii) param_estim, where the diffusion parameter is estimated, staring from an incorrect value. The diffusion parameter is estimated by augmenting the control variable to include the unknown parameter. This results in the increase of the control variable dimension from 101 to 102. The majority of the experiments are performed employing 10 ensemble members. In basic experiments, 10 observations are assimilated in each data assimilation interval. To examine the impact of increased number of observations, additional experiments are performed using 101 observations. In all experiments, observations are defined at the end of each data assimilation interval. The parameter estimation experiments are summarized in Table 1.
3) Model bias estimation
For the case of the experiment bias_estim, the estimated model bias parameter b is used to correct the erroneous model (13) via (4). The value of constant α = 0.8 is used in all bias estimation experiments, defining a smooth change from the bias of the previous cycle to the bias of the current cycle. (Note that α is not used in the parameter estimation experiments, since b is not included in the control variable.) The experiments with different values of α indicated that small variations (10%–20%) of this parameter did not have a significant impact on the experimental results.
Regarding the operator G, two different choices, representing two limiting cases, are examined: (i) G = 𝗜, the identity operator, and (ii) G = 𝗚(u), a state-dependent operator. The first choice is the most straightforward approach, implying that the model error correction does not depend on the state u and the dimension of the control vector b is equal to the dimension of the model state vector. The second choice of G is used to examine the impact of the control vector b that has a smaller size than the model state u, which may be desirable in application to complex atmospheric models. One simple way to accomplish this is to define b in the coarse spatial resolution and use an interpolation operator G to transform b into the full space of the forecast model. In this study, rather then making b tied to the grid points of the model domain, it is defined in the phase space, tied to the magnitudes of the vector u. Operator G, defined as our second choice G = 𝗚(u), is described in appendix B. In the experiments presented 𝗚 is used to map vector b (of dimension 10) into 10 categories of magnitudes of the model state vector u (of dimension 101). This implies that similar magnitudes of u have similar model bias corrections. For the case of the KdVB model this is a desirable property, since it makes the bias traveling with the waves. On the other hand, since the solution of the KdVB in our experiments consists of two waves (see Fig. 1), the specific choice of 𝗚 would occasionally result in applying the same bias correction to the u points having the same magnitude, but belonging to two different waves, which could potentially degrade the bias estimation results. Having these considerations in mind, the choices of the operator G used in this study should be interpreted only as means to examine the impact of varying size of vector b. The optimal choice of the operator G would depend on the properties of the forecast model and would require further studies. Note that in all experiments the number of DOF in the “real” bias (ηu0) was kept as large as the model state vector [bias dimension (dim) = 101] in all experiments, to simulate the conditions of complex atmospheric models. The bias estimation experiments are summarized in Table 2.
d. Validation
The experimental results will be compared and validated in terms of verification scores used in Kalman filter verifications, such as the rms errors of the state estimate, the Kolmogorov–Smirnov (K–S) test of the probability density function (PDF) of the innovations, and the χ2 test of the innovations. The rms error measures how close the calculated estimate is to the true state, while the innovation statistics provide information about the performance of the filter itself.
The K-S test (15) compares maximum absolute value of the difference between the empirical distribution function (i/Nobs) and the theoretical distribution function
Assuming that observation error covariance matrix 𝗥 is known, and the Gaussian assumptions are considered appropriate, the statistics (15) and (16) can be used as verification tools for the forecast error covariance 𝗣f .
5. Results
a. Parameter estimation
The results of the data assimilation experiments correct_model, neglect_err, and param_estim, in terms of rms errors with respect to the true solution, are presented in Fig. 2. In the experiments neglect_err and param_estim the incorrect diffusion coefficient is used (ν = 0.07), while in the experiment correct_model the correct diffusion coefficient is used (ν = 0.20). The true solution is defined by the numerical solution of the KdVB model (12), starting from the perfect initial conditions. For comparison, rms errors of the experiment using the model with the incorrect diffusion coefficient, but without data assimilation (no_assim), are also shown in Figs. 2a and 2b. In Fig. 2a only 10 observations are used in the experiments, while in Fig. 2b, the experimental results employing 101 observations in each data assimilation cycle are presented. Note that the rms errors of all experiments in Figs. 2a and 2b are smaller in the later than in the initial data assimilation cycles. This is due to the diffusive character of the KdVB model. As the figures indicate, the model error estimation experiment performs as expected, having the rms errors between the lower (neglect_err) and upper limit of success (correct_model). The results of the experiment neglect_err are not satisfactory in Fig. 2a, since the rms errors are often larger than the errors of the experiment without data assimilation (no_assim). This is slightly improved in Fig. 2b, since the rms errors of the experiment neglect_err are slightly, but consistently, smaller than the errors of the experiment no_assim. The rms errors of the experiment param_ estim are smaller than the errors of the experiment no_assim in both figures, except in the last 10 cycles in Fig. 2a, where the errors of both experiments are of the same level. As expected, the experiment correct_model shows the best results in both figures. Comparison of the two figures indicates that problems caused by the error in the diffusion coefficient cannot be eliminated by increasing the number of observations, since the experiment neglect_err shows little improvement. On the other hand, increasing the number of observations is more effective in the case when model error is estimated and corrected (param_estim), and even more in the case when model error does not exist (correct_ model). These results, if proven valid for realistic atmospheric models, may have a significant consequence on designing the future observing systems, since including additional observations, without taking a proper care of the model error, would not necessarily result in improved data assimilation results.
The innovation statistics, in terms of the PDF of the innovations, calculated for the experiments correct_ model, neglect_err, and param_estim are presented in Figs. 3 and 4. Figure 3 shows data assimilation results employing 10 observations per data assimilation cycle, and Fig. 4, employing 101 observations per data assimilation cycle. The innovations are calculated in all observation points during 90 data assimilation cycles (from cycle 10 to cycle 100) in all experiments. As Figs. 3 and 4 indicate, the results of the correct_model experiment are in best agreement with the theoretical Gaussian distribution; the param_estim experiment is also in a reasonably good agreement, while neglecting model error (neglect_err) resulted in largest departure from the expected PDF. This is an indication of a questionable quality of the analysis and forecast error covariances in the case of an erroneous model. As for the rms errors, comparison of Figs. 3 and 4 indicates that the model error problem cannot be eliminated by increasing the number of observations. On the other hand, the additional observations do provide an improvement in the cases when model error does not exist (correct_model), or it has been taken into account (param_estim).
For quantitative comparisons between the PDFs in Figs. 3 and 4 the results of the K-S test statistic D, given in (15), calculated against theoretical distribution N(0, 1), are summarized in Table 3. The statistic D indicates the largest departures from the standard normal distribution for experiments neglect_err and the smallest departures for the experiments correct_model; the K-S statistic for the experiment param_estim is between the two limiting cases. Because of larger ensemble size, experiments with 101 observations per cycle generally have smaller values of D than the experiments with 10 observations per cycle; this is also reflected in the smaller critical value α at the same significance level (5%). Further examination of Table 3 indicates that the hypothesis H0 is accepted only for the experiments with the correct model. Even though one should expect only approximate Gaussian distributions, as explained in section 4d, the fact that H0 is accepted for the experiments with correct model, but rejected for the parameter estimation experiments (and also for the experiments neglect_err), indicates that the parameter estimation process is less than perfect. In addition, Fig. 7a, discussed in more detail later, shows that the estimated values of ν are different from the true values, except for the parameter estimation experiment with 102 ensemble members and 101 observations. To find out if the incorrectly estimated value of ν is causing the rejection of the hypothesis H0, additional K-S tests are performed and summarized in Table 4. Namely, the experimental results of two param_estim experiments are given in Table 4; both experiments are using 102 ensemble members and 101 observations, but in one experiment true ν = 0.20, and in the other true ν = 0.07 (Fig. 7a). For comparison, K-S test results of the experiment neglect_err with 101 ensemble members, 101 observations, and ν = 0.07 (true ν = 0.20) are also included in Table 4 (figure is not shown for this experiment). In other words, the results of the experiments solving full rank problem and employing full set of observations are presented in Table 4. These experiments should provide the best possible results. As shown in Table 4, the hypothesis H0 is now accepted for both param_estim experiments, but it is still rejected for the experiment neglect_err. These results indicate that the results of the parameter estimation can be significantly improved (in terms of the K-S statistics) by including more observations and more ensemble members, while this is not true for the experiment neglect_err.
Finally, the K-S tests, as in Tables 3 and 4 (and also in Table 5 discussed later) have also been performed for significance levels 10% and 1%. The experimental results, regarding accepting or rejecting the hypothesis H0, remained the same.
The results of the χ2 innovation test are presented in Figs. 5 and 6, for the same three experiments. As before, the results employing 10 observations (Fig. 5) and 101 observations (Fig. 6) are presented. Note that this statistical measure, if significantly departing from the value of one, indicates the divergence of the filter. One should be aware however, that since a significance test for the χ2 statistic have not been performed, the meaning of the term “significant” has a degree of arbitrariness. Nevertheless, relative comparisons between the experiments are still valid. As in previous measures, the correct_model experiment has the best performance, and neglect_err has the worst. As expected, the parameter estimation (param_estim) results are in between the two limiting cases. One can also notice in Fig. 6a a pronounced improvement of the filter performance for the experiment neglect_err, and to some extent of the experiment param_estim, as the number of cycles increases. This could be a consequence of using a highly diffusive model (ν = 0.20) when creating the observations. Note that we consider the KdVB model highly diffusive only in the experiments with ν = 0.20 (not in all experiments).
The experimental results presented so far indicated a good performance of the proposed parameter estimation method. Let us now examine in more detail the value of the estimated diffusion parameter itself. The diffusion coefficients obtained in various parameter estimation experiments are shown in Fig. 7, plotted as functions of data assimilation cycles. Figure 7a shows the experimental results employing (i) 102 ensembles and 101 observations (full rank problem, full set of observations), (ii) 10 ensembles and 101 observations, and (iii) 10 ensembles, 10 observations; the true value of the parameter (true ν = 0.20) is also shown. As expected, the experiment with the maximum number of ensemble members and observations recovers the value that is closest to the true value. It is somewhat surprising that the experiment with 10 observations was able to recover a value that is closer to the truth than the experiment with more observations (101). Understanding this issue needs further investigation since it is related to projection of the observed information onto the ensemble subspace. Thus, a better understanding of the observability (e.g., Jazwinski 1970; Cohn and Dee 1988; Navon 1997) in the EnsDA framework is needed. Note that this experiment, even though not perfect, was clearly better than the experiment without parameter estimation (see Figs. 4 and 6; Table 3). Examination of Fig. 7b indicates that increasing the number of ensemble members (102 ensembles are used) helps to further improve the estimate of the unknown parameter (the experiments with two true values of the coefficient, 0.07 and 0.20, are presented). As shown in Table 4, this improvement is also reflected in the K-S test results.
In summary, the experimental results of this section indicated clearly beneficial results of the parameter estimation algorithm, even in the cases with little observations, employing a small number of ensemble members. For the case of the KdVB model, as used in this study, 10 observations and 10 ensemble members were sufficient for beneficial results.
b. Model bias estimation
Model bias estimation experiments are performed to evaluate the potential of the EnsDA approach to estimate and correct model errors in the form of biases, commonly present in atmospheric and other similar models. Since it was anticipated that the many degrees of freedom present in model biases could pose a serious challenge to the bias estimation problem, data assimilation experiments estimating model biases of varying DOF are performed. In Fig. 8, the rms errors for the following four data assimilation experiments are presented: (i) correct_model, (ii) neglect_err, (iii) bias_ estim (with b of dimension 101), and (iv) bias_estim (with b of dimension 10). In all four experiments 10 ensembles and 10 observations are used. For comparison, the rms errors of the experiment using the biased model, but without data assimilation (no_assim) are also shown in Fig. 8. As explained before, the bias was created by applying an additive term, defined as −ηu0, in each time step of the model integration. The same model bias is used in all experiments with the biased model (neglect_err, bias_estim, and no_assim). Comparison of the results in Fig. 8a indicates superior performance of the experiment with the nonbiased model (correct_model) and a dramatic adverse effect of the model bias on the data assimilation results when the bias was neglected (neglect_err). Note that the experiment neglect_err is close to the experiment no_assim in most data assimilation cycles. Only in the first seven data assimilation cycles in Fig. 8a is neglect_err better than no_assim. Both bias_estim experiments (with dim = 10 and dim = 101) indicate clear improvements with respect to the experiments neglect_err and no_assim, since the rms errors are smaller and are closer to the errors of the experiment correct_model in later data assimilation cycles (Fig. 8a). The rms errors of bias_estim with dim = 10 are considerably smaller than the errors of bias_estim with dim = 101, indicating positive impact of reducing the number of DOF in the bias parameter b. One can observe, however, that after cycle 90 the curves for bias_estim with dim = 10 and dim = 101 cross, suggesting a possibility that the results are opposite in later data assimilation cycles. In Fig. 8b 100 additional data assimilation cycles are included, thus presenting the results of 200 cycles for all five experiments. The results from 100 additional cycles indicate that the two bias_estim curves cross back after a few cycles, and the experiment bias_estim (dim = 10) still remains close to the experiment correct_model, while the rms errors of the experiment neglect_err continue to increase and remain close to the experiment without data assimilation (no_assim), even though there is slight improvement of the results of neglect_err with respect to no_assim after data assimilation cycle 130. Recalling the rms errors shown in Fig. 2, we conclude that little benefits of assimilation of observations could be expected if the model errors are neglected, since the results of the experiment neglect_err are close to the results of the experiment no_assim in both figures (Figs. 2 and 8). The experiment bias_estim with bias dimension of 101, even though it is relatively close to the experiment correct_model, shows a trend of filter divergence in later data assimilation cycles. These results indicate that estimation of a bias with many DOF might not work well in cases when the ensemble size is much smaller. On the other hand, reducing the number of DOF of the estimated bias, as in the experiment bias_ estim with bias dimension of 10, helps preventing filter divergence. Thus, we conclude that neglecting the model bias could result in filter divergence; however, the filter divergence could be alleviated by appropriate treatment of the model bias (as in bias_estim, dim = 10). It has also been recognized in previous studies that filter divergence could occur due to neglecting model errors (e.g., Jazwinski 1970; Hansen 2002; Mitchell et al. 2002).
In Fig. 9 the PDF innovation statistics are shown for the same four data assimilation experiments (correct_ model, neglect_err, bias_estim with dim = 10, and bias_ estim with dim = 101). Obviously, neglecting the model bias (Fig. 9a) results in considerable degradation of the filter performance. As before, the correct_model experiment (Fig. 9d) shows the best performance, thus defining the upper limit of success to be expected from the bias estimation process. Bias estimation of the bias parameter with 101 DOF (Fig. 9b), even though it shows an improvement with respect to neglect_err (Fig. 9a), has relatively large discrepancies in comparison with correct_model (Fig. 9d). The bias estimation process is clearly beneficial in the bias_estim experiment with reduced number of DOF (Fig. 9c), since the results are closer to the experiment correct_model than to the experiment neglect_err.
The results of the K-S tests performed for the experiments in Fig. 9 are summarized in Table 5. As shown in Table 5, the largest departures from the standard normal distribution are obtained for the experiment neglect_err, and the smallest for the experiment correct_ model. As before, the experimental results of the experiment bias_estim are in between the two limiting cases. The K-S test results also confirm that the bias estimation employing the control vector b of reduced dimension (10 versus 101) is beneficial. Neither of the experiments, however, passed the K-S test. This is not a surprising result, since it is not expected that the statistics should be strictly Gaussian. On the other hand, dramatic departures from the standard normal distribution are not expected either. Obviously, understanding this issue is not trivial, since it is related to the validity of the Gaussian assumption in application to biased models. Addressing non-Gaussian probability distributions is beyond the scope of this study, but it is planned for the future.
The results of the χ2 validation tests are shown in Fig. 10. Figure 10a indicates filter divergence for the experiment neglect_err (also pronounced in the rms errors in Fig. 8). Bias estimation process is clearly beneficial (Figs. 10b,c), with the values of χ2 much closer to the results of the correct_model (Fig. 10d) than to the experiment neglect_err (Fig. 10a). These results are quite encouraging, indicating the potential of the EnsDA methodologies to properly estimate and correct model biases.
6. Conclusions
A methodology for model error estimation, employing the EnsDA approach and the state augmentation method, was proposed and examined in this study. The model errors in the form of a serially correlated error (bias) and a parameter error are examined. The methodology is fully adaptive, providing estimates of the atmospheric state and the model error, as well as the uncertainties of these estimates as defined by the analysis and forecast error covariance matrices.
Data assimilation and model error estimation experiments were performed in application to the KdVB model. The experiments employing a correct and an erroneous model were used as control experiments to define the upper and lower limits of the expected success of the model error estimation process. All validation scores examined (rms errors, χ2 test, K-S test, etc.) were within the two extremes and, in general, much closer to the experiments with the correct model, thus indicating substantial improvements due to model error estimation. Neglecting model error (either in the form of incorrect diffusion or model bias), however, had detrimental effects on the data assimilation results, in some cases resulting in filter divergence. The experimental results indicated that bias estimation of the model bias with many DOF, but employing a small ensemble size could also result in filter divergence. It was demonstrated that in such cases, reducing the number of DOF in the estimated bias helps prevent filter divergence. Therefore, it may be of fundamental importance to appropriately estimate and correct model error when employing an EnsDA approach.
There are also some issues that need further attention, but have not been addressed because of the limited scope of this study. These issues include understanding observability in the EnsDA framework and addressing non-Gaussian probability distribution functions when dealing with biased models. This research is planned for the future.
The model error estimation approach was examined within the framework of the MLEF of Zupanski (2005), but it was not tied to this particular variant of EnsDA. Its application to other EnsDA approaches should be straightforward. The methodology can also be used within an ensemble smoother. The integration of a forecast model and assimilation of observations over a time interval performed in a smoother, rather then using instant observations as in a filter, may further improve the results, since more observations could better constrain the model error. Model error estimation applying an ensemble smoother approach is also planned for future work.
Acknowledgments
The authors thank Nancy Nichols, Eugenia Kalnay, and Dale Barker for helpful discussion during the preparation of this manuscript. We have also benefited from the helpful suggestions made by Arthur Hou and Sara Zhang. Inspiring discussions with Thomas Vonder Haar and Tomislava Vukicevic are very mush appreciated. The authors are also thankful to Rolf Reichle and two anonymous reviewers for providing many helpful comments that resulted in substantial improvements of the manuscript. This work was supported by the Department of Defense Center for Geosciences/Atmospheric Research Grant DAAD19-02-2-0005.
REFERENCES
Anderson, J. L., 2001: An ensemble adjustment filter for data assimilation. Mon. Wea. Rev., 129 , 2884–2903.
Anderson, J. L., 2003: A local least squares framework for ensemble filtering. Mon. Wea. Rev., 131 , 634–642.
Bennett, A. F., and P. C. McIntosh, 1982: Open ocean modeling as an inverse problem: Tidal theory. J. Phys. Oceanogr., 12 , 1004–1018.
Bennett, A. F., L. M. Leslie, C. R. Gagelberg, and P. E. Powers, 1993: Tropical cyclone prediction using a barotropic model initialized by a generalized inverse method. Mon. Wea. Rev., 121 , 1714–1729.
Bennett, A. F., B. S. Chua, and L. M. Leslie, 1996: Generalized inversion of a global numerical weather prediction model. Meteor. Atmos. Phys., 60 , 165–178.
Bishop, C. H., B. J. Etherton, and S. Majumjar, 2001: Adaptive sampling with the ensemble transform Kalman filter. Part I: Theoretical aspects. Mon. Wea. Rev., 129 , 420–436.
Buizza, R., M. Miller, and T. Palmer, 1999: Stochastic representation of model uncertainties in the ECMWF Ensemble Prediction System. Quart. J. Roy. Meteor. Soc., 125 , 2885–2908.
Cohn, S. E., and D. P. Dee, 1988: Observability of discretized partial differential equations. SIAM J. Numer. Anal., 25 , 586–617.
Daley, R., 1992: The effect of serially correlated observations and model error on atmospheric data assimilation. Mon. Wea. Rev., 120 , 164–177.
D’Andrea, F., and R. Vautard, 2000: Reducing systematic errors by empirically correcting model errors. Tellus, 52A , 21–41.
Dee, D., 1995: On-line estimation of error covariance parameters for atmospheric data assimilation. Mon. Wea. Rev., 123 , 1128–1145.
Dee, D., and A. M. da Silva, 1998: Data assimilation in the presence of forecast bias. Quart. J. Roy. Meteor. Soc., 124 , 269–295.
DeMaria, M., and R. W. Jones, 1993: Optimization of a hurricane track forecast model with the adjoint model equations. Mon. Wea. Rev., 121 , 1730–1745.
Derber, J. C., 1989: A variational continuous assimilation technique. Mon. Wea. Rev., 117 , 2437–2466.
Derber, J. C., and W-S. Wu, 1998: The use of TOVS cloud-cleared radiances in the NCEP SSI analysis system. Mon. Wea. Rev., 126 , 2287–2302.
Evensen, G., 1994: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. J. Geophys. Res., 99 , C5. 10143–10162.
Evensen, G., 2003: The ensemble Kalman filter: Theoretical formulation and practical implementation. Ocean Dyn., 53 , 343–367.
Griffith, A. K., and N. K. Nichols, 1996: Accounting for model error in data assimilation using adjoint methods. Computational Differentiation: Techniques, Applications, and Tools: Proc. Second Int. SIAM Workshop on Computational Differentiation, Santa Fe, NM, Society for Industrial and Applied Mathematics, 195–204.
Griffith, A. K., and N. K. Nichols, 2001: Adjoint techniques in data assimilation for treating systematic model error. J. Flow Turbulence Combust., 65 , 469–488.
Hamill, T. M., and C. Snyder, 2000: A hybrid ensemble Kalman filter–3D variational analysis scheme. Mon. Wea. Rev., 128 , 2905–2919.
Hansen, J. A., 2002: Accounting for model error in ensemble-based state estimation and forecasting. Mon. Wea. Rev., 130 , 2373–2391.
Heemink, A. W., M. Verlaan, and A. J. Segers, 2001: Variance reduced ensemble Kalman filtering. Mon. Wea. Rev., 129 , 1718–1728.
Hollander, M., and D. A. Wolfe, 1973: Nonparametric Statistical Methods. John Wiley and Sons, 503 pp.
Hou, A. Y., D. Ledvina, A. da Silva, S. Zhang, J. Joiner, R. Atlas, G. Huffman, and C. Kummerow, 2000: Assimilation of SSM/I-derived surface rainfall and total precipitable water for improving the GEOS analysis for climate studies. Mon. Wea. Rev., 128 , 509–537.
Hou, A. Y., S. Zhang, A. da Silva, W. Olson, C. Kummerow, and J. Simpson, 2001: Improving global analysis and short-range forecast using rainfall and moisture observations derived from TRMM and SSM/I passive microwave sensors. Bull. Amer. Meteor. Soc., 82 , 659–679.
Houtekamer, P. L., and H. L. Mitchell, 1998: Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev., 126 , 796–811.
Jazwinski, A. H., 1970: Stochastic Processes and Filtering Theory. Academic Press, 376 pp.
Jones, J. W., W. D. Graham, D. Wallach, W. M. Bostick, and J. Koo, 2004: Estimating soil carbon levels using an ensemble Kalman filter. Trans. ASAE, 47 , 331–339.
Kalman, R., 1960: A new approach to linear filtering and prediction problems. J. Basic Eng., D82 , 35–45.
Kalnay, E., 2002: Atmospheric Modeling, Data Assimilation and Predictability. Cambridge University Press, 341 pp.
Keppenne, C., 2000: Data assimilation into a primitive-equation model with a parallel ensemble Kalman filter. Mon. Wea. Rev., 128 , 1971–1981.
Kivman, G. A., 2003: Sequential parameter estimation for stochastic systems. Nonlinear Processes Geophys., 10 , 253–259.
LeDimet, F. X., and I. M. Navon, 1988: Variational and optimization methods in meteorology: A review. Tech. Rep. SCRI-144, Supercomputer Computations Research Institute, The Florida State University, Tallahassee, FL, 83 pp. [Available from Supercomputer Computations Research Institute, The Florida State University, Tallahasee, FL 32306.].
Lee, M-S., and D-K. Lee, 2003: An application of a weakly constrained 4DVAR to satellite data assimilation and heavy rainfall simulation. Mon. Wea. Rev., 131 , 2151–2176.
Marchant, T. R., and N. F. Smyth, 2002: The initial-boundary problem for the Korteweg–de Vries equation on the negative quarter-plane. Proc. Roy. Soc. London,, A458 , 857–871.
Martin, M. J., N. K. Nichols, and M. J. Bell, 1999: Treatment of systematic errors in sequential data assimilation. Tech. Note 21, Ocean Applications Division, U.K. Met Office, 45 pp.
Martin, M. J., M. J. Bell, and N. K. Nichols, 2002: Estimation of systematic error in an equatorial ocean model using data assimilation. Int. J. Numer. Methods Fluids, 40 , 435–444.
Menard, R., S. E. Cohn, L-P. Chang, and P. M. Lyster, 2000: Assimilation of stratospheric chemical tracer observations using a Kalman filter. Part I: Formulation. Mon. Wea. Rev., 128 , 2654–2671.
Mitchell, H. L., and P. L. Houtekamer, 2000: An adaptive ensemble Kalman filter. Mon. Wea. Rev., 128 , 416–433.
Mitchell, H. L., P. L. Houtekamer, and G. Pellerin, 2002: Ensemble size, balance, and model-error representation in an ensemble Kalman filter. Mon. Wea. Rev., 130 , 2791–2808.
Muccino, J. C., and A. F. Bennett, 2002: Generalized inversion of the Korteweg–de Vries equation. Dyn. Atmos. Ocean, 35 , 227–263.
Nichols, N. K., 2003: Treating model error in 3-D and 4-D Data Assimilation. Data Assimilation for the Earth System, R. Swinbank, V. Shutyaev, and W. A. Lahoz, Eds., Kluwer Academic, 127–135.
Orrell, D., 2003: Model error and predictability over different timescales in the Lorenz ‘96 systems. J. Atmos. Sci., 60 , 2219–2228.
Ott, E., B. R. Hunt, I. Szunyogh, A. V. Zimin, E. J. Kostelich, M. Corazza, E. Kalnay, and D. J. Patil, 2004: A local ensemble Kalman filter for atmospheric data assimilation. Tellus, 56A , 415–428.
Panchang, V. G., and J. J. O’Brien, 1989: On the determination of hydraulic model parameters using the strong constraint formulation. Modeling Marine Systems, Vol. 1, A. M. Davies, Ed., CRC Press, 5–18.
Reichle, R. H., D. B. McLaughlin, and D. Entekhabi, 2002a: Hydrologic data assimilation with the ensemble Kalman filter. Mon. Wea. Rev., 130 , 103–114.
Reichle, R. H., J. P. Walker, R. D. Koster, and P. R. Houser, 2002b: Extended versus ensemble Kalman filtering for land data assimilation. J. Hydrometeor., 3 , 728–740.
Sasaki, Y. K., 1970: Some basic formalisms in numerical variational analysis. Mon. Wea. Rev., 98 , 875–883.
Sasaki, Y. K., 2003: A theory of variational assimilation with Kalman filter–type constraints: Bias and Lagrange multiplier. Mon. Wea. Rev., 131 , 2545–2554.
Tarantola, A., 1987: Inverse Problem Theory: Methods for Data Fitting and Model Parameter Estimation. Elsevier, 613 pp.
Tippett, M. K., J. L. Anderson, C. H. Bishop, T. M. Hamill, and J. S. Whitaker, 2003: Ensemble square root filters. Mon. Wea. Rev., 131 , 1485–1490.
Uboldi, F., and M. Kamachi, 2000: Time–space weak-constraint data assimilation for nonlinear models. Tellus, 52A , 412–421.
van Leeuwen, P. J., 2001: An ensemble smoother with error estimates. Mon. Wea. Rev., 129 , 709–728.
Vidard, P. A., E. Blayo, F-X. Le Dimet, and A. Piacentini, 2000: 4D variational data analysis with imperfect model. Flow Turbulence Combust., 65 , 489–504.
Vvedensky, D., 1993: Partial Differential Equations with Mathematica. Addison-Wesley, 465 pp.
Whitaker, J. S., and T. M. Hamill, 2002: Ensemble data assimilation without perturbed observations. Mon. Wea. Rev., 130 , 1913–1924.
Zhang, F., C. Synder, and J. Sun, 2004: Impacts of initial estimate and observation availability on convective-scale data assimilation with an ensemble Kalman filter. Mon. Rev. Rev., 132 , 1238–1253.
Zou, X., I. M. Navon, and F. X. LeDimet, 1992: An optimal nudging data assimilation scheme using parameter estimation. Quart. J. Roy. Meteor. Soc., 118 , 1163–1186.
Zupanski, D., 1997: A general weak constraint applicable to operational 4DVAR data assimilation systems. Mon. Wea. Rev., 125 , 2274–2292.
Zupanski, D., M. Zupanski, E. Rogers, D. F. Parrish, and G. J. DiMego, 2002: Fine-resolution 4DVAR data assimilation for the Great Plains tornado outbreak of 3 May 1999. Wea. Forecasting, 17 , 506–525.
Zupanski, M., 1993: Regional four-dimensional variational data assimilation in a quasi-operational forecasting environment. Mon. Wea. Rev., 121 , 2396–2408.
Zupanski, M., 2005: Maximum likelihood ensemble filter: Theoretical aspects. Mon. Wea. Rev., 133 , 1710–1726.
Zupanski, M., D. Zupanski, D. Parrish, E. Rogers, and G. J. DiMego, 2002: Four-dimensional variational data assimilation for the blizzard of 2000. Mon. Wea. Rev., 130 , 1967–1988.
APPENDIX A
Markov Process Variable on a Coarse Time Scale
APPENDIX B
Operator 𝗚(u)
Time integration of the KdVB model and targeted observations. The triangles denote observations. The horizontal axis represents the model domain, and the ordinate axis is the amplitude. The shown cycles are (a) 1, (b) 5, (c) 10, and (d) 20. Note how the targeted observations follow the wave crests of the model solution. Also, solution decays with time.
Citation: Monthly Weather Review 134, 5; 10.1175/MWR3125.1
(a) Rms errors of the optimal model state solution with respect to the true KdVB numerical solution showing the impact of the incorrect diffusion coefficient (ν = 0.07). The errors for three data assimilation experiments—correct_model, neglect_err, and param_estim—are presented as functions of data assimilation cycles. In all three experiments 10 ensemble members and 10 observations are used. Rms errors of the experiment using the model with the incorrect diffusion coefficient, but without data assimilation (no_assim), are also shown. (b) As in (a), but using 101 observations in each data assimilation cycle. Note that the rms errors of the experiment no_assim are identical to the errors shown in (a) since this experiment does not depend on the number of observations.
Citation: Monthly Weather Review 134, 5; 10.1175/MWR3125.1
Histogram of the PDF of the innovations, calculated using all observations assimilated during 100 data assimilation cycles. In each cycle 10 observations are used (Nobs = 10). The theoretical N(0, 1) Gaussian PDF is also plotted. The impact of incorrect diffusion is examined. The results of the following experiments are shown: (a) neglect_err, (b) param_estim, and (c) correct_model.
Citation: Monthly Weather Review 134, 5; 10.1175/MWR3125.1
Chi-square (χ2) innovation test values, calculated as instant values in each data assimilation cycle (dashed) and as a 10-cycle running averages (solid) for the experiments (a) neglect_ err, (b) param_estim, and (c) correct_model. The experimental results employing Nens = 10 and Nobs = 10 are shown.
Citation: Monthly Weather Review 134, 5; 10.1175/MWR3125.1
(a) Diffusion coefficients obtained in various parameter estimation experiments: (i) Nens = 102 and Nobs= 101 (solid bold), (ii) Nens = 10 and Nobs = 101 (dashed bold), (iii) Nens = 10 and Nobs = 10 (solid thin), and (iv) true value of the coefficient (dashed thin). (b) Diffusion coefficients obtained in parameter estimation experiments with two different values of the true coefficient (0.07 and 0.20). Parameter estimation experiments in (b) were performed employing Nens = 102 and Nobs = 101.
Citation: Monthly Weather Review 134, 5; 10.1175/MWR3125.1
(a) Rms errors of the optimal model state solution with respect to the true KdVB numerical solution showing the impact of model bias. The errors for four data assimilation experiments correct_model, neglect_err, bias_estim (dim = 101), and bias_ estim (dim = 10) are presented as functions of data assimilation cycles. In all four experiments 10 ensemble members and 10 observations are used. Rms errors of the experiment using the same model as in the experiment neglect_err and both bias_estim experiments (with dim = 101 and dim = 10), but without data assimilation (no_assim), are also shown. (b) As in (a), but the experiments are performed for 100 additional data assimilation cycles (the results of 200 cycles are shown).
Citation: Monthly Weather Review 134, 5; 10.1175/MWR3125.1
Histogram of the PDF of the innovations, calculated using all observations assimilated during 100 data assimilation cycles. In each cycle 10 observations are used (Nobs = 10). The theoretical N(0, 1) Gaussian PDF is also plotted. The impact of model bias estimation is examined. The results of the following experiments are shown: (a) neglect_err, (b) bias_estim (dim = 101), (c) bias_estim (dim = 10), and (d) correct_model.
Citation: Monthly Weather Review 134, 5; 10.1175/MWR3125.1
Chi-square (χ2) innovation test values, calculated as instant values in each data assimilation cycle (dashed) and as 10-cycle running averages (solid) for the experiments (a) neglect_err, (b) bias_estim (dim = 101), (c) bias_estim (dim = 10), and (c) correct_model. The experimental results employing Nens = 10 and Nobs = 10 are shown.
Citation: Monthly Weather Review 134, 5; 10.1175/MWR3125.1
Parameter estimation experiments. Note that Nobs indicates the number of observations per data assimilation cycle.
Bias estimation experiments.
The results of K-S test statistic D (15) for the parameter estimation experiments (Figs. 3 and 4). The hypothesis H0 that the normalized innovations (14) have Gaussian distribution N(0, 1) is tested. Here, and in Tables 4 and 5, ens = ensemble members and obs = observations.
As in Table 3, but for experiments param_estim and neglect_err; in all experiments the full rank problem is solved by employing full set of observations.