1. Introduction

The quality of forecasts from dynamical nonlinear models depends both on the model and on the quality of the initial conditions. Even under the idealized conditions of a perfect model of a deterministic chaotic nonlinear system and with infinite past observations, uncertainty in the observations can make identification of the exact state impossible (Berliner 1991; Lalley 2000; Judd and Smith 2001). Such limitations make a single “best guess” prediction a suboptimal approach to state estimation—an approach that would be frustrated even in ideal cases. Alternatively, an ensemble of initial conditions can better reflect the inescapable uncertainty in the observations by capturing the sensitivity of each particular forecast. A major application of state estimation is in the production of analyses of the state of the atmosphere in order to initialize a numerical weather prediction (NWP) model (e.g., Lorenc 1986; Keppenne and Rienecker 2002; Kalnay 2003; Anderson et al. 2005; Houtekamer et al. 2005). This paper is concerned with the identification of the current state of a nonlinear chaotic system given a sequence of observations in the perfect model scenario (PMS). The use of imperfect models is discussed elsewhere (Du and Smith 2014, hereafter Part II).

In PMS, there are states that are consistent with the model’s long-term dynamics and states that are not; in dissipative systems the consistent states are often said to “lie on the model’s attractor.” Intuitively, it makes sense to distinguish those states that are not only consistent with the observations but also consistent with the model’s long-term dynamics in state estimation. The problem of state estimation in PMS is addressed by applying the pseudo-orbit data assimilation (PDA) approach (Judd and Smith 2001; Judd et al. 2008; Stemler and Judd 2009) to locate a reference trajectory (a model trajectory consistent with the observation sequence; Gilmour 1998; Smith et al. 2010) and constructing an initial condition ensemble by using the model dynamics to sample the local state space. The PDA approach is shown to be more efficient¹ and robust in finding a reference trajectory than four-dimensional variational assimilation (4DVAR). The differences between PDA and 4DVAR are discussed. Ensemble Kalman filter (EnKF) approaches provide an alternative to 4DVAR. Illustrated here both on a low-dimensional map and on a higher-dimensional flow, PDA is demonstrated to outperform one of the many EnKF approaches—that is, the ensemble adjustment Kalman filter (Anderson 2001, 2003). It is suggested that this is a general result.

The data assimilation problem of interest is defined and alternative approaches are reviewed in section 2. A full description of the methodology of the PDA approach is presented in section 3. In section 4, the 4DVAR approach and the PDA approach are contrasted using the two-dimensional Ikeda map and the 18-dimensional Lorenz96 flow. Comparisons between the EnKF approach and the PDA approach for the same two systems are made in section 5. Section 6 provides a brief summary and conclusions.

2. Problem description

The problem of state estimation is addressed within the perfect model scenario focusing only on nonlinear deterministic dynamical systems.² Let be the state³ of a deterministic dynamical system at time t ∈ ℤ. The evolution of the system is given by with , where F denotes the system dynamics that evolve the state forward in time in the system state space ℝ^m and the system’s parameters are contained in the vector . In PMS, assume the mathematical structure of F is known while uncertainty in the values of may remain. Note that PMS does not require knowing the parameters of the system. Robust approaches have been proposed to identify the parameter values (e.g., Schittkowski 1994; McSharry and Smith 1999; Pisarenko and Sornette 2004; Tarantola 2004; Smith et al. 2010; Du and Smith 2012). This paper, however, considers only the strong case where both the model class (i.e., the mathematical structure of F) and the model parameter values are identical to those of the system. In PMS, the system state and the model state share the same state space and are thus “subtractable” (Smith 2006). Define the observation at time t to be , where h(·) is the observation operator that projects the true system state into observation space. For simplicity, h(·) is taken to be the identity operator below. Full observations are made; that is, observations are available for all state variables (all components of x) at every observation time.⁴ The η_t ∈ ℝ^m represent observational noise (or “measurement error”). The statistical characteristics of the observational noise (i.e., the noise model) for η_t are known exactly.

The problem of state estimation in PMS consists of forming an ensemble estimate of the current state given (i) the history of observations s_t, for t = −n + 1, …, 0; (ii) a perfect model class with perfect parameter values; and (iii) knowledge of the observational noise model. The various approaches that have been developed to address this problem divide into two major categories: the sequential approaches and the variational approaches. Sequential approaches can be built on the foundation of Bayesian theory, which conceptually generates a posterior distribution of the state variables by updating the prior distribution with new observations (Cohn 1997; Anderson and Anderson 1999). Unfortunately, application of the complete approach is computationally prohibitive in state estimation (Hamill 2006). An approximation to the Bayesian approach, called the Kalman filter, introduced by Kalman (1960), is optimal only for linear models and a Gaussian observational noise. To better address the state estimation problem in nonlinear cases, the extended Kalman filter (Jazwinski 1970; Gelb 1974; Ghil and Malanotte-Rizzoli 1991; Bouttier 1994) uses tangent linear dynamics to estimate the error covariances while assuming linear growth and normality of errors. The extended Kalman filter performs poorly where nonlinearity is pronounced; more accurate state estimates are available from ensemble Kalman filter approaches (Evensen 1994; Houtekamer and Mitchell 1998; Burgers et al. 1998; Lermusiaux and Robinson 1999; Anderson 2001; Bishop et al. 2001; Hamill et al. 2001), which have been developed using Monte Carlo⁵ techniques. While these filters admit the non-Gaussian probability density function, only the mean and covariance are updated in these sequential approaches; information beyond the second moment is discarded. Stressing the impact of these linear assumptions, Lawson and Hansen (2004) demonstrated that inaccuracies in state estimation are to be expected if the dynamics are nonlinear owing to higher moments of the distribution of the state variables. Another well-known sequential approach, the particle filter (Metropolis and Ulam 1944), is fully nonlinear in both model evolution and analysis steps. It, however, suffers from the so-called curse of dimensionality, which prevents the particles from staying “close” to the observations in a large-dimensional system (Snyder et al. 2008). Variational approaches [e.g., four-dimensional variational assimilation (Dimet and Talagrand 1986; Lorenc 1986; Talagrand and Courtier 1987; Courtier et al. 1994)], based upon maximum likelihood estimation, consist of shooting techniques that seek a set of initial conditions corresponding to a system trajectory that is consistent with a sequence of system observations; this is a strong constraint (Courtier et al. 1994; Bennett et al. 1996). In PMS, the variational approaches both are computationally expensive and are known to suffer from local minima owing to chaotic likelihoods (where the likelihood function of initial conditions is extremely jagged for chaotic systems). This was pointed out by Berliner (1991); see also Miller et al. (1994) and Pires et al. (1996). The approach presented in this paper applies the PDA approach to locate a reference trajectory and construct an initial condition ensemble by sampling the local state space. Khare and Smith (2011) applied a similar approach with a target of indistinguishable states (Q density) to form an initial-condition ensemble. More details about PDA and indistinguishable states can be found in Du (2009). State estimation outside PMS is discussed in Part II.

3. Pseudo-orbit data assimilation in sequence space

The analytic intractability of the relevant probability distributions and the dimension of the model state space suggest the adoption of a (Monte Carlo) ensemble scheme (Lorenz 1965; see also Smith 1996 and Leutbecher and Palmer 2008) to account for the uncertainties of observations in state estimation approach. The ensemble approach is by far the most common for quantifying uncertainty in operational weather forecasting (e.g., Toth and Kalnay 1993; Leutbecher and Palmer 2008). An algorithm may generate an ensemble directly, as with the particle filter and ensemble Kalman filters, or an ensemble may be generated from perturbations of a reference trajectory. The approach presented in this paper belongs in the second category. Of course, the quality of the state estimates will vary strongly with the quality of the reference trajectory(s). The pseudo-orbit data assimilation approach (Judd and Smith 2001; Ridout and Judd 2002; Judd and Smith 2004; Stemler and Judd 2009) provides a reference model trajectory given a sequence of observations. A brief introduction to the PDA approach is given in the following paragraph.

Let the dimension of the model state space be m and the number of observation times in the window be n. The sequence space is an m × n dimensional space in which a single point can be thought of as a particular series of n states u_t, t = −n + 1, …, 0. Here each u_t is an m-dimensional vector. Some points in the sequence space are trajectories of the model and some are not. Define a pseudo orbit, ≡ {u_−n+1, …, u₋₁, u₀}, to be a point in the m × n dimensional sequence space for which u_t+1 ≠ F(u_t) for any⁶ component of . This implies that corresponds to a sequence of model states that is not a trajectory of the model. Define the mismatch as an m × (n − 1) dimensional vector (e_−n+1, …, e₋₁), where the component of the mismatch at time t is e_t = |F(u_t) − u_t+1|, t = −n + 1, …, −1.

By construction, model trajectories have a mismatch of zero. A gradient descent (GD) algorithm (details in the following paragraph) can be used to minimize the sum of the squared mismatch errors. Define the mismatch cost function to be

View Expanded

The pseudo-orbit data assimilation minimizes the mismatch cost function for in the m × n dimensional sequence space. The minimum of the mismatch cost function can be obtained by solving the ordinary differential equation:

View Expanded

where τ denotes algorithmic time. An important advantage of PDA is that the minimization is done in the sequence space: information from across the assimilation window is used simultaneously. Let the elements of corresponding to the model state at a given time be called a component of the pseudo orbit. PDA optimizes all components simultaneously.

The observations themselves projected into the model state space define a pseudo orbit—call this pseudo orbit the observation-based pseudo-orbit ≡ {s_−n+1, …, s₋₁, s₀}; with probability 1 that it will not be a trajectory. In practice, the minimization is initialized with the observation-based pseudo orbit; that is, ⁰ = , where the presuperscript 0 on denotes the initial stage of the GD. After every iteration of the GD minimization, will be updated. Recall that the pseudo orbit is a point in the sequence space; it is updated in the sense that under the GD algorithm it moves toward the manifold of trajectories. All points on the trajectory manifold have zero mismatch (each is a trajectory) and only points on the trajectory manifold have zero mismatch. To iterate the algorithm, one needs to differentiate the mismatch cost function given by

View Expanded

where d_tF(u_t) is the Jacobian of the model F at u_t. The ordinary differential equation [Eq. (2)] is solved using the Euler approximation.

The mismatch cost function has no local minima other than on the manifold⁷ for which C() = 0 (Judd and Smith 2001). Let the result of the GD minimization be ^α, where α indicates discrete algorithmic time in GD (i.e., the number of iterations of the GD minimization). As α → ∞, ^α ≡ {^αu_−n+1, … ,^αu₀} asymptotically approaches a trajectory of the model. In other words, PDA takes the observation-based pseudo orbit toward a model trajectory (i.e., toward the trajectory manifold). That trajectory need not shadow the observations in any sense (Smith et al. 2010); its merits for state estimation must be demonstrated empirically.⁸ In practice, the GD minimization is run for a finite time and thus a pseudo orbit is obtained⁹ rather than a trajectory. (In the experiments presented in the paper, the minimization is terminated after 1024 minimization iterations.) Each component of a pseudo orbit defines a model trajectory. To keep the notation clear, denote the pseudo orbit obtained from some large finite GD runs as y_−n+1, …, y₀ and iterate the middle component¹⁰ y_−n/2 forward to create a segment of model trajectory z_−n/2, …, z₀ [y_−n/2 ≡ z_−n/2 and z_t+1 = F(z_t)]. Such a model trajectory defines a reference trajectory. The middle component is expected to provide a better estimate of model state at that time than the end component at its time, as the middle component has information from both its past and its future, while the end component only has information from its past. It is important to note that although the PDA approach can be applied to any length of observation window, given finite computational resource a reference trajectory corresponding to the midcomponent will almost certainly diverge from the pseudo orbit when n is large, simply as a consequence of sensitivity to initial conditions. The midcomponent need not always be used.

To form an ensemble of initial conditions,¹¹ first generate a large number of model trajectories, called candidate trajectories, from which ensemble members can be selected. Ensemble members are drawn from the candidate trajectories according to their relative likelihood given the segment of observations. There are many ways to produce candidate trajectories; three methods of producing candidate trajectories are listed here: (i) Sample the local space around the reference trajectory. One can perturb the starting component of the reference trajectory and iterate the perturbed component forward to create candidate trajectories. (ii) Perturb the whole segment of observations s_t, t = −n + 1, …, 0 and apply PDA onto the perturbed orbit to produce the candidate trajectories—that is, the same way that the reference trajectory is produced. (iii) Similar to method (ii), perturb the reference trajectory and repeat PDA. Although methods (ii) and (iii) may produce more informative candidates, they are obviously more expensive than method (i) since the GD minimization must be repeated. To make the computational cost between PDA and other state-estimation approaches more comparable, method (i) is used to generate candidate trajectories to produce the results presented in the paper. (Details about computational costs can be found in appendix C.) The perturbations to the starting component z_−n/2 are generated using a random variable ζ, ζ is Gaussian with zero mean and standard deviation σ_p ∈ ℝ^m, and σ_p is a constant diagonal matrix estimated by standard deviation of the difference between (the truth) and z_−n/2. In practice, the value of σ_p would have to be determined empirically. (Values of σ_p for the experiments below are given in Table B1.)

Given N^cand candidate trajectories, the relative likelihood of each is then used to select ensemble members of the assimilation. To form an N^ens-member ensemble estimate of the current state, randomly draw N^ens trajectories from N^cand candidate trajectories according to their log-likelihood function over the time interval . Specifically,

View Expanded

where Γ⁻¹ is the inverse of the covariance matrix of the observational noise and denotes a candidate trajectory. The end component (at t = 0) of each selected candidate trajectory is treated as an ensemble member. In this case, each ensemble member is then of equal weight avoiding any confusion regarding how to interpret the Q density in Judd and Smith (2001).

4. Contrasting 4DVAR with PDA

4DVAR is a popular approach to noise reduction in data assimilation (Dimet and Talagrand 1986; Lorenc 1986; Talagrand and Courtier 1987; Courtier et al. 1994). 4DVAR is a shooting technique that seeks initial conditions of system trajectories consistent with a sequence of system observations. It aims to find the initial state of a model trajectory that minimizes a cost function reflecting the misfit between the trajectory and the observations. The 4DVAR cost function is

View Expanded

where x_−n+1 is the initial state and x_t+1 = F(x_t), is the background (or first guess) at t = −n + 1, is the background error covariance matrix, s_t reflects the observations at time t, and Γ is the covariance matrix of the observational noise. The second term of the cost function can be easily derived from the maximum likelihood estimate under the assumption that the observational noise model is independent and identically distributed (IID) Gaussian. The first term (background term) of the cost function aims to take account of the information from previous estimates (and any other available prior distribution of the initial state). The background is typically provided by the 4DVAR analyses from previous cycles. An estimation of is required to minimize the cost function. For 4DVAR experiments conducted here, is obtained iteratively [following Fertig et al. (2007)] as follows. Initially run 4DVAR using an arbitrary background covariance matrix ₀, then compute the covariance ₁ of the difference between the true state¹² and the background at all of the assimilation times. Next, run 4DVAR using ₁ as the background error covariance matrix and again compute the covariance ₂ of the difference between the true state and the background. Repeat this process until the estimated background covariance matrix does not change significantly.

As in PDA, the 4DVAR analysis provides a reference trajectory for use in building an initial condition ensemble. Although both PDA and 4DVAR use the information of both the model dynamics and the observations to produce the model trajectories, there are fundamental differences between them.

The PDA cost function itself does not constrain the result to stay close to the observation-based pseudo orbit [Eq. (1)]. The GD minimization is, however, initialized with the observation-based pseudo orbit.¹³ Unlike the 4DVAR approach, the PDA approach does not penalize ^αU if it strays far from the observation-based pseudo orbit; in fact, the PDA approach is almost certain to force ^αU to move away, on average, from the observation-based pseudo orbit as the minimization goes further and further (see Part II). The 4DVAR approach is derived from the maximum likelihood estimate in the case of additive IID Gaussian observational noise. For other noise models, including those non-Gaussian in distribution or with either spatial or temporal correlations (red noises), 4DVAR is expected to converge to an incorrect solution (Lu and Browning 1998). That is, the true state is not the minimum of the 4DVAR cost function even in expectation. The PDA approach itself does not impose any significant assumptions on the noise model.

Another important difference is that the 4DVAR approach considers only model trajectories, adjusting the initial condition of each model trajectory only to minimize its cost function in the m-dimensional state space [Eq. (5)]. It starts with a model trajectory and ends with a model trajectory. The PDA approach converges to a model trajectory by minimizing the mismatch cost function in the n × m dimensional sequence space. It starts from a pseudo orbit and, if run to completion, approaches at a model trajectory. In practice, given only finite computational power, only a pseudo orbit is reached and, of course, each component of a pseudo orbit will (with probability 1) define a unique trajectory.

The behavior of the 4DVAR cost function can sometimes vary so strongly with the assimilation window that Berliner (1991) dubbed it a “chaotic likelihood.” The number of local minima¹⁴ in the 4DVAR cost function increases with the length of the data assimilation window (Miller et al. 1994; Pires et al. 1996). The results trapped in the local minima are likely to be inconsistent with the observations. Gauthier (1992), Stensrud and Bao (1992), and Miller et al. (1994) have performed 4DVAR experiments with the Lorenz63 system (Lorenz 1963). They each found that performance of 4DVAR varies significantly depending on the length of the assimilation window, and difficulties arise with the extension of the assimilation window owing to the occurrence of multiple minima in the cost function. Applying the 4DVAR approach, one faces the dilemma between the impacts of local minima with a long assimilation window and the loss of information from the model dynamics given only a short window. The mismatch cost function in PDA avoids this dilemma. Although the cost function itself does not have a unique minimum, all minima of the mismatch cost function are model trajectories. The major limitation of longer assimilation windows in PDA is merely computational cost. And, as a longer assimilation window allows more information from the model dynamics and observations, the quality of the assimilation improves.

To contrast the model trajectory produced in practice by 4DVAR with that generated by PDA, both approaches are applied to the Ikeda map (Ikeda 1979; Haramel et al. 1985) and to the 18-dimensional Lorenz96 system (Lorenz 1995). Details of the systems are given in appendix A. For each system, five different length assimilation windows are tested. For the Ikeda map, assimilation windows with lengths between 4 and 16 steps are considered—for Lorenz96, lengths between 12 and 60 h. In the case of Lorenz96, 6 h indicates 0.05 time unit of the Lorenz96 system; assuming that 1 time unit is equal to 5 days, the doubling time of the Lorenz96 system roughly matches the characteristic time scale of dissipation in the atmosphere [see Lorenz (1995) for details]. PDA uses a GD minimization algorithm; the minimization terminates after 1024 GD iterations for each assimilation. 4DVAR uses a nonlinear conjugate gradient descent algorithm¹⁵ (using the Fletcher–Reeves formula; Shewchuk 1994) to minimize its cost function; the minimization terminates when the derivative of the cost function is small. (Details are given in appendix B.)

The second term of the 4DVAR cost function in Eq. (5) [specifically , which is the distance between the observation-based pseudo orbit and the model trajectory] and the distance between the true states and model trajectory are interpreted as diagnostic tools to evaluate the quality of the model trajectories generated. Results are presented in Tables 1 and 2. When the assimilation window is relatively short, both 4DVAR and PDA tend to generate model trajectories that are closer to the true states than to the observation-based pseudo orbit.¹⁶ This happens both in the Ikeda and in Lorenz96 experiments, where each approach provides effective noise reduction. Note 4DVAR outperforms¹⁷ PDA for very short windows. For the larger window lengths, however, PDA yields consistently the best results. Given the information available from the observations in a longer-window, PDA provides better state estimation than short-window 4DVAR. When the reference trajectory is “closer” to the true state of the system, a better ensemble is expected.

Table 1.

Distance between the observation-based pseudo orbits and the model trajectory generated by 4DVAR and PDA for Ikeda map, and distance between the true states and the model trajectory generated by 4DVAR and PDA for the Ikeda map. The columns show the average distance (average) and the 90% bootstrap resampling bounds (lower and upper). The noise model is N(0, 0.05²). The statistics are calculated based on 8192 assimilations and 4096 bootstrap resamples are used to calculate the resampling bounds.

View Table

Table 2.

Distance between the observation-based pseudo orbits and the model trajectory generated by 4DVAR and PDA for the Lorenz96 system, and distance between the true states and the model trajectory generated by 4DVAR and PDA for the Lorenz96 system. The columns show the average distance (average) and the 90% bootstrap resampling bounds (lower and upper). The noise model is N(0, 0.05²). The statistics are calculated based on 8192 assimilations and 4096 bootstrap resamples are used to calculate the resampling bounds.

View Table

Various “fixes” have been proposed to allow the application of 4DVAR with a long window length avoiding local minima (see Pires et al. 1996 and references therein). Voss et al. (2004) also applied a multiple-shooting approach to address the local minima problem—an initial-value approach to short windows, resembling a similar spinup procedure applied to 4DVAR using multiple short windows. The approach remains expensive and Voss’s examples show varying success. Abarbanel et al. (2009) successfully applied synchronization to smooth the (cost function) surfaces in the space of parameters and initial conditions. Abarbanel’s approach also requires extensive computations and may prove more applicable to parameter estimation. In practice, application of 4DVAR has been restricted to relatively short windows. PDA can exploit information available in longer windows (see also Judd et al. 2004). Within PMS, there is valuable information both in the observations and in the model dynamics in longer windows of observations. When the model is imperfect, this case is less easy to make; focusing on pseudo orbits, however, still holds an additional advantage over 4DVAR: the ability to diagnose model error (this point is discussed in Part II).

5. Ensemble Kalman filter versus PDA

Another well-established approach to state estimation is sequential estimation (Kalman 1960; Anderson and Moore 1979; Kaipio and Somersalo 2005). With sequential approaches, one integrates the model forward until the time that observations are available; the state provided by the model at that time is usually called the first guess. The first guess is then modified using the new observations. Sequential approaches encode all knowledge gleaned from the past in the current state information. Alternatively, when windows over time are considered, an observation inconsistent with the dynamics of any trajectory over that window can be identified as such. Sequential approaches cannot do this. This is not a question of assigning an appropriate prior for the observational noise distribution, but rather one of seeing the dynamical inconsistency of a given observation within a particular region of state space. More generally, in high-dimensional nonlinear dissipative systems, the quest for a general encoding of such information analytically is misguided:¹⁸ a given procedure must demonstrate its superiority in each case. Ensemble Kalman filter approaches (Evensen 1994; Burgers et al. 1998; Houtekamer and Mitchell 1998; Lermusiaux and Robinson 1999; Anderson 2001; Bishop et al. 2001; Hamill et al. 2001) can explore some nonlinearity. There are many different ensemble Kalman filters; the approach used here is the ensemble adjustment Kalman filter (Anderson 2001, 2003). Large ensemble sizes (512 members) have been considered in this case so as to avoid some of the complications required in operational implementations (i.e., ensemble covariance localization). Covariance inflation is adopted to improve the EnKF data assimilation results.¹⁹ For each experiment the inflation parameter value is tuned to optimize the ignorance score. Values of inflation parameter for each experiment are given in appendix B. Even after these adjustments to EnKF, its performance is inferior to PDA.

The comparison is first made in the lower-dimensional case in order to provide easily visualized evidence. Both PDA and the EnKF are applied in the two-dimensional Ikeda map and the resulting ensemble is plotted in the state space (the details of the experiments are given in appendix B). Four examples are shown in Fig. 1; in all panels the ensemble produced by the PDA approach is not only closer to the true state but also reflects the dynamical manifolds as the ensemble members lie near the system attractor. While the EnKF ensemble has its own distinctive structure, the ensemble members do not lie along the system attractor. This is expected in general, inasmuch as the EnKF approach assumes a second-moment closure, the distributions are assumed to be fully described by means and covariances (Anderson and Anderson 1999; Lawson and Hansen 2004; Hamill 2006). This may also lead to filter divergence or even catastrophic filter divergence, as reported by Harlim and Majda (2010). In the top panels of Fig. 1, the EnKF ensemble is distributed about the true state and fairly close to the model’s attractor, while in the bottom panels, the ensemble members are systematically off the attractor and not well distributed about the true state.

View Full Size

Ensemble results from both EnKF and PDA for the Ikeda map. The true state is centered in each panel (large cross), the square is the corresponding observation, and the background dots indicate samples from the Ikeda map attractor. The EnKF ensemble is depicted by 512 magenta dots. PDA ensemble is depicted by 512 green crosses. Each panel is an example of one case of state estimation.

Citation: Journal of the Atmospheric Sciences 71, 2; 10.1175/JAS-D-13-032.1

• Download Figure
• Download figure as PowerPoint slide

To assess the difference between these two approaches quantitatively, the initial-condition ensemble is translated into a predictive distribution function by standard kernel dressing (Brocker and Smith 2008). Each ensemble member is replaced by a Gaussian kernel centered on that member, providing a continuous distribution (a non-Gaussian sum of Gaussian kernels). The width of each kernel (the standard deviation of the Gaussian, called the “kernel width”) is determined by optimizing the ignorance score, introduced below.

The performance of a state-estimation technique can be evaluated with the “log p score” (ignorance score; Good 1952; Roulston and Smith 2002). The ignorance score is the only proper local score for continuous variables (Bernardo 1979; Raftery et al. 2005; Brocker and Smith 2007). Although there are other nonlocal proper scores, the authors prefer using ignorance as it has a clear interpretation in terms of information theory and can be easily communicated in terms of effective interest returns (Good 1952; Roulston and Smith 2002; Hagedorn and Smith 2009), not to mention the lack of any compelling example in favor of the general use of nonlocal scores. The ignorance score is defined by

View Expanded

where Y is the outcome and p(Y) is the probability of event Y. In practice, given K forecast–outcome pairs (p_i, Y_i, i = 1, …, K), the empirical average ignorance skill score is

View Expanded

The PDA approach is compared with the EnKF approach in both the lower-dimensional Ikeda map and higher-dimensional Lorenz96 flow (the details of the systems are given in appendix A). Noise models of the form are considered where each element of η is identically distributed. In both cases the empirical ignorance score is computed using 8192 assimilations. Table 3 shows the comparison between EnKF and PDA using the ignorance score (the optimized kernel width is also presented). To quantify the robustness of the result, 90% bootstrap resampling bounds of the ignorance score are also presented. From the table, it is clear that in both experiments the ensemble generated by PDA significantly outperforms the one generated by EnKF. In the Ikeda experiment, the relative ignorance between the two approaches is found to be around 1.4 bits for the noise model with σ_ns = 0.05 and 1.8 bits for σ_ns = 0.01. This can be interpreted as PDA placing, on average, 160% (and 250%) more probability mass on the outcome than EnKF. In the Lorenz96 experiment, relative ignorance between the two approaches is found to be around 0.8 bits for the noise model with σ_ns = 1 and 0.5 bits for σ_ns = 0.1, which can be interpreted as PDA placing, on average, 75% (and 40%) more probability mass on the outcome than EnKF. The smaller kernel width for the PDA ensemble also indicates that the PDA ensemble members are more concentrated around the true state than the EnKF ensemble. Note that although PDA seems to outperform EnKF, the computational cost for PDA is much greater in order to obtain pseudo orbit with small mismatch errors. (Computational cost is discussed in appendix C.) This cost is substantially reduced outside PMS, as with imperfect models PDA need not approach a trajectory. This issue is discussed in Part II.

Table 3.

Ignorance score and optimized kernel width of initial condition ensemble for the Ikeda map and Lorenz96 system for various noise models. The 512-member ensembles generated by the PDA approach and the EnKF approach are compared. Lower and upper are the 90% bootstrap resampling bounds of the ignorance score, the statistics are calculated based on 8192 assimilations, and 4096 bootstrap resamples are used to calculate the resampling bounds.

View Table

6. Conclusions

A new methodology for state estimation in the perfect model scenario is introduced. This pseudo-orbit data assimilation (PDA) approach aims to identify a reference trajectory about which an ensemble can be assembled.

The well-established 4DVAR approach is contrasted with the PDA approach. Results for the 4DVAR approach based on the two-dimensional Ikeda map and the 18-dimensional Lorenz96 flow are limited by the occurrence of local minima. It has been noted by Miller et al. (1994) and Pires et al. (1996) that 4DVAR suffers from the multiple local minima when applied to long windows. Long windows, on the other hand, allow benefits from having more dynamical information; PDA can exploit these benefits. The 4DVAR approach is expected to fail in practice in cases of chaotic likelihood (Berliner 1991). PDA can solve this problem posed by Berliner (H. Du and L. A. Smith 2014, unpublished manuscript).

Comparisons between the PDA approach and the EnKF approach have been made in the lower-dimensional Ikeda map and a higher-dimensional Lorenz96 model. By looking at the ensembles generated in the state space of the Ikeda map, the structure of the ensemble obtained by the PDA approach seems to be more consistent with the model dynamics (closer to the attractor) than that of the ensemble produced by the EnKF approach. By evaluating initial-condition ensembles using ignorance, in both the Ikeda map and Lorenz96 model experiments, it is demonstrated that the PDA approach systematically outperforms the EnKF approach considered (Anderson 2001, 2003). The failure of EnKF is due in large part to the loss of information beyond the second moment.

One might ask why any approach might be expected to provide better state estimation than the statistically “most likely” state given the observations. In dynamical systems with attractors, the statistically most likely state given only the observations will not lie on the attractor (with probability 1); similarly, the trajectory that generated the observations will not provide the most likely state at t = 0 (with probability 1). By allowing the use of long windows of observations, PDA gains access to more information in the dynamics; this allows more “balanced” states in the sense that relationships between components of the state vector are preserved. This includes relationships that reflect dynamically realized states (speaking loosely, states “on the attractor” and assuming one exists). If the system admits coherent structures, longer trajectories near realized states will reflect more realistic structures and their evolution, as observed by Judd et al. (2008). The key here is reducing the role of statistical distance (which does not respect such structures) and increasing the attention to the geometry of the realized flow (which does); longer windows are an advantage here (Judd et al. 2004, 2008). Within PMS, PDA might be applied to determine initial states for 4DVAR, thereby extending the window length accessible to 4DVAR. Outside the perfect model scenario, PDA finds states more consistent with the model dynamics at the cost of optimizing the statistical fit in the examples considered; this is arguably a generalization of balance to include time and coherent structure (see Part II). The aim is for coherent structures of the system to be reproduced to the extent that the model can reproduce them, and then not to be perverted to improve an inappropriate statistical fit to the observations. A description of PDA outside PMS is presented in Part II. Data assimilation for deterministic nonlinear models will always be a challenging task. PDA provides a step forward by allowing an enhanced balance between extracting information from the dynamic equations and information in the observations.