1. Introduction

Ensemble methods for data assimilation are presently undergoing rapid development. The ensemble Kalman filter (EnKF), in various forms, has been successfully applied to a wide range of geophysical systems including atmospheric flows from global to convective scales (Whitaker et al. 2004; Snyder and Zhang 2003), oceanography from global to basin scales (Keppenne et al. 2005), and the land surface (Reichle et al. 2002). Particle filters are another class of ensemble-based assimilation methods of interest in geophysical applications. [See Gordon et al. (1993) or Doucet et al. (2001) for an introduction.]

In their simplest form, particle filters calculate posterior weights for each ensemble member based on the likelihood of the observations given that member. Like the EnKF, particle filters are simple to implement and largely independent of the forecast model, but they have the added attraction that they are, in principle, fully general implementations of Bayes’s rule and applicable to highly non-Gaussian probability distributions. Unlike the EnKF, however, particle filters have so far mostly been applied to low-dimensional systems. This paper examines obstacles to applying particle filters in high-dimensional systems.

Both particle filters and the EnKF are Monte Carlo techniques—they work with samples (i.e., ensembles) rather than directly with the underlying probability density function (pdf). Naively, one would expect such techniques to require ensemble sizes that are large relative to the dimension of the state vector. Experience has shown, however, that this requirement does not hold for the EnKF if localization of the sample covariance matrix is employed (Houtekamer and Mitchell 1998, 2001; Hamill et al. 2001). The feasibility of the EnKF with ensemble sizes much smaller than the state dimension also has theoretical justification. Furrer and Bengtsson (2007) and Bickel and Levina (2008) examine the sample covariance structure for reasonably natural classes of covariance matrices and demonstrate the effectiveness of localizing the sample covariance matrix.

There is much less experience with particle filters in high dimensions. Several studies have presented results from particle filters and smoothers for very low dimensional systems, including that of Lorenz (1963) and the double-well potential (Pham 2001; Kim et al. 2003; Moradkhani et al. 2005; Xiong et al. 2006; Chin et al. 2007). Both van Leeuwen (2003) and Zhou et al. (2006), however, apply the particle filter to higher-dimensional systems. Van Leeuwen (2003) considers a model for the Agulhas Current with dimension of roughly 2 × 10⁵, and Zhou et al. (2006) use a land surface model of dimension 684. We will return to the relation of our results to their studies in the concluding section.

One might expect that particle filters, which in essence attempt to approximate the full pdf of the state, will be substantially more difficult to apply in high dimensions than the EnKF, which only involves approximation of the mean and covariance. The estimation of continuous pdfs is known to suffer from the “curse of dimensionality,” requiring computations that increase exponentially with dimension (Silverman 1986).

We argue here that high-dimensional particle filters face fundamental difficulties. Specifically, we explore the result from Bengtsson et al. (2008) and Bickel et al. (2008) that, unless the ensemble size is exponentially large in a quantity τ ², the particle-filter update suffers from a “collapse” in which with high probability a single member is assigned a posterior weight close to one while all other members have vanishingly small weights. The quantity τ ² is the variance of the observation log likelihood, which depends not only on the state dimension but also on the prior distribution and the number and character of observations. As will be discussed later, τ ² may be considered an effective dimension as it is proportional to the dimension of the state vector in some simple examples.

The tendency for collapse of weights has been remarked on previously in the geophyscial literature (Anderson and Anderson 1999; Bengtsson et al. 2003; van Leeuwen 2003) and is also well known in the particle-filtering literature, where it is often referred to as “degeneracy,” “impoverishment,” or “sample attrition.” Unlike previous studies, however, we emphasize the collapse of weights as a fundamental obstacle to particle filtering in high-dimensional systems, in that very large ensembles are required to avoid collapse even for system dimensions of a few tens or hundreds.¹

Because of the tendency for collapse, particle filters invariably employ some form of resampling or selection step after the updated weights are calculated (e.g., Liu 2001), in order to remove members with very small weights and replenish the ensemble. We do not analyze resampling algorithms in this paper but rather contend that, whatever their efficacy for systems of small dimension and reasonably large ensemble sizes, they are unlikely to overcome the need for exponentially large ensembles as τ ² grows. Resampling proceeds from the approximate posterior distribution computed by the particle filter; it does not improve the quality of that approximate posterior.

The particle filter can also be cast in the framework of importance sampling (see Doucet et al. 2001 for an introduction), which allows one to choose the proposal distribution from which the particles are drawn. All the analysis in this paper assumes that the proposal is the prior distribution, a simple and widely used approach. Although the possibility is yet to be demonstrated, clever choices of the proposal distribution may be able to overcome the need for exponentially large ensemble sizes in high-dimensional systems.

The outline of the paper is as follows. In section 2, we review the basics of particle filters. Section 3 illustrates the difficulty of particle filtering when τ ² is not small through simulations for the simplest possible example: a Gaussian prior and observations of each component of the state with Gaussian errors, both of which have identity covariance. In section 4, we derive (following Bengtsson et al. 2008) an asymptotic condition on the ensemble sizes that yield collapse when both the prior and observation errors are independent and identically distributed in each component of the state vector. Section 5 extends those results to the more general case of Gaussian priors and Gaussian observation errors. Section 6 briefly discusses the effect of a specific heavy-tailed distribution for the observation error.

2. Background on particle filters

Our notation will generally follow that of Ide et al. (1997) except for the dimensions of the state and observation vectors and our use of subscripts to indicate ensemble members.

Let x of dimension N_x be the state of the system represented in some discrete basis, such as the values of all prognostic variables on a regular grid. Since it cannot be determined exactly given imperfect observations, we consider x to be a random variable with pdf p(x).

The subsequent discussion will focus on the update of p(x) given new observations at some time t = t₀. That is, suppose that we have both a prediction p[x(t₀)] and a vector of observations y that depends on x(t₀) and has dimension N_y. {To be more precise, p[x(t₀)] is conditioned on all observations prior to t = t₀. Since all pdfs here pertain to t = t₀ and will be conditioned on all previous observations, in what follows we suppress explicit reference to t₀ and the previous observations.} We wish to estimate p(x|y), the pdf of x given the observations y, which we will term the posterior pdf.

For simplicity, let the observations have a linear relation to the state and be subject to additive random errors ϵ:

View Expanded

More general observation models are of course possible but (1) suffices for all the points we wish to make in this paper.

The particle filter begins with an ensemble of states {xfi, i = 1, . . . , Ne} that is assumed to be drawn from p(x), where the superscript f (for “forecast”) indicates a prior quantity. The ensemble members are also known as particles. The update step makes the approximation of replacing the prior density p(x) by a sum of delta functions, N−1eΣNei=1δ(x − xfi). Applying Bayes’s rule yields

View Expanded

where the posterior weights are given by

View Expanded

In the posterior, each member x^f_i is weighted according to how likely the observations would be if x^f_i were the true state.

If one of the likelihoods p(y|x^f_i) is much larger than the rest, maxw_i will be close to 1 and the particle filter approximates the posterior pdf as a single point mass. The particle-filter estimates of posterior expectations, such as the posterior mean

View Expanded

may then be poor approximations. We will loosely term this situation, in which a single member is given almost all the posterior weight, as a collapse of the particle filter. The goal of our study is to describe the situations in which collapse occurs, both through the rigorous asymptotic results of Bengtsson et al. (2008) for large N_y and N_e and through simulations informed by the asymptotics.

3. Failure of the particle filter in a simple example

We next consider a simple example, in which the prior distribution p(x) is Gaussian with each component of x independent and of unit variance and the observations y are of each component of x individually with independent Gaussian errors of unit variance. More concisely, consider N_y = N_x, 𝗛 = 𝗜, x ∼ N(0, 𝗜), and ϵ ∼ N(0, 𝗜), where the symbol ∼ means “is distributed as” and N(μ, 𝗣) is the Gaussian distribution with mean μ and covariance matrix 𝗣.

Figure 1 shows histograms for maxw_i from simulations of the particle-filter update using N_x = 10, 30, and 100, and N_e = 10³. In the simulations, x, ϵ, and an ensemble {x^f_i, i = 1, . . . , N_e} are drawn from N(0, 𝗜). Weights w_i are then computed from (3). The histograms are based on 10³ realizations for each value of N_x.

The maximum w_i is increasingly likely to be close to 1 as N_x and N_y increase. Large weights appear occasionally in the case N_x = 10, for which maxw_i > 0.5 in just over 6% of the simulations. Once N_x = 100, the average value of maxw_i over the 10³ simulations is greater than 0.8 and maxw_i > 0.5 with probability 0.9. Collapse of the weights occurs frequently for N_x = 100 despite the ensemble size N_e = 10³.

Two comparisons illustrate the detrimental effects of collapse. The correct posterior mean in this Gaussian example is given by x^a = (x^f + y)/2, where the superscript a (for “analysis”) indicates a posterior quantity and the prior mean x^f = 0 in this example. The expected squared error of x^a is E(|x^a − x|²) = [E(|x^f − x|² + E(|y − x|²)]/4 = N_x/2, while that of the observations [E(|y − x|²)] is equal to N_x. The posterior mean estimated by the particle filter,

View Expanded

has squared error of 5.5, 25, and 127 for N_x = 10, 30, and 100, respectively, when averaged over the simulations. Thus, x̂^a has error close to that of x^a only for N_x = 10. For N_x = 100, collapse of the weights is pronounced and x̂^a is a very poor estimator of the posterior mean—it has larger errors than either the prior or the observations.

As might be expected, the effects of collapse are also apparent in the particle-filter estimate of posterior variance, which is given by Σw_i|x^f_i − x̂^a|². The correct posterior variance is given by E(|x − x^a|²) = N_x/2, yet the particle-filter estimates (again averaged over 10³ simulations) are 4.7, 10.5, and 19.5 for N_x = 10, 30, and 100, respectively. Except for N_x = 10, the particle-filter update significantly underestimates the posterior variance, especially when compared with the squared error of x̂^a.

The natural question is how large the ensemble must be in order to avoid the complete failure of the update. This example is tractable enough that the answer may be found by direct simulation: for various N_x, we simulate with N_e = 10 × 2^k and increase k until the average squared error of x̂^a is less than that of the prior or the observations. We emphasize that this merely requires that the particle-filter estimate of the state is no worse than simply relying on the observations or the prior alone (i.e., that the particle filter “does no harm”). The N_e required to reach this minimal threshold is shown as a function of N_x (or N_y) in Fig. 2.

The required N_e appears to increase exponentially in N_x. The limitations this increase places on implementations of the particle filter are profound. For N_x = N_y = 90, somewhat more than 3 × 10⁵ ensemble members are needed. Ensemble sizes for larger systems can be estimated from the best-fit line shown in Fig. 2. Increasing N_x and N_y to 100 increases the necessary ensemble size to just under 10⁶, while N_x = N_y = 200 would require 10¹¹ members.

The exponential dependence on N_e is also apparent in other aspects of the problem. Figure 3 shows the minimum N_e such that maximum w_i (averaged over 400 realizations) is less than a specified value. For each of the values 0.6, 0.7, and 0.8, the required N_e increases approximately exponentially with N_x.

4. Behavior of weights for large N_y

The previous example highlights potential difficulties with the particle-filter update but does not permit more general conclusions. Results of Bengtsson et al. (2008), outlined in this section and the next, provide further guidance on the behavior of the particle-filter weights. Our discussion will be largely heuristic; we refer the reader to Bengtsson et al. for more rigorous and detailed proofs.

a. Approximation of the observation likelihood

Suppose that each component ϵ_j of ϵ is independent and identically distributed (i.i.d.) with density f (). Then for each member x^f_i, the observation likelihood can be written as

View Expanded

where y_j and (𝗛x^f_i)_j are the jth components of y and 𝗛x^f_i, respectively. An elementary consequence of (5) is that, given y, the likelihood depends only on N_y, f () and the prior as reflected in the observed variables 𝗛x. There is no direct dependence on the state dimension N_x.

Defining ψ() = log f (),

View Expanded

where V_ij = −ψ[y_j − (𝗛x^f_i)_j], the negative log likelihood of the jth component of the observation vector given the ith ensemble member. It is convenient to center and scale the argument of the exponent in (6) by defining

View Expanded

where

View Expanded

Then (6) becomes

View Expanded

where S_i has zero mean and unit variance. The simplest situation (as in the example of section 3) is when the random variables V_ij, j = 1, . . . , N_y, are independent given y, so that

View Expanded

Because S_i is a sum of N_y random variables, its distribution will often be close to Gaussian if N_y is large. When V_ij, j = 1, . . . , N_y, are independent given y, the distribution of S_i on any fixed, finite interval approaches the standard Gaussian distribution for large N_y if the Lindeberg condition holds with probability tending to 1 (see Durret 2005, section 2.4a). More generally, the approximate normality of S_i holds for any observation error density f () such that ∫f^1−ϵ(t) dt is finite for some ϵ > 0 and when the V_ij are not i.i.d. but have sufficiently similar distributions and are not too dependent (see Bengtsson et al. 2008). We note in passing that the requirement that the V_ij be not too dependent as N_y increases means that N_x must become large as well and also that the components of the state vector are not strongly dependent. The pdf of the x^f_i must also have a moment-generating function, but is otherwise unconstrained. We will return to the role of N_x in the collapse later.

Equation (8) together with the approximation S_i ∼ N(0, 1) is the basis for the asymptotic conditions for collapse derived in section 4b. They allow statements about the asymptotic behavior of likelihood, and thus of the w_i, for large sample sizes N_e and large numbers of observations N_y, using asymptotic results for large samples from the standard normal distribution.

Showing that the approximation S_i ∼ N(0, 1) is adequate for our purposes is nontrivial, since the behavior in the tails of the distribution is crucial to the derivations but convergence to a Gaussian is also weakest there. The interested reader will find details and proofs in Bengtsson et al. (2008). In fact, the approximation is adequate when the S_is are distributed as noncentral χ² variables with N_y degrees of freedom, which is exactly the case when the observations themselves are Gaussian. As the study of Bengtsson et al. (2008) shows, the adequacy of the Gaussian approximation for the S_i holds if ψ = logf has a moment generating function, for instance if f is Cauchy. In what follows, however, we will assume that S_i ∼ N(0, 1) holds in a fashion which makes succeeding manipulations valid.

b. Heuristic derivation of conditions for collapse

Using (8), the maximum weight w_(Ne) can be expressed as

View Expanded

where S_(i) is the ith-order statistic of the sample {S_i, i = 1, . . . , N_e}.² Defining

View Expanded

we then have

View Expanded

Collapse of the particle-filter weights occurs when T approaches zero.

To obtain asymptotic conditions for collapse, we next derive an expression for E(T) for large N_e and N_y by approximating E[T|S₍₁₎] and then taking an expectation over the distribution of S₍₁₎. For an expectation conditioned on S₍₁₎, the sum in (10) may be replaced by a sum over an unordered ensemble with the condition S_i > S₍₁₎. In that case the expectation of each term in the sum will be identical and

View Expanded

where S̃ is drawn from the same distribution as the S_i but with values restricted to be greater than S₍₁₎.

We now proceed with the calculation under the assumption that S_i ∼ N(0, 1). Then S̃ has the density

View Expanded

where φ() is the density for the standard normal distribution and (x) = ∫^∞_xφ(z) dz. Writing the expectation explicitly with the density of S̃ yields

View Expanded

Next, we replace φ(z) by (2π)^−1/2 exp(−z²/2) in the integrand in (13), complete the square in the exponent, and use the definition of (x) to obtain the following:

View Expanded

The behavior of Gaussian order statistics, such as the minimum of a sample, are well known (David and Nagaraja 2003). An important result is that,³ as N_e → ∞,

View Expanded

Thus, since S₍₁₎ is becoming large and negative, [S₍₁₎] approaches 1 and may be ignored in (14) when calculating the asymptotic behavior of E(T|S₍₁₎).

Now suppose that τ/log N_e → ∞ as N_e → ∞. In this limit, τ + S₍₁₎ ≈ τ(1 − 2 log N_e/τ) → ∞ and so, by the standard approximation to the behavior of for large positive values of its argument:

View Expanded

which may be easily derived with integration by parts (e.g., see section 6.3 of Bender and Orszag 1978).

Substituting (16) in (14), we conclude after some algebra⁴ that

View Expanded

But, reversing the reasoning that led to (16) gives φ[S₍₁₎]/|S₍₁₎| ≈ Φ[S₍₁₎], where Φ(x) = 1 − (x) is the cumulative distribution function (cdf) for the standard Gaussian. Thus,

View Expanded

as N_e → ∞.

Taking the expectation of (18) over S₍₁₎ then gives

View Expanded

To see this, recall that evaluating the cdf of a random variable at the value of the random variable, as in Φ(S_i), yields a random variable with a uniform distribution on [0, 1]. This property underlies the use of rank histograms as diagnostics of ensemble forecasts (Hamill 2001 and references therein) and is known in statistics as the “probability integral transform.” Thus, Φ[S₍₁₎] is distributed as the minimum of a sample of size N_e from a uniform distribution and E{Φ[S₍₁₎]} ≈ 1/N_e. In the next section, we will confirm (19) with direct simulations.

Equation (19) implies that the particle filter will suffer collapse asymptotically if N_e ≪ exp(τ ²/2). More generally, N_e must increase exponentially with τ ² in order to keep E[1/w_(Ne)] fixed as τ increases. This exponential dependence of N_e on τ ² is consistent with the simulation results of section 3, where τ ² ∝ N_y.

In contrast to the most obvious intuition, the asymptotic behavior of w_(Ne) given in (19) does not depend directly on the state dimension N_x. Instead, the situation is more subtle: τ ², a measure of the variability of the observation priors, controls the maximum weight. The dimensionality of the state enters only implicitly, via the approximation that S_i is asymptotically Gaussian, which requires that N_x be asymptotically large. One can then think of τ ² as an equivalent state dimension, in the sense that τ ² is the dimension of the identity-prior, identity-observation example (in section 3) that would have the same collapse properties.

5. The Gaussian–Gaussian case

The analysis in the previous section focused on situations in which the log likelihoods for the observations (considered as random functions of the prior) were mutually independent and identically distributed. In general, however, the observation likelihoods need not be i.i.d., since the state variables are correlated in the prior distribution and observations may depend on multiple state variables. In this section, we consider the case of a Gaussian prior, Gaussian observation errors, and linear 𝗛, where analytic progress is possible even for general prior covariances and general 𝗛.

Let the prior x ∼ N(0, 𝗣) and the observation error ϵ ∼ N(0, 𝗥). We may assume that both x and ϵ have mean zero since, if the observations depend linearly on the state, E(y) = 𝗛E(x) and p(y|x) is unchanged if y is replaced by y − E(y) and x by x − E(x).

For Gaussian observation errors ϵ, the transformation y′ = 𝗥^−1/2y also leaves p(y|x) unchanged but results in cov(ϵ′) = cov(𝗥^−1/2ϵ) = 𝗜. Further simplification comes from diagonalizing cov(𝗥^−1/2𝗛x) via an additional orthogonal transformation in the observation space. Let y″ = 𝗤^Ty′, where 𝗤 is the matrix of eigenvectors of cov(𝗥^−1/2𝗛x) with corresponding eigenvalues λ²_j, j = 1, . . . , N_y; then cov(𝗤^T𝗥^−1/2𝗛x) = diag (λ²₁, . . . , λ²_Ny), while ϵ″ = 𝗤^Tϵ′ still has identity covariance and p(y|x) is again unchanged because 𝗤 is orthogonal. [Anderson (2001) presents a similar transformation that diagonalizes the problem in terms of the state variables, rather than the observation variables.] We therefore assume, without loss of generality, that

View Expanded

and drop primes in the sequel.

a. Analysis of the observation likelihood

With the assumptions in (20), the observation errors are independent, so p(y|x^f_i) can be written in terms of a sum over the log likelihoods V_ij as in (6). In addition, the pdf for each component of the observations is Gaussian with unit variance and, given x^f_i, mean Hx^f_i. Thus,

View Expanded

The additive constant c results from the normalization of the Gaussian density and may be omitted without loss of generality, since it cancels in the calculation of the weights w_i.

We wish to approximate the observation likelihood as in (8). This requires ΣNyj=1Vij to be approximately Gaussian with mean μ and variance τ 2. Leaving aside for the moment the conditions under which the sum is approximately Gaussian, the mean and variance given y of ΣNyj=1Vij can be calculated directly using (20) together with the properties of the standard normal distribution and the fact that the Vij are independent as j varies [as in (7b)]. This yields

View Expanded

and

View Expanded

Equations (21) still depend on the specific realization y of the observations. Proceeding rigorously would require taking the expectation of (19) over y. Here, we simply assume that expectation may be approximated by replacing τ in (19) by its expectation over y. Using the fact that E(y²_j) = λ²_j + 1, we have

View Expanded

and

View Expanded

As discussed in section 3 of Bickel et al. (2008), if λ₁ ≥ λ₂ ≥ . . . , the distribution of S_i = (ΣV_ij − μ)/τ converges to a standard Gaussian as N_y → ∞ if and only if

View Expanded

That is, S_i converges to a Gaussian when no single eigenvalue or set of eigenvalues dominate the sum of squares: (23) implies that max_j(λ²_j)/Σλ²_j → 0 as N_y → ∞. The condition (23) also means that τ ² → ∞, which in turn leads to collapse if log N_e/τ² → 0.

On the other hand, in the case that (23) is not satisfied, the unscaled log likelihood converges to a quantity that does not have a Gaussian distribution. Collapse does not occur since the updated ensemble empirical distribution converges to the true posterior as N_e → ∞, whatever N_y may be.

b. Simulations

First, we check the asymptotic expression for E[1/w_(Ne)] − 1 as a function of N_e and N_y, given in (19), for the Gaussian–Gaussian case. For simplicity, let λ_j = 1, j = 1, . . . , N_y (as in the example of section 2). Then (22b) implies that E(τ ²) = 5N_y/2 and (19) becomes

View Expanded

This approximation is valid when N_e is large enough that the sample minimum follows (15) and N_y is large enough that log(N_e)/N_y is small. To capture the appropriate asymptotic regime, we have performed simulations with N_e = N^α_y, α = 0.75, 0.875, 1.0, 1.25, N_y varying over a dozen values between 600 and 3000, and E(1/w_(Ne)) approximated by averaging over 1000 realizations of the experiment. As can be seen from Fig. 4, 1 − E[1/w_(Ne)] has an approximately linear relation to, log(N_e)/N_y, though considerable scatter is present. The best-fit line to the simulation results has a slope of 0.96 with a 95% confidence interval of ±0.087, which captures the predicted slope of 4/5 ≈ 0.89.

Equation (19) also implies that asymptotic collapse of the particle filter depends only on τ rather than the specific sequence {λ_j, j = 1, . . . , N_y}. To illustrate that τ does control collapse, we consider various λ sequences by setting λ²_j = cj^−θ. In this case, the simulations fix N_y = 4 × 10³ and N_e = 10⁵ while θ takes the values 0.3, 0.5, and 0.7 and c is varied such that substituting (22b) in (19) gives 0.01 < E[1/w_(Ne)] − 1 < 0.075. These values are again chosen to capture the appropriate asymptotic regime where the normalized log likelihood S_i is approximately Gaussian. The expectation E[1/w_(Ne)] is approximated by averaging over 400 realizations of the experiment.

Figure 5 shows results as a function of 2 logN_e/τ. As predicted by (19), E[1/w_(Ne)] depends mainly on τ rather than on the specific λ sequence. The simulations thus confirm the validity of (19) and, in particular, the control of the maximum weight by τ. Nevertheless, some limited scatter around the theoretical prediction remains, which arises from weak dependence of E[1/w_(Ne)] on the λ sequence for finite τ. We defer to a subsequent study a more detailed examination of the behavior of the maximum weight for finite τ and N_e and the limits of validity of (19).

6. Multivariate Cauchy observation-error distribution

Van Leeuwen (2003) proposes the use of a multivariate Cauchy distribution for the observation error to avoid collapse and gives some numerical results supporting his claim. In Bengtsson et al. (2008), analytical arguments as well as simulations indicate that collapse still occurs but more slowly with such an observation-error distribution. Specifically, they show that, in the limit log(N_e)/N_y → 0, E(T) approaches zero at a rate given by log(N_e)/N_y log|log(N_e)/N_y|. This condition emerges from the analysis of the log likelihood of the mulivariate Cauchy in the same way as (24) emerges from analysis of the Gaussian–Gaussian case. The condition for E(T) → 0 and collapse is then identical to that in the Gaussian–Gaussian case, namely, log(N_e)/N_y → 0, but the rate is distinctly slower than those implied by (19) or (24).

Intuitively, what happens is that if ϵ has a multivariate Cauchy distribution, then ϵ can be written as

View Expanded

where z₁, . . . , z_Ny are i.i.d. N(0, 1). For given z_Ny+1 close to 0, the errors have very long Gaussian tails. This makes collapse harder because the true posterior resembles the prior, implying that the observations have relatively little information.

7. Conclusions

Particle filters have a well-known tendency for the particle weights to collapse, with one member receiving a posterior weight close to unity. We have illustrated this tendency through simulations of the particle-filter update for the simplest example, in which the priors for each of N_x state variables are i.i.d. and Gaussian, and the observations are of each state variable with independent, Gaussian errors. In this case, avoiding collapse and its detrimental effects can require very large ensemble sizes even for moderate N_x. The simulations indicate that the ensemble size N_e must increase exponentially with N_x in order for the posterior mean from the particle filter to have an expected error smaller than either the prior or the observations. For N_x = 100, the posterior mean will typically be worse than either the prior or the observations unless N_e > 10⁶.

Asymptotic analysis, following Bengtsson et al. (2008) and Bickel et al. (2008), provides precise conditions for collapse either in the case of i.i.d. observation likelihoods or when both the prior and the observation errors are Gaussian (but with general covariances) and the observation operator is linear. The asymptotic result holds when N_e is large and τ ², the variance of the observation log likelihood defined in (7b), becomes large and has an approximately Gaussian distribution. Then, in the limit that τ⁻¹N_e → 0, the maximum weight w_(Ne) satisfies E[1/w_(Ne)] ≈ 1 + τ⁻¹2 logN_e. The maximum weight therefore approaches 1 (and collapse occurs) as τ increases unless the ensemble size N_e grows exponentially with τ.

In the case that both the prior and observation errors are Gaussian, τ ² can be written as a sum over the eigenvalues of the observation-space-prior covariance matrix. The theory then predicts that collapse does not depend on the eigenstructure of the prior covariances, except as that influences τ. Simulations in section 5 confirm this result.

It is thus not the state dimension per se that matters for collapse, but rather τ, which depends on both the variability of the prior and the characteristics of the observations. Still, one may think of τ ² as an effective dimension, as it gives the dimension of the identity-prior, identity-observation Gaussian system (as in section 3) that would have the same collapse properties. This analogy is only useful, however, when the normalized observation log likelihood S_i defined in (7a) has an approximately Gaussian distribution, which requires that N_x not be too small.

Our results point to a fundamental obstacle to the application of particle filters in high-dimensional systems. The standard particle filter, which uses the prior as a proposal distribution together with some form of resampling, will clearly require exponentially increasing ensemble sizes as the state dimension increases and thus will be impractical for many geophysical applications. Nevertheless, some limitations of this study will need to be addressed before the potential of particle filtering in high dimensions is completely clear.

First, the simulations and asymptotic theory presented here have not dealt with the most general situation, namely, when the prior and observations are non-Gaussian and have nontrivial dependencies among their components. There is no obvious reason to expect that the general case should have less stringent requirements on N_e and we speculate that the Gaussian–Gaussian results of section 5 will still be informative even for non-Gaussian systems. Some support for this claim comes from the results of Nakano et al. (2007), who apply the particle filter with a variety of ensemble sizes to the fully nonlinear, 40-variable model of Lorenz (1996). Consistent with Fig. 2, they find that an ensemble size between 500 and 1000 is necessary for the posterior from the particle filter to have smaller rms errors than the observations themselves.

Second, the asymptotic theory pertains to the behavior of the maximum weight, but says nothing about how the tendency for collapse might degrade the quality of the particle-filter update. Indeed, the update may be poor long before the maximum weight approaches unity, as illustrated by Figs. 2 and 3. What is needed is practical guidance on ensemble size for a given problem with finite N_x, N_y, and τ. Though rigorous asymptotic analysis will be difficult, we anticipate that simulations may provide useful empirical rules to guide the choice of ensemble size.

Third, we have not addressed the possible effects of sequentially cycling the particle filter given observations at multiple instants in time. Overall, cycling must increase the tendency for collapse of the particle filter. The quantitative effect, however, will depend on the resampling strategy, which again makes analytic progress unlikely.

Fourth, we have not considered proposal distributions other than the prior nor have we considered resampling algorithms, which are frequently employed to counteract the particle filter’s tendency for collapse of the ensemble. We emphasize that resampling strategies that do not alter the update step are unlikely to overcome the need for very large N_e, since they do not improve the estimate of the posterior distribution, but merely avoid carrying members with very small weights further in the algorithm. It is conceivable that the required N_e might be reduced by splitting a large set of observations valid at a single time into several batches, and then assimilating the batches serially with resampling after each update step. Alternatively, one might identify states in the past that will evolve under the system dynamics to become consistent with present observations, thereby reducing the need for large ensembles of present states when updating given present observations. Gordon et al. (1993) term this process “editing,” and a similar idea is employed by Pitt and Shephard (1999). Such a scheme, however, would likely demand very large ensembles of past states.

As noted in the introduction, both van Leeuwen (2003) and Zhou et al. (2006) have applied particle filters to systems of dimension significantly larger than 100. In Zhou et al., however, each update is based on only a single observation (and only 28 observations total are assimilated); assuming that the prior uncertainty is comparable to the observation variance, τ ² < 28 in their case and their ensemble sizes of O(1000) would be adequate based on Fig. 3. Based on the characteristics of the sea surface height observations assimilated by van Leeuwen, we estimate that the particle-filter update uses O(100) observations at each (daily) analysis. Allowing for the possibility that nearby observations are significantly correlated owing to the relatively large scales emphasized by sea surface height, then van Leeuwen’s use of 500–1000 ensemble members would seem to be at the edge of where our results would indicate collapse to occur. Consistent with this, van Leeuwen notes a strong tendency for collapse.

Fundamentally, the particle filter suffers collapse in high-dimensional problems because the prior and posterior distributions are nearly mutually singular, so that any sample from the prior distribution has exceptionally small probability under the posterior distribution. For example, in the Gaussian i.i.d. case, the prior and posterior distributions have almost all their mass confined to the neighborhood of hyperspheres with different radii and different centers. The mutual singularity of different pdfs becomes generic in high dimensions and is one manifestation of the curse of dimensionality.

Another way of looking at the cause of collapse is that the weights of different members for any chosen state variable are influenced by all observations, even if those observations are nearly independent of the particular state variable. The particle filter thus inherently overestimates the information available in the observations and underestimates the uncertainty of the posterior distribution. Similar problems occur for the EnKF and, for spatially distributed systems with finite correlation lengths (e.g., most geophysical systems), can be reduced by explicitly restricting any observation’s influence to some spatially local neighborhood. This motivates the development of nonlinear, non-Gaussian ensemble assimilation schemes that perform spatially local updates, as in Bengtsson et al. (2003) or Harlim and Hunt (2007).