a. Local vectors and their covariance

We define the local state vector x(ℓℓ) at each model grid point ℓℓ by the model representation of the state within a local volume centered at location (grid point) ℓℓ. We assume that the dimension of x(ℓℓ), which is defined by the product of the number of grid points within the local volume and the number of model variables considered at each grid point, is equal to N. To simplify the notation, in what follows, we drop the argument ℓℓ from the notation of the local state vector, because all equations are valid at any arbitrary location ℓℓ.

We predict the uncertainty in the knowledge of the local state at both the analysis and the different forecast times by an ensemble-based estimate of the local error covariance matrix :

View Expanded

In Eq. (1), K is the number of ensemble members, T is the matrix transpose, and the ensemble perturbations {{x^′′(k): k == 1 …… K}} are defined by the difference

View Expanded

between the ensemble members {{x^(k): k == 1 …… K}}, and the ensemble mean:

View Expanded

b. Diagnostics for the magnitude of the uncertainties

In Satterfield and Szunyogh (2010), we focused on investigating the efficiency of the linear space defined by the range of in capturing the error

View Expanded

in the deterministic prediction, x^e started from the ensemble mean analysis . In Eq. (4), x^t is the model representation of the true state. In the present paper, our goal is to investigate (i) the accuracy of the ensemble prediction of the expected value of the magnitude of the uncertainty and (ii) the accuracy of the ensemble prediction of the spectrum of uncertainties within . To achieve this goal, we apply diagnostics to the

View Expanded

difference between the model representation of the true state and the ensemble mean state estimate, instead of ξξ. The difference δδx^t is often referred to in the literature as the error in the ensemble mean forecast. This terminology is justified when the ensemble mean is used as a deterministic forecast, which is motivated by the fact that the mean of a perfectly designed ensemble would be the most accurate deterministic forecast in the root-mean-square sense (Leith 1974). In our study, however, we consider to be the prediction of the mean of a probability distribution. Since, except for the analysis time, δδx^t is expected to be nonzero even if is a perfect prediction of the mean of the probability distribution, we refer to δδx^t as either the difference between the ensemble mean and the model representations of the true state or the local forecast uncertainty. [The notation δδx^t is consistent with the notation of Satterfield and Szunyogh (2010), where we represented all local state vectors x as the sum of the ensemble mean state estimate and a perturbation δδx.]

The motivation to apply diagnostics to δδx^t, instead of ξξ, is that a verifiable optimality condition between ‖‖δδx^t‖‖, the magnitude of δδx^t, and the ensemble variance,

View Expanded

exists at all forecast times: because V_ℓℓ is a prediction of the total variance TV_ℓℓ == E[(δδx^t)²], where (δδx^t)² == (δδx^t)^T(δδx^t) == ‖‖δδx^t‖‖², its expected value, V == E[V_ℓℓ], should satisfy the equation

View Expanded

where TV == E[TV_ℓℓ]. (Hereafter, E[··] denotes the expected value.) Verifying the relationship defined by Eq. (7), which is often referred to as the spread––skill relationship, is one of the most widely used diagnostics for the validation of an ensemble prediction system. In contrast, all we know about the magnitude ‖‖ξξ‖‖ of ξξ is that it should satisfy E[ξξ²] == V(ξξ²) == ξξ^Tξξ == ‖‖ξξ‖‖²) at analysis time and, under an ergodic hypothesis, E[ξξ²] == 2V once the forecast time is so long that predictability is completely lost (Leith 1974). That is, no verifiable diagnostic relationship exists between ξξ² and V at the intermediate forecast lead times.

We decompose δδx^t as

View Expanded

Here, δδx^t(‖‖) is the component of δδx^t that projects onto and x^t(⊥⊥) represents the component of δδx^t, which projects onto the null space of (orthogonal to the space spanned by the ensemble perturbations). Since the normalized eigenvectors associated with the first K −− 1 eigenvalues of {{u_k: k == 1, …… , K −− 1}}, define an orthonormal basis in , they provide a convenient basis to compute δδx^t(‖‖) by

View Expanded

In addition, the first K −− 1 (nonzero) eigenvalues of satisfy the following equation:

View Expanded

We introduce the notation TVS == E[(δδx^t(‖‖))²] for the variance in . In the optimal case, δδx^t would fully project onto , satisfying δδx^t == δδx^t(‖‖), leading to TVS == TV. But, when part of δδx^t is not captured by the ensemble, x^t(⊥⊥) ≠≠ 0, which leads to TVS < TV.

When the ensemble correctly represents the variance of δδx^t(‖‖), V == TVS. For a given ensemble system, V can be either smaller or larger than TVS. In the former case (V < TVS) the ensemble underestimates the magnitude of the uncertainty that can be explained by , while in the latter case the ensemble overestimates the magnitude that can be explained by . It may even happen that the ensemble satisfies the optimality condition of Eq. (7) by overestimating the true variance in to compensate for the variance lost by not capturing all true error directions. This situation occurs when the ensemble variance is tuned to satisfy Eq. (7) at a given forecast time (e.g., 48 h), but the ensemble cannot fully capture δδx^t. Such a situation can be diagnosed by verifying that V > TVS. Analyzing the results of our numerical experiment we always make a three-way comparison between TV, TVS, and V.

c. Diagnostics for the spectrum of uncertainties

While the explained variance diagnostic of Satterfield and Szunyogh (2010) quantifies the efficiency of the space in capturing the space of uncertainty in the state estimate, the comparison of TV, TVS, and V quantifies the quality of the ensemble in predicting the magnitude of the uncertainty. These diagnostics, however, do not provide information about the performance of the ensemble in distinguishing between the relative importance of the different error patterns within . To introduce a diagnostic that can measure the performance of the ensemble in quantifying the contributions of the different error patterns to the total error within , we first recall that the eigenvalue λλ_k, is the ensemble-based prediction of the variance of the uncertainty in the kth eigendirection.¹ We choose the d ratio:

View Expanded

which was first introduced in Ott et al. (2004), to measure the accuracy of the prediction of the variance in the kth eigendirection. In Eq. (11), is the kth coordinate of δδx^t(‖‖) in the coordinate system {{u_k: k == 1, …… , K −− 1}}. Since the d ratio is defined independently for each eigendirection, it is more appropriate to talk about a spectrum of the d ratio. It can be shown that if the ensemble correctly predicts, in a statistical sense, the uncertainty in the kth direction, the expected value, E[d_k], of d_k at a given time and location is equal to 1:

View Expanded

In Eq. (12) we made use of the fact that λλ_k is the ensemble-based prediction of , thus, for a correct prediction the two quantities must be equal. Since we have a verifiable optimality condition only for the expected value of d_k, we cannot use d_k to measure the performance of the ensemble at a given time and location. Instead, we collect statistical samples of d_k and obtain estimates of the expected value by computing the sample means. A sample mean smaller than 1 indicates that the ensemble tends to overestimate the uncertainty in the kth direction, while a sample mean larger than 1 indicates that the ensemble tends to underestimate the uncertainty in the kth direction.

Finally, we note that the condition E[d_k] == 1 is always satisfied when the random variable,

View Expanded

has an expected value equal to 0, , and a variance equal to 1, , since

View Expanded

A random variable similar to was first introduced for the verification of the ensemble forecast of a scalar variable by Talagrand et al. (1999) and was later named the reduced centered random variable (RCRV) by Candille and Talagrand (2005) and Descamps and Talagrand (2007). The difference between and the RCRV is that while RCRV is for a scalar atmospheric state variable, is for a vector, the local state vector, and a spectrum of scalar ratios is obtained by first projecting the centered local state vector on the principal components of the ensemble-based estimate of the background error covariance matrix and then performing the reduction by the ensemble spread only in that particular eigendirection.

a. Prediction of the magnitude of forecast error

Figure 2 shows the evolution of TV, TVS, and V. In this figure the expected value is estimated by taking the spatial average over all grid points in the NH extratropics (30°°––90°°N) and the temporal average over all forecasts started between 0000 UTC 11 January 2004 and 0000 UTC 15 February 2004.

The time evolution of TV (squares), TVS (triangles), and V (circles) for the NH extratropics. Results are shown for experiments that assimilate (top) randomly distributed simulated observations, (middle) simulated observations at the locations of conventional observations, and (bottom) observations of the real atmosphere. Note the scale is exponential. Also note the different scale in (bottom) panel.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

The time evolution of TV (squares), TVS (triangles), and V (circles) for the NH extratropics. Results are shown for experiments that assimilate (top) randomly distributed simulated observations, (middle) simulated observations at the locations of conventional observations, and (bottom) observations of the real atmosphere. Note the scale is exponential. Also note the different scale in (bottom) panel.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

The time evolution of TV (squares), TVS (triangles), and V (circles) for the NH extratropics. Results are shown for experiments that assimilate (top) randomly distributed simulated observations, (middle) simulated observations at the locations of conventional observations, and (bottom) observations of the real atmosphere. Note the scale is exponential. Also note the different scale in (bottom) panel.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

Interestingly, the difference between TVS and V at longer lead times is much larger than the difference between TV and TVS. In other words, although the linear space spanned by the ensemble perturbations provides a good representation of the space of forecast uncertainties, the ensemble severely underestimates the total variance in . Even though, this underestimation is more serious in the experiment where model errors have an effect on the total error variance TV, the underestimation in the two perfect model experiments is also significant. Thus, the underestimation of the forecast error variance cannot be fully explained by the lack of accounting for the effect of model errors in our ensemble.

To investigate whether the underestimation of TV by V at the different forecast times is primarily due to the underestimation of the analysis error by the analysis ensemble perturbations or to insufficient perturbation growth with time, we introduce the magnitude doubling time T_d to measure the rate of the error and perturbation growth. Here T_d is obtained by first fitting exponential curves x(t) == x(0) exp^at to TV, V and TVS; then, computing T_d by T_d == a⁻⁻¹ ln2. (The smaller T_d, the faster is the growth of the magnitude of the error or the ensemble perturbation.) In the two perfect-model experiments, T_d is very similar for TV and V: in the experiment with simulated observations at random locations, T_d is 25.6 h for TV and 23.1 h for V, while in the experiment with simulated observations at realistic locations, T_d is 27.0 h for both TV and V. These numbers suggest that, in the perfect-model experiments, the underestimation of the typical (average) magnitude of the forecast errors by the ensemble is primarily due to the underestimation of the typical (average) magnitude of the analysis error.

In the experiment that uses observations of the real atmosphere, T_d is 33.2 h for TV and 42.7869 h for V. We attribute the slower growth of V than that of TV to the lack of representation of the effects of model errors in the ensemble, since we did not observe a similar difference in the two perfect-model experiments. Thus, in the experiment with observations of the real atmosphere, the underestimation of the forecast error is due to a combination of the underestimation of the analysis error and the lack of representation of the effects of model errors.

Finally, we note that, interestingly, TVS grows faster than either TV or V in all 3 experiments: T_d for TVS is 19.7 h for the experiment with simulated observations at random locations, 22.4 h for the simulated observations at realistic locations, and 25.8 h for observations of the real atmosphere. In addition, the initial growth rate of TVS and V is faster than exponential growth: up until the 36-h lead time, the time evolution of the growth rate of TVS and V can be best approximated by a second-order polynomial.

Next, we turn our attention to the evolution of the spectral distribution of the forecast errors and the prediction of this property by the ensemble. In particular, Fig. 3 shows the zonal power spectrum of the meridional wind at 500 hPa averaged over all latitudes in the NH extratropics and over time. The left panels show results that were obtained by computing the power spectra for each ensemble perturbation, then taking the ensemble mean of the spectra. The right panels show the spectra for the δδx^t difference between the model representation of the truth and the ensemble mean. The shape of the spectra and the time evolution of the spectra is very similar in all panels, except for the lower-left panel, which shows the ensemble based prediction of the spectrum for the experiment with observations of the real atmosphere. Unlike in the other panels, where a dominant range of wavenumbers 6––10 emerges from a relatively flat initial spectrum, here, at analysis time, the spectrum has a well-pronounced peak at wavenumber 2. A more detailed investigation of this spectrum revealed that this peak is associated with large ensemble variance in a region over the arctic north of Russia.

The zonal power spectrum of the meridional component of the wind averaged overall latitudes in the NH extratropics and over time. Results are shown for 0––120-h forecast lead times at 12-h increments, averaged over the (left) ensemble perturbations and (right) for δδx^t for the experiments that assimilate (top) simulated observations in random locations, (middle) simulated observations in realistic locations, and (bottom) observations of the real atmosphere.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

The zonal power spectrum of the meridional component of the wind averaged overall latitudes in the NH extratropics and over time. Results are shown for 0––120-h forecast lead times at 12-h increments, averaged over the (left) ensemble perturbations and (right) for δδx^t for the experiments that assimilate (top) simulated observations in random locations, (middle) simulated observations in realistic locations, and (bottom) observations of the real atmosphere.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

The zonal power spectrum of the meridional component of the wind averaged overall latitudes in the NH extratropics and over time. Results are shown for 0––120-h forecast lead times at 12-h increments, averaged over the (left) ensemble perturbations and (right) for δδx^t for the experiments that assimilate (top) simulated observations in random locations, (middle) simulated observations in realistic locations, and (bottom) observations of the real atmosphere.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

While the qualitative behavior of the spectra of δδx^t and the ensemble perturbations is similar, as can be expected based the results discussed earlier in this section, the amplitudes are larger for δδx^t than for the ensemble perturbations. This result suggests that the underestimation of the typical magnitude of the analysis error in all three experiments and the lack of representation of the model error in the experiment with observations of the real atmosphere leads to an underestimation of the forecast errors, which is most severe at the synoptic scales, the scales where the error growth is the fastest. This result is in good agreement with that of Tribbia and Baumhefner (2004), who found that the propagation of forecast uncertainties with time in spectral space was toward the synoptic scales, where the growth rate of their magnitude became exponential.

b. Prediction of the spatiotemporal changes in the magnitude of forecast error

The usual candidate for a predictor of ‖‖δδx^t‖‖ at location ℓℓ is the ensemble spread, , at the same location. These two quantities are known to have a positive correlation, which is typically low at analysis time and asymptotes to a level of about 0.5 by about 72-h lead time (e.g., Barker 1991; Houtekamer 1993; Whitaker and Loughe 1998). Kuhl et al. (2007) found that the correlation for our system was in good agreement with those earlier results. Since a correlation of 0.5 for a sample size of N == 129 600 suggests the existence of a linear relationship between and ‖‖δδx^t‖‖, a prediction of ‖‖δδx^t‖‖ based on with a linear regression may seem to be a natural choice. Computing the correlation for our experiments, we find that it is largest for the case of simulated observations at random locations (0.58), slightly lower for the case of realistically placed simulated observations (0.52), and much lower for the realistic case (0.26). This relatively large drop in the correlation for the realistic case would itself provide an argument against using a linear regression to predict ‖‖δδx^t‖‖. An even more problematic feature of the relationship between ‖‖δδx^t‖‖ and , which is illustrated by Fig. 4, is that for larger values of , ‖‖δδx^t‖‖ varies over a much wider range of values. To better understand the problematic aspects of this result, we recall from linear statistics, that univariate regression predicts the value of a random variable y based on a predictor x by the E[y||x] conditional expectation of y given x [e.g., p. 264 in Rao (1973)]. Thus, a prediction of ‖‖δδx^t‖‖ with a prediction of its conditional expectation does not reflect the large potential magnitude of the forecast error for a large value of the spread. This motivates us to investigate the relationship between and the upper bound of ‖‖δδx^t‖‖, instead of the expectation of ‖‖δδx^t‖‖ given .

Linear regression for ensemble skill based on spread. Shown are the NH results for the actual values (gray dots) and predicted values (black line) at the 5-day lead time for experiments that assimilate (top) simulated observations in random locations, (middle) simulated observations in realistic locations, and (bottom) observations of the real atmosphere.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Linear regression for ensemble skill based on spread. Shown are the NH results for the actual values (gray dots) and predicted values (black line) at the 5-day lead time for experiments that assimilate (top) simulated observations in random locations, (middle) simulated observations in realistic locations, and (bottom) observations of the real atmosphere.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

Linear regression for ensemble skill based on spread. Shown are the NH results for the actual values (gray dots) and predicted values (black line) at the 5-day lead time for experiments that assimilate (top) simulated observations in random locations, (middle) simulated observations in realistic locations, and (bottom) observations of the real atmosphere.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

We start our investigation by breaking up the 36-day dataset of 120-h forecasts into 2 sets of 18 days, and search for a quantitative linear relationship between V_ℓℓ and the upper bound of ‖‖δδx^t‖‖² given V_ℓℓ based on the first 18 days (training period). We then use the functional relationship found for this training period to predict the upper bound of ‖‖δδx^t‖‖² based on V_ℓℓ for the second 18-day period. To dampen the effects of outliers, we use the 95th percentile of the bin values of ‖‖δδx^t‖‖² instead of the bin maximum. To obtain a qualitative prognostic relationship, we first order the values of V_ℓℓ for the training period and divide them into 100 bins, each containing an equal number of data points. Each bin provides a pair of data: a value of V_ℓℓ defined by its mean for the bin and the 95th percentile of ‖‖δδx^t‖‖². A similar technique was presented in Majumdar et al. (2001), where binned error variances were compared to ensemble sample variances.

Based on the training dataset, the correlation between the bin mean of V_ℓℓ and the bin 95th percentile of ‖‖δδx^t‖‖² is 0.9889, 0.8176, and 0.9206 at the 120-h lead time for experiments that assimilate simulated random locations, simulated observations in realistic locations, and observations of the real atmosphere, respectively, which are statistically significant at the 99.99%% level by a t test. This suggests that V_ℓℓ may be a good linear predictor of ‖‖δδx^t‖‖², even in the realistic case. Thus, we use the linear regression coefficients obtained for the training dataset to predict the 95th percentile of ‖‖δδx^t‖‖² for the second 18 days. The results are summarized in Fig. 5. The correlation values (0.9518, 0.8685, and 0.8568) between the predicted and the actual 95th percentile values of ‖‖δδx^t‖‖² indicate a linear relationship, which is statistically significant at the 99.99%% confidence level by a t test. Encouraged by the strong linear predictive relationship we find at 120 h, we turn our attention to the shorter lead times (figures for shorter lead times are not shown). At analysis time, we find lower correlation values between the bin mean of V_ℓℓ and the bin 95th percentile of ‖‖δδx^t‖‖² (0.2472, 0.7076, and 0.2182) as well as lower correlation values between the predicted and the actual 95th percentile values of ‖‖δδx^t‖‖² (0.2902, 0.6573, and 0.4161) than at 120-h lead time. As can be expected, the correlation values increase with forecast lead time, by the 48-h lead time both perfect-model experiments show correlation values greater than 0.75 for both the training period and the prediction. For the experiment that assimilates simulated random observations, the lowest values of V_ℓℓ which correspond to the highest values of ‖‖δδx^t‖‖² are found in the polar circle, where, by experiment design, we consider greater number of observations in each local region.² For the case of realistic observations, the correlation values remain relatively low until around the 96-h lead time, where we find correlation values of 0.7259 for the training period and 0.6382 for the prediction of 95th percentile values of ‖‖δδx^t‖‖². We recall that 96 h is about the lead time at which the peak in the power series of the ensemble perturbations moves into the k == 8 range, in better agreement with ‖‖δδx^t‖‖ (Fig. 3).

Mean V and the 95th percentile of TV of data divided equally into 100 bins for the NH extratropics for the first 18 days (triangles). The linear regression curve fitted to these data is shown by a solid straight line. If the prediction of the 95th percentile of TV by the linear statistical model was perfect, the actual values for the second 18 days (open circles) would fall on this line. Shown are the distributions for (left) the analysis and (right) the 120-h forecast lead time for experiments that assimilate (top) randomly distributed simulated observations, (middle) simulated observations at the locations of conventional observations, and (bottom) observations of the real atmosphere. The legends show the correlation between V and 95th percentile of TV in the training dataset (R_training) and between V and the predicted value of the 95th percentile of TV.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Mean V and the 95th percentile of TV of data divided equally into 100 bins for the NH extratropics for the first 18 days (triangles). The linear regression curve fitted to these data is shown by a solid straight line. If the prediction of the 95th percentile of TV by the linear statistical model was perfect, the actual values for the second 18 days (open circles) would fall on this line. Shown are the distributions for (left) the analysis and (right) the 120-h forecast lead time for experiments that assimilate (top) randomly distributed simulated observations, (middle) simulated observations at the locations of conventional observations, and (bottom) observations of the real atmosphere. The legends show the correlation between V and 95th percentile of TV in the training dataset (R_training) and between V and the predicted value of the 95th percentile of TV.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

Mean V and the 95th percentile of TV of data divided equally into 100 bins for the NH extratropics for the first 18 days (triangles). The linear regression curve fitted to these data is shown by a solid straight line. If the prediction of the 95th percentile of TV by the linear statistical model was perfect, the actual values for the second 18 days (open circles) would fall on this line. Shown are the distributions for (left) the analysis and (right) the 120-h forecast lead time for experiments that assimilate (top) randomly distributed simulated observations, (middle) simulated observations at the locations of conventional observations, and (bottom) observations of the real atmosphere. The legends show the correlation between V and 95th percentile of TV in the training dataset (R_training) and between V and the predicted value of the 95th percentile of TV.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

c. Spectrum of the d ratio

We now turn our attention to investigating the efficiency of the ensemble in distinguishing between the importance of the eigendirections (error patterns in physical space) in . We first compute the spectrum of d ratio d_k using the same definition of the local volume as in our calculations of E dimension and explained variance. The results are summarized in Fig. 6.

The time mean of the NH average spectrum of the ratio d_k, calculated for all assimilated variables in the local regions with energy rescaling. Results are shown for (left) the analysis time and the (right) 5-day lead time for experiments that assimilate (top) randomly distributed simulated observations, (middle) simulated observations at the locations of conventional observations, and (bottom) observations of the real atmosphere. The average is taken over all forecasts started between 0000 UTC 11 Jan 2004 and 0000 UTC 15 Feb 2004.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

The time mean of the NH average spectrum of the ratio d_k, calculated for all assimilated variables in the local regions with energy rescaling. Results are shown for (left) the analysis time and the (right) 5-day lead time for experiments that assimilate (top) randomly distributed simulated observations, (middle) simulated observations at the locations of conventional observations, and (bottom) observations of the real atmosphere. The average is taken over all forecasts started between 0000 UTC 11 Jan 2004 and 0000 UTC 15 Feb 2004.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

The time mean of the NH average spectrum of the ratio d_k, calculated for all assimilated variables in the local regions with energy rescaling. Results are shown for (left) the analysis time and the (right) 5-day lead time for experiments that assimilate (top) randomly distributed simulated observations, (middle) simulated observations at the locations of conventional observations, and (bottom) observations of the real atmosphere. The average is taken over all forecasts started between 0000 UTC 11 Jan 2004 and 0000 UTC 15 Feb 2004.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

First, we discuss the results for analysis time (left panels of Fig. 6). We find that for the experiment that assimilates simulated observations in random locations, the ensemble only slightly underestimates the error in the directions it captures (top-left panel). When realistically placed simulated observations are used, the ensemble tends to underestimate uncertainty in all captured directions, with the exception of the few leading directions (middle-left panel). In the experiment with observations of the real atmosphere, the uncertainty is underestimated in all directions captured by the ensemble. The similarity between the shape of the spectra in the two experiments that assimilate observations at realistic locations and the flat spectrum in the third experiments, where observations are nearly uniformly distributed, suggests that the larger underestimation of error variance in the trailing directions is due to the uneven distribution of observations.

As forecast time increases the underestimation of the error by the ensemble becomes increasingly more severe in all directions and in all experiments. We show results for 120-h forecast time (right panels of Fig. 6). In the experiment with randomly distributed observations, the spectrum remains flat (top-right panel), while in the experiment with realistically distributed simulated observations the slope of the spectrum does not increase. In the realistic case, in contrast, the spectrum becomes much steeper indicating an increasingly more severe underestimation toward the trailing directions. Comparing the spectra from the two experiments with realistically distributed observations, we conclude that model errors lead to a more severe underestimation of the forecast uncertainty in the trailing directions.

To obtain d-ratio figures whose meteorological (physical) meaning is easier to interpret, we now change the definition of the local volume: we investigate a single variable at a single level using 5 by 5 horizontal grid points. In these calculations N == 25 (N < K), hence, the upper bound for the E dimension in is 25. The variable and levels we choose for this analysis are the surface pressure, the temperature at 850 hPa, the two horizontal wind components at 500 hPa, and the geopotential height at 250 hPa. Figure 7 shows the time mean of this ratio in the leading direction d₁ for the temperature at 850 hPa. For the experiment that assimilates randomly placed observations, initially d₁ is highest near the Poles. The main regions of enhanced baroclinicity over Japan and off the coast of Newfoundland also show underestimation. The latter result suggests that when the distribution of the observations is nearly uniform, the use of a zonally constant covariance inflation factor in the analysis leads to an underestimation of the uncertainty in the dynamically more active regions. In contrast, for the two experiments that assimilate realistically placed observations, d₁ tends to reflect the local observation density: the uncertainty is underestimated in regions of high observation density, such as Europe, Japan, and the United States, and overestimated in regions of lower observational density, such as the Southern Hemisphere and the oceanic regions. This result is an indication that our zonally constant covariance inflation factor cannot be tuned to be optimal everywhere when there are zonal changes in observation density. Thus, we conjecture that implementing a spatially varying adaptive covariance inflation technique, such as described in Anderson (2009) or a localized version of Li et al. (2009) may lead to an improvement of the analyses and the short-term ensemble forecasts.

The time average of the ratio d_k in the leading direction for the temperature at 850 hPa. Results are shown for (left) the analysis time and (right) the 5-day forecast for experiments that assimilate (top) randomly distributed simulated observations, (middle) simulated observations at the locations of conventional observations, and (bottom) observations of the real atmosphere. The average is taken over all forecasts started between 0000 UTC 11 Jan 2004 and 0000 UTC 15 Feb 2004.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

The time average of the ratio d_k in the leading direction for the temperature at 850 hPa. Results are shown for (left) the analysis time and (right) the 5-day forecast for experiments that assimilate (top) randomly distributed simulated observations, (middle) simulated observations at the locations of conventional observations, and (bottom) observations of the real atmosphere. The average is taken over all forecasts started between 0000 UTC 11 Jan 2004 and 0000 UTC 15 Feb 2004.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

The time average of the ratio d_k in the leading direction for the temperature at 850 hPa. Results are shown for (left) the analysis time and (right) the 5-day forecast for experiments that assimilate (top) randomly distributed simulated observations, (middle) simulated observations at the locations of conventional observations, and (bottom) observations of the real atmosphere. The average is taken over all forecasts started between 0000 UTC 11 Jan 2004 and 0000 UTC 15 Feb 2004.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

The time-averaged spectrum of the d ratio for a particular grid point in a densely observed region (40°°N, 80°°W) in the northeast United States (Fig. 8) at analysis time shows that for the experiment that uses simulated observations in random locations, the ensemble, on average, overestimates the uncertainty in all directions. When simulated observations are placed in the locations of conventional observations, the leading directions of uncertainty are overestimated and the trailing directions are underestimated. For the experiment that uses observations of the real atmosphere, the ensemble underestimates uncertainty in all direction, more severely in the trailing directions. For the same grid point at the 120-h lead time the two experiments that use simulated observations show overestimation in all directions. For observations of the real atmosphere, the ensemble overestimates the uncertainty in the leading directions and underestimates uncertainty in the trailing directions. In contrast, Fig. 9 shows the same plot for a grid point chosen in a sparsely observed region, in the western Pacific Ocean (0°°, 120°°W). For this grid point, we find that the ensemble overestimates uncertainty in all directions for all experiments at analysis time and at the 120-h lead time, with the exception of a few trailing directions for the experiment that uses conventional observations where uncertainty is underestimated.

The time-averaged spectrum of the ratio d_k in a densely observed local region centered at grid point (40°°N, 80°°W) for the temperature at 850 hPa. Results are shown for (left) the analysis time and (right) the 5-day forecast for experiments that assimilate (top) randomly distributed simulated observations, (middle) simulated observations at the locations of conventional observations, and (bottom) conventional observations of the real atmosphere. The average is taken over all forecasts started between 0000 UTC 11 Jan 2004 and 0000 UTC 15 Feb 2004.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

The time-averaged spectrum of the ratio d_k in a densely observed local region centered at grid point (40°°N, 80°°W) for the temperature at 850 hPa. Results are shown for (left) the analysis time and (right) the 5-day forecast for experiments that assimilate (top) randomly distributed simulated observations, (middle) simulated observations at the locations of conventional observations, and (bottom) conventional observations of the real atmosphere. The average is taken over all forecasts started between 0000 UTC 11 Jan 2004 and 0000 UTC 15 Feb 2004.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

The time-averaged spectrum of the ratio d_k in a densely observed local region centered at grid point (40°°N, 80°°W) for the temperature at 850 hPa. Results are shown for (left) the analysis time and (right) the 5-day forecast for experiments that assimilate (top) randomly distributed simulated observations, (middle) simulated observations at the locations of conventional observations, and (bottom) conventional observations of the real atmosphere. The average is taken over all forecasts started between 0000 UTC 11 Jan 2004 and 0000 UTC 15 Feb 2004.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

As in Fig. 8, but in a sparsely observed local region centered at grid point (0°°, 120°°W).

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

As in Fig. 8, but in a sparsely observed local region centered at grid point (0°°, 120°°W).

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

As in Fig. 8, but in a sparsely observed local region centered at grid point (0°°, 120°°W).

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

d. Relationship between E dimension and d ratio

The E dimension, introduced by Patil et al. (2001) and discussed in details in Oczkowski et al. (2005), characterizes the local complexity of dynamics. The E dimension is a spatiotemporally evolving measure of the steepness of the eigenvalue spectrum, λλ₁ ≥≥ λλ₂ …… ≥≥ λλ_r …… ≥≥ λλ_K, having smaller values for a steeper spectrum (Szunyogh et al. 2007). Because Satterfield and Szunyogh (2010) found the E dimension to be a good predictor of the performance of in capturing the forecast error, we hoped to find a similar relationship between the E dimension and the quality of the prediction of the magnitude and the spectrum of uncertainties by the ensemble. While all of our attempts at finding a qualitative relationship between the E dimension and the performance of the ensemble in predicting the magnitude of the uncertainty have failed, we have found interesting differences between the spectra of d ratios for different values of the E dimension.

To explore the relationship between the E dimension and the spectrum of d ratios, we output values of the E dimension and the corresponding values for the spectrum of the d ratio. The data pairs are then ordered by E-dimension values and divided equally into 100 bins. We find that at analysis time the spectrum is better behaved for lower values of the E dimension. For instance, while for the bin with the lowest value of E dimension (Fig. 10, top-left panel) the spectrum is relatively flat and the values are near 1, for the bin with the highest values of E dimension the underestimation of the uncertainty by the ensemble is more severe. Interestingly, at 120-h lead time the spectra are better behaved for the higher values of the E dimension. For instance, for the same two examples compared at analysis time, the underestimation of the forecast, with the exception of a few leading directions, is more severe for the regions of low E dimension. These results indicate that while the spectrum of the d ratio benefits from better representation of the space of uncertainties in the low E-dimensional regions at analysis time, having a more diverse distribution of the uncertainty at a longer forecast lead time improves the representation of the forecast uncertainty. The practical implications of this result, when combined with the results of Satterfield and Szunyogh (2010), is that while a forecaster should trust the 96––120-h ensemble predictions of the possible error patterns in the lower-dimensional regions more, he or she should also keep in mind that the patterns of uncertainty associated with small eigenvalues play a more important role in reality than suggested by the raw ensemble forecast.

The time mean of the NH average spectrum of the ratio d_k, calculated for all assimilated variables in local regions with energy rescaling. Results are shown for observations of the real atmosphere for (top) the minimum bin average of E dimension, (middle) median bin average of E dimension, and (bottom) maximum bin average of E dimension for (left) the analysis time and (right) the 120-h forecast lead time.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

The time mean of the NH average spectrum of the ratio d_k, calculated for all assimilated variables in local regions with energy rescaling. Results are shown for observations of the real atmosphere for (top) the minimum bin average of E dimension, (middle) median bin average of E dimension, and (bottom) maximum bin average of E dimension for (left) the analysis time and (right) the 120-h forecast lead time.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

The time mean of the NH average spectrum of the ratio d_k, calculated for all assimilated variables in local regions with energy rescaling. Results are shown for observations of the real atmosphere for (top) the minimum bin average of E dimension, (middle) median bin average of E dimension, and (bottom) maximum bin average of E dimension for (left) the analysis time and (right) the 120-h forecast lead time.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

e. E dimension, d ratio, and forecast error

To further investigate the problem of underestimating the uncertainty with an ensemble that effectively captures the space of uncertainties, we plot the eigenvalue spectrum (normalized by the leading eigenvalue) and the percentage of TVS:

View Expanded

captured by the first k eigendirections. Figure 11, which is obtained with the same bin-averaging techniques as Fig. 10, shows the results for the minimum, median, and maximum bin values of the E dimension. At analysis time, these three values of the E dimension are 29.3679, 34.1271, and 36.3424 for the simulated observations in random locations, 14.4292, 25.9455, and 35.4192 for the simulated observations in realistic locations, and 21.8493, 31.88, and 36.9536 for observations of the real atmosphere. At the 120-h lead time, the minimum, median, and maximum bin values of the E dimension are 6.344 64, 14.0873, and 24.9761 for simulated observations in random locations; 8.22 311, 17.0315, and 26.0225 for simulated observations in realistic locations; and 10.3602, 17.7928, and 25.7089 for observations of the real atmosphere. We find that, low values of the E dimension show a quicker saturation of the percentage of TVS compared to the eigenvalue spectrum. For example, for the experiment that assimilates simulated observations in random locations, at the 120-h lead time, at the point where the eigenvalue spectrum approaches 0, only approximately 90%% of TVS has been captured by the ensemble. For the experiment that assimilates observations of the real atmosphere, at the 120-h lead time, even for the lowest bin value of the E dimension, we find that all directions captured by the ensemble are necessary to capture 100%% of TVS, but the eigenvalue spectrum saturates around n == 15 (in agreement with Fig. 10, which shows that the ensemble underestimation increases sharply for trailing directions). These results support the use of linear postprocessing techniques to increase the ensemble spread in the trailing directions at longer forecast lead times.

The eigenvalue spectrum (normalized by the leading eigenvalue) and the percentage of TVS for low (red plus signs), median (green open circles), and high (blue triangles) values of the E dimension for the NH. Shown are the results at (left) the analysis time and at (right) the 120-h forecast lead time for the experiments that assimilate (top) simulated observations in random locations, (middle) simulated observations in realistic locations, and (bottom) observation of the real atmosphere.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

The eigenvalue spectrum (normalized by the leading eigenvalue) and the percentage of TVS for low (red plus signs), median (green open circles), and high (blue triangles) values of the E dimension for the NH. Shown are the results at (left) the analysis time and at (right) the 120-h forecast lead time for the experiments that assimilate (top) simulated observations in random locations, (middle) simulated observations in realistic locations, and (bottom) observation of the real atmosphere.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide

The eigenvalue spectrum (normalized by the leading eigenvalue) and the percentage of TVS for low (red plus signs), median (green open circles), and high (blue triangles) values of the E dimension for the NH. Shown are the results at (left) the analysis time and at (right) the 120-h forecast lead time for the experiments that assimilate (top) simulated observations in random locations, (middle) simulated observations in realistic locations, and (bottom) observation of the real atmosphere.

Citation: Monthly Weather Review 139, 4; 10.1175/2010MWR3439.1

• Download Figure
• Download figure as PowerPoint slide