1. Introduction

At present, ensemble forecasting is an important product delivered by a number of operational meteorological centers around the globe. By perturbing the initial state of a high-resolution deterministic forecast, a number of ensemble members is integrated, usually at a lower resolution. From the spread in such ensembles, an estimate of the quality of the high-resolution deterministic forecast is made, and in the medium range, clusters of members are a guide to the likelihood of alternative weather scenarios. Examples of meteorological centers that produce ensemble systems on an operational basis are the National Centers for Environmental Prediction (NCEP; Toth and Kalnay 1993; Tracton and Kalnay 1993), the Canadian Meteorological Centre (CMC; Houtekamer et al. 1996), and the European Centre for Medium-Range Weather Forecasts (ECMWF; Molteni et al. 1996; Buizza et al. 1998).

An important aspect of a forecasting system in general, and for an ensemble system in particular, is the assessment of performance. Because of its probabilistic nature, verification for ensemble systems is more complex than it is for deterministic forecasts. Popular tools that measure the performance of probabilistic forecasts are rank histograms, reliability diagrams, Brier scores, receiver operating characteristic (ROC) curves, and cost–loss analyses [a detailed description may be found in Strauss and Lanzinger (1996), Wilks (1995), and Richardson (2000)]. For all of these tools, ensemble forecasts are compared to the verifying weather pattern. This truth is measured either directly by using observations, or by using verifying analysis fields. The errors in these quantities are usually neglected. The argument is that they are small compared to the errors in the forecasts and therefore will have a negligible effect on the results. However, in the short range, where forecast errors are still small, this argument may not be justified. For instance, at ECMWF, the accuracy of the 1-day forecast in the geopotential height at 500 hPa is comparable to the observation error of radiosondes (Simmons and Hollingsworth 2002).

In this paper, the impact of observation errors at verification time on verification tools is studied. The focus will be on rank histograms and reliability diagrams, that is, tools that measure the statistical aspects of ensemble forecasts. One way of doing this is to transform the probability density function (PDF) for the truth to a PDF for the verifying observations or analysis. This means that the PDF created by an ensemble system is to be convolved with the verifying observation error before it is presented to the verification tool under consideration. In practice, this can be accomplished by adding normal noise to the ensemble members, with a standard deviation given by these observation errors. The same approach has been suggested by Anderson (1996), who also points out the importance of taking into account observation errors when validating ensemble forecasts. A theoretical basis for this method is presented in section 2. In addition, a general expression is derived for the frequency of outliers (in cases when the verifying observation is found to be outside the ensemble). Special attention is given to the influence of nonnormality of the underlying distributions.

The fact that models may have systematic errors or biases can both obscure, or make it difficult to interpret, the results of verification. As a method for testing purely the impact of the sensitivity of observation errors on verification tools, a perfect model approach may be used. In such a study, the initial state of one of the ensemble members is defined as the truth. Then, because of the perfect model approach, so will its trajectory. Observations may be created by perturbing the truth according to their errors. This approach is followed in section 3 by using the three-variable Lorenz-63 model (Lorenz 1963) as a simple version of a nonlinear dynamical system.

In section 4 the effect of the incorporation of verifying observation errors is considered for the ensemble prediction system (EPS) that is operational at ECMWF. Both the verification on the basis of observations (ocean wave heights are taken as an example) and on the basis of verifying analyses (geopotential at 500 hPa) are considered. It is shown that in the short range rank histograms are quite sensitive to observation errors and that there is even a nonnegligible effect in the medium range. The effect on reliability diagrams is found to be limited.

The paper ends with some concluding remarks.

2. Analytical considerations

a. General

In the case of a perfect ensemble, each ensemble member is an independent sample from the PDF of which the truth is also a realization. Therefore, for a parameter x, the collection of the N ensemble members x₁, … , x_N plus the verifying truth x_T, forms a set of N + 1 quantities with identical statistical properties. The probablity that one given quantity satisfies a certain condition (e.g., being the smallest, second smallest, largest) must be the same for all, that is, 1/(N + 1). In particular, the probability that the ith smallest quantity is the verifying truth, that is, that the verifying truth is in between the ordered ensemble members i − 1 and i, is 1/(N + 1). For large samples, this will give rise to a flat rank histogram (see, e.g., Hamill 2001; Talagrand and Vautard 1997). This is a general result that is true for any PDF for the truth (normal or nonnormal, skewed or nonskewed).

For outliers, this can also be proved in a general analytical way. Let the PDF of the ensemble members and the verifying observation be denoted by ρ_E and ρ_O, respectively. Furthermore, in general let the standard deviation of a PDF ρ_α be denoted by σ_α and its cumulative density function by P_α(x) = ∫^x_−∞ ρ_α(y)dy. The condition that the verifying observation is an upper outlier can be expressed as

View Expanded

where H is the Heavyside function, being 0 for x < 0 and 1 for x ≥ 0. Its expectation value, denoted by FR_u can be found by weighting Eq. (1) over the joint PDF for (x_O, x₁, … , x_N):

View Expanded

where it is assumed that ensemble members and the verifying observation are mutually independent. Similarly, the frequency of lowest outliers FR_l is given by Eq. (2), if P is replaced by Q = 1 − P. In the special case that ρ_O = ρ_E, for example, when both observation and ensemble members are realizations from the PDF for the truth ρ_T, this integral can be evaluated exactly:

View Expanded

The same applies for FR_l. When observation errors are to be taken into account (though assumed to be uncorrelated with the PDF for the truth), ρ_O will be the convolution between ρ_T and the PDF for the observation error ρ_o:

View Expanded

This will have an effect on the frequency of outliers, even in the situation of a perfect ensemble, that is, ρ_E = ρ_T. This effect can be neutralized by artificially perturbing the ensemble members with respect to a PDF ρ_e with the same standard deviation as the observation error (σ_e = σ_o). In this case ρ_E → ρ_eρ_T, which will then be equal to ρ_O again, and the frequency of outliers will be restored to 1/(N + 1). A deviation from this frequency can then be attributed to incorrect statistical properties of the ensemble system.

c. Small deviations from the perfect situation

The distributions for ensemble members and the verifying observation are both (imperfect) representations of the PDF of the truth ρ_T. If the observation error does not depend on the value of the verifying truth, ρ_O is given by Eq. (4). If, in addition, ensemble members are (by hand) perturbed according to a PDF ρ_e, one has ρ_E → ρ_eρ_E. In case all distributions are normal, it follows that

View Expanded

where F_N is the function displayed in Figs. 1a and 1b.

For the general nonnormal case, such a simple expression cannot be obtained. However, if deviations of ρ_E and ρ_O from ρ_T are small, the following expansion can be made:

View Expanded

where

View Expanded

and ρ_t(x) = ρ(x/σ_T)/σ_T. Here it is assumed that the deviation of ρ_E from ρ_T is represented by a misfit in spread only, that is, the shape of both distributions is the same. The corresponding term in Eq. (6) then follows from a variable change x → (σ_T/σ_E)x and the Taylor expansion of (1 + δ)ρ[(1 + δ)x] in δ. The term that accounts for observation error can be found by inserting expression (4) into Eq. (2) and expanding with respect to ε. The term that reflects the noise added to ensemble members is found by realizing that for σ_e = σ_o and σ_E = σ_T it must counterbalance the effect of observation error exactly. It should be stressed that Eq. (6) is only valid if the individual contributions are small and of similar magnitude. Otherwise higher-order terms including interactions between the various error sources must be included.

Note that FR_u depends to lowest order linearly on a misfit in ensemble size and quadratically on σ_o and σ_e. The sensitivities α_N and β_N do not depend on the details of the observation error or the noise applied to the ensemble members. They are determined by the PDF of the underlying truth. If ρ_T is normal, α_N and β_N are equal. Numerical results are displayed by the solid line in Fig. 1c. It clearly confirms the growing sensitivity toward larger ensemble size as was observed in Figs. 1a and 1b. This property can be understood by realizing that for large N, P^N becomes a sharp transition function. In the extreme,

NρxP^Nxδxx_N

with

View Expanded

since the width of the left-hand side of Eq. (9) shrinks to 0 while its integral remains equal to 1. The value of x_N, and therefore the result of Eqs. (2), (7), and (8), depends on the details of the tail of the PDF for the truth. For a normal distribution, one has

View Expanded

which is an increasing function of N. For N = 50, x_N ≈ 2.2σ_T; for N = 500, x_N ≈ 3.0σ_T.

d. The effect of a nonnormal tail

Although for many parameters a normal distribution may give a fair representation of ρ_T within a few standard deviations of the mean, its validity in the tail may be disputable. As an illustration of the dependency of the frequency of outliers on the details of the tail of ρ_T, the Laplace distribution will be considered here:

ρ_Tx2x2(11)

Although the sharp peak at x = 0 is in practice not realistic, its tail, going much more slowly to 0 than a normal distribution, might be. The sensitivity parameters α_N and β_N can be evaluated exactly for this case:

View Expanded

where γ = 0.57721⋯ is Euler's constant, and the approximation for α_N is within 0.1% for N > 5. For the Laplace distribution, α_N and β_N differ. Their dependency on N is illustrated in Fig. 1c. It shows that β_N quickly saturates. Therefore, in contrast to the normal case [see Eq. (6)], the influence of observation errors on FR_u does not grow for N > 10. The sensitivity of FR_u to incorrect ensemble spread does continue to grow with N, although less rapidly than for the normal case. As a consequence, for large N, misfits in ensemble spread will have a larger effect on FR_u than unaccounted observation errors have.

e. Averages over large samples

The results obtained so far applied to expectation values for a single case, with one PDF for the truth. In practice, scores are evaluated over large samples, covering cases with high predictability (sharp ρ_T) and low predictability (broad ρ_T). This means that expressions for the frequency of outliers, such as Eq. (6), are to be weighted over a PDF ρ(ρ_T) expressing the temporal variability in predictability. If one assumes that for most cases the shape of ρ_T is similar, this weighting can be transformed to a weighting over σ_T. If an ensemble system obeys a good spread–skill relation, but consistently over- or underpredicts spread, it may be argued that the ratio σ_E/σ_T is constant. This means that σ_E/σ_T in the term for α in Eq. (6) can be replaced by its average values. This is not true for the term for β, since the constant observation error (and similarly for σ_e) leads to case-dependent ratios of σ_o/σ_T. Although for this term σ_T, σ_o and σ_e can be replaced by their average values, it will alter the value of β:

View Expanded

Here 〈 〉 indicates weighting over ρ(σ_T).

Methods to measure the temporal variability in predictability were explored by Kruizinga and Kok (1988) and Houtekamer (1992). They assume that σ_T ∼ exp(−γd), where d is drawn from a standard normal distribution. This leads to 〈β〉 = β exp(3γ²). The parameter γ expresses the degree of variability. For the geopotential at 500-hPa, Houtekamer (1992) obtains realistic values of γ between 0.3 and 0.4, which would lead to an enhancement of β by 30% to 60%.

3. Idealized experiments with the Lorenz model

In the previous section attention was focused on the frequency of outliers. The effect of observation errors on other parts of the rank histogram, and also on reliability diagrams, is considered in this section. For this, a synthetic dataset under idealized conditions, where initial spread, uncertainties, and observation errors could all be manipulated, was produced using the nonlinear dynamical system originally suggested by Lorenz (1963).

The dependent variables X, Y, and Z of the Lorenz model are determined by the equations

View Expanded

where σ, r, and b are constants. To solve these equations numerically the double approximation procedure suggested by Lorenz was used. The constants were taken to be σ = 10, r = 28, and b = 8/3, and the dimensionless time step Δt = 0.001. In order to generate a dataset that could later be used as initial conditions, the model was run for a period of 100 000 time steps. The result is shown as the gray lines in Fig. 2 and yields the well-known Lorenz attractor.

When generating the dataset used in the statistical analysis, randomly chosen points from the Lorenz attractor were used as the true state of the system. By assuming that the observations were subject to errors, the analysis was found by adding normally distributed noise to the true initial states. The standard deviation used for the analysis is denoted σ_a. This analysis was used to initialize the control forecasts. For the ensemble members, the initial conditions were calculated by adding normally distributed noise, with standard deviation σ_ε, to the analysis. The numerical model was then used to propagate the true state, analysis, and ensemble members forward in time. In this way the initial PDFs for the true states and ensembles evolve to PDFs for the verifying truth and ensemble members with (flow dependent) standard deviations σ_T and σ_E, respectively. To make this system resemble the EPS currently running operationally at ECMWF, 50 ensemble members in addition to the control forecast were used. In each case, the forecast range was 2000 time steps. When choosing this forecast range, we did a number of initial tests to make sure the forward integration was long enough to produce decent spread of the ensemble members and at the same time short enough to retain predictability. Two thousand time steps was found to be a reasonable compromise. An example of the trajectory of one true state is shown as the dotted line in Fig. 2. If σ_ε < σ_a, the ensemble spread will be too low, and vice versa. A system with perfect spread is obtained if σ_ε = σ_a.

Verification of operational systems are usually performed with respect to one-dimensional quantities such as 2-m temperature. Here we will use the absolute distance from the origin, normalized by the sample mean value as a measure of the system state:

View Expanded

where E ≈ 26.44 is the approximate average Euclidian distance from the origin in phase space. The reason for our choice of metric is that, in practice, most parameters verified in weather forecasting do not have a one-to-one mapping between phase space and value and that the absolute distance from origin in phase space is a single parameter that takes into account all three state variables. In principle, however, any of the state variables could have been used as a metric and would probably have yielded the same results. In a similar way as for the initial states, observation errors were simulated by adding normally distributed noise, with standard deviation σ_o, to true value at the final states. Figure 2 shows an example of an ensemble forecast using this model together with a close-up of the final state for all the ensemble members together with the true final state.

To use this system to test the effects of observation errors, 12 different experiments were defined. These experiments can be sorted into three groups: experiments with perfect spread, too low initial spread, and too large initial spread. For the cases with too low initial spread, the standard deviation for the ensemble members was σ_ε = 1/4σ_a. In the experiments with too large spread, the standard deviation for the ensemble members was σ_ε = 4σ_a. A summary of all the values used to determine the spread together with the standard deviation used for measuring the final state for each experiment is given in Table 1. For each experiment, 60 000 observations and forecasts were generated.

As was discussed in section 2 (or see, e.g., Hamill 2001), the ensemble members and the verifying observations should ideally be random draws from the same probability distribution. This means that the ranks of an observation when mixed with the ensemble members should be evenly distributed. The frequencies of such ranks (in this application ideally p = 1/52 ≈ 0.019) can be illustrated graphically in a rank histogram (Anderson 1996; Talagrand and Vautard 1997; Hamill 2001). The result from the experiment where σ_ε = σ_a and σ_o = 0 is shown in the rank histogram in Fig. 3 (experiment 1). As expected for this case, the rank histogram is almost flat.

To test the ability of the ensemble to correctly forecast probabilities of a certain event, reliability diagrams may be used (Strauss and Lanzinger 1996; Wilks 1995). Here, we will consider the event that the distance between the origin in the phase space and the final state of the system is larger than average; that is, that E > 1, where E is given by Eq. (16). The forecast probabilities are split into discrete bins ranging from 0 to 1. For each probability class, the proportion of times the event is observed, compared to the total number of ensemble forecasts in that class, called the observed frequency, is plotted against the corresponding probability class. For a perfectly reliable forecasting system, these points lie on the diagonal line. The reliability diagram for the case with perfect spread and no observation errors is shown in Fig. 4 (experiment 1).

In experiment 2 the ensemble forecasts and observations are pure random draws from a normal distribution. To obtain this, both ensembles and observations were given the same value before normally distributed noise with standard deviation of 0.25 was added. Although the rank histogram is almost flat, the reliability diagram in Fig. 4 (experiment 2) reveals no forecast skill at all.

In the first experiment, perfect ensemble forecasts were compared with observations that were taken to be the true state of the system. To simulate the effects of observation errors, normally distributed noise can be added to this final state. The effect of this on the rank histogram is shown in Fig. 3, experiment 3, where σ_o = 0.02. As can be seen, this has a rather dramatic effect on the frequencies in the two extreme ranks and can easily be misinterpreted as too low spread in the ensemble forecasts. However, in this case the ensemble spread is perfect, and the strongly u-shaped rank histogram is solely caused by observation errors. The corresponding reliability diagram seems to be much less sensitive in this respect (Fig. 4, experiment 3). The frequency of outliers (see Fig. 3) is around 40%, which for a normal distribution would imply [see Eq. (5) and Fig. 1a] σ_T = 0.082, while 〈σ_T〉 = 0.25 was observed. The large difference is attributed to a large temporal variability of predictability (see section 2e).

As argued in the introduction, the proper verification is obtained by adding noise to the ensemble members as well. The rank histogram method does not distinguish between large and small differences in the parameter to be ranked. It is of course meaningless to rank the data according to differences that are much smaller than the observation errors. Hence, this effect must be filtered out before calculating the frequencies of observations in the different ranks. In experiment 4, this has been done by adding the same amount of noise to the ensemble members as was used for the observations. The effect on rank histograms is striking (see Fig. 3): the u shape for experiment 3 has been removed completely, leaving a flat histogram, as it should. Again, the effect on the reliability diagram (Fig. 4) is much less apparent.

In Fig. 3, experiment 5, the effect of too small initial spread on the rank histogram is demonstrated. Here, the true values are used for the observations. In this case the rank histogram is also strongly u shaped and looks very similar to the result obtained in experiment 3, although in that case, this was caused by observation errors. This shows that it is not possible to distinguish between the effects of observation errors or too low ensemble spread using the rank histogram. The frequency of outliers (2 × 28%) agrees well with FR_u given by Eq. (5) with σ_E = σ_T/4.

When noise was added to the evolved ensemble member for this last case, the observed frequencies in the two extreme ranks were reduced (Fig. 3, experiment 6). However, in contrast to the case with true measurement errors, a bell shape also appeared in the center of the histogram. The combined effect of too small ensemble spread and observation errors can be seen in Fig. 3, experiment 7. In this case, the u shape of the rank histogram is even more pronounced. Experiment 8 shows the result for this last case when observation errors are filtered by adding noise to the ensemble members. Note that the frequency of outliers is considerably lower than for experiment 5. Neutralizing the observation error in this case does, in contrast to approximation (6), have a residual effect on outliers. The reason for this is that σ_E and σ_T differ too much for Eq. (6) to be valid. In the framework of normal distributions, the reduction of outliers is clear, since the addition of observation error and noise to ensemble members makes the argument of F_N in Eq. (5) deviate less from unity.

From Fig. 4 (experiments 5 to 8), it is clear that too low ensemble spread has a stronger impact on reliability diagrams than observation errors have. For the two cases without noise added to the ensemble members, the reliability curves are s shaped, underforecasting low probabilities and overforecasting high probabilities. When noise is added to the ensemble members, the curves are improved.

The effect of too large ensemble spread is demonstrated by experiments 9 to 12. The rank histogram for experiment 9 shows the Gaussian shape obtained when the true values are used for the observations. In experiment 10, the result of adding noise to the ensemble members in the case where the observations are perfect is shown. The combined effect of observation errors and too large initial spread is shown by experiment 11. Finally, the result when noise is added to the evolved ensemble members is given by experiment 12. This removes the overrepresentation in the extreme ranks, while a dome shape remains.

The reliability diagrams for the cases with too large spread (Fig. 4, experiments 9 to 12) also indicate that these are more sensitive to errors in the ensemble spread than to observation errors. The reliability curves for the four cases are almost the same, with a pronounced s shape, overforecasting low probabilities and underforecasting high probabilities.

4. The ensemble prediction system at ECMWF

In this section the impact of errors in the verifying observations on rank histograms is studied for the EPS running operationally at ECMWF (Molteni et al. 1996; Buizza et al. 2000). First the situation in which verification is performed with respect to observations is considered. As an example, the focus will be on ocean wave heights. Then the case in which rank histograms are based on verifying analyses is studied. This is done for the most commonly used parameter in this situation, which is the geopotential height field at 500 hPa.

5. Conclusions

Based on the analysis performed in section 2 and the results obtained by using the Lorenz model in section 3, it was found that observation errors may have a rather dramatic effect on rank histograms, leading to an increase in the number of observations in the lowest and highest ranks. It was shown that this sensitivity depends on the tail of the underlying distributions. For normal distributions an increasing sensitivity was found for larger ensemble size. A nonnormal tail could change this picture.

Ranking data obtained with a perfect model for both forecasts and observations resulted in u-shaped histograms when normally distributed noise was added to the observations. The results were almost identical to those obtained with perfect observations but too low ensemble spread. This false impression of too low ensemble spread is caused by the fact that the rank histogram method does not distinguish between large and small errors. Only significant differences—that is, differences larger than the observation errors—should contribute to the lowest and highest ranks when observations are outside the ensemble range. To account for this, the probability distribution from the ensemble forecast must be convolved with the observation errors. By adding normally distributed noise to each ensemble member, using the same standard deviation as for the observation errors, the flat rank histogram was restored for the perfect model case.

The reliability diagrams are less sensitive in this respect. For the case with perfect spread, the reliability diagrams were almost identical for cases with perfect observations and observations with noise added to them. Errors in the ensemble spread on the other hand had a stronger impact on the reliability curves. Too low ensemble spread resulted in underforecasting of low probabilities. Again, adding normally distributed noise to the ensemble members slightly improved the results. Too large ensemble spread resulted in s-shaped reliability curves, overforecasting for low probabilities and underforecasting for high probabilities.

Many investigations have pointed out that the ECMWF EPS does not have enough spread (Strauss and Lanzinger 1996; Evans et al. 2000; Buizza et al. 2000). Most studies of ensemble spread are based on either rank histograms or counting the number of outliers. Since the observations or verifying analysis fields are usually taken to be the truth, these results can to some extent be explained by the presence of errors in these quantities. When comparing the wave ensembles with buoy observations, we have demonstrated that the total number of outliers for the day 3 forecasts are reduced from more than 25% to less than 10% when a reasonable estimate of the observation errors is taken into account. Although the spread is still too low, the performance of the EPS is in this respect much better than often suggested.

For the Z500 forecasts, the impact of errors in the verifying analysis on ensemble spread was also considerable. Traditionally, a peak in the number of outliers has been found in the short forecast range, which was considered to be proof of lack of short-range ensemble spread. The fact that results have usually been better over western Europe and the United States, where observation networks are denser, indicates however that unaccounted errors in the verifying analysis fields may have contributed to this picture. Indeed, when noise based on estimates of such errors are added to the ensemble members, the peak in the number of outliers in the short range is found to almost vanish. Also, the numbers of outliers becomes more constant in time. Although the residual overpopulation of outliers may indicate some lack of ensemble spread, the situation is far more positive than previously believed.