Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation

a. Assessing sharpness and calibration

The goal of probabilistic forecasting is to maximize the sharpness of the forecast PDFs subject to calibration (Gneiting et al. 2003). Calibration refers to the statistical consistency between the forecast PDFs and the verifications and is a joint property of the predictions and the verifications. Briefly, a forecast technique is calibrated if meteorological events declared to have probability p occur a proportion p of the time on average. Sharpness refers to the spread of the forecast PDFs and is a property of the predictions only. A forecast technique is sharp if prediction intervals are shorter on average than prediction intervals derived from naive methods, such as climatology or persistence. The more concentrated the forecast PDF, the sharper the forecast, and the sharper the better, subject to calibration.

The principal tool for assessing the calibration of ensemble forecasts is the verification rank histogram or Talagrand diagram (Anderson 1996; Hamill and Colucci 1997; Talagrand et al. 1997; Hamill 2001). To obtain a verification rank histogram, find the rank of the verifying observation when pooled within the ordered ensemble values and plot the histogram of the ranks. Anderson (1996) and Saetra et al. (2004) suggested that when observations are used to assess an ensemble system, normally distributed noise based on the observation error could be added to the ensemble member models before the rank histogram is computed. In our experiments with the University of Washington system, the rank histograms computed from the ensembles with and without noise looked almost identical, and we retained the former.

The analogous tool for PDF forecasts is the PIT histogram. If F denotes the predictive CDF, the probability integral transform is simply the value F(x) at the verification x, a number between 0 and 1. For the Hope Airport forecast in Fig. 2bfor instance, the PIT value was 0.28. Rosenblatt (1952) studied the probability integral transform, and Dawid (1984) proposed its use in the assessment of probabilistic forecasts. The PIT histogram—that is, the histogram of the PIT values—is a commonly used tool in the econometric literature on probabilistic forecasting (see, e.g., Weigend and Shi 2000). Its interpretation is the same as that of the verification rank histogram: calibrated probabilistic forecasts yield PIT histograms that are close to uniform, while underdispersive forecasts result in U-shaped PIT histograms.

How can ensembles and PDF forecasts be fairly compared? An ensemble provides a finite, typically small, number of values only, while PDF forecasts give continuous statements of uncertainty, so this seems difficult. There are two natural approaches to a fair comparison, using either the verification rank histogram or the PIT histogram. To obtain an m-member ensemble from a PDF forecast, take the CDF quantiles at levels i/(m + 1), for i = 1, . . . , m. The verification rank histogram can then be formed in the usual way. To obtain a PIT histogram from an ensemble, fit a PDF to each ensemble forecast, as proposed by Déqué et al. (1994), Wilson et al. (1999), and Grimit and Mass (2004). The standard ensemble smoothing approach of Grimit and Mass (2004) fits a normal distribution with mean equal to the ensemble mean and variance equal to the ensemble variance. The PIT value is then computed on the basis of the fitted Gaussian CDF. Wilks (2002) proposed to smooth forecast ensembles by fitting mixtures of Gaussian distributions, an approach that allows for multimodal forecast PDFs. Multimodality may indeed be an issue for larger ensembles. For smaller ensembles, such as the University of Washington ensemble, standard ensemble smoothing using a single normal PDF suffices.

In addition to showing verification rank histograms and PIT histograms, we report the coverage of the 66⅔% central prediction interval; we chose this interval because the range of a five-member ensemble provides such. Finally, to assess sharpness, we consider the average width of the 66⅔% prediction intervals. For a five-member ensemble, this is just the average ensemble range.

For Gaussian predictive PDFs, the average width of the 100 × (1 − α)% central prediction intervals is

View Expanded

where z_1−α/2 denotes the 1 − α/2 quantile of the normal distribution with mean 0 and variance 1, and where S stands for the average standard deviation of the predictive PDFs. For instance, Table 2 shows that the average width of the central 66⅔% prediction intervals for MCE–EMOS forecasts of sea level pressure is 4.71. From (3) with α = ⅓ we find that S = 2.44. Using again (3), the average width of the 50% and 90% central prediction intervals is 3.29 and 8.01, respectively.

b. Scoring rules

Scoring rules for the verification of deterministic-style forecasts are well known and have been widely used in forecast assessment. If μ_i denotes a deterministic-style forecast and y_i is the verification, the MAE is defined as

View Expanded

where the sum is taken over the test data. A related error measure is the mean-square error (MSE), defined by

View Expanded

The RMSE is the square root of the MSE and has the advantage of being recorded in the same unit as the verifications.

We also consider two scoring rules for the assessment of predictive PDFs: the continuous ranked probability score (Unger 1985; Hersbach 2000; Gneiting and Raftery 2004) and the ignorance score (Good 1952; Roulston and Smith 2002). These scoring rules are attractive in that they address calibration as well as sharpness.

The CRPS is the integral of the Brier scores at all possible threshold values t for the continuous predictand (Hersbach 2000; Toth et al. 2003, section 7.5.2). Specifically, if F is the predictive CDF and y is the verifying observation, the continuous ranked probability score is defined as

View Expanded

where H(t – y) denotes the Heaviside function and takes the value 0 when t < y and the value 1 otherwise. Applications of the continuous ranked probability score have been hampered by a lack of closed-form expressions for the associated integral. However, when F is the CDF of a normal distribution with mean μ and variance σ², repeated partial integration in (4) shows that

View Expanded

where and the CDF, respectively, of the normal distribution with mean 0 and variance 1 evaluated at the normalized prediction error, (y − μ)/σ. We note from (4) that the average score,

View Expanded

reduces to the MAE if each F_i is a deterministic-style forecast. For this and other reasons, the CRPS can be interpreted as a generalized version of the MAE (Gneiting and Raftery 2004).

The ignorance score is the negative of the logarithm of the predictive density f at the verifying value y, that is, for a single PDF forecast,

View Expanded

In the case of a normal predictive PDF with mean μ and variance σ², we have

View Expanded

and the average ignorance is

View Expanded

When interpreting improvements in the IGN score, it is absolute rather than relative changes that are relevant (Roulston and Smith 2002).

Both CRPS and IGN are negatively oriented scores, in that a smaller value is better, and both scores are proper, meaning that they reward honest assessments. However, a key difference between the ignorance score and the continuous ranked probability score is that (5) grows linearly in the normalized prediction error, z = (y − μ)/σ, while (8) grows quadratically in z. Hence, the ignorance score assigns harsh penalties to particularly poor probabilistic forecasts, and can be exceedingly sensitive to outliers and extreme events (Weigend and Shi 2000; Gneiting and Raftery 2004). (This will become apparent in Tables 5 and 7 below.) We report both scores, but in view of the lack of robustness of the ignorance score, we prefer the continuous ranked probability score. A more detailed discussion of scoring rules is given in Gneiting and Raftery (2004).

c. Ensemble model output statistics and minimum CRPS estimation

We now describe the standard version of the EMOS method. Suppose that X₁, . . . , X_m denotes an ensemble of forecasts for a univariate weather quantity Y, and that S ² is the ensemble variance. The EMOS predictive PDF is that of the normal distribution

View Expanded

The EMOS predictive mean, a + b₁X₁ + · · · + b_mX_m, is a bias-corrected weighted average of the ensemble member forecasts; it provides a deterministic-style forecast. The EMOS predictive variance, c + dS ², is a linear function of the ensemble variance. The regression coefficients b₁, . . . , b_m in (10) reflect the overall performance of the ensemble member models over the training period, relative to the other members. They tend to be ordered in the inverse order of the forecast RMSEs, but this is not a direct relationship; they also reflect the correlations between the ensemble member forecasts. The variance coefficients c and d can be interpreted in terms of the ensemble spread and the performance of the ensemble mean forecast. All else being equal, larger values of the coefficient d suggest a more pronounced spread-error relationship. If spread and error are independent of each other, the coefficient d will be estimated as negligibly small. Hence, EMOS is robust, in the sense that it adapts to the presence as well as to the absence of significant spread-error correlation.

A classical technique for estimating the coefficients a, b₁, . . . , b_m, c, and d from training data is maximum likelihood (Wilks 1995, section 4.7). The likelihood function is defined as the probability of the training data given the coefficients, viewed as a function of the coefficients. In practice, it is more convenient to maximize the logarithm of the likelihood function, for reasons of both algebraic simplicity and numerical stability. The log-likelihood function for the statistical model (10) is

View Expanded

where the sum is taken over the training data. Here X_i1, . . . , X_im denote the ith ensemble forecast in the training set, S ²_i denotes its variance, and Y_i denotes the ith verification. Strictly speaking, (11) is the log-likelihood function under the assumption of independence. Note that the log-likelihood (11) is essentially the negative of the ignorance score (9) but is applied to the training data rather than the test data. Hence, maximum likelihood estimation is equivalent to minimizing the ignorance score for the training data.

This observation suggests a general estimation strategy: pick a scoring rule that is relevant to the problem at hand, express the score for the training data as a function of the coefficients, and optimize that function with respect to the coefficient values. We take scoring rules to be negatively oriented, so a smaller value is better, and we minimize the training score. For positively oriented scoring rules, we would maximize the training score. Such an approach is formally equivalent to MCE, a technique that has been studied in the theoretical statistics literature (Pfanzagl 1969; Birgé and Massart 1993). The minimum score approach can also be interpreted within the framework of robust M estimation (Huber 1964; Huber 1981, section 3.2) and forms a special case thereof, in that the function to be optimized derives from a strictly proper scoring rule (Gneiting and Raftery 2004). A more detailed methodological and theoretical discussion is beyond the scope of this paper.

However, we compared EMOS PDF forecasts estimated by MCE with the continuous ranked probability score, as described below, to EMOS PDF forecasts estimated by maximum likelihood. The MCE-EMOS approach clearly performed better: the predictive PDFs were sharper, and they were better calibrated. This comparison is summarized in Table 2. As a rule of thumb, it seems that predictive PDFs estimated by maximum likelihood tend to be overdispersive, resulting in unnecessarily wide prediction intervals that have higher than nominal coverage, and in inverted U-shaped PIT histograms. This latter shape is also seen in Figs. 4 and 5 of Weigend and Shi (2000), who estimate predictive densities by the maximum likelihood method in the form of the expectation maximization (EM) algorithm.

We argued in section 2b that the CRPS is a more robust and therefore more appropriate scoring rule than the ignorance score. This suggests the use of the continuous ranked probability score in minimum contrast estimation; we call this minimum CRPS estimation. The minimum CRPS estimator finds the coefficients a, b₁, . . . , b_m, c, and d in the statistical model (10) that minimize the CRPS value for the training data. Using (5) and (6), we express the training CRPS as an analytic function of the coefficients, namely

View Expanded

where

View Expanded

is a standardized forecast error, and where φ and Φ denote the PDF and the CDF, respectively, of a normal distribution with mean 0 and variance 1. We find the coefficient values that minimize (12) numerically, using the Broyden–Fletcher–Goldfarb–Shanno algorithm (Press et al. 1992, section 10.7) as implemented in the R language and environment (www.cran.r-project.org). The variance coefficients c and d are constrained to be nonnegative, which is not an issue for the parameter c. To enforce the nonnegativity of the variance coefficient d, we set d = δ² and optimize over δ. The optimization algorithm requires initial values, and starting values based on past experience usually give good solutions. However, convergence to a global maximum cannot be guaranteed, and the solution reached can be sensitive to the initial values.

The upper row in Table 3 shows the estimated EMOS coefficients for predictions on 25 May 2000, the day on which the Hope Airport forecast in Fig. 2 was issued. The training set included the forecasts and observations from the most recent 40 days that were available on 25 May 2000. The EMOS weights, b_k, reflect the overall performance of the ensemble member models over the training period, relative to the other members. They tend to be ordered in the inverse order of the forecast RMSEs, but this is not a direct relationship; they also reflect the correlations between the ensemble member forecasts, and bias effects are taken account of, too. In this example, the Aviation (AVN)-MM5, Global Environmental Multiscale Model (GEM)-MM5, and the U.S. Navy Operational Global Atmospheric Prediction System (NOGAPS)-MM5 forecasts received the highest EMOS weights and, indeed, the (linearly) bias-corrected AVN-MM5, GEM-MM5, and NOGAPS-MM5 forecasts had a smaller training RMSE than the bias-corrected Eta Model (ETA)-MM5 and Nested-Grid Model (NGM)-MM5 forecasts. The AVN-MM5, ETA-MM5, and NGM-MM5 forecasts all used NCEP initializations, and they were highly collinear. For the training period, the pairwise correlations within this group ranged from 0.93 to 0.97. EMOS assigned a substantial weight to the most skillful of these three collinear forecasts, the AVN-MM5 forecast; the ETA-MM5 and NGM-MM5 forecasts received negative and negligible weights, respectively. The correlation coefficients between NCEP- and non-NCEP-initialized member model forecasts were also high, but they reached at most 0.92. The estimated variance coefficient d turned out to be negligibly small, thereby suggesting a weak spread-skill relationship during the training period. Indeed, the correlation between the ensemble variance and the squared error of the ensemble mean forecast was only 0.11 for this particular 40-day training period, as compared to 0.33 for the entire period, January–June 2000.

It is straightforward to draw random samples from the EMOS predictive PDF. An alternative, and likely preferable, way of forming an m-member ensemble from the predictive PDF is by taking the CDF quantiles at level i/(m + 1), for i = 1, . . . , m, respectively. In this way, ensembles of any size can be obtained, and in this sense, EMOS can be viewed as a dressing method (Roulston and Smith 2002). We note that EMOS requires the ensemble member models to have individually distinguishable characteristics. This is true for the University of Washington ensemble, a multianalysis, mesoscale, short-range ensemble, and also for poor person's and multimodel ensembles. In other types of ensembles, the member models might be exchangeable. In this type of situation, the linear regression Eq. (1) can be based on the ensemble mean forecast only, which constrains the regression coefficients b₁ = · · · = b_m to be equal. In this implementation, EMOS can be applied to essentially all ensemble systems, including perturbed observations, bred, or singular vector ensembles. Jewson et al. (2004) applied such an approach to the synoptic ECMWF ensemble, using maximum likelihood estimation. However, they did not report out of sample forecasts, and consequently neither verification scores nor rank histograms. For the University of Washington ensemble, the general formulation seems preferable. In the situation of Table 2, constraining the regression coefficients in (10) to be equal, that is, using the ensemble mean only, results in MAE, RMSE, CRPS, and IGN scores up to 7% worse, as compared to the full formulation.

d. Enforcing nonnegative regression coefficients: EMOS⁺

The regression Eq. (1) allows for negative EMOS weights b₁, . . . , b_m, and in many applications of multiple linear regression negative regression coefficients are crucial. Consider, for instance, predictions of wind speed based on time series of past values. Recent changes in wind speed may have predictive power, and to take account of the changes, negative regression coefficients are required. In the context of ensemble forecasts, the negative weights seem much harder to interpret. They imply, all else being equal, that sea level pressure, say, is predicted to be lower when the forecast with the negative weight is higher. In our experiments with the EMOS technique, negative regression coefficients occurred frequently, and they were caused by collinearities between the ensemble member model forecasts. Indeed, it is well known that highly correlated predictor variables in a regression model lead to coefficient estimates that are unstable under small changes in the training data. In this sense, the negative EMOS weights could be viewed as an artifact.

To address this issue, we propose an implementation of the EMOS technique that constrains the coefficients b₁, . . . , b_m in the regression Eq. (1) to be nonnegative. We call this variant of the EMOS technique EMOS⁺. The plus sign stands for the nonnegative EMOS weights, and the idea relates to Tibshirani's (1996) least absolute shrinkage and selection operator (lasso) regression, which imposes a penalty on the absolute size of the regression coefficients and results in estimates that tend to be exactly zero for some of the coefficients, thereby improving forecast accuracy and interpretability. In our case, EMOS weights that are exactly zero can be interpreted in terms of reduced ensembles. To fit the EMOS⁺ model we proceed stepwise, as follows. We first find the unconstrained minimum of the CRPS value (12), that is, we estimate the coefficients of the standard EMOS model. If all estimated regression coefficients are nonnegative, the EMOS⁺ estimates are the same as the EMOS estimates, and the EMOS⁺ model is complete. If one or more of the regression coefficients are negative, we set them to zero and minimize the CRPS value (12) under this constraint. We also recompute the ensemble variance, using only the ensemble members that remain in the regression equation, and subsequently use the recomputed ensemble spread. This procedure is iterated until all estimated regression coefficients are nonnegative.

Table 3 illustrates this algorithm for predictions on 25 May 2000, the day on which the Hope Airport forecast in Fig. 2 was issued. The upper row shows the parameter estimates for the standard EMOS model, which include a negative coefficient for the ETA-MM5 forecast. We set this coefficient to zero and proceed with the constrained minimization, resulting in a negative weight for the NGM-MM5 forecast. The final EMOS⁺ equation uses only one of the three ensemble members initialized with NCEP models, namely the AVN-MM5 forecast, along with the GEM-MM5 and the NOGAPS-MM5 forecasts.

In our experiments with the University of Washington ensemble, which are summarized below, the EMOS and EMOS⁺ techniques had equal forecast skill. However, we found the EMOS⁺ parameters to be easier to interpret, in that the EMOS⁺ forecasts correspond to reduced ensembles. The member models with vanishing EMOS⁺ coefficients were judged not to be useful during the training period, relative to all the others, and therefore were removed from the regression equation. That said, it is important to distinguish the usefulness and the skill of an ensemble member model. Consider, for instance, a three-member ensemble. Model A has a lower RMSE than model B, and model B has a lower RMSE than model C; all three models are unbiased. If A and B are highly collinear and both are independent of C, then model C may be a more useful but less skillful ensemble member than model B.

a. Length of training period

What training period should be used for estimating the EMOS and EMOS⁺ regression coefficients and variance parameters? There is a trade-off in selecting the length of the sliding training period. Shorter training periods can adapt rapidly to seasonally varying model biases, changes in the performance of the ensemble member models, and changes in environmental conditions. On the other hand, longer training periods reduce the statistical variability in the estimation of the EMOS and EMOS⁺ coefficients. We considered sliding training periods of 19, 20, . . . , 62 days for EMOS forecasts of sea level pressure; the results for the EMOS⁺ forecasts were similar. For comparability, the same test set was used in assessing all the training periods; that is, the first 63 days on which the ensemble was operating were not included in the test period. This results in a 39-day test period, consisting of all the days between 24 April and 30 June 2000 on which the ensemble system was operational. The unit used for the sea level pressure forecasts is the millibar (mb).

The results of this experiment are summarized in Fig. 3. Figures 3a and 3b show the MAE and RMSE of the deterministic-style EMOS forecasts, respectively. These decrease sharply for training periods less than 30 days, stay about constant for training periods between 30 and 45 days, and increase thereafter. Figures 3c and 3d show the CRPS and the IGN. The patterns are similar to those for the MAE and the RMSE. The empirical coverage of the EMOS 66⅔% prediction intervals is shown in Fig. 3e along with the nominal coverage, which is represented by the horizontal line. Training periods under 30 days seem to result in underdispersive PDFs, but training periods between 30 and 60 days show close to nominal coverage. Figure 3f shows the average width of the 66⅔% prediction intervals. The average width increases with the length of the training period, but is about constant for training periods between 30 and 40 days.

To summarize these results, there appear to be substantial gains in increasing the training period beyond 30 days. As the training period increases beyond 45 days, the skill of the probabilistic forecasts declines slowly but steadily, presumably as a result of seasonally varying model biases. In view of our goal of maximizing sharpness subject to calibration, we chose a sliding 40-day training period for the EMOS and EMOS⁺ forecasts of sea level pressure. For instance, the predictive PDFs for forecasts initialized on 30 June 2000 were trained on the 40 most recent ensemble runs that had verified by this date. The earliest forecasts in this particular training set were initialized on 15 April 2000 and verified on 17 April 2000; the latest were initialized on 28 June 2000 and verified on 30 June 2000. The sliding 40-day training period worked well for temperature forecasts, too. However, distinct training periods might work best for distinct variables, forecast horizons, time periods, and regions. Ideally, we would include training data from previous years to address seasonal effects. Further research in this direction is desirable as multiyear runs of stable mesoscale ensembles become available. Ensemble reforecasting (Hamill et al. 2004) provides an attractive yet, in many cases, computationally prohibitive alternative.

b. Sea level pressure forecasts

We now give the results for EMOS and EMOS⁺ forecasts of sea level pressure, using a 40-day sliding training period and the same test set that was used to compare the different training periods. We also summarize the results for the bias-corrected ensemble member forecasts and for a climatological forecast. The bias-corrected ensemble member forecasts used least squares regression fits, α_k + β_kX_k, of the ensemble member models, X_k, on the observations, and the regression parameters were estimated on the same 40-day sliding training period. The deterministic-style climatological forecast was the average sea level pressure among the verifying observations in the training set, and the climatological predictive PDF was obtained by fitting a normal PDF to these observations.

Figure 4 shows the estimates of the EMOS coefficients for the 39 days in the test period. The estimated intercept in the multiple linear regression equation is shown in Fig. 4a. Figures 4b–f show the EMOS weights for the five ensemble member models, respectively. The weights for the AVN-MM5, CMC-MM5, and NOGAPS-MM5 forecasts were consistently positive and substantial, the weights for the ETA-MM5 forecast were consistently negative, and the weights for the NGM-MM5 forecast decreased from positive to negative. The negative regression coefficients are caused by collinearities between the ensemble member model forecasts, as discussed in sections 2c and 2d. Figures 4g and 4h show the estimated variance coefficients c and d, respectively. The estimates of the variance parameter c decreased toward the end of the test period, thereby indicating improved ensemble skill or improved atmospheric predictability, or both. The estimated values of d were mostly negligible.

The corresponding estimates of the EMOS⁺ coefficients are shown in Figs. 5a–h, respectively. The EMOS⁺ weights for the AVN-MM5, CMC-MM5, and NOGAPS-MM5 forecasts were consistently substantial, and the weights for the ETA-MM5 and NGM-MM5 forecasts were consistently zero. Hence, EMOS⁺ retained only one of the three highly collinear ensemble member models that were initialized by NCEP analyses and picked the most skillful of them, namely the AVN-MM5 forecast. The EMOS⁺ weights for this forecast were consistently higher than the respective EMOS weights. The EMOS⁺ estimates of the variance parameter c decreased during the test period, and the estimates of the variance coefficient d, shown in Fig. 5h, were mostly nonzero. The increase toward the end of the test period indicates a strengthening of the spread-error relationship or an increase in the temporal variability of the forecast skill, or both. The comparison to the EMOS estimates of the variance coefficient d, shown in Fig. 4h, seems interesting; it suggests that the spread-error relationship can be masked by collinearities between the ensemble member models.

Table 4 provides summary measures of deterministic-style forecast accuracy. Among the raw ensemble member models, the AVN-MM5 forecast performed best. Bias correction resulted in a reduction of the RSME for the ensemble member model forecasts between 4% and 17%. The improvement was most pronounced for the NOGAPS-MM5 forecast. The AVN-MM5, CMC-MM5, and NOGAPS-MM5 forecasts were more accurate than the ETA-MM5 and NGM-MM5 forecasts. The ensemble mean forecast performed considerably better than any of the ensemble member models, but the bias-corrected AVN-MM5 and NOGAPS-MM5 forecasts were more accurate than the mean of the bias-corrected ensemble. The deterministic-style EMOS and EMOS⁺ forecasts performed about equally well and were much better than any of the other forecasts. They had RMSEs between 7% and 9% less when compared to the mean of the raw ensemble and to the mean of the bias-corrected ensemble, respectively. The results in terms of the MAE were similar.

Table 5 turns to summary measures of probabilistic forecast skill. The climatological predictive PDFs showed the correct coverage, but they were too spread out to be competitive. The bias-corrected ensemble showed reduced ensemble spread, but was even more underdispersive than the raw ensemble. The EMOS and EMOS⁺ prediction intervals showed accurate coverage. The CRPS and the IGN were computed as described in section 2b, using standard ensemble smoothing for the raw and bias-corrected ensemble. The CRPS can also be computed directly, by using the empirical ensemble CDF, which takes the values 0, ⅕, . . . , ⅘, 1, with jumps at the ensemble member forecasts. This gave somewhat higher CRPS values of 1.69 and 1.72 for the raw ensemble and for the bias-corrected ensemble, respectively. The EMOS and EMOS⁺ predictive PDFs performed equally well and had by far the best scores among the forecasts. When compared to the bias-corrected ensemble, EMOS and EMOS⁺ reduced the CRPS score by 16%, and the IGN score was 3.68 points lower. The EMOS and EMOS⁺ prediction intervals were not much wider than the prediction intervals obtained from the raw ensemble. A more detailed analysis shows, perhaps surprisingly, that in 28% of the forecasts the EMOS 66⅔% prediction interval was shorter than the range of the five-member raw ensemble. In 10% of the forecasts, the EMOS 66⅔% prediction interval was shorter than the range of the bias-corrected ensemble.

The verification rank histograms for the various ensembles are shown in Fig. 6. The EMOS and the EMOS⁺ ensemble were much better calibrated than the raw ensemble or the bias-corrected ensemble and showed rank histograms that were close to being uniform but not quite uniform. Indeed, the latter was not to be expected. Sea level pressure is a synoptic variable with strong spatial correlation throughout the ensemble domain, and there were only 39 days in the evaluation period. The PIT histograms in Fig. 7 accentuate the underdispersion in the raw ensemble and in the bias-corrected ensemble.

c. Temperature forecasts

We now summarize the results for forecasts of surface temperature, a case of primary interest to the public (Murphy and Winkler 1979). The 2-m temperature forecasts were obtained as an average of the predicted lowest sigma level temperature and the predicted ground temperature. Similar to the sea level pressure forecasts, we used a sliding 40-day training period, and we considered the same region and the same test period. We omit the results for the climatological forecast, which was even less competitive than for sea level pressure, given seasonal and topographic effects. The unit used for the temperature forecasts is degrees kelvin.

Figure 8 displays the estimates of the EMOS coefficients for the 39-day test period. Figure 8a shows the estimates of the intercepts, which were consistently negative. Figures 8b–f show the estimated EMOS weights. The weights for the AVN-MM5 forecast reached a maximum of 0.57 and were consistently the highest among the five ensemble member models. The weights for the ETA-MM5 and NOGAPS-MM5 forecasts were smaller but still positive and substantial, those for the NGM-MM5 forecast oscillated about zero, and those for the GEM-MM5 forecast were initially negative, before increasing to substantially positive levels. Figures 8g and 8h show the estimated variance parameters c and d.

Figures 9a–h turn to the corresponding estimates of the EMOS⁺ coefficients. These were very similar to the EMOS estimates, except that the weights for the NGM-MM5 forecast and, initially, for the GEM-MM5 forecast, were zero. These results can, again, be interpreted in terms of the relative contributions of the ensemble member models. The correlation between the ETA-MM5 and the NGM-MM5 forecasts was the highest among the forecast pairs. To avoid collinearity, EMOS⁺ retained only one of them. The AVN-MM5 forecast was the most accurate member model and received the highest EMOS and EMOS⁺ weights.

Table 6 confirms that the AVN-MM5 forecast had the lowest RMSE among the ensemble member forecasts, both before and after bias correction. Bias correction resulted in percentage improvements in the RMSE of the ensemble member forecasts between 4% and 14%, and the NOGAPS-MM5 forecast showed the highest percentage improvement. The results in terms of the MAE were similar. The deterministic-style EMOS and EMOS⁺ forecasts performed equally well, and they were more accurate than any of the other forecasts. The percentage improvement over the bias-corrected ensemble was less pronounced than for forecasts of sea level pressure.

We now turn to a discussion of probabilistic forecast skill. Table 7 shows that the bias-corrected ensemble was slightly better calibrated than the raw ensemble. However, both the raw ensemble and the bias-corrected ensemble were strikingly underdispersive, and this was reflected in the CRPS and IGN scores, which were computed on the basis of standard ensemble smoothing. When computed directly from the ensemble CDF, the CRPS scores for the raw ensemble and for the bias-corrected ensemble were 2.13 and 1.95, respectively. The EMOS and EMOS⁺ techniques performed equally well, and they were better than any of the other forecasts, with a CRPS score that was 15% lower than for the bias-corrected ensemble, and an IGN score that was 13 points lower. The verification rank histograms and PIT histograms are shown in Figs. 10 and 11. The PIT histograms accentuate the underdispersion of the ensemble forecasts, and the histograms for the EMOS and EMOS⁺ forecasts are close to being uniform.

d. Results for reduced ensembles

In our experiments for sea level pressure and surface temperature forecasts, the EMOS and EMOS⁺ predictive PDFs were equally skillful. This comparison is summarized in Table 8, and the scores for the two techniques are almost indistinguishable. We interpreted the vanishing EMOS⁺ coefficients in terms of reduced ensembles and argued that ensemble member models with consistently vanishing EMOS⁺ weights could be removed from the ensemble, without sacrificing forecast skill.

For forecasts of sea level pressure, Fig. 5 suggests the use of a three-member ensemble, consisting of the AVN-MM5, GEM-MM5, and NOGAPS-MM5 forecasts. We applied the standard EMOS technique to the reduced three-member ensemble, and the results are shown in Table 8. The MAE, RMSE, CRPS, and IGN scores for the EMOS forecasts, the EMOS⁺ forecasts and the EMOS forecasts using the reduced ensemble are almost indistinguishable. Similarly, Fig. 9 suggests that for forecasts of surface temperature the NGM-MM5 forecast could be removed from the ensemble. Table 8 compares the EMOS predictive PDFs based on the reduced four-member ensemble to the EMOS and EMOS⁺ forecasts based on the full ensemble, and the comparison is favorable. Clearly, minimal differences in scores must not be overinterpreted, and ensemble members that contribute little to forecasting one variable might be useful for others. That said, the NGM-MM5 model was removed from the University of Washington ensemble on 1 August 2000, shortly after the end of our test period.