a. Background

An ensemble forecasting system addresses the chaotic nature of the atmosphere by providing a dynamic estimate of the prediction confidence. Such systems exploit the stochastic nature of the atmosphere by generating many solutions based on slightly perturbed initial states (Toth and Kalnay 1993). The chaotic nature of the predicted system leads the model solutions to diverge from one another with time, resulting in different realizations representing possible future atmospheric states (Epstein 1969a; Leith 1974). For well-calibrated models, closely grouped model realizations (in phase space) are assumed to indicate low uncertainty in the final atmospheric state, while widely scattered solutions indicate higher uncertainty. Ensemble forecasting is also used for climate predictions, where predictive skill may be low or negligible, but where ensembles are hoped to accurately reflect a range of possible climatic anomalies, and identify areas of potential predictability associated with boundary forcing (Barnett 1995; Stern and Miyakoda 1995; Kumar and Hoerling 1995).

Ensemble predictions generally require calibration to remove biases and to ensure that the forecast frequency of events gives a realistic representation of that in the atmosphere. Many methods have been developed to calibrate ensemble predictions to provide reliable probabilistic forecasts. The choice of an appropriate method depends on the characteristics of the ensemble forecasts and on their intended applications. Since a primary mission of the Climate Prediction Center (CPC) is seasonal climate prediction, we require a calibration method suitable for use in global climate models. Skill in climate prediction is low and varies considerably both spatially and temporally (Livezey 1990; Rowell 1998; Phelps et al. 2004; Livezey and Timofeyeva 2008; O’Lenic 2008). This may be reflected in the ensemble spread of GCM predictions, so it is critical to retain as much of this information as possible. The amount of data available to develop relationships for seasonal prediction is very limited. The National Centers for Environmental Prediction’s (NCEP) Climate Forecast System (CFS) model (Saha et al. 2006), for example, has hindcast data available only since 1981 so any calibration procedure must do well with very limited sample sizes. An additional requirement is the ability to estimate the full probability density function (PDF) of the forecast element in order to support CPC’s Probability of Exceedance product (Barnston et al. 2000).

The small amount of hindcast data available from most GCM predictions eliminates some ensemble calibration methods from serious consideration. Some calibration methods such as binning procedures (Anderson 1996; Hamill and Colucci 1997, 1998; Eckel and Walters 1998) or logistic regression (Hamill et al. 2004) divide the range of the forecast element into a series of categories (bins). The limited data available for seasonal prediction make subdivision of the data sample into multiple categories impractical. The analog approach described by Hamill and Whitaker (2006) is similarly impractical for climate prediction in view of the difficulty in finding good analogs for the limited data available for seasonal prediction (van den Dool 1994).

Regression-based approaches to the problem are appealing because of their ability to optimally “fit” data to minimize errors. When applied to continuous variables, this allows the entire dataset to simultaneously contribute to the regression relationship, enabling maximum use of small samples. Several methods based on regression have been proposed for ensemble calibration. Gneiting et al. (2005) use a pair of regression relationships to calibrate ensemble forecasts, one to correct the ensemble mean forecast and another to correct the ensemble spread. This method was among the most successful of those tested in a side-by-side comparison of a variety of ensemble calibration methods, both on an idealized model (Wilks 2006) and on GFS reforecasts of temperature and precipitation (Wilks and Hamill 2007). The method requires that the ensemble members be fit with a parametric distribution in each case. While this may be beneficial if the distribution of the residuals about the ensemble mean forecast is believed known, some aspects of atmospheric circulation are known to exhibit complex PDFs (Benzi and Speranza 1989), so a calibration method that can retain information from nonparametric distributions forecast by a model is desired for CPC’s operations, at least for model diagnostics, if not for prediction.

We propose a regression model specifically designed for use on ensemble forecasts. This “ensemble regression” (EREG) model is formulated for the commonly held assumption that individual ensemble members represent possible solutions for a given initial state (Sivillo et al. 1997). Of the many solutions, one will be “best” and if the ensembles are generated from the same model, it can reasonably be assumed that the probability of each member being best is about equal. We will show that, given this assumption, together with other assumptions usually made for linear regression, a “best member” regression equation can be estimated from the statistics of the ensemble set as a whole, with no need to explicitly identify a best member in each case. The regression model produces a calibrated set of ensemble forecasts, together with an estimated error distribution around each member that can be used to produce a PDF of the predicted variable from the ensemble forecasts.

The EREG estimates, together with their estimated errors, resemble the “dressed ensemble” approach to ensemble calibration (Roulston and Smith 2003; Wang and Bishop 2005; Fortin et al. 2006), except the ensemble members are fully calibrated for a least squares fit to the data (rather than just bias corrected as in the dressed ensemble approach) and the kernel distributions are derived from the regression estimates of the distribution of residuals about the calibrated best member.

The regression coefficients for the best-member equation are identical to those derived from the ensemble mean and are applied to each member of the ensemble. Therefore, EREG can be implemented by developing a regression relationship between the ensemble mean forecast and the observation, and applying the result to the individual members in the ensemble. This approach has recently been pragmatically explored by the National Weather Service’s Meteorological Development Laboratory (Glahn et al. 2009) on short-range forecasts. We show theoretical justification for such an approach here.

We will present the mathematical basis for the EREG procedure in section 2. EREG is applied to long-lead seasonal predictions of sea surface temperature (SST) in the Niño-3.4 region of the Pacific Ocean from NCEP’s Climate Forecast System to produce calibrated probabilistic predictions and these results are presented in section 3.

b. Terminology

For this discussion, we assume that statistics are accumulated over a sample of forecasts, such as a series of forecasts issued daily or monthly. The time average over the entire sample of M cases is indicated by angle brackets;

View Expanded

The subscript, j, represents the temporal dimension and will usually not appear in equations for individual forecast cases. For each case, a collection of N ensemble members are available, F_i, i = 1, N, and the ensemble mean is denoted as F_m. Note that over M cases, 〈F_i〉 = 〈F_m〉, since the summation over the N ensemble members is implied in averaging.

It is usually appropriate to eliminate the seasonal cycle from the predictions by expressing both forecasts and observations as departures from long-term climatology. Statistics can also be stratified by month and lead time so that data for all forecasts initialized at a given time of year and for a given lead time are pooled together to form a regression relationship.

a. Simple linear regression

Regression has been applied to the output from dynamic numerical prediction models for over 40 yr (Glahn and Lowry 1972; Glahn et al. 2009). Regression analysis usually begins with a tentative assumption of a linear relationship between the predictors (in this case the forecasts from a numerical model) and the predictand (observations), with errors represented by the term, ε. For reasons that will become clear later, this will be illustrated by the relationship between the ensemble mean, F_m, and the observation, Y:

View Expanded

Linear regression minimizes the quantity 〈(F′_m − Y)²〉 to estimate the α’s and obtains the equation, F′_m = a₀ + a₁F_m (Draper and Smith 1981), where F′_m is the regression estimate, and coefficients are given by

View Expanded

where S_Y and S_m are the sample standard deviations of Y and F_m, respectively, and R_m is the correlation coefficient between the ensemble mean forecast and the observations. The regression relationship is frequently subject to an analysis of variance with two components defined as follows:

• sum of squares due to regression,

  

  

  View Expanded

  
• sum of squares about the regression (residual),

  

  

  View Expanded

  

If the regression estimates are regarded as calibrated forecasts, the means of these two variance components, the regression and the residual, can be regarded as the sample variance of the regression forecasts, , and an associated variance in the forecast errors, S_e², respectively, and are related to R_m as shown:

View Expanded

View Expanded

View Expanded

Note that these relationships are robust and follow directly from the definitions of the variance, means, and correlation with no requirement for F_m or Y to have Gaussian distributions. Gaussian assumptions are used in significance testing, or in establishing an estimate of the forecast error distribution, but are not required for these relationships to be valid on the dependent data.

Both S_Y² and S_e² are biased estimates of each variable’s true variance, σ_Y² and σ_e², respectively. For a sample size of M cases,

View Expanded

View Expanded

View Expanded

where c represents an additional correction factor to the residual variance to account for uncertainty in R_m. It is frequently the practice in seasonal forecasting to use an estimate of σ_Y² from long-term climatology rather than to base the estimate on the sample climatology, thereby partially compensating for the biases in these relationships. The constant c compensates for the remaining bias and is given by

View Expanded

Equation (2.3a) is the explained variance, and (2.3b) gives the unexplained variance of the regression relationship. If the true relationship between F and Y is linear, and the errors are uncorrelated and Gaussian distributed, then the residual distribution will also be Gaussian. This implies that the residuals, ε, can be represented by a Gaussian distribution centered on F′_m, PDF(ε) ≈ N{F′_m, σ_Y[c(1 − R_m²)]^1/2}, following standard terminology where N(μ, σ) represent a normal distribution with mean μ and standard deviation σ. The expected value of σ_e² increases with the distance from the sample mean due to uncertainty in the regression coefficients. This effect will be neglected here for simplicity.

An important feature of the regression estimate is that its variance is reduced according to R_m². This “skill damping” effect can best be seen by expressing the regression equation in terms of a standardized departure from the mean, as shown:

View Expanded

The standardized anomaly of the regression estimate is damped toward the mean by the factor R_m.

b. Statistical constraints on an ensemble set

Rather than a single forecast, an ensemble prediction consists of a set of related forecasts all paired with a single observation. This constrains the statistics of the ensemble set according to the following series of relationships. The mean squared error of the individual ensemble members, F_i, is related to the ensemble spread and the squared error in the ensemble mean, by (see appendix A)

View Expanded

where 〈E²〉 is the mean ensemble spread,

View Expanded

The sample variance of the individual ensemble forecasts, S_I², can be related to the mean spread and variance of the ensemble mean, S_m², by a derivation similar to that shown in appendix A except substituting 〈F〉 for Y:

View Expanded

The correlation coefficients between Y and (a) the individual ensembles, R_I, and (b) the ensemble mean, R_m, are also related as shown (see appendix B):

View Expanded

Applying (2.6) and rearranging terms, 〈E²〉 is given by

View Expanded

c. Ensemble regression (EREG)

An ensemble prediction is frequently regarded as a set of possible states resulting from a given initial condition. Of the various solutions, one will be “best,” which might be regarded as either the closest to the observation, or best in some multivariate sense, as Roulston and Smith (2003) suggest. If the ensemble members are generated by the same atmospheric model, it is usually assumed that each member has an equal likelihood of being best. A linear regression model can be specifically tailored for use on ensemble prediction considering these specialized assumptions. As with any regression procedure, a tentative model is considered to describe the system, which can be rejected at a later time if these assumptions are not supported by the data.

Without actually identifying a best member, F_b, we postulate that it is related to the observation according to

View Expanded

where ε_b represents the errors only for F_b. It is further assumed that ε_b is distributed in the same manner for each potential realization.

Given our tentative regression model, the regression equation that minimizes ε_b is given by (from standard linear regression theory summarized in section 2a)

View Expanded

View Expanded

Here, R_b is the unknown expected value of the correlation between the best ensemble member and the observation. Note that given our assumption that members are equally likely to be best, on any given case, j, the expected value of F_b can be calculated from F_i, as follows:

View Expanded

Here it is reasonable to assume that the best member is determined from the closest ensemble solution after calibration by (2.10a).

The expected value of the grand mean of F_b is

View Expanded

By similar reasoning, the expected value of S_b can determined as follows:

View Expanded

Here, it must be emphasized that these relationships apply only for when our assumption that each ensemble member is equally likely to be best are met.

In subsequent discussions, quantities involving best-member statistics (subscript b, except for F_b and F′_b) refer to their expected values, and the notation will be simplified so, for example,

View Expanded

Similar to the results in section 2a, the expected values of both the explained variance, and the residual error variance, , are given by

View Expanded

View Expanded

As shown in appendix C, the expected values of the regression coefficients a₀ and a₁ are the same as the coefficients of the ensemble mean when regressed onto the observations. Thus, from (2.1),

View Expanded

Linear transformation of the forecasts does not affect the correlation between the forecast and the observation, so, after substitution from (2.7), R_b is given by

View Expanded

Because the expected value of the regression coefficients of the best-member equation are the same as those for the ensemble mean, EREG can be implemented by applying the regression equation based on the ensemble mean to each individual ensemble member. Here, R_b represents the expected value of the correlation between F′_b and the observation, provided that the regression model assumptions are accurate. In standardized anomaly form, the EREG equation is

View Expanded

indicating that EREG damps individual ensemble members to a lesser extent than the equation applied to the ensemble mean [see (2.4)].

An estimate of 〈expv(Y − F′_m)²〉 can be obtained both from (2.3b) and also from computing the expected value after substituting the individual member regression estimates:

View Expanded

Note that the expected value of the error term and the cross products involving this term is zero, and that ε_b already accounts for the expected errors in the ensemble mean (because it is derived from the residual error of a regression), so after substitution

View Expanded

The regression estimate of the residual variance about the calibrated ensemble mean is related to the regression-corrected ensemble spread, a₁²〈E²〉, as shown:

View Expanded

where ε_b² represents the remainder of the variance not accounted for by the calibrated ensemble members. Since ε_b² is nonnegative, cσ_Y²(1 − R_m²) ≥ a₁²〈E²〉.

If the above inequality is not true, then the ensemble members cannot conform to the EREG assumptions and the regression model must be rejected. This occurs when the calibrated ensemble is overdispersive (members near the ensemble mean have a higher probability of being best than those near the ensemble’s outer envelope even after applying the regression equation), and that R_b as calculated from (2.12) exceeds one. An underdispersive model presents no problems, since the regression estimate of ε_b² will adjust to account for the model’s missing variance. However, there is always a possibility that an underdispersive model can be improved by increasing the ensemble spread, shifting more weight to the dynamic prediction of the errors about the calibrated ensemble mean forecast and less to its statistical estimate. Adjustments to the ensemble spread will be addressed in section 2e.

d. Estimated PDF of the calibrated ensemble

If ε_b, in (2.9) is Gaussian distributed, then its regression estimate, ε_b, is N(0, σ_eb), σ_eb can be estimated in a manner similar to (2.3):

View Expanded

The distribution of observations around the calibrated best member can then be represented by centering the error distribution around F′_b. The forecast PDF representing the entire ensemble of N equally likely members takes the form of a series of “kernel” Gaussian distributions, each centered on the regression estimate of an individual member. The final PDF is simply the unit-normalized sum of all error distributions, each representing 1/N of the total distribution, as illustrated by the example in Fig. 1.

e. Adjustments to the ensemble spread

In this section we will examine the relationship between the ensemble spread and the EREG residual error. A linear transformation is proposed to adjust the ensemble spread, if necessary, to better fit the assumptions required for EREG. As discussed earlier, a reduction of the ensemble spread is required to produce a reasonable regression fit when the calibrated ensemble is overdispersive.

Consider a spread adjustment factor, K, applied on all cases prior to regression given by

View Expanded

where F″_i refers to the transformed ensemble forecasts. The transformation constant, K, alters the correlation between the observation and individual transformed ensemble members, R″_I, and the expected values of R_b and σ_eb, and these relationships are derived from (2.6), (2.7), and (2.14):

View Expanded

Thus,

View Expanded

View Expanded

The maximum value of K (K_max) that is consistent with regression assumptions can be computed by setting (implying that R″_b = 1), and can be calculated as shown in (2.17):

View Expanded

When K_max < 1, the EREG estimates based on the original model forecasts will be overdispersive and the spread needs to be reduced to assure that the forecast variance is less than the observed variance.

Note that K_max does not account for the sampling variability expected with a limited number of ensemble members. If an ensemble forecast is presumed to be a sample of N solutions drawn randomly from a large population of potential solutions, then the maximum value of K based on sampling considerations, K_N, is related to the ensemble size, N, as shown in appendix D:

View Expanded

Equation (2.18) expresses the maximum K value supported by an N-member ensemble randomly chosen from an infinite population of solutions and over the dependent data sample. If K_N < 1, the EREG PDF estimate is likely to be overdispersive since the ensemble spread of the original forecasts (K = 1) is greater than K_N. In this case the ensemble spread needs to be reduced by applying (2.15) with K = K_N and computing kernel distributions from (2.16a)–(2.16c).

The transformation given in (2.15) can be used to alter the ensemble for specific purposes provided that K stays within the range 0 < K < K_max. For example, K might be set to K_max to translate the ensemble forecasts into a series of calibrated point (deterministic) forecasts, equivalent to “inflation” in MOS equations (Klein et al. 1959). Inflation produces a bias-corrected set of point forecasts whose variance is the same as that of the observations over the dependent data sample. At the other extreme, setting K = 0, implies that R″_I = R_m, and ensemble regression becomes standard linear regression on the ensemble mean. This transformation would be appropriate if the data suggested that the information from individual ensemble members worsens the forecast based on the ensemble mean alone. Intermediate values of K can also be tested in conjunction with verification scores that are appropriate for use on probabilistic predictions in an attempt to improve the forecasts. The PDF estimated from an ensemble regression can be generated for a variety of K values, and the value that produces the best result when measured by a given scoring rule and on a given set of forecasts can be selected to provide an estimate of an optimum ensemble spread for that score.

a. Forecast description

EREG was tested on sea surface temperature (SST) forecasts for the Niño-3.4 region from NCEP’s CFS model (Saha et al. 2006). Niño-3.4 SSTs (mean SSTs between 5°N and 5°S and 170° and 120°W) correlate well with the state of the El Niño–Southern Oscillation (ENSO) (Barnston et al. 1997) and, therefore, are an important indicator for climate anomalies over many parts of the globe.

The CFS is an operational coupled ocean–atmosphere model that is currently run twice daily to produce forecasts for up to 6 months in advance. A CFS ensemble forecast is typically produced to support the Climate Prediction Center’s (CPC) operational climate outlooks issued in the middle of each month (Barnston et al. 1994). The CFS ensemble forecast is formed from predictions initialized at various times in the previous month, all valid for the same target periods and thus similar to the strategy used for lagged averaged forecasting (Hoffman and Kalnay 1983; Roads 1988). The use of a lagged average forecast is common in climate forecasts since perturbations in the initial state have little effect on the atmospheric seasonal forecast beyond 1 month (Phelps et al. 2004, and about 2 months for oceanic predictions (Vialard, et al. 2005). Because there is usually close to a 1-month lag between the latest data used for the CPC seasonal forecasts and the start of the first 3-month target season, the effects of different lead times of the ensemble members are expected to be minor in most circumstances, although they may have an impact on early leads.

A retrospective archive of the CFS model is available from three sets of five consecutive daily forecasts initialized near the start, middle, and end of each month between 1981 and 2004. Beginning in August 2004, the CFS model became operational and was run daily, so the ensemble was obtained from the 15 most recent daily runs available early in each month. Lead time is defined as the approximate amount of time, in months, between the data used for the latest CFS model run and the start of the target period. Three-month averages (referred to here as seasonal averages) of SSTs are formed from the monthly means from the CFS.

We have translated the CFS ensemble forecasts for 3-month-mean Niño-3.4 SSTs into a cumulative probability distribution function (CPDF) in a standardized format for ease of data handling. The standard format expresses values of SST that are expected to be equal to or exceed 2%, 5%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 95%, and 98% of the time, so that the forecast precision is the same regardless of how many ensembles were used, or how the data distribution was obtained.

The continuous ranked probability score (CRPS) was used to verify the probabilistic forecasts (Matheson and Winkler 1976; Hersbach 2000). The CRPS measures the squared difference between the forecast CPDF and the CPDF of the observation over the complete range of the observations, −∞ < x < ∞, where x refers to the range of the forecast values. Note that the CPDF of the observation, y, is simply F(x) = 0 for x < y and F(x) = 1 for x ≥ y. A CRPS skill score (CRPSS) was formed by comparison with the scores obtained from climatological probabilities, CRPS_CL:

View Expanded

The CPDF for the standard forecast format was produced by assuming a linear increase in probabilities between the specified probability values (2%–98%). The distribution tails were supplied by assigning points representing the values where the CPDF reaches 0 and 1. These points were assigned a value that minimizes the CRPS for a linearly increasing CPDF outside of the forecaster-specified interval, assuming that the actual distribution of observations outside the interval is Gaussian.

A CPDF forecast was generated from the ensemble by three different methods. For one method, the ensembles were translated directly into CPDF form assuming a linear increase between the N-ordered ensemble members. The CFS prediction was assumed to be at the median of that member’s forecast distribution, with each member representing 1/N of the total. The tails of the distribution were obtained by applying the CRPS-minimizing linearly increasing CPDF to the ends of the distribution as defined above (see Fig. 2). The piecewise linear CPDF obtained from the N ensemble members was then interpolated to the standard format and the CRPS was computed from that forecast as described above. We referred to these as the “original” forecasts.

A second method of translation used only the information in the forecast ensemble means, processed by standard linear regression (REG) as described in section 2a with the CPDF obtained from a high-resolution integration of the PDF and then expressed in standard format. In the third method, the EREG method was applied to individual ensemble members as described in sections 2c and 2d.

We processed the historical forecasts for both the REG and EREG using cross validation (Michaelsen 1987), in which each target year was removed from the equation development sample, together with two additional years, chosen randomly. Climatology for a given cross-validation trial was always computed from the 30 most recent remaining years.

b. Results

Results for all forecasts made between January 1982 and December 2005 are shown in Table 1. The original CFS ensemble members are not competitive with the regression-calibrated forecasts in the CRPS evaluation at any lead time. Differences in skill between the two regression-based postprocessing methods are very small, which is to be expected since they are both based on the same forecast.

The differences in CRPSS between the various methods were tested for significance. The score differences between the models exhibit much less month-to-month variability than the scores themselves do and they have some month-to-month dependence. A rough estimate of the effective sample size was obtained by examination of lag correlations (Thiébaux and Zwiers 1984) and suggests that an effective sample size of about 100 might be appropriate for these data. While accurate assessment of significance would require Monte Carlo resampling tests, these rough tests indicate that the differences between the scores for REG and EREG are not significant at the 5% level at any lead time.

Forecasts were translated into the probabilities that the observation would fall within one of three categories: below, near, or above normal Niño-3.4 SSTs based on the lower, middle, or upper third of the climatological distribution, respectively. This is a common format for seasonal forecasts and is measured here by a three-category ranked probability skill score (RPSS; Epstein 1969b; Murphy 1970) (see the columns under RPSS-3 in Table 1). Results hint that the EREG is slightly favored over REG on lead 1 and beyond when measured by RPSS-3, although again, differences are not significant at the 5% level.

Even though these results show the EREG and REG to be nearly identical for most lead times, EREG makes more direct use of the ensembles and, thus, better represents the information from the CFS. There is some suggestion that EREG improves the three-category probabilities.

The reliability diagrams for the 0- and 5-month lead forecasts are shown in Fig. 3. Because forecasts were generated from a variable width interval with fixed probability thresholds, the sample size is the same for all probability bins in Fig. 3. This is in contrast with most reliability diagrams in the literature, which show the reliability for specific events, and therefore, some bins have more data than others. Reliability diagrams applied to forecasts specified for fixed probability thresholds effectively measure the same information contained in ranked histograms (Anderson 1996; Talagrand et al. 1997; Hamill and Colucci 1997) and are subject to many of the same cautionary issues in their interpretation as discussed by Hamill (2000).

The reliability of the forecasts shows that the poor performance of the original CFS is due primarily to poor calibration leading to an overconfident forecast (a slope of less than one crossing the perfect reliability line near the median). Both regression methods produced reliable probabilities as evidenced in Fig. 3.

The CRPSS for varying K (2.15) for leads of 1 and 5 stratified by initial time are displayed in Figs. 4 and 5, respectively. Here, K expresses the ratio of the ensemble spread between the transformed and original ensembles prior to the regression. Values summarized in Table 1 and Fig. 3 are for K = 1, although the yearly average of the data in Fig. 3 is slightly different from the corresponding numbers in Table 1 because this sample includes additional data from February–December 1981. Results displayed are for Niño-3.4 SST forecasts initialized in the months of December, January, or February (D, J, or F); March, April, or May (M, A, or M); June, July, or August (J, J, or A); and September, October, or November (S, O, or N). In general, the CRPSS remains nearly constant from K = 0 to about K = 0.8, and then falls steadily until K = K_max [see (2.16)]. The value of K_max varied for each initial month and was usually between 1.5 and 2 for these data. Where K in Figs. 4 and 5 exceeded K_max, scores were obtained from a kernel width of near zero, duplicating the results obtained from the K value where the kernel width first approached zero. The CPDF for K = K_max is a step function increasing about 1/N each time an ensemble member’s forecast value is passed. Note that the scale of the plots in Figs. 4 and 5 varies according to the CRPS score range, and that the scores are not dramatically lower than the optimum even for large K. Because the forecast when K = K_max is essentially a series of calibrated point forecasts, a comparison of these values with the original forecasts (orig in Table 1) shows the benefits of using calibration as opposed to the benefit kernel smoothing. The calibrated ensembles are considerably better than the original forecasts, with the yearly average CRPSS for 1- and 5-month leads of 0.535 and 0.274, respectively, compared to corresponding values for the original (uncalibrated) ensemble in this sample of 0.410 and 0.184.