Combining Spatial Statistical and Ensemble Information in Probabilistic Weather Forecasts

1. Introduction

Ensemble prediction systems have been developed to generate probabilistic forecasts of weather quantities that address the two major sources of forecast uncertainty in numerical weather prediction: uncertainty in initial conditions, and uncertainty in model formulation. Originally suggested by Epstein (1969) and Leith (1974), ensemble forecasts have been operationally implemented on the synoptic scale (Toth and Kalnay 1993; Houtekamer et al. 1996; Molteni et al. 1996) and are under development on the mesoscale (Stensrud et al. 1999; Wandishin et al. 2001; Grimit and Mass 2002; Eckel and Mass 2005). In a wide range of applications, probabilistic forecasts based on ensembles provide higher economic and societal value than a single deterministic forecast (Richardson 2000; Palmer 2002; Gneiting and Raftery 2005).

While showing significant spread–error correlations, ensemble forecasts are often biased and underdispersive (Buizza 1997; Hamill and Colucci 1997; Grimit and Mass 2002; Scherrer et al. 2004; Eckel and Mass 2005). Hence, to realize the full potential of an ensemble forecast it is necessary to apply some form of statistical postprocessing, with the goal of generating probabilistic forecasts that are calibrated and yet sharp. In the spirit of the pioneering work of Glahn and Lowry (1972), who introduced regression-type model output statistics approaches to a meteorological audience, various statistically based ensemble postprocessing techniques have been proposed. In this paper, we introduce a postprocessing technique that combines two of these methods, Bayesian model averaging (Raftery et al. 2005) and the geostatistical output perturbation technique (Gel et al. 2004a), to generate calibrated probabilistic forecasts of whole weather fields simultaneously, rather than just weather quantities at individual locations.

Bayesian model averaging (BMA) is a statistical technique originally developed for social and health science applications in situations with several competing statistical models (Hoeting et al. 1999). Raftery et al. (2005) proposed the use of BMA to calibrate forecast ensembles and generate predictive probability density functions (PDFs) for future weather quantities. The BMA predictive PDF is a weighted average of predictive PDFs associated with each individual ensemble member, with weights that reflect the member’s relative skill. However, each location in the forecast domain is considered individually, and spatial correlations among errors are ignored.

The geostatistical output perturbation (GOP) method dresses a single deterministic weather field forecast with simulated error fields, to obtain statistical ensembles of weather fields that take spatial correlations into account (Gel et al. 2004a). This resembles the perturbation approach in Houtekamer and Mitchell (1998, 2001), but in the GOP technique spatially correlated perturbations are applied to the outputs of numerical weather prediction models, rather than the inputs.

In essence, the BMA technique honors ensemble information but ignores spatial correlation. The GOP method takes spatial dependencies into account, but applies to a single deterministic forecast, rather than to an ensemble of weather field forecasts, and fails to honor the flow-dependent spread that derives from the nonlinear evolution of the atmosphere and is characteristic for dynamical ensembles.

Spatial BMA addresses these shortcomings by combining the two techniques. As in the original BMA technique, the spatial BMA predictive PDF is a weighted average of forecast PDFs centered at bias-corrected versions of the ensemble member models, with weights that relate to each member’s performance. However, in spatial BMA the forecast PDFs are multivariate densities with covariance structures designed to honor the spatial structure of weather observations. The spatial BMA technique can be used to generate statistical ensembles of whole weather fields simultaneously, of any size, and at minimal computational cost. At any location individually, spatial BMA reduces to the original BMA technique.

The paper is organized as follows. In section 2, we review the BMA and GOP methods and describe the spatial BMA technique in detail. In section 3 we give an example of spatial BMA forecasts of surface temperature over the North American Pacific Northwest, using the University of Washington mesoscale ensemble (Grimit and Mass 2002; Eckel and Mass 2005). Section 4 presents verification results for spatial BMA forecasts in the calendar year 2004, focusing on spatial and composite quantities. The paper ends with a discussion in section 5, in which we compare the spatial BMA technique to the dressing approaches of Roulston and Smith (2003) and Wang and Bishop (2005).

2. Methods

We now describe the BMA, GOP, and spatial BMA techniques, and we explain our approach to parameter estimation.

a. Bayesian model averaging

We consider an ensemble of K weather field forecasts. In our examples, this is the eight-member University of Washington mesoscale ensemble (UWME; Eckel and Mass 2005), but BMA applies to all forecast ensembles with physically distinguishable member models, such as the poor person’s or multimodel ensembles. With small modifications, BMA also applies to ensembles with exchangeable members, including bred and singular-vector ensembles (Raftery et al. 2005).

We write y for the weather quantity of interest, and f₁, . . . , f_K for the respective ensemble member forecasts. With each ensemble member, we associate a conditional PDF, g_k(y| f⁰_k), which we interpret as the conditional PDF of y given that member k is the best among the ensemble member forecasts, as indicated by the superscript. The BMA predictive PDF for the weather quantity then is

View Expanded

where w_k is the probability of ensemble member k being the best. In the implementation of Raftery et al. (2005), which applies to forecasts of surface temperature and sea level pressure, the conditional PDFs are univariate normal densities centered at a linearly bias-corrected forecast. Hence, g_k(y| f⁰_k) is a univariate normal PDF with mean a_k + b_kf_k and standard deviation σ₀, assumed to be constant across ensemble members. We denote this situation by

View Expanded

The BMA weights in (1) and the bias and variance parameters in (2) are estimated from training data using a two-stage procedure. The bias parameters a_k and b_k are estimated for each ensemble members separately via linear least squares regression: they are the values of a_k and b_k that minimize the residual sum of squares over the entire domain. As such, they are domain specific and do not vary with location. The BMA weights w₁, . . . , w_k and the BMA variance σ²₀ are estimated simultaneously for all the K ensemble members using the Expectation Maximization (EM) algorithm (Dempster et al. 1977). The BMA weights reflect the relative performance of the ensemble member models during the training period; since they are probabilities, they are nonnegative and their sum is equal to 1. The best member interpretation is intuitively appealing, but it should be noted that the model in (1) is also a mixture model, where w_k represents the weight of the kth mixture component.

The BMA method as specified by (1) is implemented in the ensembleBMA package for the R language (Ihaka and Gentleman 1996), which is available online at http://cran.r-project.org.

In its original formulation, BMA yields predictive PDFs for one location at a time, and thus ignores correlations between the errors in forecasts of the same weather quantity at different locations. Typically, however, there are strong spatial correlations between these errors. If we seek the predictive PDF of a spatially aggregated quantity such as the average or minimum temperature across a region, then the spatial correlation is important. One way to proceed using BMA output would be to obtain the predictive PDF assuming that forecast errors at different locations are statistically independent, and thus uncorrelated. However, this would give an erroneous predictive PDF for an aggregated quantity if the spatial correlation was strong, as it often is.

b. The geostatistical output perturbation technique

The GOP technique dresses a single deterministic weather field forecast with Gaussian error fields that are generated using geostatistical methods (Gel et al. 2004a). Here, the deterministic weather field forecast is taken to be a member of the dynamical ensemble.

Specifically, let S denote a possibly large but finite set of distinct model grid points or scattered observation sites. If our intention is to produce postage stamp maps of weather field forecasts, this set is the model grid. For verification purposes, it is a collection of observation locations, and the forecasts are bilinearly interpolated from the model grid to the observation sites. We write

View Expanded

for the weather field at the sites of interest, and F_k = {f_k(s): s ∈ S} for the corresponding deterministic weather field forecast. The GOP technique employs a statistical model, which assumes that

View Expanded

where 1 is the vector with all components equal to 1. The right-hand side of (3) denotes a multivariate normal PDF centered at the bias-corrected member forecast, a_k1 + b_kF_k, with covariance matrix Σ_k, whose entries are specified in (4). Superficially, one might think of (3) as a spatial version of (2), but the relationships differ fundamentally: in (3), we consider F_k as a single deterministic forecast without reference to any of the other ensemble members; in (2), we consider f_k(s) conditionally on this member being the best among the ensemble member forecasts. This latter assumption of forecast k being the best generally implies a deflated variance in (2), when compared to (3), as will be seen below. For surface temperature and sea level pressure, the use of a multivariate normal PDF seems reasonable as an approximation, but this may not be true for other weather variables, such as precipitation or wind speed.

From here on, we refer to the difference between the observation and the bias-corrected forecast as the error. The covariance matrix in (3) describes the spatial structure of the error field and needs to be estimated from training data. Gel et al. (2004a) used a parametric, stationary, and isotropic geostatistical model, which assumes that the (i, j)th element of the covariance matrix Σ_k is

View Expanded

where ||s_i − s_j|| denotes the Euclidean distance between the respective locations, s_i and s_j, and δ_ij equals 1 if s_i = s_j and is 0 otherwise. In geostatistical terminology, ρ²_k is called the nugget effect and represents the variance of the measurement error as well as small-scale variability, ρ²_k + τ ²_k is known as the sill, and r_k is called the range and indicates the rate at which the spatial correlations of the errors decay (Cressie 1993; Chilès and Delfiner 1999). In meteorological terminology, measurement error is often referred to as instrument error, and representativeness errors correspond to small-scale variability. Covariance structures that are more complex can be accommodated, and we discuss some of the options in section 5.

Note that (3) and (4) give a fully specified, multivariate normal predictive PDF for the weather field Y. To generate statistical ensembles from this PDF, we express (3) and (4) in the form of the stochastic representation

View Expanded

where F_k is the deterministic weather field forecast, a_k and b_k are scalar bias parameters, and E_1k = {ϵ_1k(s): s ∈ S} and E_2k = {ϵ_2k(s): s ∈ S} are independent random vectors with mean zero, satisfying

View Expanded

and cov[ϵ_2k(s_i), ϵ_2k(s_j)] = ρ²_kδ_ij, respectively. In this representation, E_1k is a spatially correlated error field that varies continuously with distance, and we refer to it as the continuous component of the error field. In contrast, E_2k is a noise vector that stands for instrument and representativeness errors, and we refer to it as the discontinuous component of the error field. Statistical GOP ensembles of any size can be obtained by simulating E_1k and E_2k from their respective multivariate PDFs, and adding the simulated errors to the bias-corrected forecast, as directed by (5). For the simulations, we use the circulant embedding technique (Wood and Chan 1994; Gneiting et al. 2006) as implemented in the RandomFields package for the R language (Schlather 2001). The GOP method is itself implemented in the ProbForecastGOP package for the R language. (All R packages are available online at http://cran.r-project.org.)

c. Spatial BMA

We now show how to combine the BMA and GOP methods into the spatial BMA technique. Again, we consider a weather field Y = {Y(s): s ∈ S} at a possibly large but finite collection S of locations, but now conditionally on an ensemble,

View Expanded

of K weather field forecasts simultaneously, rather than just a single deterministic weather field forecast. The spatial BMA predictive PDF for the weather field is

View Expanded

where w_k is the BMA weight, equal to the probability that member k is the best among the ensemble member forecasts, and g_k(Y|F⁰_k) is the conditional PDF of Y given that member k is the best, as indicated by the superscript. In our implementation, the conditional PDFs are multivariate normal densities centered at the bias-corrected ensemble member forecast, a_k1 + b_kF_k, and having a spatially structured covariance matrix, Σ⁰_k. By analogy to (2), we denote this situation by

View Expanded

In (7),

View Expanded

where σ²₀ is the BMA variance in (2), Σ_k is the spatially structured GOP covariance matrix with entries specified in (4), and ρ²_k and τ ²_k are the respective GOP covariance parameters. The quantity

View Expanded

is the ratio of the BMA variance to the GOP variance for the errors, and we call it as the deflation factor for member model k, where k = 1, . . . , K. Spatial BMA generalizes both the original BMA method and the GOP technique: it reduces to the former when the set S consists of a single location only, and it reduces to the latter for an ensemble of size K = 1, that is, a deterministic weather field forecast.

Similarly to GOP, the spatial BMA Eqs. (6)–(8) give a fully specified, multivariate predictive PDF for the weather field. However, it is more practical to generate a statistical ensemble of weather field forecasts, by sampling from the spatial BMA predictive PDF. Conditionally on ensemble member k being the best, we can write (7) as

View Expanded

where E⁰_1k and E⁰_2k denote the continuous and the discontinuous parts of the conditional error field, respectively, with multivariate normal PDFs equal to those described in section 2 for the unconditional counterparts, E_1k and E_2k, except that the covariance matrix is rescaled by the deflation factor, α_k.

The following algorithm generates a member of the spatial BMA ensemble:

1. Sample a number k ∈ {1, . . . , K}, with probabilities given by the BMA weights, w₁, . . . , w_K. This specifies the member of the dynamical ensemble to be dressed.

2. Simulate realizations of the continuous and discontinuous parts, E⁰_1k and E⁰_2k, of the conditional error field from the respective conditional PDFs.

3. Use the right-hand side of (9) to dress the bias-corrected weather field forecast, a_k1 + b_kF_k, with the simulated conditional error fields, E⁰_1k and E⁰_2k.

Proceeding in this manner, we obtain spatial BMA ensembles of weather field forecasts, of any desired ensemble size, and at minimal computational cost.

d. Parameter estimation

The estimation of a spatial BMA model for an underlying dynamical ensemble requires the fitting of a BMA model as well as GOP models for the individual ensemble members. This is done using prior observations and ensemble forecasts for the same prediction horizon and forecast cycle, with forecasts that are bilinearly interpolated from the model grid to the observation sites. We use a sliding training period consisting of the recent past. In deciding how long this training period should be, there is a trade-off: with a short training period the method adapts more quickly to changes in the ensemble and in its component members as well as seasonal changes. With a longer training period, on the other hand, estimation tends to be less variable. Raftery et al. (2005) showed that for 48-h BMA forecasts of surface temperature in the North American Pacific Northwest there are substantial gains in increasing the length of the training period to 25 days, but there is little gain beyond. In the examples below, we adopt this choice of a sliding 25-day training period. Other weather variables, domains, and forecast lead times may require different choices.

To fit the BMA models (1) and (2), we follow Raftery et al. (2005) in estimating the bias parameters, a_k and b_k, by linear least squares regression of the observations on the respective ensemble member forecast. The BMA weights, w_k, and the BMA variance, σ²₀, are estimated using the maximum likelihood technique in the form of the EM algorithm (Dempster et al. 1977). The estimate of the BMA variance σ²₀ is then refined by searching numerically for the value of σ²₀ that minimizes the continuous ranked probability score (CRPS; Hersbach 2000; Gneiting et al. 2005; Wilks 2006, his section 7.5.1) of BMA over the training data. This is done keeping all the other BMA parameters fixed, while searching over a range of values of σ²₀ centered around the maximum likelihood estimate given by the EM algorithm.

It remains to fit the GOP models for the weather field forecasts using member model k, where k = 1, . . . , K. Estimation of the spatial covariance parameters, ρ²_k, τ ²_k and r_k in (4), is based on the fact that the GOP error field, ϵ_k(s) = ϵ_1k(s) + ϵ_2k(s), satisfies

View Expanded

where E denotes expectation. In geostatistical language, the error field has variogram

View Expanded

where d = ||s_i − s_j|| denotes the Euclidean distance between two distinct observation sites, and γ_k(d) is one-half the expected squared difference between errors at stations that are distance d apart.

We now compute the sample version of the variogram, γ̂_k(d), using data from the sliding training period, as follows:

1. Use the estimates of the bias-correction terms a_k and b_k previously obtained by fitting a linear least squares regression to the data from the 25-day training period.

2. For each day in the training period, find the empirical error field, by subtracting the bias-corrected forecast field from the corresponding field of verifying weather observations.

3. For each day in the training period, and for all pairs of observation locations on that day, find the distance between the sites, and compute one-half the squared difference between the errors.

4. Group the distances into bins B_l with midpoints d_l.

5. Compute the empirical variogram value γ̂_k(d_l) at distance d_l, by averaging the respective one-half squared differences over the distance bin B_l.

With this, we apply the weighted least squares technique to estimate the GOP parameters. Specifically, if n_l denotes the total number of pairs of observation sites whose distance falls into bin B_l, the weighted least squares estimates of the covariance parameters ρ²_k, τ ²_k and r_k are the values that minimize

View Expanded

To solve this optimization problem, we use the quasi-Newton and conjugate-gradient techniques described by Byrd et al. (1995) and implemented in the R language (Ihaka and Gentleman 1996).

Following the estimation of the spatial covariance parameters for ensemble members k = 1, . . . , K, we combine the GOP and BMA models into the spatial BMA model, using (6)–(8). We do the estimation using the previously mentioned R packages, ensembleBMA and ProbForecastGOP.

3. Example

We now give an example of 48-h spatial BMA forecasts of surface temperature over the North American Pacific Northwest, which includes Oregon, Washington, southern British Columbia, and part of the northeastern Pacific Ocean, using the UWME (Grimit and Mass 2002; Eckel and Mass 2005). In the 2004 version used here, the UWME is an eight-member multianalysis ensemble. The members use the fifth-generation Pennsylvania State University–National Center for Atmospheric Research (PSU–NCAR) Mesoscale Model (MM5; Grell et al. 2004) driven by initial and lateral boundary conditions supplied by eight distinct global models. Specifically; the AVN member uses initial and lateral boundary conditions from the Global Forecast System run by the National Centers for Environmental Prediction (NCEP); the CMCG member is based on the Global Environmental Multiscale model run by the Canadian Meteorological Center; the Eta Model member uses the limited-area mesoscale model run by NCEP; the GASP member is based on the Global Analysis and Prediction model run by the Australian Bureau of Meteorology; the JMA member is based on the Global Spectral Model run by the Japan Meteorological Agency; the NGPS member uses the Navy Operational Global Atmospheric Prediction System run by the Fleet Numerical Meteorology and Oceanography Center; the TCWB member is based on the Global Forecast System run by the Taiwan Central Weather Bureau; and the UKMO member derives from the Unified Model run by the Met Office. Eckel and Mass (2005) give a detailed description of UWME.

Our example is for 48-h forecasts of the surface (2 m) temperature field over the North American Pacific Northwest, initialized at 0000 UTC 14 February 2004. To deal with nonstationarities in the error fields, we divided the 12-km UWME forecast grid into two subdomains, land and ocean, and estimated separate spatial BMA models for the two domains, using the aforementioned 25-day sliding training period.

Table 1 shows estimates of the BMA variance, σ²₀, the BMA weights, w_k, and the additive and multiplicative bias, a_k and b_k, respectively, for the eight UWME members. The BMA weights differed substantially between land and ocean. The member with the highest BMA weight on land was the NGPS model, and the CMCG model had the highest weight over the ocean. The GASP and TCWB models performed poorly relative to the other members during the training period and received negligible weights in both domains. The two domains also differed in terms of the BMA variance, which was smaller over the Pacific Ocean, likely because of a decrease in the representativeness error.

Table 2 shows estimates of the GOP covariance parameters, ρ²_k, τ ²_k and r_k, for the error fields, along with estimates of the deflation factor, α_k. The estimates of the nugget effect, ρ²_k, which subsumes instrument and representativeness errors, were much larger on land than over the Pacific Ocean. The estimates of τ ²_k were generally somewhat larger on land than over ocean. The range, r_k, corresponds to the correlation length of the continuous component of the error field, with spatial correlations decaying to about 0.05 at distance 3r_k. The estimates of the range were larger over the ocean than on land, suggesting stronger correlations over water.

The deflation factor α_k reflects the skill of each ensemble member, with the more accurate members receiving the higher estimates. Indeed, if a member model generally performs well, then its unconditional error variance will not be very different from its conditional error variance given that it is the best among the ensemble member forecasts, and its deflation factor will be close to 1. Still, caution is needed in interpreting estimates of deflation factors. For instance, the estimated deflation factors in Table 2 were generally higher on land than they were over the ocean, and the land deflation factor for the NGPS model was larger than 1, counter to intuition. These patterns can be explained by Fig. 1, which illustrates the estimation of the GOP covariance parameters for the NGPS member on land and the CMCG member over the ocean. Each panel shows both the empirical variogram of the error field, composited over the training period, and the fitted exponential variogram. The intercept of the fitted exponential variogram equals the estimate of the nugget effect, ρ²_k, and corresponds to the variance of instrument and representativeness errors. The horizontal asymptote is at the estimated sill, ρ²_k + τ ²_k, and equals the estimated marginal variance of the GOP error field. The weighted least squares technique seems to underestimate the sill for the NGPS member on land, resulting in a deflation factor that exceeds 1.

The exponential variograms fit quite well over the first 400 km, and the fit deteriorates thereafter. This is quite typical of geostatistical applications, and is not a matter of concern. Generally, when fitting a parametric variogram model, attention is focused on the smaller distances, which are particularly relevant in characterizing the spatial statistical properties of the error fields.

Figures 2 and 3 illustrate the generation of a member of the spatial BMA ensemble on land and over the ocean, respectively. Figures 2a and 3a show the bias-corrected member of the dynamical ensemble that is to be dressed. On land, this is the NGPS member, and over the ocean it is the CMCG member. Figures 2b,c and 3b,c show simulated realizations of the continuous and discontinuous components of the error field, respectively. Figures 2d and 3d show the member of the spatial BMA ensemble as the sum of the three components. Repeating this process, statistical ensembles of any size can be generated.

A characteristic feature of Figs. 2 and 3, and in general of the spatial BMA (and GOP) ensemble member fields, is an increase in roughness compared to weather fields generated by numerical weather prediction models. This stems from spatial BMA aiming to reproduce the spatial structure of weather observations, including instrument and representativeness errors, represented by the discontinuous component of the error field (Figs. 2c and 3c). The discontinuous component can be ignored, if desired, and the spatial BMA technique can be implemented by adding the bias-corrected weather field forecast (Figs. 2a and 3a) and the continuous component of the simulated error field (Figs. 2b and 3b) only. This is an implementation decision that needs to be made depending on the prediction problem at hand. In our implementation, we added both components of the error to the bias-corrected forecast field.

The continuous component of the error field generally contributes more than the discontinuous component, since the estimates of the covariance parameter τ ²_k, which represents the marginal variance of the continuous component, are substantially larger than the estimates of the nugget effect ρ²_k, the marginal variance of the discontinuous component.

For each region, we generated a spatial BMA ensemble of 19 weather fields. These could be displayed in the form of a postage stamp plot, but this would be likely to overwhelm users, and plots that summarize the spatial BMA ensemble are likely to be more useful. Ensemble forecasts for all types of composite quantities can be derived from the statistical ensemble. For instance, we might be interested in predicting the empirical variogram of the temperature field verifying at 0000 UTC 16 February 2004. We computed the empirical variogram for each of the 19 members of the spatial BMA ensemble, using 300 distance bins. At each bin, the minimum and the maximum of the respective 19 values envelop a 95% prediction interval for the verifying variogram value, which we computed from the observed temperature field. Figure 4 shows the results of this experiment. The prediction intervals generally cover the verifying empirical variogram values.

4. Verification results

In calendar year 2004, the 0000 UTC cycle for the 12-km domain of the eight-member UWME was run on 245 days. For each day, we fitted BMA, GOP, and spatial BMA models for 48-h forecasts of surface (2 m) temperature over the North American Pacific Northwest, separately on land and over the ocean, using a sliding 25-day training period. We then generated original BMA, GOP, and spatial BMA forecast ensembles for each day. The original BMA ensembles were created by sampling from the univariate original BMA predictive PDFs at each location separately, incorrectly assuming spatial independence of the error fields. The GOP ensembles were based on the UWME UKMO model, which had the best aggregate performance among the ensemble member models, both on land and over the ocean. In the interest of a fair comparison to the eight-member UWME, our GOP, original BMA, and spatial BMA ensembles also had eight members only. However, the statistical ensembles allow for ensembles of any size, and larger ensembles frequently show better verification results.

We now assess and rank the performance of the UWME, GOP, original BMA, and spatial BMA ensembles, emphasizing spatial and composite quantities. On average, observations of surface temperature were available at 761 stations on land and 196 stations over the Pacific Ocean. We verified bilinearly interpolated ensemble forecasts against the temperature observations.

In contrast to the statistical ensembles, UWME is not designed to take instrument and representativeness errors into account. Hence, we consider a fifth ensemble, which we call UWME + noise. To create the UWME + noise ensemble, we added Gaussian noise to each of the eight UWME members, at each site independently, with mean zero and a variance that equals the estimated nugget effect, ρ²_k, for the corresponding member model.

a. Temperature forecasts at individual sites

We begin by assessing surface temperature forecasts at individual sites. For forecasts at single sites, spatial BMA, and original BMA are equivalent; hence the results for the two ensembles are essentially identical, with any differences due to chance variability in the generation of the ensemble members. All verification statistics were spatially and temporally composited over the Pacific Northwest and the 2004 calendar year.

Table 3 shows the mean absolute error (MAE) and the average CRPS (Hersbach 2000; Gneiting et al. 2005; Wilks 2006, his section 7.5.1) for the various ensemble methods. The MAE assesses the accuracy of the deterministic forecasts. The UWME and UWME + noise deterministic forecast is the raw ensemble mean; for the GOP method it is the bias-corrected UWME UKMO forecast; and for the original BMA and spatial BMA techniques it is a weighted average of the bias-corrected ensemble member forecasts. The CRPS is a scoring rule for predictive PDFs that addresses calibration as well as sharpness, and is proper, that is, discourages hedging. The CRPS generalizes the absolute error, to which it reduces for deterministic forecasts; it is also reported in degrees Celsius, and average CRPS values can be directly compared to the MAE (Gneiting et al. 2005). A clear rank order can be observed, in that the BMA ensembles showed substantially lower CRPS values than the GOP ensemble, followed by the UWME + noise and UWME ensembles.

To assess the calibration of the ensemble forecasts, we use the verification rank histogram (Anderson 1996; Talagrand et al. 1997; Hamill and Colucci 1997; Hamill 2001). Figure 5 shows the histograms for the various ensembles. We also computed the respective discrepancy from uniformity,

View Expanded

where K = 8 is the number of ensemble members and p_j is the observed relative frequency of rank j. The smaller the discrepancy, the smaller the deviation from a uniform rank histogram, and the better the calibration. In both domains, the UWME and UWME + noise ensembles were underdispersive, while the GOP, original BMA, and spatial BMA ensembles had rank histograms that were nearly uniform. The slight overdispersion of the rank histograms of the original and spatial BMA ensembles over ocean can be attributed to the smaller number of cases available over ocean, compared to land.

5. Discussion

We have introduced the spatial BMA method, a statistical postprocessing technique for calibrating forecast ensembles of whole weather fields simultaneously. Spatial BMA generalizes and combines Bayesian model averaging (BMA) and the geostatistical output perturbation (GOP) technique, and it honors ensemble as well as spatial statistical information. The spatial BMA predictive PDF for the weather field is a weighted average of multivariate normal PDFs centered at bias-corrected members of the dynamical forecast ensemble. At any single location, spatial BMA reduces to the original BMA technique. It is computationally inexpensive and can be used to generate statistical ensembles of any size.

In experiments with the University of Washington mesoscale ensemble, the Spatial BMA ensemble compared favorably to the raw dynamical ensemble, the raw ensemble with added observational noise, the GOP ensemble, and the original BMA ensemble. In particular, the minimum spanning tree histogram, a key tool in assessing the calibration of ensembles of weather field forecasts (Smith and Hansen 2004; Wilks 2004), was closest to being uniform for the spatial BMA ensemble. For forecasts of composite quantities, such as temperature averages over NWS forecast zones and minimum temperature along the Cascades corridor of Interstate 90, the GOP ensemble and the spatial BMA ensemble showed similar performances, and outperformed the other types of ensembles. While our experiments were with surface temperature fields, spatial BMA in its present form applies to all weather variables with forecast error distributions that are approximately Gaussian, including sea level pressure. Further research is needed to extend spatial BMA to other weather variables, such as precipitation or wind speed. Sloughter et al. (2007) presented a non-Gaussian version of BMA that yields calibrated quantitative probabilistic precipitation forecasts at individual sites, but not for weather fields.

There are several directions in which the spatial BMA technique could be developed. One is bias correction. In the current implementation, we use a simple linear bias correction that does not take altitude, land use, latitude, longitude, or distance from the ocean into account. More sophisticated regression based bias removal techniques might include some or all of these quantities as predictor variables. Another possibility is to use a nearest-neighbor approach based on distance, altitude, and land use categories.

Another possibility would be to reduce the bias correction to a simple additive bias correction, including only the term a_k and fixing b_k to 1. This would reduce the number of parameters to estimate. In their implementation of the original BMA technique for the Canadian Ensemble System, Wilson et al. (2007) found that for training periods of up to 50 days and for forecast lead times up to 7 days, using only a_k in the bias removal step performed as well as using both a_k and b_k.

Another way to reduce the number of parameters to estimate on each day would be to estimate the covariance parameters for each ensemble member only once, using data from a previous year. The regression parameters a_k and b_k, the BMA weights, w₁, . . . , w₈, and the BMA standard deviation σ₀ would still have to be estimated using the training period, and the new estimate of the BMA variance would be employed to compute the deflaction factor α_k for each ensemble member.

In modeling the covariance structure of the error fields, we used a stationary and isotropic exponential correlation function. There are several ways in which more complex, and potentially more realistic, covariance structures could be used. Stationary and isotropic correlation functions that are more versatile than an exponential function are available (Mitchell et al. 1990; Gneiting 1999). Anisotropic covariance structures could also be used (Purser et al. 2003); however, in the case of surface temperature over the Pacific Northwest, Gel et al. (2004b) did not find any significant differences between longitudinal and latitudinal empirical variograms of the forecast error fields. Finally, nonstationary covariance models, that is, models that are not translation invariant, could be used. In our experiments, we dealt with nonstationarities between the land and the ocean by fitting and generating two distinct spatial BMA ensembles, each of which used a stationary and isotropic covariance structure. This was a fairly simple way to resolve nonstationarities, and yet produced good results. The methods of Paciorek and Schervish (2006) could be used to fit valid covariance structures that are stationary on homogeneous domains, yet nonstationary globally, thereby allowing for the generation of a single spatial BMA ensemble over all domains simultaneously, without incurring discontinuities along the boundaries.

An issue not explicitly considered in the spatial BMA approach is that of phase or displacement errors. These could perhaps be addressed by partitioning the errors of the ensemble member weather field forecasts into displacement, distortion, amplitude, and residual fields, as in Du et al. (2000), and applying the spatial BMA technique to the residual component only, while developing parametric statistical models for displacement, distortion, and amplitude errors. This would be an interesting avenue for future research, with potential rewards in the form of sharper yet calibrated forecast PDFs, but may require impracticably large sets of training data.

Another issue that calls for discussion is the choice of the training period. In the current implementation, we use forecast and observation data from a sliding window consisting of the 25 most recent days available to estimate the spatial BMA parameters. This allows the method to adapt rapidly to seasonal changes in the atmosphere as well as changes in the design of the ensemble, but limits the availability of training data. However, even with limited data, we did not have the problem of overfitting when estimating all the spatial BMA parameters, as our results on the quality of the spatial BMA out-of-sample predictions indicate. In addition, the estimates of the parameters vary smoothly with time, and the variability from day to day is not great. A potential way of increasing the amount of training data is to also use training data from the same season in previous years; this could be done using ensemble reforecasts, as proposed by Hamill et al. (2004). However, reforecasts put high demands on computational and human resources, and they were not available to us.

We close by comparing spatial BMA to other ensemble postprocessing techniques. Wilks (2002) proposed fitting mixtures of multivariate normal densities to ensemble forecasts of multivariate weather quantities. This resembles the spatial BMA technique, but does not take bias and calibration adjustments into account. Roulston and Smith (2003) proposed combining statistical and dynamical ensembles, and suggested the use of hybrid ensembles, in which the members of the dynamical ensemble are dressed with errors drawn from an archive of the best member errors. A difficulty in this approach is the identification of the best members. Wang and Bishop (2005) showed that under a wide range of scenarios the best member dressing method fails to be calibrated. They proposed a modified dressing technique, in which statistical perturbations are generated, with flexible covariance structures that are estimated from training data. This is similar to the spatial BMA technique, in that the Wang and Bishop (2005) predictive PDF is also a weighted average of multivariate normal densities, each centered at a bias-corrected member of the dynamical forecast ensemble, but the weights are all equal and do not depend on the member’s skill. Fortin et al. (2006) proposed dressing kernels that depend on the rank of the member within the ensemble. This method is tailored to ensembles with exchangeable members, as opposed to the University of Washington mesoscale ensemble, or any poor person’s ensemble, for which the members are not exchangeable. Raftery et al. (2005, p. 1170) discuss an adaptation of BMA to the case of exchangeable members.