a. Bias correction

Systematic biases are substantial in dynamic modeling systems (Atger 2003; Mass 2003), and bias correction is an essential and well-established step in weather forecasting. The predominant approach is based on regression, using model output statistics (MOS) schemes based on multiple linear regression for linear variables, such as temperature or pressure, and logistic regression for binary variables, such as precipitation occurrence or freezing (Glahn and Lowry 1972; Wilks 2006a). For wind, the traditional approach is to develop separate MOS equations for the zonal and meridional components, and derive single-valued forecasts of wind speed and wind direction from them. However, this does not take the dependencies between the wind components into account and can lead to biases (Carter 1975; Glahn and Unger 1986). Thus, we take a different approach to predicting wind direction, and propose the use of a state-of-the-art circular–circular regression technique.

Specifically, let f and υ denote the predicted and observed wind direction, respectively. Let θ( f ) and θ(υ) denote the associated points on the unit circle in the complex plane, as described above. Downs and Mardia (2002) and Kato et al. (2008) propose a regression equation of the form

View Expanded

where β₀ is a complex number with modulus |β₀| = 1, β₁ is any complex number and the bar denotes complex conjugation. The mapping from θ( f ) to θ(υ) is a Möbius transformation in the complex plane, which is one-to-one, and maps the unit circle to itself. The regression parameters β₀ and β₁ need to be estimated from training data. While β₀ is a rotation parameter, β₁ can be interpreted as pulling a direction toward a fixed angle, namely the point β₁/|β₁| on the unit circle, with the concentration about this point increasing as |β₁| increases, except that the antipodal point remains fixed (Kato et al. 2008). Figure 1 provides an illustration. Figure 1a shows twelve equally spaced nontransformed angles. Figure 1b illustrates the Möbius transformation (2) when β₀ = 1 (i.e., no rotation) and β₁ = 0.3i (i.e., a contraction toward the imaginary unit in the complex plane, which corresponds to a direction of 90°). In Fig. 1c, the parameter values are β₀ = 1 and β₁ = 0.6i, resulting in a stronger contraction. Figure 1d uses β₀ = −i and β₁ = 0.6i, thereby compositing the contraction with a counterclockwise rotation by 90°.

In our circular–circular regression approach to bias correction for wind direction, we estimate the Möbius transformation (2) from training data by numerically minimizing the sum of the circular distances between the fitted bias-corrected forecasts and the respective verifying directions as a function of the regression parameters.

For comparison, we consider two reference techniques. The first is median-angle correction, which arises as the special case of circular–circular regression in which the parameter β₁ = 0 is fixed. Then the Möbius transformation (2) is simply a rotation. In our minimum circular distance approach to estimation, the rotation parameter β₀ becomes the circular median of the directional errors in the training data. The second reference technique is mean-angle correction; that is, a rotation by the circular mean of the directional errors in the training data.

Recall here that if the training data comprises the pairs ( f₁, υ₁), … , ( f_n, υ_n) of predicted and observed directions, the median of the directional errors is the angle m that minimizes

View Expanded

which is equivalent to the definition in Eq. (2.32) of Fisher (1993, p. 36). The circular mean is obtained by forming the vector sum of the directional errors, each of which is represented as a unit vector in the complex plane, and rescaling to the unit circle, or equivalently, by applying Eq. (2.9) of Fisher (1993, p. 31). Lund (1999) proposes a fitting criterion of this latter type in the context of circular regression.

b. Von Mises distribution for angular data

The von Mises distribution is a natural baseline for modeling angular data, such as wind directions, and it may be viewed as a circular analog of the Gaussian distribution (Fisher 1993, p. 49). Specifically, an angular variable is said to have a von Mises distribution with mean direction μ and concentration parameter κ ≥ 0 if it has density

View Expanded

on the circle, where I₀ is a modified Bessel function of the first kind and order zero. As the concentration parameter κ gets close to zero, the von Mises distribution becomes a uniform distribution on the circle. In the appendix, we review maximum likelihood (ML) estimation for the concentration parameter of the von Mises distribution, which can be viewed as a limiting case of Bayes estimation under weak prior information (Guttorp and Lockhart 1988).

c. Bayesian model averaging

During the past decade, the ability of ensemble systems to improve deterministic-style forecasts and to predict forecast skill has been convincingly established (Palmer 2002; Gneiting and Raftery 2005). However, forecast ensembles are typically biased and underdispersive (Hamill and Colucci 1997; Eckel and Walters 1998), and thus some form of statistical postprocessing is required. Wilks (2006b), Wilks and Hamill (2007), and Bröcker and Smith (2008) review and compare techniques for doing this.

Bayesian model averaging (BMA) was introduced by Raftery et al. (2005) as a statistical postprocessing method that generates calibrated and sharp predictive probability density functions (PDFs) from ensemble forecasts. The BMA predictive PDF of any future weather quantity of interest is a weighted average of PDFs associated with the member forecasts, where the weights reflect the members’ predictive skill over a training period. The initial development was for linear weather quantities, such as surface temperature, quantitative precipitation, and wind speed (Raftery et al. 2005; Sloughter et al. 2007; Wilson et al. 2007; Sloughter et al. 2009), for which the component PDFs are probability distributions on the real line. For all variables considered and on both the synoptic scale and the mesoscale, the BMA postprocessed PDFs outperformed the unprocessed ensemble forecast and were calibrated and sharp.

Here we extend the BMA approach to accommodate wind direction, which is an angular variable and thus requires component PDFs that are probability distributions on the circle. Let f₁, … , f_m denote an ensemble of bias-corrected forecasts. We then take the BMA predictive PDF to be a mixture of the form

View Expanded

where the components are von Mises distributions with mean direction f_j and concentration parameter κ_j. The BMA weights w₁, … , w_m are probabilities and so they are nonnegative and add up to 1; that is, . Our standard BMA specification uses a common concentration parameter, so that

View Expanded

The common concentration parameter simplifies and stabilizes estimation and, in our experience with the University of Washington Mesoscale Ensemble over the Pacific Northwest, does not deteriorate the predictive performance appreciably. However, in other types of situations the more general formulation with distinct concentration parameters might be more adequate.

The BMA weights, w₁, … , w_m, and the common concentration parameter κ of the component PDFs are estimated by maximum likelihood from training data. Typically, the training set comprises a temporally and/or spatially composited collection of past, bias-corrected ensemble member forecasts, f_1k, … , f_mk, and the corresponding verifying direction, υ_k, where k = 1, … , n, with n the number of cases in the training set. The likelihood function ℓ is then defined as the probability of the training data, viewed as a function of the w_j’s and κ; that is,

View Expanded

where the product extends over all instances in the training set. The ML estimates are those values of the w_j’s and κ that maximize the likelihood function; that is, the values under which the verifying directions were most likely to materialize.

The likelihood function typically cannot be maximized analytically, and so it is maximized using the expectation-maximization (EM) algorithm (Dempster et al. 1977; McLachlan and Krishnan 1997). The EM algorithm is iterative and alternates between two steps, the E (or expectation) step, and the M (or maximization) step. It uses unobserved quantities z_jk, which can be interpreted informally as the probability of ensemble member j being the most skillful forecast for verification υ_k. The z_1k, … , z_mk are nonnegative and sum to 1 for each instance k in the training set.

In the E step, the z_jk are estimated given the current values of the BMA weights and component PDFs. Specifically,

View Expanded

where the superscript l refers to the lth iteration of the EM algorithm, and thus w_q^(l) and κ^(l) refer to the estimates at the lth iteration. In the M step we obtain updated estimates

View Expanded

of the BMA weights. Furthermore, an updated estimate, κ^(l+1), of the common concentration parameter is obtained by optimizing the expected complete-data log likelihood given the latent variables; that is, by maximizing

View Expanded

over κ ≥ 0. For implementation details, see the appendix.

An ensemble forecast can occasionally be poor, in the sense that all member forecasts turn out to be substantially different from the verifying direction. This possibility motivates a BMA specification with a uniform mixture component. Under this more general specification, to which we refer as BMA⁺, the predictive density becomes

View Expanded

where u is the density of a uniform distribution on the circle; that is, a von Mises distribution with concentration parameter κ = 0, and where the BMA weights are nonnegative and add up to 1, so that . The intention here is similar to that of the addition of a climatological mixture component, as proposed and implemented for linear variables by Rajagopalan et al. (2002) and Bröcker and Smith (2008). The adaption of the EM estimation algorithm from the BMA to the BMA⁺ specification is straightforward.

d. Forecast verification

Wind direction is an angular variable, and standard scoring rules for linear variables do not apply. Instead, we use circular analogs of the absolute error and the continuous ranked probability score, as introduced by Grimit et al. (2006).

From a probabilistic forecast for an angular quantity, we can create a single-valued forecast by determining the circular median of the predictive distribution, as described above and by Fisher (1993, pp. 35–36). To assess the quality of this forecast, we use the mean circular distance or circular absolute error, AE_circ( f, υ), between the single-valued forecast f and the verifying direction, υ, as given by (1) in the unit of degrees.

To assess probabilistic forecasts for an angular quantity, we use the angular or circular continuous ranked probability score, which is defined by

View Expanded

where P is a forecast distribution on the circle, υ is the verifying direction, V and V* are independent copies of an angular random variable with distribution P, and denotes the expectation operator. Note that when P is a uniform distribution on the circle, then CRPS_circ(P, υ) equals 45°, independently of the verifying direction. The circular continuous ranked probability score is proper and reduces to the circular absolute error when the forecast is single-valued, just as the linear continuous ranked probability score generalizes the absolute error (Grimit et al. 2006). It takes the unit of degrees and allows for the direct comparison of deterministic (single-valued) forecasts, discrete ensemble forecasts, and postprocessed ensemble forecasts that can take the form of a predictive density.

The general goal in probabilistic forecasting is to maximize the sharpness of the forecast distributions subject to calibration (Gneiting et al. 2007). For probabilistic forecasts of a linear variable, it is straightforward to assess calibration via rank or probability integral transform (PIT) histograms, and sharpness via the mean width of the corresponding prediction intervals. For an angular variable, such as wind direction, there are no established ways of assessing calibration and sharpness. For example, there is no direct analog of the verification rank or PIT histogram; because there is no natural ordering on the circle if a forecast distribution is uniform or multimodal. Similarly, there is no generally accepted measure of dispersion for circular distributions (Fisher and Lee 1992, p. 666).

To address these challenges, we introduce a novel measure of sharpness for a circular distribution P, namely

View Expanded

where V and V* are independent copies of an angular random variable with distribution P. The quantity S(P) has two complementary interpretations. By the first equality in (9), it equals one-half times the expected circular distance between two directions drawn independently and at random from the forecast distribution. Thus, S(P) is a natural measure of sharpness on the circle, attaining a value of zero for a point measure, and a value of 45° for a uniform distribution. The smaller S(P), the sharper the forecast distribution, and the sharper, the better, subject to calibration.

The second equality in the defining (9) is immediate from (8) and shows that S(P) equals the expected value of the circular continuous ranked probability score when the forecast distribution is P, and the verifying wind direction is drawn at random from this distribution. Thus, if a forecasting method is calibrated, we expect the mean sharpness measure (9) and the mean circular continuous ranked probability score to be roughly equal. Approximate equality can be checked informally or via significance tests, as proposed by Held et al. (2010).

For some predictive distributions, such as mixtures of von Mises densities, the circular continuous ranked probability score cannot be computed analytically. In such cases, we approximate it by simulating a Monte Carlo sample υ₁, … , υ_N from the predictive distribution, and computing

View Expanded

which agrees with (8) when the predictive distribution, P, assigns mass 1/N to each of υ₁, … , υ_N. Similarly, the sharpness measure (9) can be approximated by the rightmost term in (10). In order for the approximation to be accurate, the sample size N needs to be large, and we generally use N = 1 000.

a. The University of Washington mesoscale ensemble

The University of Washington ensemble system is a mesoscale, short-range ensemble based on the fifth-generation Pennsylvania State University–National Center for Atmospheric Research (PSU–NCAR) mesoscale model (MM5; Grell et al. 1995). It forms an integral part of the Pacific Northwest regional environmental prediction effort (Mass et al. 2003). The original five-member mesoscale ensemble was designed as a single-model, multianalysis system that uses MM5 with a nested, limited-area grid configuration focusing on the states of Washington and Oregon (Grimit and Mass 2002). Beginning in the autumn of 2002, the size of the mesoscale ensemble was increased to eight members, using additional global analyses and forecasts, and named the University of Washington Mesoscale Ensemble (UWME; Eckel and Mass 2005). Table 1 shows acronyms and the sources of the initial and lateral boundary conditions for the member forecasts.

The evaluation period of this study begins 1 January 2003 and extends through 31 December 2003, in which the UWME system provided 48-h forecasts beginning at 0000 UTC each day, with the verifying wind directions being recorded 48 h later. Model 10-m wind component forecasts at the four grid-box centers surrounding each station were bilinearly interpolated to the observation location and then rotated from grid-relative to north-relative. No adjustment was made for any vertical displacement of the model surface level from the real terrain. Station-based observations of near-surface wind were acquired in real time from 54 surface airway observation (SAO) stations in the United States and Canada. Our verification results include forecast–observation cases only when the verifying wind speed was at least 5 kt (2.57 m s⁻¹), since wind direction observations are unreliable at lower wind speeds. In view of this constraint, forecast–observation cases at the individual stations were available for a minimum of 201, median of 219, and maximum of 264 days in calendar year 2003.

Before showing composite verification results, we give a specific example of 48-h BMA and BMA⁺ forecasts of wind direction at Castlegar Airport, British Columbia (station code CYCG), valid 0000 UTC 26 August 2003. The member-specific circular–circular regression schemes for bias correction and the BMA parameters were estimated on a 28-day training period immediately preceding the initialization date, using data at Castlegar only. Table 2 shows the eight raw and bias-corrected UWME member forecasts and the respective BMA and BMA⁺ weights for this site and particular training period. The bias correction technique results in member-specific counterclockwise rotations, which range from 2° to 12°. The UKMO member received the highest BMA and BMA⁺ weights, but the weights do not differ much between the ensemble members.

Figure 2 illustrates the raw and bias-corrected UWME ensemble forecasts and the BMA and BMA⁺ density forecasts at Castlegar along with two reference forecasts, to which we refer as climatology and median error climatology (MEC), respectively. The climatology forecast uses the 28 observed wind directions during the sliding training period, giving them equal weights in a discrete probability mass function. This is a short-term climatology, which can adapt to seasonal changes as well as to changes in atmospheric regimes. The MEC technique takes the form of a von Mises density that is centered on the circular median of the bias-corrected ensemble members, with a concentration parameter that is estimated (using ML) on the same 28-day training period as the other methods. This resembles the mean error climatology method of Grimit et al. (2006), but the density is centered on the circular ensemble median, rather than the circular mean, and the estimation method is different. Each panel shows the respective forecast distribution, taking the form of either a discrete probability mass function or a continuous probability density function, along with the verifying wind direction, which was westerly at 280 degrees. The circular continuous ranked probability score (CRPS) is smallest (i.e., best) for the BMA⁺ forecast distribution, at 17.5°, followed by the BMA, MEC, bias-corrected UWME, raw UWME, and climatology forecasts.

b. Bias correction

We turn to composite verification results for bias correction. In section 2a we proposed three methods for bias correcting angular variables, namely circular–circular regression, which employs a state of the art regression approach tailored to circular data, and two benchmarks, median-angle correction and mean-angle correction. As noted before, we fit the bias correction schemes for each ensemble member individually.

There are two choices to be made here, namely about the method used and the length of the sliding training period. Table 3 shows the mean circular absolute error for each method, averaged over the eight UWME member forecasts, calendar year 2003, and the 54 stations we consider, for sliding training periods that range from 7 to 42 days. In choosing the length of the training period, there is a trade-off, and no automatic way of making it. Both weather patterns and model specifications change over time, so that there is an advantage in using a short training period to adapt to such changes. On the other hand, the longer the training period is, the less the estimation variance. The training sets are constrained to cases at the location at hand, and the periods are extended if there are missing data. For example, the 7-day training period always uses the seven most recent available forecast cases.

At a 7-day training period the simpler methods outperform the more complex method, namely circular–circular regression. However, as the training period grows, circular–circular regression becomes the method of choice. This is not surprising, and can readily be explained by the bias–variance trade-off, in that more complex statistical methods require larger training sets, to avoid overfitting. Overall, circular–circular regression with training periods of 28 days or more performs the best. On average, it reduces the circular absolute error by 2° or 3°, as compared to the raw forecast. In the subsequent ensemble postprocessing experiments, we thus use circular–circular regression to bias-correct the UWME member forecasts, where the regression parameters in (2) are fit on a member- and location-specific 28-day sliding training period.

c. Bayesian model averaging

With an effective bias-correction technique now at hand, we proceed to discuss ensemble postprocessing techniques for wind direction. All results below are based on the same 28-day sliding training period that we use for bias-correction via circular–circular regression, and are insensitive to changes in the length of the training period. We compare the various methods using the mean circular continuous ranked probability score. Furthermore, we reduce the forecast distributions to the corresponding circular medians, and compute the mean circular absolute error for these single-valued forecasts.

Specifically, Table 4 shows the verification statistics for the discrete UWME (raw) and UWME (bias-corrected) forecast distributions, with the bias correction using circular–circular regression, the BMA and BMA⁺ forecasts (based on the bias-corrected UWME), and the two reference forecasts introduced and described in section 2a, namely climatology and MEC, where the latter is also based on the bias-corrected UWME. The results are averaged over calendar year 2003 and the 54 stations we consider. Bias-correction via circular–circular regression yields a reduction of the circular absolute error for the ensemble median forecast of 3° to 4° on average. As expected, ensemble calibration does not result in any further reduction of the circular absolute error, because MEC, BMA, and BMA⁺ address calibration errors only. However, the latter methods result in a much decreased mean circular continuous ranked probability score, with BMA⁺ performing the best, while BMA is a close competitor. Indeed, the BMA and BMA⁺ forecast distributions are sharper than those for MEC or climatology, and they are calibrated, because the corresponding mean sharpness measure (9) equals roughly the mean circular continuous ranked probability score.

If the forecasts are stratified by verifying wind speeds (results not shown), we see differing error characteristics, in that the mean circular absolute error and the mean circular continuous ranked probability score are lower at the stronger winds when the wind direction observations tend to be more stable. However, the relative ranking of the forecast methods remains unchanged from that in Table 4.

When averaged over calendar year 2003 and the 54 stations, the mean BMA and BMA⁺ weights for the UWME members show little difference. For BMA, the smallest mean weight is 0.122 and the highest is 0.129; for BMA⁺ the corresponding range is from 0.108 to 0.116, with the uniform component contributing a weight of 0.108 on average. The individual weight estimates are quite stable, with a median absolute change in a weight from one training period to the next of 0.002.

Turning now to results at individual stations, Fig. 3 shows the Pacific Northwest domain for the UWME system, along with the locations of the 54 SAO stations considered in this study. The color at each station location indicates what forecast method had the lowest mean circular continuous ranked probability score in calendar year 2003. At 46 of these stations the BMA⁺ method performed best, and at six stations the BMA forecasts showed the lowest score. MEC and climatology performed best at one station each.