1. Introduction
Ensemble perturbations are designed to sample the distribution of analysis and forecast errors. Initial ensemble perturbations that are designed to represent the initial condition error distribution are added to the best available analysis to create the ensemble of initial conditions from which the ensemble forecast is made using one or more nonlinear (possibly stochastic) models (Toth and Kalnay 1997; Toth et al. 2001; Palmer et al. 1998; Houtekamer et al. 1996). Hence, the variance of the ensemble provides a prediction of the error variance of the ensemble mean or, possibly, the high-resolution forecast that is often made in conjunction with a coarser-resolution ensemble forecast. In particular, ensemble forecast variances are designed to predict flow-dependent and observational network dependent aspects of the error variance. Ensemble-based probabilistic forecasts of events are typically based on the relative frequencies of events in the ensemble and have been shown to provide information that can increase the socioeconomic value of weather and climate forecasts by, among others, Richardson (2000), Zhu et al. (2002), Palmer (2002), Mylne (2002), and Cai et al. (2011).
Ensemble forecasts are imperfect: their initial conditions imperfectly sample the distribution of true states given observations, the number of members is typically too few to accurately represent the second and third moments of the distribution of errors, and they imperfectly represent the effects of poorly understood and poorly quantified systematic and stochastic sources of model error. Postprocessing techniques have proven useful in correcting some of the deficiencies of the ensemble. The goal of postprocessing is to describe the distribution of true states given an ensemble of forecasts (including extreme but rare events).
A variety of approaches have been proposed for statistical postprocessing of ensemble forecasts. Nonhomogeneous Gaussian regression (NGR; Gneiting et al. 2005) assumes a Gaussian distribution and performs bias correction of the mean and a variance correction by maximizing a log-likelihood function to obtain coefficients. Dressing methods (Roulston and Smith 2003; Wang and Bishop 2005; Fortin et al. 2006) and Bayesian model averaging (BMA; Raftery et al. 2005) address ensemble dispersion by generating a kernel distribution around each raw ensemble member. The raw ensemble members are typically bias corrected based on historical data and can be weighted, as in BMA, if the ensemble is constructed from different model forecasts or generated in some way that creates ensemble members that do not represent equally likely states. In BMA, the weights and variances are chosen by maximum likelihood from the training data. While BMA was originally formulated to combine forecasts from different models, this method is closely aligned with the “best-member dressing” method of Roulston and Smith (2003) and the approach of Wang and Bishop (2005), which ensures that mean ensemble variance agrees with the forecast error variance. These dressing formulations increase the variance of the ensemble and hence are appropriate for underdispersive ensembles but inappropriate for overdispersive ensembles. For an ensemble prediction system with exchangeable members, Wang and Bishop (2005), Roulston and Smith (2003), and BMA only differ in implementation details (Fortin et al. 2006). Fortin et al. (2006) formulated a dressing method that is appropriate for overdispersive ensembles. Krzysztofowicz and Evans (2008) introduced the Bayesian processor of forecasts (BPF) in which climatological information is incorporated via Bayes’s rule after distributions have been transformed to be normally distributed. Further, Bishop and Shanley (2008) argue that prior climatological information should be incorporated into BMA ensemble postprocessing through Bayes’s theorem.
The method presented in the present study includes climatological information following Bayes’s rule as in Krzysztofowicz and Evans (2008). The inclusion of prior climatological information acts to push the deterministic forecast toward the climatological mean in the same way that a regression-based correction would. However, the method presented here features a climatological weighting that changes with day-to-day changes in forecast error variance. In this way, this method accounts directly for the fact that how close we draw to climatology is itself uncertain, and that uncertainty is flow dependent.
Eckel et al. (2012) discussed the uncertainty associated with ensemble variance prediction and described ambiguity as the uncertainty in the prediction of forecast uncertainty (or in the forecast probability of a specific event) arising from finite sampling and deficiencies in simulation of various sources of forecast uncertainty. Eckel et al. (2012) quantified this type of uncertainty using total ambiguity, the 90% confidence interval of calibrated ensemble-based forecast probabilities. Eckel et al. (2012) constructed distributions of ensemble-based forecast probabilities through bootstrapping, ensemble of ensembles, and calibrated error sampling. The posterior distribution of error variances in Bishop and Satterfield (2013) and Bishop et al. (2013) show how one can quantify this ambiguity using long archives of innovation, ensemble variance pairs.
In Bishop and Satterfield (2013) an analytic approximation to the posterior distribution of error variances, given an imperfect ensemble prediction and prior climatological information, was introduced and tested using a simple dynamical model. In Bishop et al. (2013), equations were developed that enable all parameters defining the prior, likelihood, and posterior distributions of Bishop and Satterfield’s analytical model to be deduced from a long archive of innovation and ensemble variance pairs. These distributions quantify the nonhomogeneous nature of the true error variances or error heteroscedasticity given an ensemble variance. In what follows, we develop a method for using error heteroscedasticty information in ensemble postprocessing.
The outline of this paper is as follows: to assess the value of the postprocessing and discover the sensitivity of this value to changes in parameter regimes, in section 2 we apply a hierarchy of ensemble postprocessing techniques to synthetic data. The hierarchy of ensemble postprocessing techniques is ordered according to how much information they incorporate from the known distribution of error variances given an imperfect ensemble variance prediction. In section 3, we apply the same postprocessing techniques and diagnostics to operational ensembles produced by the Fleet Numerical Meteorology and Oceanography Ensemble Forecast System (FNMOC EFS). Results are compared to homoscedastic ensemble postprocessing techniques as well as to Raftery el al.’s (2005) BMA method in section 4. Conclusions follow in section 5.
2. Utilization of error variance distributions in ensemble postprocessing
In this section, a new heteroscedastic postprocessing algorithm is described that incorporates information about (i) the climatological prior distribution of error variances and (ii) the distribution of ensemble sample variances given an error variance. To isolate the value of accounting for heteroscedasticity in postprocessing, we also develop a hierarchy of homoscedastic ensembles that vary in the amount of information used in the ensemble postprocessing.
The hierarchy of postprocessing schemes is then tested using synthetic data consistent with the analytic model described in detail in Bishop et al. (2013). The use of synthetic data serves two purposes. First, it allows us to test our postprocessing algorithm in a framework in which all the assumptions of our theory are satisfied. Second, using synthetic data allows us to easily modify parameter values and document how the benefits of postprocessing depend on underlying parameter values.
a. Distributions of error variance given an ensemble variance
b. Heteroscedastic ensemble postprocessing
Using (8) to create a 1000-member ensemble ensures that this ensemble optimally combines climatological and forecast information given that the error variance of the forecast is uncertain. The climatological mean of the true state, 
Equation (8) represents a heteroscedastic postprocessing method because it assumes a range of possible true error variances given an ensemble variance. Equation (8) shows that the weight given to the climate mean changes for each of the 
c. “Non-FP” homoscedastic ensembles
To isolate the value of heteroscedastic postprocessing in a simplified context, we now generate partially postprocessed non-FP homoscedastic ensembles using three different methods, designed to gradually increase the amount of information the ensemble has about the posterior distribution of error variances. In the first method, we completely ignore the ensemble variance 
In the second non-FP experiment, we set the posterior error variance equal to the corrected error variance prediction 
For the third non-FP ensemble, we set 
The FP method arose directly from theoretical considerations outlined in Bishop and Shanley (2008), Bishop and Satterfield (2013), and Bishop et al. (2013). These considerations incorporated ideas from the “shift and stretch,” NGR, and BPF methods. In addition, they lent an entirely new aspect to ensemble postprocessing; namely, the random sampling of the distribution of hidden error variances given an imperfect ensemble variance. The non-FP ensemble postprocessing approaches (invariant, MSS, and informed Gaussian) are homoscedastic in that they all assume that there is just a single forecast error variance associated with each forecast 
d. Rank frequency histograms for FP and non-FP ensembles
To create rank frequency histograms (RFHs) (also known as “Talagrand diagrams;” Anderson 1996; Hamill and Colucci 1997; Talagrand et al. 1997) one takes an M-member ensemble forecast and the observed verification state and orders the M forecasts and observed state from lowest to highest value. If the verification state is lower than all the forecasts, it is assigned rank 1, if it is lower than all but one of the ensemble members, it is said to have rank 2, and so on. The RFH plots the frequency with which the verifying state takes each possible rank over a large number of forecasting trials. If the verification state is drawn from the same distribution as the ensemble members then it would take each rank with equal frequency. This is a sufficient but not necessary condition for a flat histogram [see Bishop and Shanley (2008) for an example of a flat RFH from an ensemble that is not drawn from the distribution of truth given the forecast]. Since our experiments have a known “true” state, we simply rank the true state among the ordered ensemble members.
Figure 1 compares the results of the RFHs for the four methods used to postprocess the ensemble. For these experiments, the prescribed parameters were 
Rank frequency histogram obtained from an M = 1000 member postprocessed ensemble. In this figure, the known true state is ranked among the ordered ensemble members. The true state is taken from a normal distribution with mean 
Citation: Monthly Weather Review 142, 9; 10.1175/MWR-D-13-00286.1
e. Weather roulette
To translate the enhanced RFH flatness due to heteroscedastic ensemble postprocessing to a recognizable “value” to users of ensemble forecasts, we implement a version of Hagedorn and Smith’s (2009) weather roulette diagnostic. Weather roulette is a hypothetical gambling game in which two players, A and B, each open a weather roulette casino in which one can bet that a verifying observation (or for the case of synthetic data, a known true state) will fall into one of 
To be clear about what we mean by “payout ratio”, suppose that player B bets $10 in player A’s casino that the verifying observation will fall into the third climatological bin. If the verifying observation actually does fall into the third climatological bin, player B will receive 
Thus, if in the geometric average 
f. Parameter regime dependence
To study the sensitivity of the value of heteroscedastic postprocessing to error regime, we first perform our tests using synthetic data in which the parameter values can be easily controlled and modified. The models and methods used to generate the synthetic data for these tests are described in detail in Bishop and Satterfield (2013) and Bishop et al. (2013). For our control experiment, we define the climatological distribution of innovation variances given by setting 
The mean, maximum, minimum, and standard deviation of the effective daily interest rate (in percent) earned by the FP ensemble when the (top) invariant, (middle) MSS, and (bottom) informed Gaussian are played for 10 independent trials, each consisting of 100 000 forecast events. Effective ensemble sizes of 
Citation: Monthly Weather Review 142, 9; 10.1175/MWR-D-13-00286.1
In considering Fig. 2, note that each of the compared ensemble forecasts are based on the same deterministic forecast 
We see that the gamblers positive effective daily interest rate per trial in the informed-Gaussian casino decreases markedly as 
The relative variance of the assumed prior climatological inverse-Gamma distribution of error variances is given by 
As in Fig. 2, but (left) 1/(α2) increased by a factor of 10 and (right) 1/(α2) decreased by a factor of 10.
Citation: Monthly Weather Review 142, 9; 10.1175/MWR-D-13-00286.1
Aside from the parameters that define the prior and posterior distributions, the results are also sensitive to the bin width used in weather roulette. Recall that we had set 
As in Fig. 2, but only 
Citation: Monthly Weather Review 142, 9; 10.1175/MWR-D-13-00286.1
3. Application to 500-hPa virtual temperature fields
a. Datasets
We now test our postprocessing algorithms on the raw ensemble forecasts produced by the FNMOC EFS. This ensemble employs the ensemble transform (ET) technique (Bishop and Toth 1999; McLay et al. 2008) to produce initial conditions for 20 forecasts made using the Navy Operational Global Atmospheric Prediction System (NOGAPS). The NOGAPS model used for the ensemble forecasts has a spectral resolution of T159 and 42 vertical levels. For this study, we focus on 500-hPa virtual temperatures. We apply postprocessing techniques to ensemble forecasts made at 0000 and 1200 UTC for lead times at 24-h increments from 24 to 168 forecast hours and consider the time period of 0000 UTC 1 March–1200 UTC 31 May 2012. For this time period, the resolution of the deterministic forecast was T319 with 42 vertical levels. For the results presented in this paper, a 1° × 1° gridpoint representation of these fields was used.
b. Recovering observation error variance
To compute the mean of the climatological distribution of error variances 
The Desroziers method is based on computing the expected value of the analysis residual (observation minus analysis) and innovation (observation minus background). It can be shown that, if the data assimilation system properly specifies the background and observation error variance, this expected value will result in the observation error variance. Desroziers et al. (2005) suggested an iterative method for converging on the true value of the observation error variance if the observation error covariance matrix is inappropriately specified. In the present study, however, we do not apply an iterative procedure, since this would be computationally costly. Instead, we simply compare the result from the Desroziers diagnostic to those recovered from the Hollingsworth–Lönnberg method. Figure 5 shows a comparison of the recovered values of observation error variance for radiosonde observations of temperature in the Northern Hemisphere. The data for use in Fig. 5 came from NOGAPS analyses at T319 spectral resolution for the period of 1 February–1 April 2011. Both methods were applied at eight vertical levels (1000, 850, 500, 300, 250, 100, 50, and 10 hPa) and the recovered values were interpolated to all model levels using a cubic spline interpolation. The value of R prescribed within the data assimilation system is also shown. At lower and middle pressure levels we see a better agreement between both recovered values of R than the value prescribed, the exception is the stratospheric levels where data are sparse. For the 500-hPa level, which is the focus of this study we find good agreement: 0.5518 for Desroziers and 0.5735 from Hollingsworth–Lönnberg. In what follows, we simply use the value obtained from Hollingsworth–Lönnberg method.
A comparison of the recovered observation error variance using the Derozier method (dotted line) and the Hollingsworth–Lönnberg method (dashed line) data used from temperature measurements of radiosondes and NOGAPS analyses at T319 spectral resolution for 1 Feb–1 Apr 2011. The prescribed value is also shown (solid line). Values were recovered at eight pressure levels and a cubic spline interpolation was used to plot the vertical profile.
Citation: Monthly Weather Review 142, 9; 10.1175/MWR-D-13-00286.1
c. Application of parameter recovery equations and heteroscedastic postprocessing
To recover the parameters 
The parameter recovery equations, derived in Bishop et al. (2013) and summarized in the appendix, assume that the forecasts and observations are unbiased and that observation error and forecast error are uncorrelated. If a global bias exists (i.e., the model has the wrong climatology), that bias should be removed prior to the application of the recovery equations and postprocessing. In the present dataset, the mean bias was found to be small enough (maximum of −0.17 K at a 24-h lead time), that no improvement was shown when an out of sample bulk bias correction was applied. Therefore, in what follows no systematic bias correction is included. Equation (8) acts to push the deterministic forecast toward the climatological mean in the same way that a regression-based correction would. However, the climatological weighting changes with day-to-day changes in forecast error variance.
The values of parameters recovered using (A1)–(A6) from two time periods, 1 March–30 April 2012 and 1 April–3 May, are summarized in Table 1. Values of recovered parameters are shown for lead times between 24 and 168 h at 24-h increments. The value of a is less than 1 for all lead times and for both time periods, indicating that the ensemble is typically underdispersive. The nonzero values of k corresponds to effective ensemble size (M = 2k + 1) larger than one, which indicates that the raw FNMOC ensemble has at least some skill in predicting the flow-dependent error variance. The value of k, and hence the effective ensemble size increases with lead time. This behavior is consistent with the findings of Satterfield and Szunyogh (2010), who showed that ensembles span a greater proportion of the vector space of forecast errors as the lead time increases. The recovered values are in fairly good agreement between the two time periods. This agreement makes the idea of recovering parameters during a training period of the previous two months plausible. In what follows, we use recovered parameters from the training period of 1 March–30 April 2012 and apply those parameters to postprocess ensemble data for the month of May 2012.
Hidden error variance parameters recovered from time series of innovation and ensemble variance pairs. Forecasts with lead times ranging from 24 to 168 h, valid during the period (top) 1 Mar–30 Apr 2012 and (bottom) 1 Apr–31 May 2012, were considered.
Section 2f discussed the sensitivity of the benefits associated with the use of the FP ensemble to the parameter regime. Guidance from synthetically generated data indicated that the FP ensemble outperformed the informed-Gaussian ensemble up to values of 
4. Comparison to non-FP homoscedastic ensembles and Bayesian model averaging
To create an M = 1000 postprocessed ensemble from the FNMOC ensembles we use the same approach as that for the synthetic data following (8) to obtain a 1000-member sample from our estimate of the distribution of truth given the forecast. Then, since we use observations for verification we add, to each of the 1000 ensemble members, random normal numbers with mean zero and variance equal to the observation error variance R, following Bowler (2006). This second step yields an estimate of the distribution of observations given the forecast. To define the climatological mean and standard deviation, 
a. Comparison to homoscedastic ensembles
We compare the results of our heteroscedastic postprocessing methods with the three homoscedastic methods described in section 2b. Again, we first assess the performance of these postprocessing methods using RFHs. Figure 6 shows the rank frequency histograms for the raw ensemble and the FP ensemble, as well as the ensembles postprocessed using homoscedastic methods. For this figure, all postprocessing parameters were derived out of sample. This figure considers all 48-h ensemble forecasts for the Northern Hemisphere (30°–90°N) valid during the month of May 2012 and uses 500-hPa radiosonde observations of temperature as the verifying observation. For the month of May, there are N = 23 927 observations, to achieve a perfectly flat rank frequency histogram with M = 1000 ensemble forecasts each bin would have a frequency of 23–24. For the raw ensemble with 20 members, the optimal frequency would be 1139–1140. The raw ensemble shows a high bias, with observations falling below the minimum ensemble value more frequently. When the raw ensemble is recentered on the higher-resolution deterministic forecast (shown in Fig. 6b) the bias is largely removed.
RFH obtained from an M = 1000 member postprocessed FNMOC EFS ensemble. Results are shown for (a) the raw ensemble, (b) the raw ensemble recentered on the higher-resolution deterministic forecast, (c) FP ensemble, (d) invariant ensemble, (e) MSS ensemble, and (f) informed-Gaussian ensemble. Shown are 48-h forecasts valid for May 2012 postprocessed using hidden error variance parameters derived from the training period 1 Mar–30 Apr 2012.
Citation: Monthly Weather Review 142, 9; 10.1175/MWR-D-13-00286.1
The FP ensemble gives a RHF with overpopulated center bins. The overpopulation in the center bins is most likely due to the recovered value of the ensemble sensitivity parameter 
Figure 7 shows the mean, minimum, maximum, and standard deviation of the effective daily interest rate (in terms of percent) achieved over five trials when the FP ensemble plays the homoscedastic non-FP ensembles. Each trial has the same original FNMOC EFS ensemble and recovered parameters, but different random draws from the posterior and a different set of 10 000 randomly drawn verifying observations. Equally probable weather roulette bins were determined from the full set of observations. At the 24-h lead time the invariant ensemble performs very similarly to the informed Gaussian due to small values of effective ensemble size and that 
The mean, maximum, minimum, and standard deviation of the effective daily interest rate (in percent) earned by the FP ensemble when the (top) invariant, (middle) MSS, and (bottom) informed Gaussian are played for five independent trials, each consisting of 10 000 randomly chosen forecast events. All forecasts are valid for May 2012 and postprocessed using hidden error variance parameters derived from the training period 1 Mar–30 Apr 2012.
Citation: Monthly Weather Review 142, 9; 10.1175/MWR-D-13-00286.1
b. Comparison to Bayesian model averaging
Here we compare our results with the fairly well-established BMA ensemble postprocessing method of Raftery et a1. (2005). We tried to follow the method described in Raftery et al. (2005) as closely as possible while preserving consistency, where possible, with the other methods used in this paper.
We use a linear regression–based correction, where the regression coefficients are determined from the previous two months and vary latitudinally based on 5° latitude bands centered on the latitude of interest, to correct the 20 raw ensemble members following,
Following the same procedure as in step 1, regression coefficients are determined for the deterministic forecast.
The 20 regression-corrected ensemble members combined with the single regression-corrected deterministic forecast from a 21-member ensemble are used as input to the expectation and maximization algorithm.
We calculate the weights
We create a M = 1000 member postprocessed ensembles by drawing a random number between [1, …, k = 21] with probabilities
(top) RFH obtained from an M = 1000 member FNMOC EFS ensemble postprocessed using the BMA method shown for the 48-h forecast lead time. (bottom) Weather roulette interest rate earned by the FP ensemble when the BMA ensemble is played. All forecasts are valid for May 2012 and postprocessed using hidden error variance parameters derived from the training period 1 Mar–30 Apr 2012.
Citation: Monthly Weather Review 142, 9; 10.1175/MWR-D-13-00286.1
Another score of interest is the binned spread-skill plot (e.g., Wang and Bishop 2003). We divide all data points into 
Binned spread-skill diagram shown for the 48-h lead time for (a) the raw ensemble, (b) FP, (c) BMA, (d) invariant, (e) MSS, and (f) informed Gaussian. All forecasts are valid for May 2012 and postprocessed using hidden error variance parameters derived from the training period.
Citation: Monthly Weather Review 142, 9; 10.1175/MWR-D-13-00286.1
We also compute the continuous ranked probability score (CRPS) for the postprocessed ensemble, as well as the decomposition into reliability, resolution, and uncertainty following Hersbach (2000). The CRPS compares the cumulative distribution functions between forecasts and observations, having an optimal value of zero. A perfectly reliable forecast is indicated by a reliability value of zero and this computation of reliability is closely related to the rank histogram (Hersbach 2000). Positive values of resolution indicate that the ensemble performs better than the climatological probabilistic forecast.
This score was chosen since we want to focus on the full distribution, not on a particular event. For our computation, we computed the CRPS for five independent trails, each trial randomly choosing N = 10 000 observations as verification. Shown in Fig. 10 are the CRPS (top panels), reliability (middle panels), and resolution (bottom panels) computed as the mean values over those five trials. The standard deviation computed over the five independent trials is also shown. In terms of CRPS, initially, the BMA ensemble performs similarly to the FP ensemble; however, beyond the 48-h lead time a benefit is seen from the FP ensemble. Beyond the 48-h lead time the FP ensemble provides the best performance, converging with the MSS and informed Gaussian around the 144-h lead time. Although the BMA ensemble shows the highest values for resolution (bottom panels of Fig. 9), beyond the 24-h lead BMA shows the poorest performance for reliability. The most reliable forecasts are given by the FP ensemble for all lead times. In terms of resolution, the FP ensemble performs similarly to invariant ensemble and informed-Gaussian ensembles at shorter lead times, and similarly to the MSS and informed-Gaussian ensembles at longer lead times.
(top) CRPS and the corresponding decomposition into (middle) reliability and (bottom) resolution shown for the FP ensemble (black), informed-Gaussian ensemble (blue), invariant ensemble (green), MSS ensemble (cyan), and the BMA ensemble (red). Lead times are shown at 24-h increments (left) between 24 and 72 h and (right) between 96 and 168 h. All forecasts are valid for May 2012 and postprocessed using hidden error variance parameters derived from the training period 1 Mar–30 Apr 2012.
Citation: Monthly Weather Review 142, 9; 10.1175/MWR-D-13-00286.1
5. Conclusions
A heteroscedastic ensemble postprocessing technique has been introduced that incorporates aspects of the BPA, shift and stretch, and the NGR postprocessing methods. Unlike previous approaches, it allows climatological information to be incorporated in the forecast in a way that takes advantage of the best available estimate of the distribution of true error variances given an ensemble variance. Here, we employed estimates of the distribution of true error variances given an ensemble variance that are based on Bishop and Satterfield (2013) and Bishop et al.’s (2013) theory of hidden error variances. We also introduced a hierarchy of homoscedastic ensemble postprocessing techniques in which a single error variance is assigned to each forecast that, in differing ways, partially captured the information in Bishop and Satterfield’s analytical model of the distribution of true error variances given an ensemble variance. The homoscedastic hierarchy included techniques very similar to the previously proposed BPA, shift and stretch, and NGR postprocessing techniques. Our experiments with synthetic data demonstrated that ignoring any aspect of the distribution of true error variances given an imperfect error variance prediction degrades the quality of probabilistic forecasts. Using real data and real ensemble forecasts from FNMOC, we again found that the more extensively the postprocessing technique incorporated information about the estimated posterior distribution of hidden error variances given an ensemble variance, the greater the improvement to probabilistic skill scores. In the real data test, heteroscedastic postprocessing outperformed our implementation of the BMA method.
In this paper, heteroscedastic postprocessing was applied to virtual temperatures at 500 hPa. This variable was chosen because 1) the distribution is expected to be Gaussian and 2) the bias is lower than that of variables closer to the surface. These simplifications allowed for a proof-of-concept demonstration of this method in a semi-idealized framework. For nonnormally distributed variables, one would have to apply appropriate transformations as in Krzysztofowicz and Evans (2008) and Hamill (2008). In a companion paper, we will focus on applying this algorithm to the less idealized situations where the distribution of errors is nonnormal, as well as to cases where bias correction must be applied as a first step. In addition, this study was based on a three-month period, in practice an optimal running or historical training set would need to be defined. Finally, we note that various aspects of the postprocessing methods presented in this paper could be combined, for example BPF and BMA could be combined by including climatology in BMA through (8) as opposed to using a linear regression–based correction. Conversely, a linear regression–based correction could be applied to heteroscedastic methods prior to parameter recovery. One aspect of BMA and similar kernel dressing methods that could potentially offer a benefit for bimodal distribution is that the original structure of the ensemble is retained; for this reason in a companion paper, we will also present a kernel formulation of heteroscedastic postprocessing.
Acknowledgments
We thank the three anonymous reviewers for their valuable comments that helped to improve this manuscript. EAS gratefully acknowledges support from the Jerome and Isabella Karle Fellowship at NRL and from the NRL Base Program through BE033-03-45. CHB gratefully acknowledges support from the U.S. Office of Naval Research Grants 4304-D-O-5 and 6681-0-4-5.
APPENDIX
Estimating Parameters of Error Variance Distributions
REFERENCES
Anderson, J. L., 1996: A method for producing and evaluating probabilistic forecasts from ensemble model integrations. J. Climate, 9, 1518–1530, doi:10.1175/1520-0442(1996)009<1518:AMFPAE>2.0.CO;2.
Bishop, C. H., and Z. Toth, 1999: Ensemble transformation and adaptive observations. J. Atmos. Sci., 56, 1748–1765, doi:10.1175/1520-0469(1999)056<1748:ETAAO>2.0.CO;2.
Bishop, C. H., and K. T. Shanley, 2008: Bayesian model averaging’s problematic treatment of extreme weather and a paradigm shift that fixes it. Mon. Wea. Rev., 136, 4641–4652, doi:10.1175/2008MWR2565.1.
Bishop, C. H., and E. A. Satterfield, 2013: Hidden error variance theory. Part I: Exposition and analytic model. Mon. Wea. Rev., 141, 1454–1468, doi:10.1175/MWR-D-12-00118.1.
Bishop, C. H., E. A. Satterfield, and K. T. Shanley, 2013: Hidden error variance theory. Part II: An instrument that reveals hidden error variance distributions from ensemble forecasts and observations. Mon. Wea. Rev., 141, 1469–1483, doi:10.1175/MWR-D-12-00119.1.
Bowler, N. E., 2006: Explicitly accounting for observation error in categorical verification of forecasts. Mon. Wea. Rev., 134, 1600–1606, doi:10.1175/MWR3138.1.
Cai, X., M. Hejazi, and D. Wang, 2011: Value of probabilistic weather forecasts: Assessment by real-time optimization of irrigation scheduling. J. Water Resour. Plann. Manage., 137, 391–403, doi:10.1061/(ASCE)WR.1943-5452.0000126.
Daley, R., 1991: Atmospheric Data Analysis.Cambridge University Press, 457 pp.
Desroziers, G., L. Berre, B. Chapnik, and P. Poli, 2005: Diagnosis of observation, background and analysis-error statistics in observation space. Quart. J. Roy. Meteor. Soc., 131, 3385–3396, doi:10.1256/qj.05.108.
Eckel, F. A., M. S. Allen, and M. C. Sittel, 2012: Estimation of ambiguity in ensemble forecasts. Wea. Forecasting, 27, 50–69, doi:10.1175/WAF-D-11-00015.1.
Fortin, V., A.-C. Favre, and M. Saïd, 2006: Probabilistic forecasting from ensemble prediction systems: Improving upon the best-member method by using a different weight and dressing kernel for each member. Quart. J. Roy. Meteor. Soc., 132, 1349–1369, doi:10.1256/qj.05.167.
Gneiting, T., A. E. Raftery, A. H. Westveld, and T. Goldman, 2005: Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation. Mon. Wea. Rev., 133, 1098–1118, doi:10.1175/MWR2904.1.
Hagedorn, R., and L. A. Smith, 2009: Communicating the value of probabilistic forecasts with weather roulette. Meteor. Appl., 16, 143–155, doi:10.1002/met.92.
Hamill, T. M., cited 2008: Krzysztofowicz and Evans' “Bayesian processing of forecasts” evaluation with GFS reforecasts. National Oceanic and Atmospheric Administration. [Available online at http://www.esrl.noaa.gov/psd/people/tom.hamill/BPF_review_hamill.pdf.]
Hamill, T. M., and S. J. Colucci, 1997: Verification of Eta–RSM short-range ensemble forecasts. Mon. Wea. Rev., 125, 1312–1327, doi:10.1175/1520-0493(1997)125<1312:VOERSR>2.0.CO;2.
Hersbach, H., 2000: Decomposition of the continuous ranked probability score for ensemble prediction systems. Wea. Forecasting, 15, 559–570, doi:10.1175/1520-0434(2000)015<0559:DOTCRP>2.0.CO;2.
Hollingsworth, A., and P. Lönnberg, 1986: The statistical structure of short-range forecast errors as determined from radiosonde data. Part I: The wind field. Tellus, 38A, 111–136, doi:10.1111/j.1600-0870.1986.tb00460.x.
Houtekamer, P. L., L. Lefaivre, J. Derome, H. Ritchie, and H. L. Mitchell, 1996: A system simulation approach to ensemble prediction. Mon. Wea. Rev., 124, 1225–1242, doi:10.1175/1520-0493(1996)124<1225:ASSATE>2.0.CO;2.
Johnson, C., and N. Bowler, 2009: On the reliability and calibration of ensemble forecasts. Mon. Wea. Rev., 137, 1717–1720, doi:10.1175/2009MWR2715.1.
Krzysztofowicz, R., and W. B. Evans, 2008: Probabilistic forecasts from the National Digital Forecast Database. Wea. Forecasting, 23, 270–289, doi:10.1175/2007WAF2007029.1.
McLay, J. G., C. H. Bishop, and C. A. Reynolds, 2008: Evaluation of the ensemble transform analysis perturbation scheme at NRL. Mon. Wea. Rev., 136, 1093–1108, doi:10.1175/2007MWR2010.1.
Mylne, K. R., 2002: Decision-making from probability forecasts based on forecast value. Meteor. Appl., 9, 307–315, doi:10.1017/S1350482702003043.
Palmer, T. N., 2002: The economic value of ensemble forecasts as a tool for risk assessment: From days to decades. Quart. J. Roy. Meteor. Soc., 128, 747–774, doi:10.1256/0035900021643593.
Palmer, T. N., R. Gelaro, J. Barkmeijer, and R. Buizza, 1998: Singular vectors, metrics, and adaptive observations. J. Atmos. Sci., 55, 633–653, doi:10.1175/1520-0469(1998)055<0633:SVMAAO>2.0.CO;2.
Raftery, A. E., T. Gneiting, F. Balabdaoui, and M. Polakowski, 2005: Using Bayesian model averaging to calibrate forecast ensembles. Mon. Wea. Rev., 133, 1155–1174, doi:10.1175/MWR2906.1.
Richardson, D., 2000: Skill and relative economic value of the ECMWF ensemble prediction system. Quart. J. Roy. Meteor. Soc., 126, 649–667, doi:10.1002/qj.49712656313.
Roulston, M. S., and L. A. Smith, 2003: Combining dynamical and statistical ensembles. Tellus, 55A, 16–30, doi:10.1034/j.1600-0870.2003.201378.x.
Satterfield, E., and I. Szunyogh, 2010: Predictability of the performance of an ensemble forecast system: Predictability of the space of uncertainties. Mon. Wea. Rev., 138, 962–981, doi:10.1175/2009MWR3049.1.
Talagrand, O., R. Vautard, and B. Strauss, 1997: Evaluation of probabilistic prediction systems. Proc. ECMWF Workshop on Predictability, Reading, United Kingdom, ECMWF, 1–26.
Toth, Z., and E. Kalnay, 1997: Ensemble forecasting at NCEP and the breeding method. Mon. Wea. Rev., 125, 3297–3319, doi:10.1175/1520-0493(1997)125<3297:EFANAT>2.0.CO;2.
Toth, Z., Y. Zhu, and T. Marchok, 2001: The use of ensembles to identify forecasts with small and large uncertainty. Wea. Forecasting, 16, 463–477, doi:10.1175/1520-0434(2001)016<0463:TUOETI>2.0.CO;2.
Wang, X., and C. H. Bishop, 2003: A comparison of breeding and ensemble transform Kalman filter ensemble forecast schemes. J. Atmos. Sci., 60, 1140–1158, doi:10.1175/1520-0469(2003)060<1140:ACOBAE>2.0.CO;2.
Wang, X., and C. H. Bishop, 2005: Improvement of ensemble reliability with a new dressing kernel. Quart. J. Roy. Meteor. Soc., 131, 965–986, doi:10.1256/qj.04.120.
Zhu, Y., Z. Toth, R. Wobus, D. Richardson, and K. Mylne, 2002: The economic value of ensemble-based weather forecasts. Bull. Amer. Meteor. Soc., 83, 73–83, doi:10.1175/1520-0477(2002)083<0073:TEVOEB>2.3.CO;2.
