1. Introduction
Ensemble forecasts have now been generated for almost three decades and generally provide a better basis for decision-making processes than single forecasts (Toth and Kalany 1993; Molteni et al. 1996; Richardson 2000). In practice the scenarios are often given a probabilistic interpretation, but model inadequacies tend to limit their direct use. Statistical methods are therefore widely applied to account for deficiencies and make forecasts reliable in a probabilistic sense [see, e.g., Vannitsem et al. (2018) for an overview].
One possible way to classify the methods is whether distributional assumptions on the forecasts are made or not. For the former type the parameters of the distributions are often assumed to be simple functions of the ensemble output and estimated by optimizing the likelihood or the continuous ranked probability score. In recent years the tendency has been toward more flexible distributions or mixture of simpler distributions in order to better describe the relation between ensemble output and what is being observed. For example, Scheuerer and Hamill (2015) use a censored, shifted gamma distribution for precipitation forecasting, while Baran and Lerch (2016, 2018) use a mixture of truncated normal and lognormal distributions for wind speed forecasting. Mixtures with one component distribution for each ensemble member have also been applied for many years (Raftery et al. 2005).
There is also a variety of methods not assuming any form of the predictive distribution. In analog forecasting, cases similar to the present forecast situation are searched for in long historical archives and applied as forecast (van den Dool 1989; Hamill and Whitaker 2006). Methods based on regression trees have recently also received increasing attention due to their flexibility and ease of use; for example, Taillardat et al. (2016) use quantile regression forest, a slight modification of random forest. A very different strategy is to make linear adjustments of each ensemble member (e.g., van Schaeybroeck and Vannitsem 2015; Wilks 2018). Quantile regression methods, the topic of this article, have been proposed by several authors. Bremnes (2004) introduced local quantile regression which gives noncrossing, nonlinear quantiles. Censored linear quantiles were studied by Friederichs and Hense (2007) for mixed distributed variables like precipitation. Highly adaptable quantile regression functions by means of neural networks were proposed by Cannon (2011). Wahl (2015) used quantile regression in a Bayesian context and considered penalization methods for variable selection. Penalized quantile regression was also applied by Ben Bouallegue (2017). In a related field, Nielsen et al. (2006) used quantile regression with spline functions based on single model NWP output for wind power forecasting.
The main motivation behind this article is that nature imposes few restrictions on the shape of forecast distributions, which can be confirmed by studying the variation in ensemble forecast distributions over time (see section 2). Statistical methods should therefore allow for sufficiently flexible predictive distributions in order to provide well calibrated forecasts in all weather situations. Another motive is that not all information about the future is contained in the ensemble mean and standard deviation, and one aim is to make use of all ensemble members in a versatile manner. To this end, a method based on constrained quantile regression splines (CQRS) is proposed. In CQRS each quantile is represented by a spline function of the ensemble model output, but not necessarily the same output for all quantiles. The latter allows fitted quantiles to better resemble the raw ensemble distribution. Since splines are very adaptable functions, the main challenge is to find appropriate constraints for best possible predictions and in particular for moderate to extreme situations where data are sparse. The method is tested on ensemble wind speed forecasting data at 125 Norwegian sites.
The remainder of this article is structured as follows. The data are presented in section 2. Section 3 describes the constrained quantile regression spline method, two other benchmark methods and the forecast evaluation approach. Section 4 describes experiments and the outcome of these, while further discussion and conclusions are given in sections 5 and 6.
2. Data
To test the CQRS method, forecasts of wind speed at 10 m height at 125 Norwegian synoptic stations from the European Centre for Medium-Range Weather Forecasts Ensemble Prediction System (ECMWF ENS) and corresponding measurements of hourly maximum 10-min average wind speed are considered. The measurement stations were selected from the Norwegian Meteorological Institute’s network of automatic synoptic stations with the requirement of at least 97.5% data availability. Most, if not all, of the measurement data have been processed by a simple automatic quality control system and some also manually assessed. For the period under study, the ECMWF ensemble prediction system had a horizontal resolution of about 32 km and consisted of 51 members; one unperturbed (control) and 50 exchangeable, pairwise symmetrically perturbed members. All forecasts were generated at 0000 UTC (coordinated universal time) with lead times +12, +36, +60, +84, and +108 h and bilinearly interpolated to the locations of the stations. The observations and corresponding forecasts are from 21 November 2013 to 31 December 2015. Almost the same data were used in Eide et al. (2017).
The locations of the stations are shown in Fig. 1 along with the topography, the maximum observed wind speed, and the correlation between the control run for lead time +36 h and the observed wind speed. Norway has a varied topography, with mountain ranges with peaks up to almost 2500 m above the sea level and plateaus, and valleys and fjords as well as lowlands. The strongest wind speeds are observed along the coast and in mountainous areas, while the more sheltered stations in valleys and lowlands are exposed to considerably weaker wind speeds. The skill of the ECMWF ENS forecasts also varies strongly with the locations. In terms of correlation the highest values are seen close to the coast line, while the lowest mostly occur in narrow valleys that are not well resolved in the ECMWF model.
(left) Maximum observed wind speed (m s−1) for each of the 125 sites. (middle) Elevation with the darkest gray level representing about 2500 m above sea level. (right) Linear correlation between the ensemble control run (%) and the observation for each site for lead time +36 h.
Citation: Monthly Weather Review 147, 5; 10.1175/MWR-D-18-0420.1
In many statistical methods assumptions on the predictive distributions are made at an early stage in the modeling process. A key aspect is to consider flexibility versus tractability in view of the amount of training data. Despite limited number of ensemble members, studying ensemble forecast distributions over longer time periods gives indications of possible shapes of predictive distributions. Obviously, the ensemble mean and variance/uncertainty vary depending on the weather situations, but further statistics may provide additional information. In Fig. 2 the ensemble skewness, the average of (member − mean)3/standard deviation3 over the ensemble, is plotted against discretized standardized ensemble means for lead time +60 h. Standardization was obtained by subtracting the temporal mean and dividing by the temporal standard deviation. Clearly, for weak wind cases the majority of forecast distributions are skewed upward, while for strong wind situations the distributions are more often skewed toward zero. Thus, statistical methods should ideally be able to reflect this variability.
Boxplot of ensemble skewness against standardized ensemble mean for lead time +60 h. There are approximately 1900 forecasts in each of the 50 categories. Outliers are not plotted.
Citation: Monthly Weather Review 147, 5; 10.1175/MWR-D-18-0420.1
For all experiments later the data are split into training and test sets by 1 January 2015, which implies about 13 months of data for training and 12 months for testing and evaluation.
3. Methods
In this section, splines and their properties are first briefly introduced before they are applied in quantile regression. The section then describes ensemble postprocessing by means of the truncated normal and lognormal distributions, while the last subsection outlines approaches to forecast verification.
a. Splines
if the spline coefficients
the spline function
b. Constrained quantile regression splines
A potential problem in quantile regression is that quantiles may cross and be invalid since they are fitted separately. Here, this is handled as follows. First, each quantile is assumed to depend only on one covariate which is linearly transformed to the interval 
An example of fitted quantiles is shown in Fig. 3. For each quantile the corresponding ensemble quantile is used as the covariate and for clarity only 11 out of 51 quantiles are visualized. The upper plot shows fitted quantiles with covariates linearly transformed to 
Example of fitted splines at a coastal site for lead time +60 h. For clarity only every 5th quantile is shown. (top) Fitted quantiles on the 
Citation: Monthly Weather Review 147, 5; 10.1175/MWR-D-18-0420.1
c. Benchmark methods
By using CQRS, forecast distributions can vary considerably in shape and it is of particular interest to compare CQRS with statistical methods restricting forecasts to certain parametric distributions. As benchmarks, the lognormal EMOS (Baran and Lerch 2015) and the mixture EMOS of truncated normal and lognormal distributions (Baran and Lerch 2016) for wind speed forecasting are chosen. Methods based on other distributions could have been applied as well. Thorarinsdottir and Gneiting (2010) use the truncated normal distribution, while Lerch and Thorarinsdottir (2013) apply the generalized extreme value distribution. Both articles also include intercomparisons of the methods and approaches combining these distributions. Baran and Lerch (2016) conclude that the mixture of truncated normal and lognormal distributions is to be preferred.
d. Forecast verification methods
In this article all forecasts are given in terms of a set of quantiles and the overall aim is to assess the quality of these. As with other types of probabilistic forecasts, quantile forecasts should be reliable and at the same time as sharp as possible. Both properties need to be examined in detail separately. In addition, a summarizing measure is essential, for example to be able to rank forecast models. For quantile forecasts the quantile score (QS) is the obvious choice, see Gneiting and Raftery (2007), and references therein. Let y denote the observation and 
A fundamental requirement for probabilistic forecasts is that they are reliable. By definition the probability of observing a value below the τ quantile is τ. For a set of quantile forecasts the proportion of measurements below the given quantile is therefore a suitable statistic for reliability. The closer the proportion is to τ the more reliable the forecast model is.
Sharpness refers to the spread of the probability mass and depends only on the forecasts. In most cases sharpness is defined by the length of the interval between two quantiles. Here, sharpness is assessed by calculating the average widths of central intervals with 50% and 88% coverage probabilities. This definition favors unimodal forecast distributions, but in some weather situations the underlying uncertainty may better be represented by multimodal distributions. A supplemental metric for sharpness is therefore also proposed and computed as follows. Assume a forecast in terms of M ordered quantiles is given and that these form 
All verification statistics are computed for the whole test period for each station and also averaged over all stations. To further increase the insight of the performance of the forecast models, the data are split into groups by the ensemble mean, ensemble standard deviation, ensemble skewness, and the Hartigan’s dip test statistic of unimodality (Hartigan and Hartigan 1985) of the ensemble. Several ways of grouping has been tested, but in section 4 the assessment is restricted to three groups (low, medium, and high) defined by site-dependent 10th and 90th percentiles. Attention is especially paid to reliability, since all forecasts should be reliable and too much averaging may conceal systematic deviations from reliability.
4. Experiments and results
In this section the constrained quantile regression method is tested on the wind speed forecasting data. First, details concerning the splines and the estimation of quantiles are examined. Then, the choice of quantile covariate is considered followed by an intercomparison with the benchmark methods. In the end some experiments for further refinements of the CQRS method are looked into.
a. Spline configurations
Splines are in general highly adaptable functions that would easily overfit the training data unless they somehow are restricted. Several ways of finding the right balance are here tested:
Number of interior knots. The number of knots determines the flexibility of splines. Only equidistant knot sequences with from zero and to three knots in the interval
Upper constraint. Only tests with and without an upper constraint set to 30% higher than the maximum for each station in the training set are carried out. A lower constraint at 0 m s−1 is always included. The upper limit was chosen based on a short extreme value analysis on the training data.
Increasing constraint. It seems natural to require quantiles to be nondecreasing with increasing covariate values. With the spline defined in Eq. (3) the requirement is attained if the spline coefficients are in nondecreasing order. In practice there are two ways to achieve this. First, the constraints on the spline coefficients can be added to the linear program optimization. Second, the spline coefficient can be sorted in increasing order after the optimization is done, that is, with no constraints in the optimization step. Both options are tested. Initially, splines without this constraint were also considered, but it did not work out well and was not investigated further.
Extrapolation. Extrapolation refers to predictions at covariate values beyond the range of the training data. Note that extrapolation is not part of the training process, but appears only in the prediction step. Predictions with and without extrapolation is offered. No extrapolation implies that covariates exceeding the training maximum are set equal to the maximum. In practice extrapolation only involves very few cases. Consequently, barely any impact on the summary scores can be expected, so the choice mostly concerns the preferable degree of caution in the forecast models.
Spline order. The order of the B-splines determines how smooth the spline functions are. Linear and cubic splines are here tested.
Quantile scores (
The overall best model is for simplicity applied to all sites in the experiments to follow. A sensitivity study was carried out to assess the impact of not using the best individual model for each site. On average the quantile scores were approximately from 1% to 9% worse depending on the lead time. The largest degradation was seen for the shortest lead time.
b. Covariate selection for quantile predictions
In the previous subsection the jth ensemble quantile was applied as covariate for predicting the jth quantile, but this may not necessarily be optimal. Several other covariate definitions are here examined:
jth ensemble quantile;
ensemble control member;
ensemble median;
ensemble mean;
the average of the jth ensemble quantile and the ensemble mean;
- the Bernstein polynomial estimator of the jth quantile defined aswhere
the average of the jth Bernstein quantile estimator and the ensemble mean.
Quantile skill scores (%) for a selection of quantile covariates with the ensemble quantile as reference.
c. Comparison with other ensemble postprocessing methods
CQRS is in this section compared to the LN model and the TN-LN model described in section 3c. Two CQRS models are fitted; one based on the ensemble quantile covariate (CQRSq) and the other based on the average of ensemble quantile and ensemble mean (CQRSqm). All models were trained separately for each site and lead time and predictions of 51 quantiles were made on the test data. In Table 3 quantile skill scores with TN-LN as reference are given. First, it can be noticed that the raw ensemble forecasts are far worse than all calibrated forecasts. Second, LN was slightly less skillful than TN-LN which confirms the findings of Baran and Lerch (2016). Overall the CQRSqm generated the best forecasts, but only marginally better than TN-LN with the largest improvement (0.7%) at +12 h.
Quantile skill scores (%) for all models and lead times with the TN-LN as reference. RAW denotes the raw ECMWF ENS forecasts.
The statistical significance of the latter was tested by means of bootstrap (Efron and Tibshirani 1994) as follows. First, the difference in quantile score between CQRSqm and TN-LN was calculated for each point in time and space (45 477 in total). Brief investigations of dependencies revealed that autocorrelations were more or less insignificant at lag one for the majority of sites, while for some they were clearly significant. For sites close to each other spatial correlation was evident. Based on this, bootstrap sampling with replacement was carried out in blocks of two subsequent points in time and all sites. In total 25000 bootstrap replications were performed. The fractions of these where CQRSqm was better than TN-LN were 100%, 99.9%, 97.7%, 94.8%, and 75.6% for lead times +12 h, +36 h, … , +108 h. Thus, apart from the longest and maybe second longest lead times, it can be concluded that CQRSqm is better than TN-LN in a statistical sense with respect to the quantile score.
Quantile scores averaged over all quantile levels may hide interesting details. For example, for the statistical models the 50% quantile contributed about seven times more to the overall quantile score than the 98% quantile. That is, the overall quantile score is focused on the central part of the forecast distributions. The quantile skill scores for CQRSqm relative to TN-LN were here larger for the low and high quantiles than on average. For the 98% quantile the QSS varied between 2% and 4% depending on the lead time, while the numbers for the 2% quantile were between 1% and 2%.
Averaging over all forecast situations may also conceal useful information. In Fig. 4 forecasts are grouped according to whether the ensemble mean is low, medium, or high. For both low and high ensemble means the QSS is in clear favor of CQRS with up to nearly 6% improvement, while for the medium group TN-LN is roughly 0.4% better than CQRSqm. A possible explanation could be that the simple linear relation in the location parameters of TN-LN is too restrictive. The LN model is clearly the worst model for low wind speeds, but slightly better than TN-LN for high wind speeds. Grouping the forecasts according to other ensemble statistics gave only minor variations and no obvious patterns. Last, it was investigated whether there were any significant variations in the QSS for CQRS when stations were grouped by maximum observed wind speed and the correlations as in Fig. 1 into low, medium, and high values. Also in this case there were no notable variations.
Quantile skill scores (%) as function of lead time grouped by whether the raw ensemble mean is low (less than the 10th percentile), medium (between the 10th and 90th percentiles), or high (larger than 90th percentile). QSS for the models LN, CQRSq, and CQRSqm are shown based on TN-LN as reference.
Citation: Monthly Weather Review 147, 5; 10.1175/MWR-D-18-0420.1
Reliability was first assessed on an overall basis, but that did not reveal any major differences between the models although the CQRS seemed to be the most reliable. Larger differences were seen when forecasts were grouped by the ensemble mean. In Fig. 5 deviations in quantile reliability (observed proportion minus nominal probability) are shown for lead times +12 and +108 h. Clearly, for low and high ensemble means TN-LN and especially LN has large deviations, while CQRS seems reasonably good. The deviations are largest for the mid quantiles. In general reliability improves with increasing lead time for all models.
Deviation from quantile reliability grouped by whether the raw ensemble mean is low (less than the 10th percentile), medium (between the 10th and 90th percentiles), or high (larger than 90th percentile) for lead times (top) +12 h and (bottom) +108 h. The statistic is shown for the models LN, TN-LN, CQRSq, and CQRSqm.
Citation: Monthly Weather Review 147, 5; 10.1175/MWR-D-18-0420.1
Scores for sharpness are shown in Table 4. The ECMWF ENS underestimates uncertainty severely and, consequently, has too narrow forecast intervals. For the 50% coverage interval the LN model has on average the shortest central intervals among the calibrated models. Also the TN-LN produced slightly shorter central intervals than the CQRS methods. For composite intervals, however, the CQRS models are superior. The average length is reduced by roughly 1 m s−1 (up to almost 50%) indicating frequent multimodal forecast distributions or clustered quantiles. Since quantiles are estimated separately in CQRS they could perhaps be too clustered by design. The result should therefore be used with caution. For the 88% coverage interval, CQRSqm generated shorter intervals than the other methods both for central and composite intervals. As expected the difference between the two types is less due to fewer configurations to assign the large probability mass. Finally, it can be noticed that the interval lengths somewhat surprisingly did not vary much by lead time. A possible explanation is that the predictability/correlation at some stations is almost independent on lead time. This affects in particular the shortest lead times.
Average length of forecasts intervals with 50% and 88% coverage probabilities for the models grouped by interval type and lead time. RAW denotes the raw ECMWF ENS forecasts.
For operational use it is vital that extreme forecasts are physically realistic. By construction the CQRS forecasts are constrained to not exceed wind speeds greater than 130% of the maximum in the training set at each site. For both LN and TN-LN there are no upper limits. As a first check, the number of forecasts with at least one quantile above 45 m s−1, which is approximately the highest wind speed ever recorded in the station network of the Norwegian Meteorological Institute, is computed. For the LN and TN-LN models 52 (0.11%) and 6 (0.01%) forecasts exceeded the 45 m s−1 criterion. The highest predicted quantile for TN-LN was about 62 m s−1 and similar for the LN model. Next, the number of forecasts with quantiles twice as high as the station maxima are counted. These were 81 (0.18%) and 4 (0.01%) for the LN and TN-LN models. With a threshold of 1.5 times the observed maxima these increased to 432 (0.95%) and 144 (0.32%), respectively. Although the occurrences are rare they indicate that the upper tails in forecast distributions generated by these methods need further investigations.
d. Refinements
Even though the CQRS approach guarantees noncrossing valid quantiles, separate estimation of many quantiles will in practice often lead to small bumps in the underlying forecast density function usually without any physical reasoning. It is therefore of interest to investigate whether all quantiles need to be estimated. One possibility is to just estimate some quantiles by CQRS and compute the rest simply by interpolation. A brief study where only every 2nd, 5th, 10th, and 25th quantile were estimated by CQRS, while those between were obtained by linear interpolation, was carried out. The lowest and highest quantiles were always estimated by CQRS. Thus, every 2nd quantile, for example, referred to quantiles 1/52, 3/52, 5/52, etc. and similar for the other step sizes. In Table 5 the quantile skill scores with all quantiles estimated by CQRS as reference are shown. Based on these, clearly only every 5th quantile needs to be estimated by CQRS without any loss in forecast skill. A positive side effect is that the computational burden could be reduced by a factor of almost 5 also. Another possibility to achieve smoother underlying distributions is to smooth the regression coefficients, see Koenker (2005, chapter 5) for an overview. This is left for future studies.
Quantile skill score (%) when estimation of quantiles is partly replaced by linear interpolation. The scores are given for 26, 11, 6, and 3 evenly spaced CQRS estimated quantiles, while the remaining quantiles are obtained by interpolation. The quantile score for CQRS estimation of all 51 quantiles is used as reference.
The results from section 4a show that almost the same score could be achieved by several spline configurations. To further increase robustness forecasts from these could potentially be combined. Averaging forecasts generated by splines with one and two interior knots gave approximately 0.1% improvement relative to the former. Thus, only minor improvements on the overall performance could be expected. The bootstrap is another averaging technique which is applied in many situations to reduce estimation uncertainty and thereby also increasing the forecast skill. A test with 50 resamples of the training data improved the quantile score of CQRS by only about 0.15% which may indicate that the estimated splines are already quite stable.
5. Discussion
In contrast to methods like the TN-LN mixture CQRS does not provide fully specified probability distributions, but only a finite set of quantiles. This limitation may, however, not be of much practical concern. As seen, intermediate quantiles can be obtained by interpolation, while quantiles beyond the range of estimated quantiles could for example be attained by fitting generalized Pareto distributions for the tails (see, e.g., Friederichs 2010). It should also be added that CQRS is not limited to the prediction of the quantiles chosen here. Any set is possible. One only needs to define reasonable covariates. Probabilities for events below or above certain thresholds could be calculated by inverting interpolated quantiles.
There are several studies showing that including several variables from the NWP ensemble may improve the forecasts (see, e.g., Taillardat et al. 2016; Messner et al. 2017; Eide et al. 2017). With the CQRS method it is unfortunately not straightforward to include more variables and further research is required. For two covariates splines in terms of tensor products of B-splines could be used, but further generalization would likely lead to too many parameters to estimate. An alternative would be to take an additive approach with univariate splines, but imposing constraints on the fits may be challenging. It is, however, always possible to reorder quantiles in the end if needed. A third alternative would be two split the process in two stages. First, any regression method for the mean, say, could be applied to make predictions for each ensemble member which at the next step are applied as input to the CQRS.
In future studies it would be interesting to compare CQRS with other quantile regression approaches applied to ensemble weather forecasting, in particular those of Bremnes (2004), Cannon (2011), and Wahl (2015). Concerning computational demand the CQRS likely compares favorably. An R implementation based on the packages quantreg and splines is computationally very fast. The training of 51 quantiles with nearly 400 training cases took in this study around 0.1 s on a single core on an Intel i7-3770 3.4 GHz CPU (2012 model). Since all quantiles can be estimated in parallel, the time for training could in principle be reduced to a few milliseconds on suitable hardware. Computer codes are available from the author.
6. Conclusions
In this article it is demonstrated how quantile regression with constrained spline functions can be applied to turn NWP ensemble forecasts into well calibrated quantile forecasts. Since assumptions on the shape of the underlying predictive distribution are not made, the CQRS can better adapt to variations in forecast uncertainty than methods based on simple parametric distributions. It also implies that CQRS could be applied to other meteorological variables with at most minor modifications. Further, all ensemble members can be fully utilized which implies that forecast quantiles can resemble the raw ensemble distribution if there is support in the data for this. Another convenient feature is that CQRS is computationally cheap. For future work the main challenge will likely be to find adequate ways of including more than one ensemble variable to the covariates.
The CQRS was here tested on wind speed forecasting data at 125 Norwegian synop stations. On these, results show that CQRS was slightly better, less than 1% in terms of quantile skill score, and more reliable than the truncated normal and lognormal mixture method proposed by Baran and Lerch (2016). For strong wind speed forecasts improvements up to about 4.5% were seen.
Acknowledgments
The author wish to thank Sándor Baran for sharing details on the implementation of the TN-LN mixture method.
REFERENCES
Baran, S., and S. Lerch, 2015: Log-normal distribution based EMOS models for probabilistic wind speed forecasting. Quart. J. Roy. Meteor. Soc., 141, 2289–2299, https://doi.org/10.1002/qj.2521.
Baran, S., and S. Lerch, 2016: Mixture EMOS model for calibrating ensemble forecasts of wind speed. Environmetrics, 27, 116–130, https://doi.org/10.1002/env.2380.
Baran, S., and S. Lerch, 2018: Combining predictive distributions for statistical post-processing of ensemble forecasts. Int. J. Forecast., 34, 477–496, https://doi.org/10.1016/j.ijforecast.2018.01.005.
Ben Bouallegue, Z., 2017: Statistical postprocessing of ensemble global radiation forecasts with penalized quantile regression. Meteor. Z., 26, 253–264, https://doi.org/10.1127/metz/2016/0748.
Bremnes, J. B., 2004: Probabilistic forecasts of precipitation in terms of quantiles using NWP model output. Mon. Wea. Rev., 132, 338–347, https://doi.org/10.1175/1520-0493(2004)132<0338:PFOPIT>2.0.CO;2.
Cannon, A. J., 2011: Quantile regression neural networks: Implementation in R and application to precipitation downscaling. Comput. Geosci., 37, 1277–1284, https://doi.org/10.1016/j.cageo.2010.07.005.
Cheng, C., 1995: The Bernstein polynomial estimator of a smooth quantile function. Stat. Probab. Lett., 24, 321–330, https://doi.org/10.1016/0167-7152(94)00190-J.
Efron, B., and R. J. Tibshirani, 1994: An Introduction to the Bootstrap. Monogr. on Statistics and Applied Probability, No. 57, Chapman and Hall/CRC, 456 pp.
Eide, S. S., J. B. Bremnes, and I. Steinsland, 2017: Bayesian model averaging for wind speed ensemble forecasts using wind speed and direction. Wea. Forecasting, 32, 2217–2227, https://doi.org/10.1175/WAF-D-17-0091.1.
Friederichs, P., 2010: Statistical downscaling of extreme precipitation events using extreme value theory. Extremes, 13, 109–132, https://doi.org/10.1007/s10687-010-0107-5.
Friederichs, P., and A. Hense, 2007: Statistical downscaling of extreme precipitation events using censored quantile regression. Mon. Wea. Rev., 135, 2365–2378, https://doi.org/10.1175/MWR3403.1.
Gneiting, T., and A. E. Raftery, 2007: Strictly proper scoring rules, prediction, and estimation. J. Amer. Stat. Assoc., 102, 359–378, https://doi.org/10.1198/016214506000001437.
Hamill, T. M., and J. S. Whitaker, 2006: Probabilistic quantitative precipitation forecasts based on reforecast analogs: Theory and application. Mon. Wea. Rev., 134, 3209–3229, https://doi.org/10.1175/MWR3237.1.
Hartigan, J. A., and P. M. Hartigan, 1985: The dip test of unimodality. Ann. Stat., 13, 70–84, https://doi.org/10.1214/aos/1176346577.
Koenker, R., 2005: Quantile Regression. Econometric Society Monogr., No. 38, Cambridge University Press, 366 pp.
Koenker, R., and J. Bassett, 1978: Regression quantiles. Econometrica, 46, 33–50, https://doi.org/10.2307/1913643.
Lerch, S., and T. L. Thorarinsdottir, 2013: Comparison of non-homogeneous regression models for probabilistic wind speed forecasting. Tellus, 65A, 21 206, https://doi.org/10.3402/tellusa.v65i0.21206.
Messner, J. W., G. J. Mayr, and A. Zeileis, 2017: Nonhomogeneous boosting for predictor selection in ensemble postprocessing. Mon. Wea. Rev., 145, 137–147, https://doi.org/10.1175/MWR-D-16-0088.1.
Molteni, F., R. Buizza, T. N. Palmer, and T. Petroliagis, 1996: The ECMWF ensemble prediction system: Methodology and validation. Quart. J. Roy. Meteor. Soc., 122, 73–119, https://doi.org/10.1002/qj.49712252905.
Murphy, A. H., 1988: Skill scores based on the mean square error and their relationships to the correlation coefficient. Mon. Wea. Rev., 116, 2417–2424, https://doi.org/10.1175/1520-0493(1988)116<2417:SSBOTM>2.0.CO;2.
Nelder, J. A., and R. Mead, 1965: A simplex method for function minimization. Comput. J., 7, 308–313, https://doi.org/10.1093/comjnl/7.4.308.
Nielsen, H. A., H. Madsen, and T. S. Nielsen, 2006: Using quantile regression to extend an existing wind power forecasting system with probabilistic forecasts. Wind Energy, 9, 95–108, https://doi.org/10.1002/we.180.
Raftery, A. E., T. Gneiting, F. Balabdaoui, and M. Polakowski, 2005: Using Bayesian model averaging to calibrate forecast ensembles. Mon. Wea. Rev., 133, 1155–1174, https://doi.org/10.1175/MWR2906.1.
R Core Team, 2018: R: A language and environment for statistical computing. R Foundation for Statistical Computing, https://www.R-project.org/.
Richardson, D. S., 2000: Skill and relative economic value of the ECMWF ensemble prediction system. Quart. J. Roy. Meteor. Soc., 126, 649–667, https://doi.org/10.1002/qj.49712656313.
Scheuerer, M., and T. M. Hamill, 2015: Statistical postprocessing of ensemble precipitation forecasts by fitting censored, shifted Gamma distributions. Mon. Wea. Rev., 143, 4578–4596, https://doi.org/10.1175/MWR-D-15-0061.1.
Schumaker, L., 2007: Spline Functions: Basic Theory. 3rd ed. Cambridge University Press, 600 pp., https://doi.org/10.1017/CBO9780511618994.
Taillardat, M., O. Mestre, M. Zamo, and P. Naveau, 2016: Calibrated ensemble forecasts using quantile regression forests and ensemble model output statistics. Mon. Wea. Rev., 144, 2375–2393, https://doi.org/10.1175/MWR-D-15-0260.1.
Thorarinsdottir, T. L., and T. Gneiting, 2010: Probabilistic forecasts of wind speed: Ensemble model output statistics by using heteroscedastic censored regression. J. Roy. Stat. Soc., 173A, 371–388, https://doi.org/10.1111/j.1467-985X.2009.00616.x.
Toth, E., and E. Kalany, 1993: Ensemble forecasting at NMC: The generation of perturbations. Bull. Amer. Meteor. Soc., 74, 2317–2330, https://doi.org/10.1175/1520-0477(1993)074<2317:EFANTG>2.0.CO;2.
van den Dool, H. M., 1989: A new look at weather forecasting through analogues. Mon. Wea. Rev., 117, 2230–2247, https://doi.org/10.1175/1520-0493(1989)117<2230:ANLAWF>2.0.CO;2.
Vannitsem, S., D. Wilks, and J. Messner, Eds., 2018: Statistical Postprocessing of Ensemble Forecasts. 1st ed. Elsevier, 362 pp.
van Schaeybroeck, B., and S. Vannitsem, 2015: Ensemble post-processing using member-by-member approaches: Theoretical aspects. Quart. J. Roy. Meteor. Soc., 141, 807–818, https://doi.org/10.1002/qj.2397.
Wahl, S., 2015: Uncertainty in mesoscale numerical weather prediction: Probabilistic forecasting of precipitation. Bonner meteorologische abhandlungen, heft 71, University of Bonn, 108 pp.
Wilks, D. S., 2018: Enforcing calibration in ensemble postprocessing. Quart. J. Roy. Meteor. Soc., 144, 76–84, https://doi.org/10.1002/qj.3185.
