1. Introduction
Increasingly, meteorological and hydrological ensemble forecasts have been used by decision makers to judge whether or not to take an action to protect against a possible loss. These ensemble forecasts allow users to estimate the probability distribution function of forecasts states, or, in other words, to compute the probability of occurrence of events. The increased use of these new, mainly probabilistic, forecasts has increased the demand for information on their quality. Assessing the quality of ensemble forecasts is rather complex and involves the use of assessment metrics that are different from the ones normally used to assess single forecasts. The reader is referred to, for example, Wilks (1995), for a comprehensive discussion of the issue of the evaluation of the accuracy of probabilistic forecasts.
Ensemble forecasts have been used in hydrological applications within the European Commission’s Prevention, Information and Early Warning project (PREVIEW; information online at http://www.preview-risk.com). Within this project, Buizza et al. (2007a) have discussed issues linked to the validation of the overall forecast value of probabilistic meteorological and hydrological forecasts, and have introduced a general framework that can be used to assess both their technical and functional qualities. Pappenberger et al. (2009) have made some specific recommendations on verification methods that are more relevant for hydrological applications, so that the information can be easily interpreted by hydrologists who use meteorological data to drive hydrological models.
This paper applies some of the ideas of Pappenberger et al. (2009) to European Centre for Medium-Range Weather Forecasts (ECMWF) ensemble forecasts of meteorological variables relevant for hydrological applications for the river Danube. Forecasts for two variables, 2-m temperature and total precipitation, are verified on three spatial scales: the upper Danube (which is upstream of Bratislava, Slovakia), the entire Danube catchment, and the whole of Europe. The seasonality of the forecast performance is also discussed. In the first part of this paper, the characteristics of the Danube catchment are described, and the skill scores and performance measures used are introduced. This is followed first by an analysis first of a 3-month period, and then of a longer 5.5-yr period. Finally, the average model performance over lead time will be discussed, and some conclusions will be drawn.
a. Catchment description
In this study we focus on three spatial scales (Fig. 1, Table 1): the upper Danube (which is upstream of Bratislava), the entire Danube catchment, and the whole of Europe. The Danube catchment was selected because it is part of the PREVIEW project, which investigates the performance of flood forecasting using the July–August 2002 floods in the Danube catchment as an example. This reflects the need to perform catchment-based evaluations for hydrologists, and to encompass an area with enough observations to perform a robust analysis. In Table 1 the area, number of ECMWF Ensemble Prediction System (EPS) grid points, and the number of observations in each catchment are listed. Table 1 shows that even the Danube, which is the largest European catchment, is still small compared to the continental scale on which meteorological verification is performed traditionally.
2. Description of skill measures and variables
Two variables (2-m temperature and precipitation) will be analyzed in this study. The evaluation of the forecasts will be performed with three methods: eyeball verification, the Nash–Sutcliffe measure, and the Brier skill score. Moreover, the water balance of water years will be computed as it is of hydrological interest.
Eyeball verification is often used as a standard verification tool for meteorological forecasts (Mariani et al. 2005) and tests for consistency between numerical value and forecasters’ experience and opinion. Cloke and Pappenberger (2008) have used this method to gain additional understanding in the behavior of more traditional skill measures. In this paper, it will be used to evaluate time series of precipitation and temperature (e.g., by analyzing peak behavior) and to support the findings of all other scores.
The Nash–Sutcliffe measure will be used to compare accumulation periods of precipitation over 24 h as well as over the entire lead time throughout this paper.
In this paper, all Brier skill scores will be computed on precipitation accumulated over 24 h. We have refrained from calculating scores that use precipitation results accumulated over entire lead times, as this would require an assumption about the threshold distribution. For example, one could assume a threshold of 5 mm for the first 24 h, of 10 mm for a lead time of 48 h, and of 15 mm for a lead time of 72 h. However, this cannot be justified by the nonlinear nature of the score.
The Brier skill score and Nash–Sutcliffe efficiency show similar trends and allow for identical interpretations. Therefore, results are displayed with only one of the two scores to keep this paper to a reasonable length, although all analysis includes both scores.
Closing the water balance is an important aspect in hydrological modeling and an essential hydrological property (Beven 2006). Therefore, it is of interest for hydrological applications to evaluate the bias of the ECMWF forecasts in predicting the water balance. The bias is computed by accumulating precipitation amounts of a given lead time over a water year (October–September) and dividing it by the observed amounts. A value of 1 indicates that the forecast predicted the same amount of water as the observations. A value below 1 indicates that the forecast predicts less water than actually falls and a value above 1 vice versa. The observations of each water year are compared to a long-term average of observations in a similar fashion to indicate whether it was a dry (fraction is below 1) or wet year (fraction is above 1).
3. ECMWF model description
The past two decades have seen an expanding use of ensemble systems in numerical weather prediction. These systems provide forecasts and estimates of the probability distribution of forecast states, which can be used not only to identify the most likely outcome, as was done in the past using single, high-resolution forecasts, but also to assess the probability of occurrence of weather event of interest (see, e.g., Buizza 2008). More recently, hydrological ensemble prediction models have been developed either to directly use global meteorological ensemble forecasts as input, or to use dynamically downscaled meteorological ensemble forecasts (Thielen et al. 2008).
The ECMWF was one of the first two centers, together with the National Centers for Environmental Prediction in Washington, D.C., to include an ensemble prediction system as part of its operational suite in 1992. The rainfall and temperature forecasts used in this study are based on the latest version of the ECMWF ensemble forecasts (known as the Ensemble Prediction System or EPS). These forecasts provide 51 realizations for a 15-day lead time (in this paper we use only 10 as the results are shown over a longer time period). The ECMWF EPS simulates the effects of uncertainties in the initial conditions by starting the 50 perturbed forecasts from initial conditions generated by adding to the control (unperturbed) analysis perturbations that are in turn generated using singular vectors, and it simulates the effects of model uncertainties using a stochastic physics scheme. A detailed description of the ECMWF numerical weather prediction model can be found in a number of references (e.g., Buizza et al. 1999, 2007). The forecasts have been run with spectral triangular truncation T399 with linear grid (which is equivalent to approximately 50-km resolution at mid latitudes) and 40 vertical levels, with initial time of 1200 UTC.
4. Results of the evaluation of ECMWF forecasts for hydrological applications
The evaluation of the results will be split into three parts. In the first part the predicted and observed values for a time period of 3 month in 2002 will be analyzed. This is followed by an investigation of the performance of ECMWF forecasts over a longer time frame (for the period between January 2002 and June 2007). Finally, in the third part, the evaluation will focus on different lead times. We will compare the three spatial scales in the first half of this paper and concentrate only on the Danube in the latter as this reflects the general trends.
a. Model performance over 3 months (April–June 2002)
The limited time frame of 3 months allows us to plot the actual predicted and observed values and the supporting eyeball verification. First, the analysis will concentrate on precipitation accumulated over the whole lead time. Whole lead time means an accumulation starting from time 0; for example, a 196-h forecast is analyzed based on the precipitation accumulating between 0 and 196 h. Then, the results will be compared to an accumulation period of 24 h. Finally, the results of the temperature evaluation will be shown.
1) Precipitation (accumulated over whole lead time)
In Fig. 2 the hyetograph for the upper Danube catchment is shown (a filter of 30 days is applied to all graphs in this section to show trends). The plot at the top of Fig. 2 shows the 24-h predictions of the control and the high-resolution forecast. The bottom plot of Fig. 2 shows the median of the ensemble and the 10th and 90th percentiles of the ensemble distribution. The observations are marked as dashed lines. Figure 2 shows that the predictions of 24-h accumulated precipitation with 1-day lead time have correct timing. The Nash–Sutcliffe result is below 0.6 and thus very low. However, all forecasts (control, high resolution, and ensemble) can still be classified as skillful in comparison to a climatological mean prediction. A significant proportion of the observations lie outside the uncertainty bounds representing the 10th and 90th percentiles of the predictions, a total of 72%: in a perfectly reliable system, one should expect this to happen 20% of the time. This result indicates that for 24-h precipitation over this region and during this verification period, the ensemble spread is too narrow. Furthermore, the highest peaks of precipitation are often underestimated.
In Figs. 3a–c forecasts with a lead time of 5 days and accumulated over the entire forecast period are shown for the area of Europe (Fig. 3a), the Danube catchment (Fig. 3b), and the upper Danube subcatchment (Fig. 3c). Overall, the predictions for the area of Europe are better and have significantly higher Nash–Sutcliffe values than do the other two areas. For example, the Nash–Sutcliffe value for the high-resolution forecast for Europe is 0.58 and 0.21 for the Danube. The Nash–Sutcliffe value for the upper Danube subcatchment is 0.41. One would expect that averaging over larger areas would result in better scores as more smoothing is applied. However, this cannot be seen in a comparison between the Danube catchment and its upstream subcatchment: in this case, the subcatchment achieves higher scores. The observed hyetographs show that the precipitation in the entire Danube catchment is dominated by the precipitation falling in the upstream subcatchment. More detailed analysis reveals that, in this particular time period, forecasts for the eastern part of the Danube catchment in the Tisza River catchment have been of lower quality. The percentages of simulations above and below the 10th and 90th percentiles over Europe are significant lower, only 28% for the entirety of Europe compared to 39% and 44% for the Danube and its upstream catchment, respectively. A smaller area will naturally exhibit a higher variance in any response variable and thus produce a larger number of outliers. Unfortunately, neither the Danube catchment nor its subcatchment upstream of Bratislava exhibits much skill. Nevertheless, the timing between the predicted and forecasted peaks is excellent and individual peaks are predicted very well. For example, the high precipitation amounts on 10 June are well predicted, although the storm around 5 May has been missed.
2) Precipitation (accumulated over 24 h)
Traditionally, precipitation forecasts are verified with 24-h accumulated values. However, this is only hydrologically useful if the catchment memory is 24 h. In Fig. 4, the 24-h accumulated precipitation for a lead time of 5 days is shown for the upstream Danube subcatchment (the other domains follow a similar pattern). This 24-h accumulated forecast yields lower performance than Fig. 3c, with worse Nash–Sutcliffe values. The width of the ensemble distribution (bottom plot in Fig. 3c) is considerably larger for the forecast accumulated over the entire lead time in comparison to the forecast accumulated only over 24 h. There is no significant bias in the peak predictions for the forecast accumulated over the longer time period. The timing of the forecasts with respect to the observations has considerably deteriorated for the 24-h accumulated precipitation.
3) Temperature
Figures 5a–c show 2-m temperature with a lead time of 5 days for the three evaluation areas. It can be seen that the Nash–Sutcliffe value is very high even at a lead time of 5 days, with values of 0.94, 0.79, and 0.65 for the high-resolution forecast of the entirety of Europe, the Danube, and its upstream subcatchment, respectively. The ensemble encompasses the observations well; for example, only 4% of all observations are above (below) the upper Danube subcatchment’s 90th (10th) percentile. This indicates that for this variable the ensemble is more reliable than for 24-h precipitation [see section 4a(1)]. A small negative bias can be detected over Europe and the Danube as both have significantly more outliers above 90% than below 10%. This is less apparent for the smaller upper Danube subcatchment. Moreover, the upper Danube subcatchment has a lower Nash–Sutcliffe score of 0.65 for the high-resolution forecast than does the Danube, with a Nash–Sutcliffe value of 0.79, while the whole of Europe has the largest scores (Nash–Sutcliffe value for the high-resolution forecast of 0.94). This can again be explained by smoothing. In general, the ensemble range is very large (sometimes above 10°C), which would have a significant impact on the results of hydrological models, for example, snowmelt-driven flood predictions.
In summary, the analysis of the 3-month period shows that precipitation forecasts follow qualitatively the pattern of the observations, but underestimate its peaks. A large proportion of the observations fall outside the 10th and 90th percentiles given by the EPS. Although the timing of the peaks between the forecasted and observed precipitation and temperature is good, the observed values often lay outside the ensemble forecast range. The model’s performance over Europe is in general better than the performance over smaller subcatchments. The Nash–Sutcliffe value of the high-resolution forecast, and the control and the median of the ensembles values for precipitation are very low even for a 1-day forecast. Accumulation over lead time improves the skill score in comparison to a short accumulation period, such as 24 h. In contrast, temperature forecasts are very good even with a 5-day lead time.
b. Model performance over 5.5 yr (January 2002–June 2007)
In this section the trends of a 5.5-yr period will be investigated with the Brier skill score using the Ensemble Prediction System. Our conclusions can be replicated with the Nash–Sutcliffe score. The sensitivity of the forecast quality to the catchment size will be discussed in the first part of this section. This is followed by an analysis of the time series of the Brier skill score with four different thresholds for the 5.5 yr. Afterward, the trend in skill for three different lead times over the Danube catchment will be analyzed. Finally, an evaluation of the water balances concludes this section.
1) Catchment comparison (precipitation)
Catchment scale has a significant influence on the performance values of weather forecasts. Figure 6 shows that forecasts in the Danube area and upper Danube subcatchment are on average less skillful in winter, have a higher variance, and have significantly larger peaks and troughs in comparison to the continental scale. Effects as the ones observed in Fig. 6 are explained by the smaller climatic range over which a catchment usually spans. Additionally, the size of the catchment (and thus the size of the verification domain) plays an important role as larger catchments usually have a larger density of data and more of a smoothing effect. Moreover, the resolution of the weather forecast model is often too coarse to resolve the small catchment scale. The larger a catchment or verification area, the greater the climatological coverage of the regions will be and the more data there will be to build reliable averages. Kann and Haiden (2005) observed a reduction in mean absolute error with increasing verification area. The total and main sensitivities of the discharge predictions will depend on the spatial covariance structure of the variables and the nonlinear transformation through the hydrological model. For example, in the case of precipitation, the sensitivity of the river flow hydrograph toward the uncertainty in precipitation on catchment response decreases with catchment scale (Rodriguez-Iturbe and Mejia 1974; Segond 2006).
Other thresholds indicate a similar picture, although the absolute values of the Brier skill score are lower. In what follows, the analysis will concentrate on the entire Danube catchment and we will omit further comparisons with the continental or subcatchment scales as similar conclusions can be drawn from the different scales.
2) Comparison of Brier skill score for four thresholds (precipitation)
Figure 7 illustrates a long-term picture (January 2001–June 2007) for the Brier skill score of the ensemble forecasts with four thresholds for a 1-day lead time and a 24-h accumulation period over the Danube catchment. The skill is computed in comparison to climatologically averages based on a 20-yr period of observations (for details see Rodwell 2005). In general, skill deteriorates with increasing thresholds, but there are notable exceptions, such as spring 2003, which can be mainly explained by event frequency (there was a strong drought in the Danube basins and therefore the skill scores of large thresholds have rarely been exceeded). Figure 7 shows that forecasts above 1 and 5 mm (24 h)−1 are largely skillful over the entire forecast period. Predictions above 10 and 20 mm (24 h)−1 can be skillful, especially in winter. In Fig. 7, the precipitation skill seems to have no long-term (e.g., 5 yearly) trend (despite changing forecast models). A small positive trend can be detected in winter 2006/07 for small thresholds: this is likely linked to the implementation of the higher-resolution ECMWF Variable Resolution EPS (VAREPS) system in September 2006 (Buizza et al. 2007b) and introduction of the new model cycle 31r1 (Untch et al. 2006), although one cannot rule out the possible influence of seasonal variations.
The Brier skill score is not only influenced by the choice of threshold, but also by the length of the lead time, as is discussed in the next section.
3) Comparison of Brier skill scores for three lead times (precipitation)
In Fig. 8 the skill scores of predications above 1 mm (24 h)−1 for three different lead times are shown over the Danube catchment. Overall, it is not surprising that forecasts with a shorter lead time show a higher skill score than forecasts with a longer lead time. However, notable exceptions exist (e.g., summer 2006). This is more likely to happen for an underperforming forecast (indicated by very low skill scores), but can also be observed for forecasts of high quality [see, e.g., the study by Grazzini (2007) and the discussion in Pappenberger et al. (2009)]. This is on the one hand a property of the formulation of the Brier skill score and on the other hand simply due to the fact that for individual forecasts long-term predictions can outperform short-term predictions (especially when the forecast is accumulated over a longer time period). Figure 7 shows (as Fig. 6) no clear upward trend in skill score from 2001 till the end of 2007, and there is slight evidence of improvement after December 2007. The skill levels for the 1- and 5-day lead times are mostly positive, although prolonged periods of negative skill can be observed. The 10-day lead time is sometimes positive in this catchment.
4) Bias
Figures 6–and 8 showed time series of performances continuously from January 2002 to June 2007. However, for hydrological applications, the verification of meteorological forecasts of precipitation has to be extended to the water year (staring in October and finishing in September) because it allows for an evaluation that is based on the annual cycle associated with the natural progression of the hydrologic seasons. In Fig. 9 the mass balances (equivalent to a model bias) of precipitation amounts are computed for each water year computed monthly for a year to date. A value of 1 indicates that the forecast for this particular lead time predicted the same amount of water as the observations. A value below 1 indicates that the forecast predicts less water than actually falls and vice versa for a value above 1. The observations of each water year are compared to a long-term average in a similar fashion to indicate whether it was a dry (fraction is below 1) or wet year (fraction is above 1).
No relationship between bias and precipitation amount could be established. In more detail, Fig. 9 shows that the ensembles tend to underpredict the total amount of precipitation falling in the Danube catchment. The underestimation ranges between 5% and 10% for the precipitation accumulated over 10 days. On average the high-resolution forecast and the ensemble mean also underestimate the precipitation amounts. However, during the last water years the high-resolution forecast overestimates the water volume. The relationship between wet and dry years and the over- or underestimation is weak. The last water year seems to have been particularly well predicted.
5) Time series of Nash–Sutcliffe for 2-m temperature
The 2-m temperature analysis is based on the Nash–Sutcliffe skill score instead of the Brier skill score, as temperature is a more normally distributed variable and thus lends itself to a continuous skill score. In Fig. 10, the trend over the past 5.5 yr is shown for a lead time of 5 days.
The three different forecasts perform very similarly and as for the precipitation no long-term trend can be observed. The Nash–Sutcliffe is above 0.5 in a majority of the cases and thus is better than a climatological mean. However, a large proportion of the forecasts is below the Nash–Sutcliffe value of 0.6, which means that they have to be classified as nonskillful. No clear seasonal signal can be observed, as in the precipitation forecasts.
6) Summary of 5.5-yr model performance
The analysis of ECMWF forecasts over 5.5 yr has shown that catchment scale has a large influence on the variance as well as the extremes on the calculations of a skill score through time. The smaller a catchment, the larger is the variance. The analysis of the Brier skill score for four thresholds and three lead times suggests no trend in skill between 2002 and the end of 2006. An improvement seems to materialize for all lead times and small thresholds from the end of 2006 onward. The water balance analysis suggests a 10% underestimation by the ensemble mean and an overestimation by the high-resolution forecast over the last year. Temperature shows no trend in skill in the years between 2002 and the middle of 2007.
c. Average model performance over lead times
The analysis of the previous section concentrated on the evolution of scores for 24- and 120-h forecast lead times only. However, it is also important to see the development of scores over increasing lead times. This analysis will first be performed for 2-m temperature. This is followed by an evaluation of the 24-h accumulated precipitation and precipitation predictions accumulated over the entire lead time. Finally, the percentage of simulations above or below the 10th and 90th percentiles, the spread, and the mean error will be analyzed with respect to increasing lead time.
1) Temperature
In Fig. 11, the Nash–Sutcliffe score for the Danube catchment is plotted against the lead time in the form of box and whisker plots. Each box has lines at the lower quartile, median, and upper quartile. The whiskers are lines extending from each end of the box to show the extent of the rest of the data. A majority of the forecasts are skillful up to day 5, with a large proportion of forecasts still achieving a Nash–Sutcliffe value above 0.5 at 6 days. Figure 11 shows large uncertainty bounds.
2) Precipitation (accumulated over 24 h)
The Brier skill score as well as the Nash–Sutcliffe criteria will illustrate the performance of precipitation forecasts accumulated over 24 h.
(i) Brier skill score
Figure 12 shows the Brier skill score over the Danube catchment for exceeding 1 mm (24 h)−1 for the entire year and split into summer (June–August) and winter (November–January) seasons. Figure 12 deliberately focuses only on a threshold of 1 mm (24 h)−1 as the behavior for higher thresholds can be seen later in the analysis of the Nash–Sutcliffe criteria. Forecasts in winter are significantly more skillful, with a positive mean Brier skill score up to day 7. In summer, this mean positive score can only be achieved up to day 6. The performance over the entire year also allows skillful forecasts only up to day 6. The large uncertainty ranges in Fig. 11 indicate significant variance in the data (as seen previously in Fig. 6). The uncertainty bounds show that a large proportion of the forecasts are still skillful up to day 10. However, this also means that many forecasts are not skillful even when the mean Brier skill score for this particular lead time is positive.
(ii) Nash–Sutcliffe
In Fig. 12 the Nash–Sutcliffe criteria is plotted against the lead time in the form of box and whisker plots. Precipitation accumulated over 24 h is shown in Fig. 13. In Fig. 13 the median of the Nash–Sutcliffe criteria is above 0.5 for forecasts with up to a lead time of 72 h. The Nash–Sutcliffe criteria indicate fewer days of skillful forecasts than the Brier skill score (Fig. 12). This is due to the numerical formulation of the two scores. The Brier skill score is probabilistic and in Fig. 12 only a threshold of 1 mm (24 h)−1 is shown (effectively a rain–no-rain event). The Brier skill score and the Nash–Sutcliffe efficiency criteria achieve similarly skillful lead times when the Brier skill score is computer with a threshold of 5–10 mm (24 h)−1, which is a value closer to the mean precipitation of the catchment and thus similar to the Nash–Sutcliffe measure. Figure 13 also contains subplots, which split the year into summer (June–August) and winter (November–January). On average the winter scores are better than the summer scores (see also the Brier skill score figures above), with larger uncertainties in the winter months, which is linked to the summer–winter precipitation variations observed in the Danube catchment.
The previous analysis only concentrates on precipitation accumulated over 24 h. In what follows, the accumulation over thee entire lead times is analyzed.
3) Precipitation (accumulated over lead times)
Figure 14 shows the Nash–Sutcliffe criteria for precipitation accumulated over the entire lead time and the skillful predictions stretch up to days 4 and 5. The forecasts accumulated over a longer time are largely more skillful than forecasts accumulated over short time periods (see Fig. 13). The use of accumulated predictions is important as, for example, large error at the beginning of a forecast may trigger the hydrological process of saturation excess overland flow at a lead time of 120 h, whereas the forecast on day 4 alone (ignoring antecedent conditions) may lead to precipitation to infiltrate rather than flow overland. Figure 13 indicates that on average a successful forecast will be possible with a lead time of 4 to 5 days. The figures (Figs. 13 and 14) show clearly that skillful forecasts can exist for the entire forecast range.
The Nash–Sutcliffe criteria only analyzes the mean or median of an ensemble distribution. However, any probabilistic forecast also includes spread and a percentage of simulations beyond, for example, the predicted 10th and 90th percentiles.
4) Ensemble spread and percentage of outliers
The ensemble spread and percentage of outliers are also important quantities to make a meteorological forecast useful for hydrological predictions. The percentage of outliers is classified as the proportion of observations outside the 10th and 90th percentiles. Meteorological skill scores exist to evaluate this property. However, many hydrologists will be unfamiliar with those and therefore in Fig. 15 the spread, the percentage of outliers, and the error of the mean for the ensemble precipitation forecasts in the catchment are plotted against lead time for the Danube catchment. The average number of outliers drops over the lead time significantly faster for precipitation accumulated over 24 h than for accumulations over the entire lead time. Figure 15 suggests that the precipitation forecast for this particular catchment does not represent the full uncertainties in at least the first 3 days as the percentage of outliers is significantly above the anticipated 20%. The spread and error of a forecast accumulated over the entire lead time is naturally very high, but reflects the fact that the error gets propagated through the forecast.
5. Conclusions
In this paper the accuracy of ECMWF forecasts for hydrological applications is evaluated. Attention was focused on two variables, 2-m temperature and total precipitation, and on three spatial scales: the upper Danube (which is upstream of Bratislava), the entire Danube catchment, and the whole of Europe. The evaluation of the forecasts is performed via three methods: eyeball verification, the Nash–Sutcliffe measure, and the Brier skill score. Moreover, the yearly water balance is assessed. The evaluation is split into three parts. In the first part the predicted and observed values for a time period of 3 months in 2002 are analyzed. Then, the performance of ECMWF forecasts over a longer time (for the period between January 2002 and June 2007) is investigated. Finally, the evaluation focuses on the average performance over various lead times.
The analysis of the 3-month period shows that precipitation forecasts follow largely in pattern the observations although it is often underestimated. The timing of the peaks between the forecasted and observed precipitation and temperature shows a slight discrepancy. The Nash–Sutcliffe of the high-resolution forecast, the control, and the median of the ensembles values for precipitation are all very low even for a 1-day forecast (the reader is reminded that scores above 0.6 are usually seen as acceptable; see the discussion above). A large proportion of the observations fall outside the 10th and 90th percentiles given by the Ensemble Prediction System. Accumulation over lead time improves the skill score in comparison to a short accumulation period, such as 24 h. Compared to total precipitation, the 2-m temperature forecast is very good even with a 5-day lead time.
The analysis of ECMWF forecasts over 5.5 yr has shown that catchment scale has a large influence on the variance as well as on extremes. Results indicate that the smaller a catchment, the larger is the variance of the scores. The analysis of the Brier skill score for four thresholds and three lead times suggests no trend in skill between 2002 and the end of 2006. An improvement can be detected for all lead times and for small thresholds from the end of 2006 onward linked to the increased resolution and revision of the cloud scheme (Bechtold et al. 2008), in line with results documented in Richardson et al. (2008). The water balance analysis suggests a 10% underestimation by the ensemble mean and an overestimation by the high-resolution forecast over the last year. Temperature predictions are skillful up to day 6, with no trend in skill shown in the years between 2002 and the middle of 2007. Precipitation predictions are skillful up to day 7 if measured with a Brier skill score [exceeding 1 mm (24 h)−1]. The Nash–Sutcliffe criteria indicates skillful predictions up to days 4 and 5. Forecasts accumulated over a longer time period are largely more skillful than forecasts accumulated over shorter time periods. The percentage of outliers for the subcatchment of the Danube is very high, which suggests that the ensemble does not encompass the full uncertainty of the observed rainfall fields. It should be noted that the analysis in this paper ignores the uncertainty in the observations, which can have a significant influence on model performance and will need to be incorporated into future analyses (see Pappenberger et al. 2009 for discussion).
Acknowledgments
The work of this paper has been funded by the PREVIEW project. We thank Hannah Cloke (King’s College, London, United Kingdom), Martyn Clark (NIWA, Christchurch, New Zealand), and John Schaake (NOAA) for their comments that greatly improved the quality of this manuscript.
REFERENCES
Bechtold, P., Koehler M. , Jung T. , Doblas-Reyes F. , Leutbecher M. , Rodwell M. J. , Vitart F. , and Balsamo G. , 2008: Advances in simulating atmospheric variability with the ECMWF model: From synoptic to decadal time-scales. ECMWF RD Tech. Memo. 556, 24 pp.
Beven, K., 2006: Searching for the Holy Grail of scientific hydrology: Q(t) = H((S)under-left-arrow, (R)under-left-arrow, Delta t)A as closure. Hydrol. Earth Syst. Sci., 10 , 609–618.
Buizza, R., 2008: The value of probabilistic prediction. Atmos. Sci. Lett., 9 , 36–42. doi:10.1002/asl.170.
Buizza, R., Miller M. , and Palmer T. N. , 1999: Stochastic representation of model uncertainties in the ECMWF Ensemble Prediction System. Quart. J. Roy. Meteor. Soc., 125 , 2887–2908.
Buizza, R., and Coauthors, 2007a: EURORISK/PREVIEW report on the technical quality, functional quality and forecast value of meteorological and hydrological forecasts. ECMWF Research Department Tech. Memo. 516, ECMWF, Shinfield Park, Reading, United Kingdom, 63 pp.
Buizza, R., and Coauthors, 2007b: The new ECMWF VAREPS (Variable Resolution Ensemble Prediction System). Quart. J. Roy. Meteor. Soc., 133 , 681–695.
Cloke, H. L., and Pappenberger F. , 2008: Evaluating forecasts of extreme events for hydrological applications: An approach for screening unfamiliar performance measures. Meteor. Appl., 15 , 181–197.
Freer, J. E., McMillan H. , McDonnell J. J. , and Beven K. J. , 2004: Constraining dynamic TOPMODEL responses for imprecise water table information using fuzzy rule based performance measures. J. Hydrol., 291 , 254–277.
Grazzini, F., 2007: Predictability of a large-scale flow conducive to extreme precipitation over the western Alps. Meteor. Atmos. Phys., 95 , 123–138.
Kann, A., and Haiden T. , 2005: The August 2002 flood in Austria: Sensitivity of precipitation forecast skill to areal and temporal averaging. Meteor. Z., 14 , 369–377.
Mariani, S., and Coauthors, 2005: A limited area model intercomparison on the “Montserrat-2000” flash-flood event using statistical and deterministic methods. Nat. Hazards Earth Syst. Sci., 5 , 565–581.
Nash, J. E., and Sutcliffe J. V. , 1970: River flow forecasting through conceptual models, Part 1: A discussion of principles. J. Hydrol., 10 , 282–290.
Pappenberger, F., and Beven K. , 2004: Functional classification and evaluation of hydrographs based on multicomponent mapping. Int. J. River Basin Manage., 2 , 89–100.
Pappenberger, F., Scipal K. , and Buizza R. , 2007b: Hydrological aspects of meteorological verification. Atmos. Sci. Lett., 9 , 43–52. doi:10.1002/asl.171.
Pappenberger, F., Buizza R. , Bodis K. , and Ghelli A. , 2009: The skill of probabilistic precipitation forecasts under observational uncertainties within the generalized likelihood uncertainty estimation framework for hydrological applications. J. Hydrometeor., 10 , 794–806.
Richardson, D. S., and Coauthors, 2008: Verification statistics and evaluations of ECMWF forecasts in 2007–2008. ECMWF RD Tech. Memo. 578, 47 pp.
Rodriguez-Iturbe, I., and Mejia J. M. , 1974: The design of rainfall network in time and space. Water Resour. Res., 10 , 713–728.
Rodwell, M. J., 2005: Comparing and combining deterministic and ensemble forecasts: How to predict rainfall occurrence better. ECMWF Newsletter, No. 106, ECMWF, Reading, United Kingdom, 17–23.
Schaefli, B., and Gupta H. , 2007: Do Nash values have value? Hydrol. Processes, 21 , 2075-2080.
Segond, M-L., 2006: Stochastic modelling of space–time rainfall and the significance of spatial data for flood runoff generation. Ph.D. thesis, Imperial College London, London, United Kingdom, 222 pp.
Thielen, J., Schaake J. , Hartman R. , and Buizza R. , 2008: Aims, challenges and progress of the Hydrological Ensemble Prediction Experiment (HEPEX)—A summary of the 3rd HEPEX workshop held in Stresa 27–29th June 2007. Atmos. Sci. Lett., 9 , 29–35. doi:10.1002/asl.168.
Untch, A., Miller M. , Hortal M. , Buizza R. , and Janssen P. , 2006: Towards a global meso-scale model: The high-resolution system TL799L91 & TL399L62 EPS. ECMWF Newsletter, No. 108, ECMWF, Reading, United Kingdom, 6–13.
Wilks, D. S., 1995: Statistical Methods in the Atmospheric Sciences: An Introduction. International Geophysics Series, Vol. 59, Academic Press, 467 pp.
Outline of the study region. The extents of the European window, the Danube catchment, and the upper Danube are shown.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Outline of the study region. The extents of the European window, the Danube catchment, and the upper Danube are shown.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Outline of the study region. The extents of the European window, the Danube catchment, and the upper Danube are shown.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Hyetographs for 24-h accumulated precipitation of the upper Danube. (top) The high-resolution and control forecasts. (bottom) The ensemble mean and the maximum and minimum of the ensemble distribution. Nash–Sutcliffe results and the percentage of outliers are also shown.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Hyetographs for 24-h accumulated precipitation of the upper Danube. (top) The high-resolution and control forecasts. (bottom) The ensemble mean and the maximum and minimum of the ensemble distribution. Nash–Sutcliffe results and the percentage of outliers are also shown.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Hyetographs for 24-h accumulated precipitation of the upper Danube. (top) The high-resolution and control forecasts. (bottom) The ensemble mean and the maximum and minimum of the ensemble distribution. Nash–Sutcliffe results and the percentage of outliers are also shown.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
(a)–(c) Hyetographs for 120-h accumulated precipitation over (a) Europe, (b) the Danube, and (c) the upper Danube. (top) The high-resolution and control forecasts. (bottom) The ensemble mean and the minimum and maximum of the ensemble distribution. Nash–Sutcliffe results and the percentage of outliers are also shown.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
(a)–(c) Hyetographs for 120-h accumulated precipitation over (a) Europe, (b) the Danube, and (c) the upper Danube. (top) The high-resolution and control forecasts. (bottom) The ensemble mean and the minimum and maximum of the ensemble distribution. Nash–Sutcliffe results and the percentage of outliers are also shown.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
(a)–(c) Hyetographs for 120-h accumulated precipitation over (a) Europe, (b) the Danube, and (c) the upper Danube. (top) The high-resolution and control forecasts. (bottom) The ensemble mean and the minimum and maximum of the ensemble distribution. Nash–Sutcliffe results and the percentage of outliers are also shown.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Hyetographs for 24-h accumulated precipitation at a lead time of 5 days for the upper Danube catchment. (top) The high-resolution and control forecasts. (bottom) The ensemble mean and the minimum and maximum of the ensemble distribution. Nash–Sutcliffe results and the percentage of outliers are also shown.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Hyetographs for 24-h accumulated precipitation at a lead time of 5 days for the upper Danube catchment. (top) The high-resolution and control forecasts. (bottom) The ensemble mean and the minimum and maximum of the ensemble distribution. Nash–Sutcliffe results and the percentage of outliers are also shown.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Hyetographs for 24-h accumulated precipitation at a lead time of 5 days for the upper Danube catchment. (top) The high-resolution and control forecasts. (bottom) The ensemble mean and the minimum and maximum of the ensemble distribution. Nash–Sutcliffe results and the percentage of outliers are also shown.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
(a)–(c) Temperature graphs for a lead time of 5 days over (a) Europe, (b) the Danube, and (c) the upper Danube. (top) The high-resolution and control forecasts. (bottom) The ensemble mean and the minimum and maximum of the ensemble distribution. Nash–Sutcliffe results and the percentage of outliers are also shown.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
(a)–(c) Temperature graphs for a lead time of 5 days over (a) Europe, (b) the Danube, and (c) the upper Danube. (top) The high-resolution and control forecasts. (bottom) The ensemble mean and the minimum and maximum of the ensemble distribution. Nash–Sutcliffe results and the percentage of outliers are also shown.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
(a)–(c) Temperature graphs for a lead time of 5 days over (a) Europe, (b) the Danube, and (c) the upper Danube. (top) The high-resolution and control forecasts. (bottom) The ensemble mean and the minimum and maximum of the ensemble distribution. Nash–Sutcliffe results and the percentage of outliers are also shown.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Comparison of the verification areas of Europe and the Danube catchment and its subcatchment by a Brier skill score of precipitation with a threshold of >1 mm (24 h)−1 and a 5-day lead time (after Pappenberger et al. 2007), using a 3-month filter.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Comparison of the verification areas of Europe and the Danube catchment and its subcatchment by a Brier skill score of precipitation with a threshold of >1 mm (24 h)−1 and a 5-day lead time (after Pappenberger et al. 2007), using a 3-month filter.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Comparison of the verification areas of Europe and the Danube catchment and its subcatchment by a Brier skill score of precipitation with a threshold of >1 mm (24 h)−1 and a 5-day lead time (after Pappenberger et al. 2007), using a 3-month filter.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Comparison of different precipitation thresholds with the Brier skill score for a lead time of 1 day between January 2002 and June 2007 in the Danube catchment, using a 3-month filter.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Comparison of different precipitation thresholds with the Brier skill score for a lead time of 1 day between January 2002 and June 2007 in the Danube catchment, using a 3-month filter.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Comparison of different precipitation thresholds with the Brier skill score for a lead time of 1 day between January 2002 and June 2007 in the Danube catchment, using a 3-month filter.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Comparison of different lead times with the Brier skill score for a threshold of 1 mm (24 h)−1 between January 2002 and June 2007 over the Danube catchment, using a 3-month filter.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Comparison of different lead times with the Brier skill score for a threshold of 1 mm (24 h)−1 between January 2002 and June 2007 over the Danube catchment, using a 3-month filter.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Comparison of different lead times with the Brier skill score for a threshold of 1 mm (24 h)−1 between January 2002 and June 2007 over the Danube catchment, using a 3-month filter.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Water balance precipitation for the duration of the lead time for each water year within the evaluation period.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Water balance precipitation for the duration of the lead time for each water year within the evaluation period.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Water balance precipitation for the duration of the lead time for each water year within the evaluation period.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Nash–Sutcliffe scores for temperature using the high-resolution, control, and median of the ensemble forecasts for a lead time of 5 days over the Danube catchment.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Nash–Sutcliffe scores for temperature using the high-resolution, control, and median of the ensemble forecasts for a lead time of 5 days over the Danube catchment.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Nash–Sutcliffe scores for temperature using the high-resolution, control, and median of the ensemble forecasts for a lead time of 5 days over the Danube catchment.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Box and whisker plots of the Nash–Sutcliffe efficiency criteria in comparison to forecast lead times for temperature forecasts accumulated over 24 h.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Box and whisker plots of the Nash–Sutcliffe efficiency criteria in comparison to forecast lead times for temperature forecasts accumulated over 24 h.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Box and whisker plots of the Nash–Sutcliffe efficiency criteria in comparison to forecast lead times for temperature forecasts accumulated over 24 h.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Brier skill score of exceeding 1 mm (24 h)−1 for different lead times for the Danube catchment. The performance results over the entire year, for the winter months (November–January), and for the summer months (June–August) are shown.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Brier skill score of exceeding 1 mm (24 h)−1 for different lead times for the Danube catchment. The performance results over the entire year, for the winter months (November–January), and for the summer months (June–August) are shown.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Brier skill score of exceeding 1 mm (24 h)−1 for different lead times for the Danube catchment. The performance results over the entire year, for the winter months (November–January), and for the summer months (June–August) are shown.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
As in Fig. 11, but for (top) the performance over an entire year, (middle) the performance over winter, and (bottom) the range of Nash–Sutcliffe values over summer.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
As in Fig. 11, but for (top) the performance over an entire year, (middle) the performance over winter, and (bottom) the range of Nash–Sutcliffe values over summer.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
As in Fig. 11, but for (top) the performance over an entire year, (middle) the performance over winter, and (bottom) the range of Nash–Sutcliffe values over summer.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
As in Fig. 11, but for (top) the performance over an entire year, (middle) the performance over winter, and (bottom) the range of Nash–Sutcliffe values over summer.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
As in Fig. 11, but for (top) the performance over an entire year, (middle) the performance over winter, and (bottom) the range of Nash–Sutcliffe values over summer.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
As in Fig. 11, but for (top) the performance over an entire year, (middle) the performance over winter, and (bottom) the range of Nash–Sutcliffe values over summer.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Percentage of outliers, spread, and RMSE of the mean for precipitation forecasts with different lead times for an accumulation period of 24 h and for the entire lead time for the Danube catchment.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Percentage of outliers, spread, and RMSE of the mean for precipitation forecasts with different lead times for an accumulation period of 24 h and for the entire lead time for the Danube catchment.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Percentage of outliers, spread, and RMSE of the mean for precipitation forecasts with different lead times for an accumulation period of 24 h and for the entire lead time for the Danube catchment.
Citation: Weather and Forecasting 24, 3; 10.1175/2008WAF2222120.1
Description of the catchment/area characteristics used in this study.
