1. Introduction
The intensity of precipitation events has increased since the 1950s, and the world has to brace itself for more extreme precipitation events in the future (IPCC 2021). Global warming will increase atmospheric moisture availability, resulting in more intense and prolonged rainfall events (Fowler et al. 2021; Hodnebrog et al. 2019; Kahraman et al. 2021; Pendergrass 2018). Flood disasters with fatalities are often related to extreme precipitation events and fast-developing flash floods. For example, the 2021 western Europe floods resulted in more than 200 fatalities (Tradowsky et al. 2023) and 46 billion euros of economic losses (Munich Re 2022). In Italy, the May 2023 Emilia–Romagna floods caused approximately 8.8 billion euros in damage and 15 fatalities (Arrighi and Domeneghetti 2024).
Research indicates that hydrometeorological early warning systems hold the potential to save lives and reduce asset losses (Global Commission on Adaptation 2019; Hallegatte 2012). However, current systems often underperform due to various bottlenecks in the early warning process, including short lead times, missed signals, and communication failures (Jonkman et al. 2023; Kreibich et al. 2021, 2017; Merz et al. 2021; Thieken et al. 2023; Mohr et al. 2023). Addressing existing shortcomings requires a systematic assessment of the potential of precipitation forecasts for early response (Cornwall 2021; Kreibich et al. 2017; Thieken et al. 2023). This translation of precipitation forecasts to early action triggers remains a large knowledge gap. Existing forecast evaluation studies often lack a translation to early action and use precipitation thresholds that do not necessarily reflect impacts (e.g., the 95th percentile; Haiden et al. 2023; Rivoire et al. 2023).
This study addresses this challenge by deploying two early warning indicators for extreme rainfall from the European Centre for Medium-Range Weather Forecasts (ECMWF): the extreme forecast index (EFI) and shift of tails (SOT). These indicators provide a concise summary of the relationship between the 51-member ensemble forecast distribution and the reforecast equivalent. We evaluate whether these two indicators can be used to prompt timely actions to reduce the impacts of extreme rainfall events in Europe. Initially, we compute the potential economic value (PEV) of these forecasts across Europe for lead times of up to 5 days. In this approach, the economic value is a function of forecast skill, the cost of early action, and the benefit of damage prevention. This analysis reveals regions where the early warning indicators could be particularly valuable and identifies actionable lead-time windows. We apply the optimal warning thresholds found in this analysis to forecasts produced ahead of the 2021 western Europe floods. By examining the precipitation forecasts issued at that time, we assess how accurately the early warnings anticipated the areas that would be affected and discuss the opportunities they provided for effective action.
The methodology section begins with an overview of the methodological framework, followed by a detailed explanation. In the results, we provide estimates of the potential value of EFI and SOT to trigger early action across Europe, with a focus on the 2021 floods in western Europe. Finally, we discuss our results and present our conclusions and recommendations.
2. Methods
Figure 1 outlines the methodological steps taken in this study. We used two early warning indicators for rainfall provided by ECMWF: EFI and SOT (Fig. 1, top left). We applied a Europe-wide 8-yr verification (2016–23) to these forecasts by comparing them to extreme rainfall observations from the E-OBS gridded rainfall dataset (Fig. 1, top right). This revealed their value for users taking early action up to 5 days in advance (Fig. 1, center). To explore whether the performance of EFI and SOT also holds for truly exceptional events, we tested them on the western Europe floods in July 2021 (Fig. 1, bottom). We assessed whether the forecasted conditions were sufficiently extreme and accurate in pinpointing the correct locations, based on comparisons with Emergency Events Database (EM-DAT) flood impact data, to justify issuing early warnings (Fig. 1, bottom right). Further details of each of these steps are discussed below.
The framework illustrating the methodological steps used in the study. We used the EFI and SOT warning indicators for extreme rainfall (top left). The (middle) PEV of these forecasts was estimated by comparison to observed records (top right). These forecast indicators were (bottom left) then applied to the 2021 western European flood event at multiple lead times and (bottom right) compared with reported impact data from EM-DAT to demonstrate their capability to trigger timely mitigation actions.
Citation: Bulletin of the American Meteorological Society 106, 3; 10.1175/BAMS-D-24-0073.1
a. Input data.
1) Rainfall observations.
(i) The E-OBS dataset.
We used the Europe-wide E-OBS dataset (Cornes et al. 2018) as ground-truth rainfall observations for the period 2016–23, downloaded at a 0.25° × 0.25° resolution. The gridded dataset is created by the interpolation of the European Climate Assessment and Dataset (ECA&D) station network, containing more than 23 000 stations (Fig. A1). Pixels with more than 10% of days without data were excluded. The high number of stations, the long time series (1950–2023), and coverage of entire Europe make this dataset unique (Cornes et al. 2018). The dataset was shown to accurately represent rainfall patterns, with generally higher accuracy for extreme precipitation events compared to ERA5 (Hu and Franzke 2020). However, accuracy strongly depends on station density (Bandhauer et al. 2022), which is relatively low in certain parts of eastern and southern Europe (Fig. A1). Moreover, very localized precipitation extremes will not be well represented on a 0.25° grid, especially if located in between rainfall stations.
(ii) Definition of extreme precipitation.
In this study, daily extreme precipitation is defined as exceeding a 5-yr return period at a given location. The 5-yr rainfall event map was derived from the E-OBS dataset and was downloaded from the Copernicus Climate Data Store (Mercogliano et al. 2020). The return periods for each grid point were estimated by fitting a generalized extreme value (GEV) distribution to the annual maxima of daily precipitation over a 30-yr period (1989–2018) (Mercogliano et al. 2020). The gridded dataset, originally at a 0.1° × 0.1° horizontal resolution, was resampled to 0.25° × 0.25° to match the resolution we used for E-OBS and the ECMWF indicators, using linear interpolation. The resulting thresholds (mm day−1) (Fig. A2, left) and the total number of events per pixel (Fig. A2, right) are shown in the appendix. A higher threshold (e.g., 10- or 20-yr return period) would be better aligned with the case study (2021 western Europe floods) and impacts. However, these higher rainfall thresholds do not allow an accurate estimate due to the shortness of the EFI and SOT data record. For example, a 10-yr return period threshold returns an average of one extreme rainfall event per pixel, with large parts of Europe recording none.
2) ECMWF early warning indicators.
We used two of ECMWF’s early warning indicators for extreme rainfall: the EFI and SOT (Tsonevsky 2016). These two indicators summarize the severity of the 51-member rainfall ensemble forecast, compared to the model climatological distribution (Haiden et al. 2023)—the latter here referred to as “M-climate.” The M-climate is calculated from a large set of reforecasts. These reforecasts span a 20-yr historical period and always use the same model version as the real-time forecasts with which they are compared. Furthermore, the M-climate is calculated separately for different calendar days, locations, and lead times. Both indicators have been produced operationally for over 10 years. The indicators were downloaded from the operational archive of ECMWF’s Meteorological Archival and Retrieval System (MARS) data catalog (ECMWF 2023a). Although both indicators have been archived since 2012, we only used the forecasts after 8 March 2016, when an increase in horizontal resolution from 32 to 18 km was implemented (Hólm et al. 2016). Homogeneity of the resolution of the forecast system is considered to be helpful for calculating statistical characteristics that can be considered representative for the entire time window. We interpolated the ECMWF grid to a horizontal resolution of 0.25° × 0.25° to match with the E-OBS grid, using linear interpolation.
(i) The EFI.
(ii) The SOT.
The EFI and SOT combined provide valuable information on the potential severity of expected rainfall. High EFI values and low SOT values indicate that an extreme event is likely but that the chance of a very extreme situation is limited. High EFI and high SOT values indicate a high probability of an extreme event, with a chance that the event will be very extreme. Low EFI and high SOT values indicate a large spread (i.e., uncertainty) in the ensemble prediction but with some indications for a very extreme event. This is often the earliest pointer to an upcoming very extreme event. Visual illustrations and training materials for the EFI and SOT indicators are available in Tsonevsky (2016) and the ECMWF forecast user guide (Hewson and Owens 2018).
b. The “value” of a forecast.
To estimate the value of the EFI and SOT forecasts for triggering early action, we used the PEV method (Richardson 2000). The approach has been widely applied to assess the potential of translating ensemble weather forecasts into early action (e.g., Bischiniotis et al. 2019; Buizza 2001; Busker et al. 2023; Portele et al. 2021; Ravuri et al. 2021; Zhu et al. 2002). This theory is instrumental in the design of an early action plan, as it estimates the expected savings in case the forecasts are used to trigger early action. For a given climatological frequency of the event
1) Forecast skill analysis.
The EFI and SOT forecasts were evaluated against extreme rainfall observations (>5-yr return period) over the period 2016–23, for each season (winter–summer–spring–autumn) separately. For a 1–5-day lead time, the forecasts were translated to yes/no warnings using >400 decision thresholds on both EFI and SOT. For every threshold, these warnings were compared to the extreme rainfall observations to determine the number of hits, misses, false alarms, and correct negatives. This approach of evaluating EFI and SOT is based on ECMWF’s methodology deployed to evaluate EFI forecasts (see Haiden et al. 2023). To our knowledge, SOT forecasts are not routinely validated, which makes the SOT analysis in this study particularly relevant.
2) Costs and prevented damage of early actions.
Action characteristics are as relevant as forecast skill to assess the value of forecasts for decision-making. Even inaccurate forecasts can have value for very cheap actions, and the value of accurate forecasts can disappear for expensive actions. Thus, an estimation of the action costs C and prevented damage L is vital to the design of an early action system. These are often summarized as a ratio: the cost–loss (C/L) ratio (Richardson 2000).
Surprisingly little is known about the C/L ratio of preventive disaster risk reduction measures (Mechler 2016; Richardson 2000). This often hampers concrete estimations of the value of a forecast for a specific case (e.g., Richardson 2000, 2003; Verkade and Werner 2011; Zhu et al. 2002). Using survey data from the 2021 western Europe flood (Endendijk et al. 2023b,a), we were able to provide realistic estimates of the prevented damages of emergency flood mitigation measures. Due to implementing early actions such as the use of water pumps, flood barriers, and elevating possessions, studies found a substantial damage reduction to the buildings and their contents (29% and 44%, respectively) (Endendijk et al. 2023b). Elevating household possessions only reduces content damage, and we assumed that the water pumps and flood barriers contributed equally to the 29% reduction in building damage. Estimates of the absolute amount of damage reduction have been derived by applying the above percentages to the median of the damage per household in the affected area (€25,000 for building damage and €17,000 for household contents; Table 3, Endendijk et al. 2023a). Subsequently, we calculated the C/L ratio by comparing these prevented damages to the costs of the emergency measures: €500 for a water pump (waterpompshop 2023) and €200 for both a flood barrier/shield (Aerts 2018, Table A2) and elevating household possessions. The latter we estimated based on the missed income of 4 h of work for two persons, using the average hourly wage of €25 (CBS 2023). Subsequently, we calculated the C/L ratios by dividing the action costs by the prevented losses. To make these estimations more robust, we included another study (Kreibich et al. 2012, 2011) on mobile flood barriers in Germany. They found benefit–cost ratios (BCRs) between 5.62 (2 uses in 20 years) and 56.2 (20 uses in 20 years). We included these estimates as well, as upper (1/5.62) and lower (1/56.2) C/L estimates, respectively.
These studies suggest that the costs of emergency flood actions are low compared to the prevented damage, which results in low C/L ratios (Table 1). Our best estimate of C/L is 0.08 (range 0.02–0.18). These C/L ratios are of the same order of magnitude as the few existing estimates in other applications (Richardson 2003). Nonetheless, the estimates are context specific, and the reality is that there is considerable context-specific variability (see discussion). This is especially true for actions that could reduce the loss of life, such as evacuation measures, an aspect that we did not include here.
The ratios between costs C and prevented losses L of a selection of flood emergency measures taken by households in the Netherlands (2021 western Europe flood) and Germany (Danube and Elbe floods in 2002, 2005, and 2006). This resulted in the C/L ratio used to evaluate early actions in this study.
3) PEV.
c. Case study: The July 2021 western Europe floods.
1) Event introduction.
The 2021 western Europe floods were one of the most disastrous floods observed in Europe over the last decades (Jonkman et al. 2023). It led to record-high discharges in the Rhine and Meuse Rivers and their tributaries. Most damage was caused by streams and creeks in steep-sided valleys turning into raging torrents, widely exceeding protection standards (de Bruijn et al. 2023). The event was triggered by extreme precipitation across extensive areas (de Bruijn et al. 2023; Junghänel et al. 2021). The rain began in the early morning of 14 July (0500 UTC in the Ahr valley) and intensified over the course of the day (Mohr et al. 2023). Total rainfall surpassed 160 mm within 48 h and local peaks of more than 250 mm (24 h)−1 were recorded (de Bruijn et al. 2023; Junghänel et al. 2021). The rainfall amount was ranked as the fifth most extreme event recorded in Germany in the past 70 years (Ludwig et al. 2023). Local return period estimates vary significantly but are generally estimated to be around once every several hundred years (Mohr et al. 2023; Ludwig et al. 2023). The distinctive synoptic conditions during the event were marked by a slow-moving low pressure system, with rainbands almost parallel to the movement direction (Mohr et al. 2023). The Baltic Sea exhibited anomalously warm temperatures, which contributed to a northeasterly flow of warm and moist air toward the low pressure system (Copernicus 2021). Combined with orographic enhancement on the northern flank of the High Fens area, these factors resulted in the exceptionally high rainfall amounts observed (Hewson 2022; Copernicus 2021). Total damages exceeded 46 billion euros (Munich Re 2022) and over 200 fatalities were recorded (Tradowsky et al. 2023). The impact was highest in Belgium (mainly in Pepinster and Verviers) and Germany (mainly in the Ahr catchment)—the only countries with registered fatalities (Jonkman et al. 2021). This was not only related to the higher rainfall amounts compared to other areas but also to the hilly terrain with fast runoff (Jonkman et al. 2021) and a significant share of people who were not reached by warnings (Thieken et al. 2023).
2) An evaluation of SOT and EFI warnings for the 2021 floods.
We extend our evaluation of EFI and SOT by applying the long-term PEV results to the 2021 floods. For the affected area, we calculated the PEV and derived the optimal warning thresholds. To calculate the location-specific PEV and optimal thresholds, we spatially pooled the 8-yr verification statistics over the affected area (shown in Fig. 3, bottom right) for the summer season. We only included the pixels with observed extreme rainfall events (25 independent events in total, including the 2021 event). The optimal warning thresholds, leading to the highest PEV, were subsequently applied to the operational forecasts as produced in the days prior to the 2021 floods. This allows to identify where and when the optimal warning thresholds were exceeded and action would be triggered. In case these actions lead to a very small economic value in the long term (PEV < 0.2), the quality of the forecasts is not high enough, and action triggers are not considered reliable. We extended the event evaluation by including on-the-ground impacts. We overlaid the forecasts and action triggers with postdisaster impacts recorded in the EM-DAT disaster dataset (CRED and UCLouvain 2023) in the four hardest hit countries: Luxembourg, the Netherlands, Germany, and Belgium (Copernicus 2021). Some database entries (eight in total) were discarded as their locations could not be geolocated or were identified to not be the result of the specific event. In total, 27 located impacts were retained and used to characterize the 2021 flood event.
3. Results
a. Early action to extreme rainfall in Europe.
The EFI and SOT indicators have high PEV if used to trigger early actions ahead of extreme rainfall. Figure 2 shows that SOT forecasts have high value across most of Europe (PEV > 0.5) for lead times up to 3 days. This figure represents an average PEV across all seasons, using seasonal warning thresholds (see methods). Clear hotspots with very high value (PEV approaching or exceeding 0.8) are Portugal, Spain, England, Denmark, Finland, and Belgium. Furthermore, the Alps and Dolomites are clear hotspots. Orographic precipitation processes seem to enhance prediction skill over these mountain ranges. No significant value is found in Greece and (most of) middle and southern Italy. Results for EFI (Fig. A3) show a faster decrease in forecast value (PEV) than for SOT as forecast lead time increases, which is expected since the SOT indicator is designed to give long-lead indications of potentially very extreme events (see methods). For a lead time of 5 days, both EFI and SOT have no practical value for decision-making anymore over most of Europe, as the PEV is negligible.
The PEV of the SOT forecasts for anticipating extreme rainfall events (>5-yr return period) at lead times between 1 and 5 days. The PEV is calculated using an estimation of the costs C and prevented damages L of emergency flood mitigation measures (C/L = 0.08). The delineated rectangle highlights the affected region during the 2021 western Europe floods (used in the case study). The corresponding figure for the EFI forecasts is presented in Fig. A3.
Citation: Bulletin of the American Meteorological Society 106, 3; 10.1175/BAMS-D-24-0073.1
b. The July 2021 western Europe floods: Predictions and action triggers.
1) Long-term (2016–23) forecast verification: Finding optimal action triggers.
We continue with a long-term evaluation (2016–23) of the EFI and SOT forecasts in the area affected by the July 2021 western Europe floods (Fig. 3, bottom right). We limit this part of the analysis to this specific area, and to the summer season, to be able to apply the findings directly to the operational forecasts issued for the July 2021 floods (outlined in the next section).
The forecast value (expressed as PEV) of (left) EFI and (right) SOT forecasts of extreme rainfall events (>5-yr return period event) at lead times varying from 1 to 5 days. The PEV is based on a forecast verification for summer during the 2016–23 period over the shown region (bottom right). The peaks of the PEV curves are marked with colored dots. The red dashed line illustrates the C/L ratio found for emergency flood mitigation measures (0.08; see Table 1). For this C/L ratio, the optimal trigger thresholds of 0.88 (EFI) and 1.85 (SOT) give the largest PEV on a 1-day lead time. The (top) contingency metrics show the number of pixels recorded over the area as hits, misses, false alarms, and correct negatives in case these thresholds are used on a 1-day lead time. The optimal warning thresholds for all lead times and seasons can be found in Table A1.
Citation: Bulletin of the American Meteorological Society 106, 3; 10.1175/BAMS-D-24-0073.1
Figure 3 shows that decision-makers can extract significant long-term economic value (PEV) from using EFI and SOT forecasts in the area affected by the July 2021 floods. These estimates result from using optimal warning thresholds, which lead to the highest economic value (PEV). The optimal warning thresholds for EFI and SOT for a 1-day lead time in summer are, respectively, 0.88 and 1.85 (Fig. 3, top). The thresholds for all seasons and lead times can be found in Table A1. Note that these thresholds are all based on a C/L ratio of 0.08.
For different C/L ratios, different estimates of the PEV are found. The highest value for each lead time is found for cheap (no regret) actions with a C/L ratio of 0.003. However, this C/L ratio is substantially lower compared to the range found for emergency flood mitigation measures (0.02–0.18, Table 1). The PEV strongly decreases with higher C/L ratios (Fig. 3). Although the PEV at the peaks of the curves (at C/L = 0.003) is comparable for EFI and SOT, the SOT forecasts generally show higher value when triggering actions with higher C/L ratios (Fig. 3). This especially holds for longer lead times (Figs. 2 and 3, A3). The high PEV values for the SOT indicator illustrate its ability to generate hits with a relatively low number of false alarms and misses, using only the 10% most extreme ensemble members. For both EFI and SOT, one hit comes at the expense of approximately 2–3 false alarms (Fig. 3, top). The total number of pixels (n) with false alarms (350 for EFI and 433 for SOT; Fig. 3, top) is low compared to the total number of forecasts in which no extreme event occurs (176 305). This yields a very low false alarm rate of 0.002, which further underscores the accuracy of the forecasts.
2) ECMWF warnings for the 2021 floods.
In this section, we showcase the EFI and SOT forecasts produced in the 5 days prior to the July 2021 western Europe floods. Additionally, we apply the long-term PEV estimates and optimal warning thresholds found in the previous section (Fig. 3 and Table A1) to this specific event, to show where and when action would be recommended. Since the identified optimal action triggers are season dependent (Table A1), we used the summer season’s thresholds.
The ECMWF rainfall early warning indicators—EFI and SOT—warned for a truly exceptional extreme event several days ahead of the 2021 western Europe floods (Fig. 4). The forecasts were the second highest recorded for the plotted region since their operational use began in 2012. Only the forecast issued for the 2018 Mullerthal flood (Mathias 2019) was slightly more extreme. This is illustrated by the CDF of all EFI and SOT values ever forecasted in this region (Fig. 4, upper and lower panels), relative to the values forecasted for the 2021 event (Fig. 4, dashed lines in the CDFs). This shows the extremity of the forecasts issued prior to the 2021 floods.
Predictions of extreme rainfall on 14 Jul 2021 that triggered the 2021 floods in western Europe, as indicated by the (top) EFI and (bottom) SOT early warning indicators, with lead times of 1–5 days. Blue circles represent flood disaster impacts as registered in EM-DAT. The yellow outline shows the location of the Ahr catchment. Black and green contours show exceedance of the optimal warning thresholds for C/L = 0.08 and C/L = 0.18, respectively (see Table 1). Contours are not shown for lead times with negligible value (PEV < 0.2). The CDFs illustrate the severity of the 1-day lead EFI and SOT forecasts for 14 Jul 2021 (dashed line) relative to all values forecasted over the displayed region since 2012.
Citation: Bulletin of the American Meteorological Society 106, 3; 10.1175/BAMS-D-24-0073.1
These extreme forecasts exceeded the optimal warning thresholds as found in the previous section (Fig. 3 and Table A1) over large areas (Fig. 4). Affordable emergency flood mitigation actions (C/L = 0.08) would have been triggered 5 days in advance for SOT and 3 days in advance for EFI. This shows that SOT can give actionable long-lead information of a potentially very extreme event, with some ensemble members predicting rainfall far beyond the limits of the model climatology (M-climate). For a 2–3-day lead time, the warning indicators also exceeded the thresholds to trigger more expensive actions (C/L = 0.18, Fig. 4). The areas that exceeded the optimal warning thresholds are spatially well aligned with flood impacts registered in the EM-DAT database (Fig. 4, blue circles). Note that triggers are not shown for cases where the threshold was exceeded, but the PEV of the action was negligible (PEV < 0.2, see methods). Although these triggers would of course have had value for this particular event, these were likely a “lucky shot” as the long-term forecast performance (2016–23) was generally not good enough (i.e., too many false alarms and missed events).
This case study confirms the higher value of the SOT indicator, compared to EFI, for such extreme events. Most striking is the spatial accuracy of the SOT forecasts. The Ahr catchment (Fig. 4, yellow outline)—the area with 133 fatalities (Apel et al. 2022)—was accurately predicted as the hardest hit area at least 1 day in advance. At this location, SOT values as high as 7.5 were produced, indicating that the ensemble prediction tail was at least 7.5 times larger than the upper tail of the model climatology. This provided a clear warning of an exceptional event. The EFI indicator was saturated at this location, as it does not capture the extremity of the ensemble tail beyond the M-climate limit. The highly exceptional SOT warning values enabled a preparation window of at least 16 h to prepare for the incoming disastrous precipitation in the Ahr valley and its surrounding areas (see discussion).
4. Discussion
a. From early warning to early action.
This study shows that EFI and SOT forecasts provide actionable warnings to extreme rainfall events, which can significantly reduce damages across Europe. The results presented here aid the decision-making process: rather than a subjective interpretation of the forecasts, we showcase how decision-makers can determine optimal warning thresholds for a specific location and lead time. This study demonstrates the value of this early action framework in the 2021 western Europe flood event. The forecasts widely exceeded the warning thresholds at a 2–3-day lead time. At a 1-day lead time (initialized at 0000 UTC 14 July), the forecasts accurately predicted the Ahr valley as the most impacted area. This can be considered a feasible window of opportunity for effective action. These forecasts were available to users in the early morning of 14 July (see ECMWF dissemination schedule; ECMWF 2023c). The first houses in the Ahr valley were flooded around 2240 UTC 14 July (Thieken et al. 2023). In theory, a time window of 16 h is sufficient to facilitate the evacuation of villages (Cornwall 2021).
Although the study shows the potential of EFI and SOT early warnings, operational early actions can always be triggered using a wide range of information sources. Sources available include river and flash flood forecasts from the European Flood Awareness System (EFAS) (Alfieri and Thielen 2015; Smith et al. 2016), which also possess high value for decision-makers (Pappenberger et al. 2015). High-resolution precipitation forecasts can provide additional and valuable information, as shown by the timely and accurate weather alarms for the 2021 floods from the Dutch (KNMI; Jonkman et al. 2021) and German meteorological services (Deutscher Wetterdienst; DWD 2021). These high-resolution limited-area models better represent local-scale processes and features, like convection and orography. The translation of such warnings into action also depends on forecast communication and dissemination (Golding 2022), an important pillar in the early warning chain (UNDRR 2022). Unfortunately, the 2021 floods show that warnings lose their effect when they are not received, understood or trusted, or if the ability to respond is limited (Fekete and Sandholz 2021; Thieken et al. 2023).
This study emphasizes that the value (PEV) that can be extracted from the forecasts is not only highly dependent on the forecast skill but also on the action costs C and prevented damages L. We estimated C and L of emergency flood mitigation measures. Limited evidence and knowledge on the characteristics of such preventive measures (Mechler 2016; Richardson 2000) makes these estimates uncertain. One of the major challenges is that C and L for flood mitigation measures are very region and context dependent (Poussin et al. 2015). The prevented losses depend on the effectiveness of the measure in place, and on its reliability. One of the measures we investigated was a mobile flood wall. The effectiveness of this measure is close to zero in case it breaks or is overtopped. Nonetheless, the C/L ratios estimated in this study are in line with other studies on the costs and benefits of early actions for floods (Global Commission on Adaptation 2019; Hallegatte 2012; Pappenberger et al. 2015) and other applications (Kolb and Rapp 1962; Murphy 1977; Thornes and Stephenson 2001). The Global Commission on Adaptation (Global Commission on Adaptation 2019) found that just 24 h of warning for an upcoming precipitation event can lead to a 30% reduction of damages. We are confident that the costs of actions are, overall, much smaller than the damages prevented. However, providing accurate C/L estimates remains a major challenge.
The highest PEV for extreme rainfall events (>5-yr return period event) was obtained for very low C/L ratios (C/L = 0.003). For related actions, the forecasts would provide the maximum savings of total expenses. However, given our estimates of the C/L ratio for emergency flood mitigation measures (Table 1), such low ratios are likely unrealistic. Moreover, achieving such high PEV values would be commensurate with triggering an unacceptable number of false alarms. To obtain the highest PEV of 0.82 for EFI at a C/L of 0.003 (Fig. 3, left), every hit would be accompanied by 20 (1-day lead) to 57 (5-day lead) false alarms. Acting on forecasts with such a high number of false alarms is often unrealistic and undesirable. It decreases trust in forecasts and can lead to inferior decision-making and reluctance to trigger action, known as the “cry wolf effect” (Breznitz 1984; LeClerc and Joslyn 2015).
b. Limitations and future research.
Our study builds on the existing but limited body of research on the use of precipitation forecasts (e.g., Magnusson et al. 2014; Richardson 2000), including warning indicators such as EFI and SOT (Bouallègue 2024), to trigger early actions. In our study, we advance the field by offering, for the first time, spatial forecast value estimates for early action measures across Europe. In contrast to existing studies, we provide estimates of the costs and benefits associated with these actions and link the forecasts and optimal action triggers directly to the recorded impacts for the 2021 flood event. Nevertheless, several limitations need to be mentioned.
The EFI and SOT warning indicators are based solely on precipitation forecasts. This makes them more suited to warning for local flash floods in smaller catchments but less effective for floods in larger catchments driven by upstream precipitation extremes. Moreover, the exceedance of a rainfall-related threshold of EFI or SOT, or even of rainfall itself, does not necessarily imply (extensive) damage. Many other factors determine the impact of the rainfall event. In large parts of Europe, antecedent soil moisture is an important preconditioning factor that can determine where and when heavy rainfall leads to flooding (Berghuijs et al. 2019). Extreme rainfall on saturated soils will ordinarily cause a greater impact than extreme rainfall on soils with more storage capacity. The precipitation-based EFI and SOT forecasts do not account for variations in soil moisture. Besides hydrometeorological factors, exposure and vulnerability dynamics play a key role in determining the impact. In the Ahr valley, the steep topography along the river forced communities to settle close to the river and left very little space for the floodwaters to inundate (Mohr et al. 2023). Similar rainfall quantities in flat areas would undoubtedly have caused less impact. Future studies could create impact-based forecasts using ECMWF’s warning indicators (especially SOT) and including the factors described above.
We used operational forecasts from ECMWF since March 2016, specifically from the moment the ensemble forecast resolution increased from 32 to 18 km (Hólm et al. 2016). The ECMWF ensemble prediction system has further improved since 2016 due to multiple upgrades in their systems. An important upgrade was implemented in June 2023, which increased the horizontal resolution of the ensemble system from 18 to 9 km and led to some improvements in the precipitation prediction (Lang et al. 2023). Extreme precipitation events are sometimes small scale and can result from atmospheric processes within a grid box, such as during severe convection. Even at 9-km resolution, these subgrid events can be strongly underestimated by the ECMWF ensemble system, which predicts “gridbox average” rainfall. ECMWF has now developed a postprocessing method to better represent extreme rainfall events within a grid box, at the point scale. This postprocessing approach is called “ecPoint” and shows promising improvements for the prediction of local rainfall extremes, compared to the conventional ensemble forecasts (Hewson and Pillosu 2021). Future studies should investigate whether these products lead to improved opportunities for early action, particularly for more localized, highly convective events, which are often difficult to predict.
5. Conclusions and recommendations
This study showcases an early action framework to anticipate floods triggered by extreme rainfall events in Europe. We translate the extreme forecast index (EFI) and shift of tails (SOT) forecasts, two extreme weather indicators from ECMWF, to concrete action triggers that can aid operational decision-making. The value of the forecasts for triggering early action, expressed as the potential economic value (PEV), is high (PEV > 0.5) over most of Europe on short lead times of up to 2–3 days. Their value decreases over longer lead times and disappears over most of Europe on a 5-day lead time. While EFI is most widely known, we show that SOT often has a higher value for decision-making. This is especially evident on longer lead times. Our estimates are based on emergency flood mitigation measures with a ratio between costs and prevented losses (C/L) of 0.08. The peak of the PEV curves, representing the maximum economic benefits, occurs at much lower C/L ratios. Although these economic benefits are theoretically achievable, realizing them in practice is challenging due to the high number of false alarms and unnecessary actions involved.
We further showcase the value of EFI and SOT forecasts for the 2021 western Europe flood. At least 2–3 days in advance, the derived optimal warning thresholds were exceeded over a large area. At a 1-day lead time, the SOT forecasts were highly exceptional (SOT > 7) and pinpointed exactly to the Ahr catchment, where more than 100 fatalities were registered. These forecast values represented the second highest ever issued for the region since the indicator became operational in 2012.
These results illustrate that the EFI and SOT forecasts can reliably serve as a baseline for triggering preparedness actions for localized severe floods in areas where a strong relation exists between precipitation and local flash flood risk. The framework and results from this study should be included in operational protocols to aid decision-making on triggering early action. National meteorological and hydrological services can use this framework to determine their own optimal warning thresholds for their area, tailored on the costs (C) and prevented losses (L) of the early actions and the forecast skill. Future research should continue to expand the evaluation of these indicators to other rare and high-impact rainfall events.
Acknowledgments.
This project was supported by the COASTMOVE ERC advanced grant (Grant 884442). We acknowledge the E-OBS dataset from the EU-FP6 project UERRA (https://www.uerra.eu) and the Copernicus Climate Change Service and the data providers in the ECA&D project (https://www.ecad.eu). We acknowledge the support of Gerard van der Schrier (KNMI) in better understanding the uncertainties in E-OBS. We would like to express our gratitude to ECMWF, especially to two persons: Paul Dando, for general technical support, and Ivan Tsonevsky, for his help on the EFI and SOT indicators. We also thank colleagues Thijs Endendijk and Maurice Schmeits for useful discussions. Last, we recognize the BAZIS service for the use of the high-performance computing cluster.
Data availability statement.
All Python scripts used in this research for the analysis and figures are available on Zenodo (https://doi.org/10.5281/zenodo.14587737).
APPENDIX Supporting Figures and Table
Figures A1–A3 and Table A1 provide supporting information for the main manuscript. Figures A1 and A2 show the station locations and the 5-yr return period events from the E-OBS dataset, respectively. Figure A3 presents the potential economic value (PEV) of the EFI early warning indicator over Europe. Table A1 lists the warning thresholds that yield the highest PEV for the case study area.
The locations of the precipitation stations over Europe included in E-OBS, collected as part of the ECA&D initiative (Klein Tank et al. 2002). Downloaded from https://www.ecad.eu/.
Citation: Bulletin of the American Meteorological Society 106, 3; 10.1175/BAMS-D-24-0073.1
(left) The 5-yr return map for 24-h rainfall over Europe as downloaded from the Copernicus Climate Data Store (Mercogliano et al. 2020). (right) The number of 24-h rainfall events from E-OBS exceeding the 5-yr return period over the period of the analysis (2016–23).
Citation: Bulletin of the American Meteorological Society 106, 3; 10.1175/BAMS-D-24-0073.1
As in Fig. 2, but for the EFI early warning indicator.
Citation: Bulletin of the American Meteorological Society 106, 3; 10.1175/BAMS-D-24-0073.1
The early warning thresholds that generate the highest PEV for each season. Warning thresholds with PEV < 0 are removed. These statistics have been generated over the bounding box of the case study area (Fig. 3, red box) for the estimated C/L ratio of flood mitigation measures (C/L = 0.08). The seasons contain a different amount of time steps with a pixel recording an extreme event (autumn = 11, winter = 5, summer = 25, and spring = 10).
References
Aerts, J. C. J. H., 2018: A review of cost estimates for flood adaptation. Water, 10, 1646, https://doi.org/10.3390/w10111646.
Alfieri, L., and J. Thielen, 2015: A European precipitation index for extreme rain-storm and flash flood early warning. Meteor. Appl., 22, 3–13, https://doi.org/10.1002/met.1328.
Apel, H., S. Vorogushyn, and B. Merz, 2022: Brief communication: Impact forecasting could substantially improve the emergency management of deadly floods: Case study July 2021 floods in Germany. Nat. Hazards Earth Syst. Sci., 22, 3005–3014, https://doi.org/10.5194/nhess-22-3005-2022.
Arrighi, C., and A. Domeneghetti, 2024: Brief communication: On the environmental impacts of the 2023 floods in Emilia-Romagna (Italy). Nat. Hazards Earth Syst. Sci., 24, 673–679, https://doi.org/10.5194/nhess-24-673-2024.
Bandhauer, M., and Coauthors, 2022: Evaluation of daily precipitation analyses in E-OBS (v19.0e) and ERA5 by comparison to regional high-resolution datasets in European regions. Int. J. Climatol., 42, 727–747, https://doi.org/10.1002/joc.7269.
Berghuijs, W. R., S. Harrigan, P. Molnar, L. J. Slater, and J. W. Kirchner, 2019: The relative importance of different flood-generating mechanisms across Europe. Water Resour. Res., 55, 4582–4593, https://doi.org/10.1029/2019WR024841.
Bischiniotis, K., B. van den Hurk, E. Coughlan de Perez, T. Veldkamp, G. G. Nobre, and J. Aerts, 2019: Assessing time, cost and quality trade-offs in forecast-based action for floods. Int. J. Disaster Risk Reduct., 40, 101252, https://doi.org/10.1016/j.ijdrr.2019.101252.
Bouallègue, Z. B., 2024: Seamless prediction of high-impact weather events: A comparison of actionable forecasts. Tellus, 76A, 14–28, https://doi.org/10.16993/tellusa.3262.
Breznitz, S., 1984: Cry Wolf: The Psychology of False Alarms. Psychology Press, 280 pp.
Buizza, R., 2001: Accuracy and potential economic value of categorical and probabilistic forecasts of discrete events. Mon. Wea. Rev., 129, 2329–2345, https://doi.org/10.1175/1520-0493(2001)129<2329:AAPEVO>2.0.CO;2.
Busker, T., H. de Moel, B. van den Hurk, and J. C. J. H. Aerts, 2023: Impact-based seasonal rainfall forecasting to trigger early action for droughts. Sci. Total Environ., 898, 165506, https://doi.org/10.1016/j.scitotenv.2023.165506.
CBS, 2023: StatLine – Employment; jobs, wages, working hours, SIC2008; key figures. Accessed 12 February 2024, https://opendata.cbs.nl/#/CBS/en/dataset/81431ENG/table.
Copernicus, 2021: Widespread European flooding, July 2021. Copernicus EMS – European Flood Awareness System, accessed 23 January 2024, https://www.efas.eu/en/news/widespread-european-flooding-july-2021.
Cornes, R. C., G. van der Schrier, E. J. M. van den Besselaar, and P. D. Jones, 2018: An ensemble version of the E-OBS temperature and precipitation data sets. J. Geophys. Res. Atmos., 123, 9391–9409, https://doi.org/10.1029/2017JD028200.
Cornwall, W., 2021: Europe’s deadly floods leave scientists stunned. Science, 373, 372–373, https://doi.org/10.1126/science.373.6553.372.
CRED and UCLouvain, 2023: EM-DAT. Accessed 12 December 2023, www.emdat.be.
de Bruijn, K., B. van den Hurk, K. Slager, G. Rongen, M. Hegnauer, and K. J. van Heeringen, 2023: Storylines of the impacts in the Netherlands of alternative realizations of the western Europe July 2021 floods. J. Coastal Riverine Flood Risk, 2, 8, https://doi.org/10.59490/jcrfr.2023.0008.
DWD, 2021: Bericht zum Ablauf und Umfang der operationellen Warn- und Beratungstätigkeit des Deutschen Wetterdienstes im Vorlauf und während des Unwetters 12–15 Juli 2021 in NRW und Rheinland-Pfalz, ausgelöst durch das Tief “Bernd.” 10 pp., https://www.dwd.de/DE/presse/hintergrundberichte/berichts_warnchronologie.pdf.pdf?__blob=publicationFile&v=2.
ECMWF, 2023a: The Mars Catalogue – Operational archive. Accessed 1 December 2023, https://apps.ecmwf.int/mars-catalogue/?class=od.
ECMWF, 2023b: ECMWF | Charts. Accessed 16 January 2024, https://charts.ecmwf.int/.
ECMWF, 2023c: Dissemination schedule – Data and charts – ECMWF confluence Wiki. Accessed 24 January 2024, https://confluence.ecmwf.int/display/DAC/Dissemination+schedule.
Endendijk, T., W. J. W. Botzen, H. de Moel, J. C. J. H. Aerst, S. J. Duijndam, K. Slager, B. Kolen, and M. Kok, 2023a: Experience from the 2021 floods in the Netherlands: Household survey results on impacts and responses. J. Coastal Riverine Flood Risk, 2, 9, https://doi.org/10.59490/jcrfr.2023.0009.
Endendijk, T., W. J. W. Botzen, H. de Moel, J. C. J. H. Aerts, K. Slager, and M. Kok, 2023b: Flood vulnerability models and household flood damage mitigation measures: An econometric analysis of survey data. Water Resour. Res., 59, e2022WR034192, https://doi.org/10.1029/2022WR034192.
Fekete, A., and S. Sandholz, 2021: Here comes the flood, but not failure? Lessons to learn after the heavy rain and pluvial floods in Germany 2021. Water, 13, 3016, https://doi.org/10.3390/w13213016.
Fowler, H. J., and Coauthors, 2021: Anthropogenic intensification of short-duration rainfall extremes. Nat. Rev. Earth Environ., 2, 107–122, https://doi.org/10.1038/s43017-020-00128-6.
Global Commission on Adaptation, 2019: Adapt now: A global call for leadership on climate resilience. World Resources Institute, 90 pp., https://gca.org/reports/adapt-now-a-global-call-for-leadership-on-climate-resilience/.
Golding, B., Ed., 2022: Towards the “Perfect” Weather Warning: Bridging Disciplinary Gaps through Partnership and Communication. Springer International Publishing, 270 pp.
Haiden, T., M. Janousek, F. Vitart, Z. Ben-Bouallegue, and F. Prates, 2023: Evaluation of ECMWF forecasts, including the 2023 upgrade. ECMWF Tech. Memo. 911, 60 pp., https://www.ecmwf.int/en/elibrary/81389-evaluation-ecmwf-forecasts-including-2023-upgrade.
Hallegatte, S., 2012: A cost effective solution to reduce disaster losses in developing countries: Hydro-Meteorological Services, early warning, and evacuation. Policy Research Working Paper 6058, World Bank, 22 pp., https://doi.org/10.1596/1813-9450-6058.
Hewson, T. D., 2022: European floods in summer 2021: The extreme rainfall of 14–15 July. Forecast user – ECMWF Confluence Wiki. https://confluence.ecmwf.int/display/FCST/European+Floods+in+Summer+2021%3A+the+Extreme+rainfall+of+14-15+July.
Hewson, T. D., and R. G. Owens, 2018: ECMWF forecast user guide – Section 8.1.9 Extreme Forecast Index – EFI, and Shift of Tails – SOT. https://confluence.ecmwf.int/display/FUG/Section+8.1.9+Extreme+Forecast+Index+-+EFI%2C+and+Shift+of+Tails+-+SOT.
Hewson, T. D., and F. M. Pillosu, 2021: A low-cost post-processing technique improves weather forecasts around the world. Commun. Earth Environ., 2, 132, https://doi.org/10.1038/s43247-021-00185-9.
Hodnebrog, Ø., and Coauthors, 2019: Intensification of summer precipitation with shorter time-scales in Europe. Environ. Res. Lett., 14, 124050, https://doi.org/10.1088/1748-9326/ab549c.
Hólm, E., R. Forbes, S. Lang, L. Magnusson, and S. Malardel, 2016: New model cycle brings higher resolution. ECMWF Newsletter, No. 147, ECMWF, Reading, United Kingdom, 46 pp., https://www.ecmwf.int/en/newsletter/147/meteorology/new-model-cycle-brings-higher-resolution.
Hu, G., and C. L. E. Franzke, 2020: Evaluation of daily precipitation extremes in reanalysis and gridded observation-based data sets over Germany. Geophys. Res. Lett., 47, e2020GL089624, https://doi.org/10.1029/2020GL089624.
IPCC, 2021: Summary for policymakers. Climate Change 2021: The Physical Science Basis, V. Masson-Delmotte et al., Eds., Cambridge University Press, 3–32.
Jonkman, S. N., and Coauthors, 2021: Hoogwater 2021: Feiten en Duiding. Expertise Netwerk Waterveiligheid, 214 pp., https://pure.tudelft.nl/ws/portalfiles/portal/100347108/fact_finding_hoogwater_2021_versie_1_2_1_.pdf.
Jonkman, S. N., H. de Moel, R. Moll, and K. Slager, 2023: Editorial for the Special issue on “2021 Summer Floods in Europe”. J. Coastal Riverine Flood Risk, 2, 11, https://doi.org/10.59490/jcrfr.2023.0011.
Junghänel, T., and Coauthors, 2021: Hydro-klimatologische Einordnung der Stark- und Dauerniederschläge in Teilen Deutschlands im Zusammenhang mit dem Tiefdruckgebiet “Bernd” vom 12. bis 19. Juli 2021. DWD, 16 pp., https://www.dwd.de/DE/leistungen/besondereereignisse/niederschlag/20210721_bericht_starkniederschlaege_tief_bernd.pdf?__blob=publicationFile&v=10.
Kahraman, A., E. J. Kendon, S. C. Chan, and H. J. Fowler, 2021: Quasi-stationary intense rainstorms spread across Europe under climate change. Geophys. Res. Lett., 48, e2020GL092361, https://doi.org/10.1029/2020GL092361.
Klein Tank, A. M. G., and Coauthors, 2002: Daily dataset of 20th-century surface air temperature and precipitation series for the European Climate Assessment. Int. J. Climatol., 22, 1441–1453, https://doi.org/10.1002/joc.773.
Kolb, L. L., and R. R. Rapp, 1962: The utility of weather forecasts to the raisin industry. J. Appl. Meteor., 1, 8–12, https://doi.org/10.1175/1520-0450(1962)001<0008:TUOWFT>2.0.CO;2.
Kreibich, H., S. Christenberger, and R. Schwarze, 2011: Economic motivation of households to undertake private precautionary measures against floods. Nat. Hazards Earth Syst. Sci., 11, 309–321, https://doi.org/10.5194/nhess-11-309-2011.
Kreibich, H., S. Christenberger, and R. Schwarze, 2012: Corrigendum to “Economic motivation of households to undertake private precautionary measures against floods” published in Nat. Hazards Earth Syst. Sci., 11, 309–321, 2011. Nat. Hazards Earth Syst. Sci., 12, 391–392, https://doi.org/10.5194/nhess-12-391-2012.
Kreibich, H., M. Müller, K. Schröter, and A. H. Thieken, 2017: New insights into flood warning reception and emergency response by affected parties. Nat. Hazards Earth Syst. Sci., 17, 2075–2092, https://doi.org/10.5194/nhess-17-2075-2017.
Kreibich, H., P. Hudson, and B. Merz, 2021: Knowing what to do substantially improves the effectiveness of flood early warning. Bull. Amer. Meteor. Soc., 102, E1450–E1463, https://doi.org/10.1175/BAMS-D-20-0262.1.
Lang, S., D. Schepers, and M. Rodwell, 2023: IFS upgrade brings many improvements and unifies medium-range resolutions. ECMWF Newsletter, No. 176, Reading United Kingdom, ECMWF, 21–28, https://www.ecmwf.int/en/newsletter/176/earth-system-science/ifs-upgrade-brings-many-improvements-and-unifies-medium.
LeClerc, J., and S. Joslyn, 2015: The cry wolf effect and weather-related decision making. Risk Anal., 35, 385–395, https://doi.org/10.1111/risa.12336.
Ludwig, P., and Coauthors, 2023: A multi-disciplinary analysis of the exceptional flood event of July 2021 in central Europe—Part 2: Historical context and relation to climate change. Nat. Hazards Earth Syst. Sci., 23, 1287–1311, https://doi.org/10.5194/nhess-23-1287-2023.
Magnusson, L., D. Richardson, and T. Haiden, 2014: Verification of extreme weather events: Discrete predictands. ECMWF Tech. Memo. 731, 29 pp., https://www.ecmwf.int/en/elibrary/75517-verification-extreme-weather-events-discrete-predictands.
Mathias, L., 2019: Major flood event in the Mullerthal region on 1 June 2018: Event analysis and predictability. MeteoLux, 17 pp., https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.meteolux.lu/fr/filedownload/595/floodmullerthal_fullreport_en_final-1.pdf/type/pdf&ved=2ahUKEwj62-_CgO2FAxXJ3QIHHTWpBD8QFnoECA4QAQ&usg=AOvVaw2hkEGOSEcrj2_Cz-OvQ7HE.
Mechler, R., 2016: Reviewing estimates of the economic efficiency of disaster risk management: Opportunities and limitations of using risk-based cost–benefit analysis. Nat. Hazards, 81, 2121–2147, https://doi.org/10.1007/s11069-016-2170-y.
Mercogliano, P., G. Rianna, A. Reder, M. Raffa, M. Mancini, M. Stojiljkovic, C. de Valk, and G. van der Schrier, 2020: Extreme precipitation risk indicators for Europe and European cities from 1950 to 2019. Copernicus Climate Change Service (C3S). Climate Data Store (CDS), accessed 1 December 2024, https://doi.org/10.24381/cds.3a9c4f89.
Merz, B., and Coauthors, 2021: Causes, impacts and patterns of disastrous river floods. Nat. Rev. Earth Environ., 2, 592–609, https://doi.org/10.1038/s43017-021-00195-3.
Mohr, S., and Coauthors, 2023: A multi-disciplinary analysis of the exceptional flood event of July 2021 in central Europe—Part 1: Event description and analysis. Nat. Hazards Earth Syst. Sci., 23, 525–551, https://doi.org/10.5194/nhess-23-525-2023.
Munich Re, 2022: Hurricanes, cold waves, tornadoes: Weather disasters in USA dominate natural disaster losses in 2021. Accessed 3 January 2024, https://www.munichre.com/en/company/media-relations/media-information-and-corporate-news/media-information/2022/natural-disaster-losses-2021.html.
Murphy, A. H., 1977: The value of climatological, categorical and probabilistic forecasts in the cost-loss ratio situation. Mon. Wea. Rev., 105, 803–816, https://doi.org/10.1175/1520-0493(1977)105<0803:TVOCCA>2.0.CO;2.
Pappenberger, F., H. L. Cloke, D. J. Parker, F. Wetterhall, D. S. Richardson, and J. Thielen, 2015: The monetary benefit of early flood warnings in Europe. Environ. Sci. Policy, 51, 278–291, https://doi.org/10.1016/j.envsci.2015.04.016.
Pendergrass, A. G., 2018: What precipitation is extreme? Science, 360, 1072–1073, https://doi.org/10.1126/science.aat1871.
Portele, T. C., C. Lorenz, B. Dibrani, P. Laux, J. Bliefernicht, and H. Kunstmann, 2021: Seasonal forecasts offer economic benefit for hydrological decision making in semi-arid regions. Sci. Rep., 11, 10581, https://doi.org/10.1038/s41598-021-89564-y.
Poussin, J. K., W. J. Wouter Botzen, and J. C. J. H. Aerts, 2015: Effectiveness of flood damage mitigation measures: Empirical evidence from French flood disasters. Global Environ. Change, 31, 74–84, https://doi.org/10.1016/j.gloenvcha.2014.12.007.
Ravuri, S., and Coauthors, 2021: Skilful precipitation nowcasting using deep generative models of radar. Nature, 597, 672–677, https://doi.org/10.1038/s41586-021-03854-z.
Richardson, D. S., 2000: Skill and relative economic value of the ECMWF ensemble prediction system. Quart. J. Roy. Meteor. Soc., 126, 649–667, https://doi.org/10.1002/qj.49712656313.
Richardson, D. S., 2003: Predictability and economic value. Seminar on Predictability of Weather and Climate, Shinfield Park, Reading, United Kingdom, ECMWF, 321–332, https://www.ecmwf.int/node/11922.
Rivoire, P., O. Martius, P. Naveau, and A. Tuel, 2023: Assessment of subseasonal-to-seasonal (S2S) ensemble extreme precipitation forecast skill over Europe. Nat. Hazards Earth Syst. Sci., 23, 2857–2871, https://doi.org/10.5194/nhess-23-2857-2023.
Smith, P. J., and Coauthors, 2016: On the operational implementation of the European Flood Awareness System (EFAS). Flood Forecasting: A Global Perspective, T. E. Adams and T. C. Pagano, Eds., Academic Press, 313–348.
Thieken, A. H., P. Bubeck, A. Heidenreich, J. von Keyserlingk, L. Dillenardt, and A. Otto, 2023: Performance of the flood warning system in Germany in July 2021—Insights from affected residents. Nat. Hazards Earth Syst. Sci., 23, 973–990, https://doi.org/10.5194/nhess-23-973-2023.
Thornes, J. E., and D. B. Stephenson, 2001: How to judge the quality and value of weather forecast products. Meteor. Appl., 8, 307–314, https://doi.org/10.1017/S1350482701003061.
Tradowsky, J. S., and Coauthors, 2023: Attribution of the heavy rainfall events leading to severe flooding in western Europe during July 2021. Climatic Change, 176, 90, https://doi.org/10.1007/s10584-023-03502-7.
Tsonevsky, I., 2016: New EFI parameters for forecasting severe convection. ECMWF Newsletter, No. 144, ECMWF, Reading, United Kingdom, 29–32, https://www.researchgate.net/publication/292151482_New_EFI_parameters_for_forecasting_severe_convection.
Tsonevsky, I., C. A. Doswell III, and H. E. Brooks, 2018: Early warnings of severe convection using the ECMWF extreme forecast index. Wea. Forecasting, 33, 857–871, https://doi.org/10.1175/WAF-D-18-0030.1.
UNDRR, 2022: Early warnings for all. Accessed 17 January 2024, https://www.undrr.org/early-warnings-for-all.
Verkade, J. S., and M. G. F. Werner, 2011: Estimating the benefits of single value and probability forecasting for flood warning. Hydrol. Earth Syst. Sci., 15, 3751–3765, https://doi.org/10.5194/hess-15-3751-2011.
waterpompshop, 2023: Waterpomp kopen? Het grootste aanbod waterpompen! Accessed 9 February 2024, https://www.waterpompshop.nl/.
Zhu, Y., Z. Toth, R. Wobus, D. Richardson, and K. Mylne, 2002: The economic value of ensemble-based weather forecasts. Bull. Amer. Meteor. Soc., 83, 73–83, https://doi.org/10.1175/1520-0477(2002)083<0073:TEVOEB>2.3.CO;2.
