What is EFAS Postprocessing?
.
| What: | Developers and users of the postprocessed flood forecast product of the European Flood Awareness System (EFAS) collaborated to determine priorities for future co-developments in line with user requirements and available resources |
| When: | 27 September 2022 |
| Where: | Joint Research Centre Conference Centre, Ispra, Italy |
With flood damage increasing, the effective use of mitigation instruments is essential (WMO 2021). Flood Early Warning Systems are cost-effective tools to increase preparedness for floods and reduce the resulting damage (Pappenberger et al. 2015; Verkade and Werner 2011). The European Flood Awareness System (EFAS), part of the Copernicus Emergency Management Services (CEMS), is a pan-European flood forecasting system established to complement national systems by producing medium-range probabilistic flood forecasts particularly for large transnational rivers (Thielen et al. 2009; Demeritt et al. 2013; De Roo et al. 2011; Smith et al. 2016). EFAS is co-produced to ensure that the forecasts created provide decision-relevant information that can be incorporated into the procedures of the forecast users while fully utilizing the knowledge and resources of all involved (Lienert et al. 2022; Bierens et al. 2020). In practice, several mechanisms are employed to facilitate the required collaboration including webinars, online feedback forms, training sessions, user surveys, workshops, working groups, and the EFAS annual meetings.
The focus of the workshop discussed here was the co-design of the EFAS postprocessed forecasts with the workshop forming part of a larger iterative co-production process (Fig. 1). The EFAS postprocessed forecasts were initially introduced following a consultation with users which found that hydrographic forecast products would be beneficial. As with all forecasting systems, EFAS forecasts are subject to errors and uncertainties from several sources including the initial conditions, the meteorological forcings, and the hydrological model. Therefore, hydrographs created from the raw (or non-postprocessed) hydrological model output must be considered carefully. The postprocessed forecast product aims to provide a bias-corrected hydrograph and to quantify the total predictive uncertainty (De Roo et al. 2011). The postprocessed forecasts are available at river gauge locations for which historic and near-real-time river discharge observations are available (Matthews et al. 2022). The number of locations has increased from three stations during initial testing (Bogner and Pappenberger 2011) to 1,608 stations as of 13 September 2022 across the EFAS domain.
Co-production cycle for the EFAS postprocessed forecasts and workshop agenda. The workshop was an opportunity to co-explore the limitation of the postprocessed forecasts and co-design potential developments.
Citation: Bulletin of the American Meteorological Society 104, 5; 10.1175/BAMS-D-23-0061.1
During the 17th EFAS Annual Meeting in September 2022, a dedicated workshop was organized for producers and users of the EFAS postprocessed forecasts to discuss its future evolution. Here, we summaries the outcomes of the workshop and present the next steps in the co-developmental cycle. We also reflect on the broader workshop findings that should be applied to other EFAS forecast products and forecasting systems. Finally, we discuss the lessons learnt regarding stakeholder engagement in a “post-pandemic” world.
Workshop organization
The 17th EFAS Annual Meeting was held on 27–28 September 2022. The organizers and host (the Joint Research Centre of the European Commission) opted for a hybrid format to encourage face-to-face discussions, as it was the first meeting since social and travel restrictions had eased, while still engaging with those unable to travel. The meeting was open to representatives from the EFAS partners (hydrometeorological authorities across Europe mandated to provide flood early warnings for their respective regions), and co-organized by all EFAS operational centers (organizations contracted to run EFAS). To complement the plenary sessions, four workshops were run focusing on specific topics voted for by attendees at registration, including a workshop on the EFAS postprocessed forecasts.
The workshop on the EFAS postprocessed forecasts was held on the first day of the meeting and lasted 1 h and 15 min, with a total of 21 in-person attendees (∼27% of present) and 20 online attendees (∼40% of registered). The primary objectives of the workshop were to
- 1)present the operational EFAS postprocessed forecasts and forecast products including a recent increase in temporal resolution,
- 2)determine the current usage of the EFAS postprocessed forecasts,
- 3)identify limitations and barriers to the use of the EFAS postprocessed forecasts, and
- 4)determine priorities for future developments of the EFAS postprocessed forecasts.
The usability of forecasts is dependent on the relevance of the forecast information content, the communication channel, the forecast visualization (also known as the forecast product), the quality of the forecast, and the expertise of the user (Vincent et al. 2020; WMO 2022). Therefore, the workshop was organized around these five topics (Table 1). The format of the workshop and its position within the wider co-production process of the EFAS postprocessed forecast is shown in Fig. 1. A pre-workshop questionnaire was distributed to all registered attendees one week before the workshop to comprehend the current usage and understanding of the postprocessed forecasts. A post-workshop questionnaire was also distributed allowing participants to provide anonymous and individual feedback.
Questions provided as prompts to facilitate the group discussions.
The workshop consisted of a presentation followed by a participatory group activity to encourage dialogue (Fig. 1). The presentation gave an overview of each of the five topics. Prompts were provided to guide group discussions (Table 1), but participants were encouraged to consider other ideas inspired by their own systems and experience using the EFAS postprocessed forecasts in an operational setting. Mind-maps prepared ahead of the workshop were used to record the discussions. Online participants used Webex (www.webex.com/) and Miro (https://miro.com/) to participate in the workshop with breakout rooms used to allow for group discussions.
EFAS postprocessed forecast
A key aim of the presentation was to ensure all participants had an understanding of the postprocessed forecasts even if they did not regularly use the forecasts. The presentation also allowed the changes due to a recent increase in the temporal resolution to be presented (Mazzetti et al. 2020). The full presentation is available at www.efas.eu/en/news/17th-efas-annual-meeting and details regarding each of the five topics are documented on the “CEMS-Flood Wiki” (https://confluence.ecmwf.int/display/CEMS/CEMS-Flood).
The EFAS postprocessed forecasts use recent river discharge observations (provided by EFAS data providers, which include several EFAS partners) to adjust the EFAS multimodel medium-range ensemble forecasts to account for errors and uncertainties in the ensemble forecasts. The method of postprocessing has changed over time (Matthews et al. 2022; Bogner and Kalas 2008; Bogner and Pappenberger 2011; Bogner et al. 2012; Raftery et al. 2005; Smith et al. 2016; De Roo et al. 2011), with the current method using a combination of the Model Conditional Processor (MCP; Todini 2008; Coccia and Todini 2011) and the Ensemble Model Output Statistics method (EMOS; Gneiting et al. 2005). The MCP requires an offline calibration procedure which is performed twice a year to incorporate the most recent river discharge observations. The MCP quantifies the hydrological uncertainty and corrects biases due to the hydrological model. The EMOS method quantifies the uncertainty due to the meteorological forcings. The outputs from both methods are combined using the recursive Kalman filter (Kalman 1960).
The postprocessed forecast is available to users via the EFAS Sensor Observation Service (SOS), a web-based application programming interface (API). The EFAS-SOS allows users to visualize each percentile of the postprocessed forecast as well as to download the forecast data. Additionally, the postprocessed forecast is transformed into a forecast product, called the “Real-Time Hydrograph,” available on the EFAS website (Fig. 2). The postprocessed forecast product has been available since 2012 and in 2020 the product was merged with the medium-range reporting point layer so that all medium-range forecast information regarding a specific location was available in a single pop-out window.
The so-called “Real-time Hydrograph” forecast product accessed via the EFAS website. Forecast from 20 Sep 2022 for the Veľké Kapušany station, Slovakia.
Citation: Bulletin of the American Meteorological Society 104, 5; 10.1175/BAMS-D-23-0061.1
The postprocessed forecast product consists of three panels (pop-out window in Fig. 2): a hydrograph (left) where darker shaded percentiles are closer to the median, and two bar charts showing the probability of exceeding the mean flow (MQ, lower right) and the mean annual maximum (MHQ, upper right), respectively, at each lead time. The flood thresholds are calculated from river discharge observations—a key distinction between the postprocessed forecast product and other EFAS medium range forecast products which use thresholds calculated from simulated reference climatologies.
Results of the latest forecast evaluations were presented, showing that the postprocessed forecasts could predict the exceedance probability of the flood thresholds more reliably (i.e., the forecast probabilities more accurately represented the exceedances of the thresholds) than the raw ensemble forecasts (Matthews et al. 2022), and that the update to 6-hourly time steps increased the skill of the forecast probability distribution at shorter lead times.
Current usage
All participants stated that, in the context of EFAS, postprocessing is useful while 50% said it is essential. Most participants selected the key aims of postprocessing to be the reduction and quantification of forecast uncertainties. Over 70% of participants primarily use the EFAS postprocessed forecasts to predict upcoming floods in the next 5 days whereas only 14% (2/14) use the forecasts to predict floods more than 10 days in the future. All participants indicated they use the EFAS postprocessed forecasts in conjunction with other flood and meteorological forecast products.
Co-identifying limitations and future developments
Potential future development and limitations were discussed for each of the five topics. Here, we discuss the key responses from the questionnaires and the mind-map-based group discussions (Fig. 3).
Mind-maps from two group discussions.
Citation: Bulletin of the American Meteorological Society 104, 5; 10.1175/BAMS-D-23-0061.1
Access.
All participants indicated a preference for accessing the postprocessed forecast via the EFAS website, with most indicating a high confidence in accessing the product in this manner. Locating stations by name or identification number rather than by municipality was suggested as a more efficient search function. The EFAS SOS is useful for comparing EFAS forecasts with a user’s own system. However, it can be slow and difficult to use, indicating the need for improved efficiency of the EFAS SOS system and clearer guidelines on data access through this service.
Method.
Improving the postprocessing method was most frequently selected as a top priority for future developments, with both the MCP and the EMOS components of the method identified as possible limitations. For example, the MCP component requires a time series of recent (currently the previous 40 days) river discharge observations on which to condition the forecast probability. Incorporating local knowledge of the catchments may optimize the number of observations necessary for each station.
Water level is the main variable of interest for some EFAS partners. Therefore, extending the postprocessing method to create forecasts of water level would be beneficial for some users. However, the suggestion was not unanimous so water level should not replace discharge and would need to be included as an additional product.
Product design.
Two key changes to the forecast product were proposed during the workshop: the flood event thresholds and the representation of the probability distribution. There was a consensus that locally defined thresholds (provided by EFAS partners) would be more useful than the current flow-based thresholds. Alternative thresholds include frequency-based magnitudes (e.g., return periods), physically based thresholds (e.g., defined using hydraulic models), or other locally defined threshold. Locally defined thresholds would allow more consistency with local decision making and for direct comparison with users’ own systems.
The slowly changing color gradient in the forecast product (Fig. 2) makes it difficult to identify a specific percentile. Boxplots were suggested as a potential alternative to the current form of the hydrograph. Boxplots would also make the postprocessed forecast product consistent with other EFAS forecast products allowing for direct comparison between the postprocessed and raw forecasts.
Training and documentation.
In the post-workshop survey, most attendees said the workshop was useful, but that additional training was desired, with a webinar being the preferred format. Overall, participants were satisfied with the level of detail in the documentation provided on the EFAS Wiki pages but stated that it was not easy to find specific information on the postprocessed forecasts. It was suggested that a direct link from the forecast product on the EFAS website would be beneficial. Additionally, some participants requested more practical guidance regarding the data requirements and station specific information regarding the calibration process (e.g., the calibration period).
Evaluation.
Participants indicated a desire for the postprocessed forecasts to be evaluated routinely with each calibration exercise performed (twice a year). There was a clear preference for the evaluation results to be station specific and available alongside the forecast products on the EFAS website as an “evaluation product.” The importance of skill of different features of the forecasts varied between participants with some prioritizing the reliability of the forecast exceeding a threshold and others prioritizing the peak magnitude and timing of the forecast median. This indicates that the evaluation product should be designed using a multimetric approach.
Going forward
A set of potential future developments of the EFAS postprocessed forecasts, to be researched in the co-investigation phase, has been defined from the workshop discussions (Fig. 4). During this process three key principles were found to overlap with many of the tasks:
Potential future developments of the EFAS postprocessed forecasts ordered by complexity. Square color indicates the principle of interest: locally relevant products (purple), user-focused auxiliary information (green), and ease of access (yellow). Gray indicates feedback that does not explicitly follow a guiding principle. Circle color indicates intended timeframe.
Citation: Bulletin of the American Meteorological Society 104, 5; 10.1175/BAMS-D-23-0061.1
- 1) Locally relevant forecast products: The information content of the forecast product should be relevant to the task of the users. For example, locally defined flood thresholds allow users to connect the forecast to their local procedures.
- 2) User-focused auxiliary information: Auxiliary information should allow users to extract the information that is applicable to them, their usage of the forecast, and their region of interest. For example, forecast evaluations should be conducted at station level.
- 3) Ease of access: All forecast and auxiliary information should be easy and quick for the user to access. For example, providing links to the documentation alongside the forecast product means users can check the information without significantly detouring from their task.
These three principles can be applied to other EFAS forecast products and other early warning systems. In addition to user requirements, future developments are guided by available resources. Additionally, due to the number of locations being postprocessed in EFAS, each step must be automated once in operations.
EFAS partners will be invited to participate in the co-investigation phase, led by the CEMS Hydrological Forecast Computational Centre, to ensure that communication channels between developers and users remain open throughout the process. During the co-investigation phase feasibility studies will be conducted for each development in Fig. 4. The study results will be discussed with EFAS partners to identify any changing needs and to ensure the solutions are appropriate. Once a preferred solution has been selected it will be presented to all stakeholders to provide the opportunity to raise concerns and offer alternative solutions before implementation in the operational system.
Reflections on a hybrid workshop
This was the first hybrid EFAS annual meeting following the pandemic. The aim was to allow in-person and online attendees to participate in the workshop equally. However, there were three key challenges in running the workshop discussed here. First, time constraints limited the workshop to 1 h and 15 min, limiting opportunity for open discussion. While extending the time within the annual meeting was not possible, a standalone workshop would allow more time but may not be attended by as many EFAS partners. Second, the dynamic differed between the in-person groups who discussed the topics, working through the prompts together, and online groups who worked individually. When split into breakout rooms many online participants did not communicate with fellow participants. This could have been for several reasons including lack of technical equipment, being in a shared environment, multitasking, or lack of engagement. Additionally, due to the explicit split between online and physical attendees there was no discussion between the two. Therefore, for the workshop to be more beneficial for both the organizers and the online participants, rather than replicating the group mind-mapping activity online, the activity could differ between online participants and in-person attendees. Finally, the number of responses to the questionnaires was low compared to participants. This is likely due to competing priorities outside of the workshop and as such, if time allows, questionnaires should be conducted during the workshop.
Concluding remarks
A workshop was held during the 17th EFAS Annual Meeting to co-produce future developmental priorities for the EFAS postprocessed forecasts. Despite difficulties that emerged due to time constraints and the hybrid nature of the session, all workshop objectives were achieved. While the specific outcomes from the workshop are applicable to the EFAS postprocessed forecast product, the three derived principles are applicable to other EFAS forecast products and those of other forecasting systems, such as the Global Flood Awareness System (GloFAS; Alfieri et al. 2013).
Acknowledgments.
This work was supported by the Engineering and Physical Sciences Research Council (Grant EP/R513301/), the Natural Environment Research Council (Grant NE/S015590/1), the NERC National Centre for Earth Observation and the European Centre for Medium-Range Weather Forecasts. We thank all participants for their insightful comments and the technical team at the JRC Conference Centre for their help. We also thank Sue Todd, Karen O’Regan, Stefania Grimaldi, and Vera Thiemig for their help before, during, and after the workshop.
References
Alfieri, L., P. Burek, E. Dutra, B. Krzeminski, D. Muraro, J. Thielen, and F. Pappenberger, 2013: GloFAS – Global ensemble streamflow forecasting and flood early warning. Hydrol. Earth Syst. Sci., 17, 1161–1175, https://doi.org/10.5194/hess-17-1161-2013.
Bierens, S., K. Boersma, and M. J. van den Homberg, 2020: The legitimacy, accountability, and ownership of an impact-based forecasting model in disaster governance. Polit. Governance, 8, 445–455, https://doi.org/10.17645/pag.v8i4.3161.
Bogner, K., and M. Kalas, 2008: Error‐correction methods and evaluation of an ensemble based hydrological forecasting system for the Upper Danube catchment. Atmos. Sci. Lett., 9, 95–102, https://doi.org/10.1002/asl.180.
Bogner, K., and F. Pappenberger, 2011: Multiscale error analysis, correction, and predictive uncertainty estimation in a flood forecasting system. Water Resour. Res., 47, W07524, https://doi.org/10.1029/2010WR009137.
Bogner, K., F. Pappenberger, and H. L. Cloke, 2012: Technical note: The normal quantile transformation and its application in a flood forecasting system. Hydrol. Earth Syst. Sci., 16, 1085–1094, https://doi.org/10.5194/hess-16-1085-2012.
Coccia, G., and E. Todini, 2011: Recent developments in predictive uncertainty assessment based on the model conditional processor approach. Hydrol. Earth Syst. Sci., 15, 3253–3274, https://doi.org/10.5194/hess-15-3253-2011.
Demeritt, D., S. Nobert, H. L. Cloke, and F. Pappenberger, 2013: The European Flood Alert System and the communication, perception, and use of ensemble predictions for operational flood risk management. Hydrol. Processes, 27, 147–157, https://doi.org/10.1002/hyp.9419.
De Roo, A. P. J., and Coauthors, 2011: Quality control, validation and user feedback of the European Flood Alert System (EFAS). Int. J. Digital Earth, 4, 77–90, https://doi.org/10.1080/17538947.2010.510302.
Gneiting, T., A. E. Raftery, A. H. Westveld III, and T. Goldman, 2005: Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation. Mon. Wea. Rev., 133, 1098–1118, https://doi.org/10.1175/MWR2904.1.
Kalman, R. E., 1960: A new approach to linear filtering and prediction problems. J. Basic Eng., 82, 35–45, https://doi.org/10.1115/1.3662552.
Lienert, J., J. C. M. Andersson, D. Hofmann, and F. Silva Pinto, and M. Kuller, 2022: The role of multi-criteria decision analysis in a transdisciplinary process: Co-developing a flood forecasting system in western Africa. Hydrol. Earth Syst. Sci., 26, 2899–2922, https://doi.org/10.5194/hess-26-2899-2022.
Matthews, G., C. Barnard, H. Cloke, S. L. Dance, T. Jurlina, C. Mazzetti, and C. Prudhomme, 2022: Evaluating the impact of post-processing medium-range ensemble streamflow forecasts from the European Flood Awareness System. Hydrol. Earth Syst. Sci., 26, 2939–2968, https://doi.org/10.5194/hess-26-2939-2022.
Mazzetti, C., D. Decremer, and C. Prudhomme, 2020: Major upgrade of the European Flood Awareness System. ECMWF Newsletter, No. 166, ECMWF, Reading, United Kingdom, 26–32, www.ecmwf.int/en/newsletter/166/meteorology/major-upgrade-european-flood-awareness-system#:∼:text=On%2014%20October%202020%2C%20the,a%20step%2Dchange%20in%20EFAS.
Pappenberger, F., H. L. Cloke, D. J. Parker, F. Wetterhall, D. S. Richardson, and J. Thielen, 2015: The monetary benefit of early flood warnings in Europe. Environ. Sci. Policy, 51, 278–291, https://doi.org/10.1016/j.envsci.2015.04.016.
Raftery, A. E., T. Gneiting, F. Balabdaoui, and M. Polakowski, 2005: Using Bayesian model averaging to calibrate forecast ensembles. Mon. Wea. Rev., 133, 1155–1174, https://doi.org/10.1175/MWR2906.1.
Smith, P. J., and Coauthors, 2016: On the operational implementation of the European Flood Awareness System (EFAS). Flood Forecasting: A Global Perspective, T. E. Adams and T. C. Pagano, Eds., Academic Press, 313–348, https://doi.org/10.1016/B978-0-12-801884-2.00011-6.
Thielen, J., J. Bartholmes, M. H. Ramos, and A. de Roo, 2009: The European Flood Alert System – Part 1: Concept and development. Hydrol. Earth Syst. Sci., 13, 125–140. https://doi.org/10.5194/hess-13-125-2009.
Todini, E., 2008: A model conditional processor to assess predictive uncertainty in flood forecasting. Int. J. River Basin Manage., 6, 123–137, https://doi.org/10.1080/15715124.2008.9635342.
Verkade, J. S., and M. G. F. Werner, 2011: Estimating the benefits of single value and probability forecasting for flood warning. Hydrol. Earth Syst. Sci., 15, 3751–3765, https://doi.org/10.5194/hess-15-3751-2011.
Vincent, K., and Coauthors, 2020: Re-balancing climate services to inform climate-resilient planning–A conceptual framework and illustrations from sub-Saharan Africa. Climate Risk Manage., 29, 100242, https://doi.org/10.1016/j.crm.2020.100242.
WMO, 2021: WMO atlas of mortality and economic losses from weather, climate, and water extremes (1970–2019). WMO-1267, 90 pp., https://library.wmo.int/doc_num.php?explnum_id=10989.
WMO, 2022: Assessment guidelines for end-to-end flood forecasting and early warning systems. WMO-1286, 36 pp., https://library.wmo.int/doc_num.php?explnum_id=11379.
