1. Introduction
Lives continue to be lost and widespread damage created by weather-related hazards globally, despite the issuance of weather warnings. The economic impact of weather events is increasing due to changes in societal factors, such as population growth and demographic shifts, increased vulnerability, as well as a changing climate (Changnon et al. 2000). The World Meteorological Organization (WMO) has suggested a potential solution is for national meteorological and hydrological services (NMHSs) to use impact-based warnings to bridge the gap between forecasts and possible impacts of impending hazards (WMO 2015). This supports the shift toward multihazard early warning systems becoming “people centered” [Basher 2006; United Nations International Strategy for Disaster Reduction (UNISDR) 2015], empowering “individuals and communities threatened by hazards to act in sufficient time and in an appropriate manner to reduce the possibility of personal injury and illness, loss of life and damage to property, assets and the environment” (WMO 2018, p. 3). The vast majority of European NMHSs surveyed by Kaltenberger et al. (2020) expect to be using impact-based criteria for their public weather warnings in the near future.
Developing or reviewing a warning system requires understanding its foundation—whether it is designed around phenomena/hazards, underlying hazard processes, impacts, or actions—and the associated challenges and benefits of such systems (Potter et al. 2014, 2018a). Investigating the effectiveness of impact-based weather warnings has been a focus of the WMO High Impact Weather research program (Taylor et al. 2018; Zhang et al. 2019). But there remains a gap in our understanding of what the benefits and challenges are of implementing an impact-based forecasting and warning (IBFW) system for severe weather from an institutional perspective. A key step in implementing IBFW services involves consulting with stakeholders and understanding challenges and benefits of the system (WMO 2015; Kaltenberger et al. 2020).
This research aims to understand the benefits and challenges of IBFWs for hydrometeorological hazards, from the perspective of meteorologists, hydrologists, and emergency managers who have been or could be using the new warning approach. This understanding will lead to defining a way forward for ensuring IBFWs are beneficial and worthwhile pursuing and will help to address implementation challenges.
2. Background
a. Overview of impact-based forecasts and warnings
Early warning systems are the last line of defense against oncoming hazards. Thus, their operation and success are crucial as recognized by the Hyogo Framework for Action 2005–15 and the Sendai Framework for Disaster Risk Reduction 2015–30 (Sendai Framework; UNISDR 2005, 2015). However, considering historic events, one criticism of many early warning systems is their failure to effectively communicate information to their audiences to prompt appropriate and timely protective action (Basher 2006; WMO 2015; Uccellini and Ten Hoeve 2019). As such, early warning systems that are more “people centered” have been proposed (Basher 2006; Scolobig et al. 2015). The Sendai Framework also advocates for risk-informed decision-making, with “understanding disaster risk” a priority of the framework (UNISDR 2015, p. 14). IBFWs have been proposed by the WMO to attempt to add meaningful context to severe weather warnings (WMO 2015). The WMO proposes that IBFWs can incite more appropriate responses to warnings by including specific social, economic, and environmental impacts of a hazard in the warning (WMO 2015). This would allow audiences to adjust their plans and actions to better adapt to, and cope with, the consequences of the hazard (Kox et al. 2018a; WMO 2015).
Traditionally, the thresholds for issuing warnings have been solely based on phenomena (e.g., wind speeds and snowfall depth), but IBFWs consider the timing and location of the impending hazards to identify the vulnerability of people and assets, and therefore what the impacts may be (WMO 2015). It is argued that warnings built on hazard-based thresholds may misrepresent the risks of the hazard(s) and lack messaging about risk, potentially leading to inaction (F. Sai et al. 2018, unpublished manuscript; https://doi.org/10.5194/nhess-2018-26).
b. Benefits and challenges of impact-based forecasts and warnings
The primary perceived benefit of IBFW systems is to increase the public’s understanding of the impending hazard and associated impacts (WMO 2015). IBFW systems research has focused on assessing public response and perceptions to warning messages containing impact information [defined by Kaltenberger et al. (2020) as “impact-oriented warnings”], rather than focusing on thresholds for triggering warnings based on potential impacts. Results of the studies are mixed in terms of their effectiveness in inciting appropriate behaviors; some found that including impact messaging promotes a behavioral response (or intention to respond) (e.g., Baker 1991; Casteel 2016, 2018; Morss et al. 2016, 2018; Weyrich et al. 2018), some found that impact messaging had no effect on behavioral response (Perreault et al. 2014; Potter et al. 2018b), and others had internally mixed results (Ripberger et al. 2015; Morss and Hayden 2010).
Little research has been conducted to assess the feasibility of implementing IBFW systems from an operational and institutional perspective. Forecasters and practitioners in Europe see the value added from impact information in building situation awareness and planning mitigation and response efforts (Kox et al. 2018a,b; Terti et al. 2019). Emergency managers, responders, forecasters, and other infrastructure managers in Germany indicated that receiving impact-focused warnings and forecasts would allow them to better prepare for oncoming severe weather (Kox et al. 2018a,b). Similarly, researchers and practitioners involved in weather communication in Finland and France who participated in a role-playing game found the additional impact information helped form a more “holistic” view of the situation that increased their confidence in decision-making (Terti et al. 2019). However, during the games the practitioners appeared to use the impact information less than traditional hydrometeorological data, which the authors speculated to be due to unfamiliarity with the new information, limited understanding of how to use it, lack of trust in the new developments, and incapability of new visualizations to convey useful information (Terti et al. 2019). This raises questions and challenges with obtaining and effectively using the appropriate information to support IBFWs.
IBFW systems require more than physical hydrometeorological information to inform decision-making. As these new warnings introduce the human element to the system, forecasters and warning services need information about historic and future potential impacts (e.g., injuries, psychosocial impacts, damage, disruption, economic impacts) and time- and space-variable vulnerability and exposure (Kox et al. 2018a; Obermeier and Anderson 2014; WMO 2015). Practitioners and researchers question how and where to obtain this information, and how to share it across disciplines, for effective implementation of IBFW systems (Hemingway and Robbins 2019; Kox et al. 2018a,b; Wei et al. 2018; Wilhelmi and Morss 2013).
More understanding is needed around the decision-making process behind issuing IBFWs and selecting the appropriate thresholds for various receivers (Harrison et al. 2014; Kox et al. 2018b; WMO 2015). This decision-making process requires linking specialized knowledge from various stakeholders, such as forecasters (Obermeier and Anderson 2014) and local emergency management practitioners and responders (Harrison et al. 2014; Kox et al. 2018b; WMO 2015). Furthermore, community involvement and local knowledge of past impacts can help determine impact thresholds for various groups (Kox et al. 2018b) (F. Sai et al. 2018, unpublished manuscript; https://doi.org/10.5194/nhess-2018-26).
IBFW systems have yet to be implemented in all member countries of the WMO. New Zealand (NZ) is one such country exploring perceived benefits and challenges and feasibility of implementing IBFWs for severe weather.
c. Severe weather warnings in New Zealand
National meteorological and hydrological services (often separate agencies) are responsible for forecasting and monitoring hydrometeorological hazards (Golnaraghi 2012; Rogers and Tsirkunov 2013). Simultaneously, civil defense and emergency management (CDEM) agencies are typically responsible for managing risks and emergency preparedness and response by conducting risk assessments, building mitigation, preparedness, and response plans (Golnaraghi 2012; Komendantova et al. 2014). The nature of the severe weather warning system varies from country to country.
The Meteorological Service of New Zealand (MetService) is responsible for issuing weather forecasts and warnings for land, sea, and air in NZ [Ministry of Civil Defence and Emergency Management (MCDEM) 2015; Williamson 1998]. The land-based severe weather warning thresholds are based on physical characteristics of the hazard (e.g., “Widespread rainfall greater than 50 mm within 6 hours or 100 mm within 24 hours”; MetService 2020). However, recent changes involve the consideration of the “expected severity and impact of the event” (MetService 2020) when issuing warnings, with labels of “orange” and “red” (for the most extreme events) added. There is a small amount of regional variation of these thresholds, based on research into regional variation of impact. In 2019, MetService introduced color-coded warnings, which consider the “expected severity and impact of the event” (MetService 2020).
The 16 regional councils of NZ hold most of the responsibility for flood monitoring, forecasting, and warning, with support from MetService, the National Institute of Water and Atmospheric Research, and CDEM groups (Ministry for the Environment 1991; Rouse 2011). The 16 CDEM groups in NZ are responsible for “disseminating national warnings to local communities and maintaining local warning systems” (section 25.4); they are the lead agencies for emergency management of hydrometeorological hazards locally and regionally (MCDEM 2015). New Zealand’s National Emergency Management Agency manages the response to those hydrometeorological events that approach national scale, in close collaboration with the local agencies and MetService. New Zealand does not have a national agency that coordinates hydrological event response at a smaller scale. Due to differences in population in the regions, and structures of the local government bodies, there are large variations in the resources available to support all aspects of the hydrometeorological value chain.
That many agencies are involved in different elements of the warning system for hydrometeorological hazards, both in NZ and in other countries, demonstrates one of the potential challenges of issuing informative, useful, and timely warnings. Conducting an assessment into the benefits and challenges of implementing an IBFW system will help inform countries doing this and focus efforts on addressing the challenges.
3. Methods and materials
The exploratory, open-ended research question guiding this research is, “What are the benefits and challenges of impact-based forecast and warning systems for hydrometeorological hazards from an institutional perspective?” We used qualitative methods to investigate underlying meanings placed on systems, processes, and events and to understand “why” a process occurs, rather than a snapshot of “what” or “how many,” as seen in quantitative studies (Blumer 1969; Miles and Huberman 1994; Patton 2002).
We first conducted interviews in Europe in December 2014 and January 2015 to understand organizational experiences with IBFW systems, and the participants’ perspectives of the system (Fig. 1). We then ran three workshops in NZ between April and August 2015 to understand what stakeholders thought the challenges and benefits of IBFW systems might be. Already familiar with the messages issued by MetService regarding broadscale and local-scale severe weather, workshop participants were shown two examples of IBFWs: one that had been prepared for a New Zealand commercial client, and the Vigilance Météorologique web page published by Météo-France (https://vigilance.meteofrance.fr/fr). Participants were also reminded of the improvements they had previously requested regarding MetService’s warning system over the three or so years preceding the workshop, namely, graphical depictions; structured, predictable text; consistent use of location names and areas; common alerting protocol compliance; a simplified set of warning messages; and availability in a georeferenced format. The aim of the workshops was to explore, with participants, the idea of impact-based warnings and whether it was viable in the New Zealand setting. Afterward, participants filled in feedback forms with questions aligning with workshop discussions, to give participants the opportunity to provide their opinions outside of the group setting.
Timeline and locations of interviews and workshops conducted in this research.
Citation: Weather, Climate, and Society 13, 2; 10.1175/WCAS-D-20-0110.1
a. Interviews
We conducted six interviews throughout December 2014 and January 2015 with nine forecast meteorologists, flood forecasters, governance officials, and communicators. Each interview included one to three participants and were conducted in the United Kingdom (four interviews), France (one), and Switzerland (one). These national meteorological services were chosen as they were known to use impact-based weather warnings. Most participants were purposefully sampled due to their roles (Miles and Huberman 1994), with others invited on the suggestion of their colleagues (reputational case selection). The interviews were between 26 and 92 min long and were audio recorded using a mobile telephone application. The open-ended interview questions enabled a natural flow of conversation onto topics that the participants thought would be valuable for the study. Key topics included
benefits and challenges of the IBFW systems,
effective interagency communication for rapid warnings,
the development of thresholds, and
whether the impacts considered are on individuals or larger groups of people.
The interviews were transcribed and thematically analyzed in NVivo 10 software. Thematic analysis is a qualitative method used in psychology research, involving identifying, analyzing, and reporting themes (patterns) in the data (Braun and Clarke 2006). An inductive analysis approach was followed, allowing for themes to be identified from the data that were different from the interview questions and analytic preconceptions. Our findings from the scoping interviews informed topics to investigate further in the New Zealand workshops.
b. Hydrology workshop
We held a workshop at Tasman District Council in Nelson, NZ, with the national flood warning group in April 2015 (duration of 1.75 h). There were 40 participants (39 local government hydrologists and 1 from the CDEM sector), representing every agency responsible for issuing flood warnings in NZ. A MetService representative gave a short presentation to introduce the concept of impact-based warnings, using a hydrological example. The participants broke into five groups to discuss guiding questions on the strengths and weaknesses of impact-based warnings. These questions included the following:
What do you think of this impact-based approach?
What are the strengths and weaknesses?
What might your flood warnings look like if you received impact-based weather warnings?
Where might the weather warnings be issued (e.g., which website)?
How could your flood/impact knowledge be integrated with the MetService hazard information?
c. Emergency manager workshop
A workshop was held in July 2015 at the Wellington Region Emergency Management Office (WREMO), NZ, to understand emergency management perspectives on the appropriateness, and the strengths and weaknesses of an IBFW system (duration of 2 h). There were nine participants, mainly emergency management advisors. Following a presentation on impact-based warnings by a MetService representative, the participants had a group discussion, guided by Table 1.
The layout of the table used in the WREMO workshop to record the perspectives of participants on the strengths and weaknesses of various types of weather warning systems.
A discussion on how local CDEM knowledge could be integrated with MetService’s knowledge was recorded by the researchers on a PowerPoint slide to enable visibility of notes for all participants. Feedback forms were filled in by seven of the nine participants after the workshop.
d. MetService workshop
A 1-h workshop was held at MetService in Wellington, NZ, in August 2015 with nine forecast meteorologists. Initial presentations by the researchers outlined the concept and described the preliminary findings of the previous workshops and interviews. A group discussion was facilitated to discuss those findings and identify further benefits or challenges of IBFWs. The guiding questions targeted two key challenges; whether a warning should be triggered in a rural area despite less overall impact (i.e., the element of exposure), and how best to include likelihood in the decision to warn. The researchers took notes for thematic analysis, and two feedback forms were later returned from the participants. Detailed comments on warning thresholds were also received from several participants.
A “low risk” ethics notification was lodged with the Massey University Human Ethics Committee prior to data collection. Participants received information sheets and signed consent forms. Their identities remain anonymous in this research to protect them from potential risks; we use identifying codes alongside quotes, where “Gov.” refers to participants in the role of governance, “Hyd.” refers to hydrologists and flood forecasters, “Met.” refers to meteorologists and weather forecasters, “Int.” refers to international interviews, and “NZ” refers to NZ workshops.
4. Results and discussion
We present the results and discussion together in this section. It is split into two major parts: perceived benefits, and perceived challenges of impact-based forecasts and warnings. Each contain further subsections with more detail.
a. Benefits of impact-based forecasts and warnings
Key themes that we identified from discussions of benefits of IBFW systems are a perception of improved public understanding of warnings, added awareness of antecedent conditions and cascading hazards, perceived reduction in “false alarms” and missed events, and increased interagency communication. These are discussed below.
1) Improved understanding by the public
As such, the participants indicated that impact information would provide added meaning and context that would help the public’s decision-making. As hydrologists from an NZ-based workshop said, “the more information and context” out there, the better, “particularly for flood warning purposes.” This entails describing the impacts of the hazard using common and understandable terminology with a timeline for expected impacts.Warnings themselves, we have noticed over and over again, it’s been proven in many catastrophic events that they don’t really make people do anything, because they don’t understand what they mean. So that’s why we need to move further than that now (Gov. Int. A).
The participants’ perceptions that the impact-based warnings would increase the public’s understanding of the weather and its impacts are generally supported by the literature. Previous studies have identified that including impact messaging helps people understand the impacts of the event (e.g., Potter et al. 2018b) and is related to prompting the desired behavior, for example, evacuating from a hurricane (Baker 1991; Morss and Hayden 2010).
2) Added awareness of antecedent conditions and cascading hazards
Thus, if issuing IBFWs, the forecasters and meteorologists might issue warnings based on the ground saturation from previous days of rainfall accompanied with flood hazard and impact information. This also allows for warnings about cascading hazards and impacts, such as flooding leading to traffic congestion, which obstructs emergency services responses, leading to citywide impacts (Gov. Int. A).If you have had a steady amount of rainfall for weeks and weeks and weeks, even if it’s just a few millimetres per day. At the end of the day, the ground is saturated, it cannot absorb water anymore, so there’ll be flooding . . . We don’t really create an urgent need for people to take action; when you give them a figure of five millimetres is going to be your rainfall today, but we haven’t told them that on the ground . . . because it is so saturated, you may eventually tip the scale and that’s going to cause flooding or the river’s going to burst its bank (Gov. Int. A).
The need to take into account antecedent conditions was also identified in the development of the French Heat Health Index (Pascal et al. 2006). Pascal et al. identified that the characteristics of the preceding summer, and the vulnerability of the population, should be considered when assigning warning thresholds. Furthermore, the WMO identifies simultaneous, cascading, and cumulative hazards and impact as key considerations in multihazard early warning systems (WMO 2018).
3) Reduction in “false alarms” and missed events
Participants suggested that this would result in fewer false alarms by warning only when the community will be affected (Met. Int. A, B, C). This would be based on understanding frequently occurring, “normal” hazards in an area (thus, not particularly impactful) versus what is abnormal and impactful (Met. Int. C). This deeper understanding of a community’s vulnerability, exposure, and history of impacts requires agencies and stakeholders across levels and scales to collect the appropriate data and share it more systematically.For us but more importantly for the user, so the public and our emergency responder community, [the impact-based system] works so much better, because it, we’re a small island, but still a seventy mile per hour wind in Shetland, that’s nothing particularly unusual really. Whereas if that happens in the summer months in London, that’s a very different scenario. So that’s just one small example of where the impacts-based system works a lot better (Met. Int. C).
Future research should test our participants’ perceptions of fewer false alarms and warning fatigue due to more tailored thresholds in an IBFW system. Previous research has shown that even if the rate of false alarms decreases slightly with IBFWs, there is little evidence that compliance improves (e.g., LeClerc and Joslyn 2015).
4) Increased interagency communication
Hydrologists and meteorologists in New Zealand agreed, describing “more collaboration between forecasters and users,” and the promotion of “better communication between CDEM groups” as strengths of IBFWs (Met., Hyd. NZ). IBFW implementation offers an opportunity to improve interagency communication and cooperation in NZ. This supports previous findings in the literature on the importance of interagency communication during the warning, response, and recovery phases of an event, particularly in the dissemination of advice (e.g., Becker et al. 2019; Pascal et al. 2006; Paton et al. 1998, 1999; Uccellini and Ten Hoeve 2019; Wein et al. 2016).I think an impact-based system means that you have to collaborate to get additional bits of information . . . to come up with something that is more informative rather than just the threshold-based thing, which may or may not have any impact at all (Met. Int. A).
b. Challenges with impact-based warnings
Our participants were supportive of implementing an IBFW system for severe weather. However, several challenges remain in designing and using the system.
1) Designing the system for individuals or society
This is an interesting perspective and highlights the significance of needing to further investigate solutions to this challenge. Another example given by a participant is that the public perceived a European meteorological service to have failed in warning for an event that resulted in a child unfortunately being killed by a falling tree at a camping ground (Met. Int. A). Despite issuing a lower-level warning (“amber alert”) for the area (which is usually an unpopulated field), the public felt unwarned and questioned why the warning was not classed as a higher impact (Met. Int. A). This raises questions around how meteorological services can effectively communicate the risk, which may still be high at an individual level, but has been classed as a lower level due to the large-scale thresholds used for stakeholder decision-making (Met. Int. A, B, D, Hyd. Int. A). This also highlights the challenge of how a meteorological service could forecast such localized impacts (Met. Int. A, D).So we’ve had some issues this week with people living in North West Scotland arguing quite strongly that an amber warning we had out should have been a red warning. We’ve had to point out quite strongly why it shouldn’t have been. So it’s an issue from a public perception and a media perspective. . . . I think this is one of the biggest challenges, ‘cause we’ve developed a service that we know works really, really well to responders, but for the, it hasn’t been designed with the public in mind, specifically. It’s maybe not been shaped for them and I don’t think that’s a massive problem (Met. Int. B).
While careful messaging is one solution suggested by the participants, another is to design the IBFW system so that it considers the potential impacts at an individual level. In the CDEM NZ workshop, we gave the following example and question:If we said “minor impacts” to a member of the public, that might [be perceived to] mean their garden being wet, but actually for us a “minor impact” will be a few houses [flooded] . . . So when we’re writing for the public, rather than just say “minor impacts,” we should say “there’ll be flooding for low lying fields, minor roads, isolated properties, or individual properties” (Hyd. Int. A).
A person’s roof is blown off their house in a city. The storm moves over a rural area, and a farmer’s roof is also blown off their house. Should those two areas receive the same level of warning? Even though in the country just one roof came off, but in the city, maybe a thousand roofs come off?
The answer was a resounding “yes,” even when probed with follow-up questions on impacts to wider society and the economy. This was also supported by most meteorologists and forecasters at the workshop with NZ’s MetService, with the added comment that it was due partly to “liability issues.” These participants perceived that the purpose of warnings is for people to take precautionary actions at an individual level to minimize the impacts. This aligns with the people-centered approach to designing early warning systems (e.g., UNISDR 2015). Therefore, perhaps further investigation is needed into how IBFW systems can be targeted at an individual level as much as possible.
User-defined thresholds could minimize the potential for overwarning by considering the vulnerability and capacities of individuals, assets, and infrastructure in the area. Vulnerability is a key driver of the impacts resulting from severe weather; understanding the nature of it is required to design effective mitigation strategies (e.g., Changnon et al. 2000; Hosterman et al. 2019; Wilhelmi and Morss 2013). Yet vulnerability is often overlooked when designing warnings—perhaps due to the challenges associated with quantifying vulnerability. Our research findings support the movement toward conducting integrated assessments of vulnerability to inform targeted mitigation and response efforts (Wilhelmi and Morss 2013).
Furthermore, geographical areas that are warned could be as contained as technically possible, to help reduce overwarning. The ability to do this will likely vary between hazards. NZ CDEM participants suggested that warnings should be issued at catchment (or district) level if possible, to prevent cities from receiving weather warnings targeted at nearby mountainous areas. They also stated that location names should be clear in warnings, especially when the region and city names are the same (as is the case for the wider “Wellington region,” which contains “Wellington city”). This requirement was communicated to MetService meteorologists responsible for issuing land-based severe weather warnings. Partly to increase location specificity, MetService has also implemented the issuing of warnings in the Common Alerting Protocol (CAP). Warnings expressed in CAP contain a precise geographic description of location.
This variation in triggering thresholds across different regions and/or countries is thought to confuse members of the public, particularly those who travel. Nonetheless, developing warning systems with local vulnerabilities and exposure in mind is important for developing an effective warning system (UNISDR 2015; WMO 2015), but challenging in practice. The benefits of a standardized early warning system include simplification of many variables by using common frameworks and language to increase understanding; clarity for emergency managers; reduced workload for monitoring, education, and outreach; and the interoperability of equipment and systems across hazards, agencies, and countries (Potter et al. 2018a). One could argue that IBFW systems could be standardized across borders or regions so that each level in the system has a common severity of impact—however, this may not be realistic, because of differences in local contexts (legal, cultural, etc.). Finding a way to harmonize the warning levels across boundaries would enable members of the public to travel between regions/countries with confidence in understanding the likely impacts associated with the warning levels. It would also have the benefit of allowing a comparison of events over time. Further investigation of this concept would be beneficial.If a French person, German person, [or] Italian person who lives in a mountainous area then comes to the UK and sees a red warning for snow or something, they might be thinking, “Oh my God, twenty foot of snow, we’re not going to be able to move for three weeks.” But actually our [UK] impacts are very different, and you might be out and about within two or three days. So that brings challenges as well. So comparing apples with pears and also different perceptions of what red means (Met. Int. B).
This participant indicated that the design of the system must be made from the start, but the process can be evolutionary. For example, the system may start with two agencies such as a CDEM group and hydrologists to provide impact-based flood warnings or launch a pilot project confined to a vulnerable area, to test it for one or two years. After testing, the system could be rolled out to the rest of the country (Gov. Int. A).That is why talking to your stakeholders, and that also includes your district or your community, mayors . . . the council leaders or whatever they are, they really need to say what is important for us, and how much rain or wind we can stand—after that we cannot operate anymore. And okay, there is [only] a handful of us here, but it is so important for us. So . . . that’s where the difficulty of this system comes in. Not difficulty, but a challenge, I would say (Gov. Int. A).
2) Verification of impact-based forecasts and warnings
Verification is a standard practice warning services use to measure and improve the quality of their warnings. Verifying hazard-based warnings uses observations and reports to measure the timeliness (e.g., lead time) and accuracy (e.g., hits, misses, false alarms) of the forecast or warning (Sharpe 2010; Wilson and Giles 2013). IBFW systems introduce the need to verify how well the impacts were forecasted or warned based on ground observations, actions taken, and severity of impacts (Met. Int. A; Robbins and Titley 2018). Participants identified “subjective verification” of IBFWs as an implementation challenge (Met. Int. A, B), with technical limitations preventing IBFWs from being truly objective (Met. Int. A).
These concerns echo those of Hemingway and Robbins (2019) who questioned how to verify forecasted impacts if and when impacts are reduced, as is the goal of the IBFWs. Forecasters must develop new verification methodologies to account for the observed impacts in addition to the observed hazard (e.g., Obermeier and Anderson 2014; Robbins and Titley 2018).The challenge you have with verification is that actually if we’ve got the warning right, we should be minimising or helping to reduce the impacts of the weather. So sometimes it’s difficult to tell whether you’ve got it right (Met. Int. B).
The need to consider the human response to weather warnings is becoming increasingly common globally. For example, the Forecasting a Continuum of Environmental Threats (FACETs) concept in the United States incorporates human response as a component of verification, along with meteorological-based criteria including false alarm duration and area, and lead times (Rothfusz et al. 2014).There’s a good example . . . from Scotland a couple of years ago. We were expecting very strong winds coming through during the day and the decision was taken to close schools so there were no kids out travelling around. And what made the media was that a school bus did overturn but there were no children on it, they were just literally moving the bus. So we’d reduced the impact . . . but it was verified as good advice because they’d made the decision not to let the school children travel (Met. Int. B).
3) Lack of data and knowledge
Impact-based forecasts and warnings require a multitude of datasets, primarily hazard, impacts, vulnerability, and exposure (WMO 2015). Participants, in agreement with previous research (e.g., Changnon et al. 2000; Kaltenberger et al. 2020), indicated that a challenge with implementing IBFW systems is the lack of impact data.
Meteorologists have limited knowledge about impacts, for which data collection tends to be the responsibility of the CDEM sector (Hyd., CDEM, Met. NZ). This makes it difficult to accurately forecast impacts. Nonetheless, our meteorological participants stated that their organizations do collect impact information for significant events, including from media and social media, “comments . . . press release, pictures, videos, whatever we can grab” (Met. Int. B, D). Social media, crowdsourcing, and other unconventional data sources are increasingly used data sources for understanding postevent impacts (Spruce et al. 2020; Kaltenberger et al. 2020). Using these voluntary sources of information can empower and include members of society who contribute the data (UNISDR 2015).
We found that in many cases a “mature” loss database did not yet exist to store impact data in (e.g., Met. Int. A). The participants indicated that their impact databases were “less organised . . . than a technical database with fields” (Met. Int. D). The Sendai Framework (UNISDR 2015) encourages participating nations to “systematically evaluate, record, share and publicly account for disaster losses and understand the . . . impacts” (p. 15) including exposure and vulnerability information. Establishing a national loss database is underway in NZ and aligns with the objective to “improve the information and intelligence system that supports decision-making in emergencies to enable informed, timely, and consistent decisions by stakeholders and the public” in the National Disaster Resilience Strategy (MCDEM 2019). The database is then referred to “when something similar is expected, and we can make reference to . . . events when dealing with the press, with the general public, with the media . . . and that’s quite good” (Met. Int. D). Our findings strongly support the need for collecting and storing impact information and related hydrometeorological information, to help inform future impact-based warnings (e.g., Gov. Int. A; Hemingway and Robbins 2019; Kaltenberger et al. 2020; Kox et al. 2018a,b; Wei et al. 2018). The sharing of data, including warnings, between databases may also assist with verification of warnings, and would require standardization of data and protocols. More research is needed to understand the various data sources of impact information, and how it can be collected, stored, and used for IBFWs. Additionally, it would be beneficial to focus research on the development of internationally applicable impact scales.
The rapid integration of meteorological information and impact thresholds was seen (mainly by NZ participants) as a potential challenge, as the interagency communication required would create delays in issuing warnings (Hyd., Met., CDEM NZ). Our international participants described how meteorological agencies could share office spaces with hydrological agencies, and have representatives in offices of key stakeholders, for quickly sharing information (e.g., Gov. Int. A, Met. Int. A). Other potential solutions were engaging with appropriate agencies to set up trigger points at which discussions occur, and using past event data (weather, impacts, and actions) to develop user-defined thresholds (CDEM NZ). A spatial web portal could support sharing impact information between CDEM groups, flood management agencies, and the meteorological service for the latter to develop “high level impact statements” (Hyd. NZ). Another suggestion was for CDEM groups and hydrologists to populate an impact threshold database to help integrate data and build a foundation of knowledge (Met. NZ). However, it was noted that even if an event were repeated in terms of meteorological conditions and location affected, human responses and therefore impacts would differ, causing a challenge in using impacts from previous events as a basis for decision-making (Met. Int. A).
4) Roles and responsibilities
Our participants highlighted the need to clarify the roles and responsibilities of all organizations that would be involved, which is supported by NZ’s National Disaster Resilience Strategy (MCDEM 2019). Organizations’ operational remits may prevent communication of risk and impact information and prescriptive actions (Hyd. Int. A, Met. Int. D) or may increase liability due to providing incorrect impact information (Met., Hyd., CDEM NZ).
Participants showed concern that IBFWs implementation might require a shift in existing roles (Gov. Int. A; Met. Int. A, B, D; Met. NZ; CDEM NZ). Many participants felt that meteorologists may not be best-suited to communicate impacts, risks, and advice due to their lack of expertise in modeling vulnerabilities and exposure; CDEM groups, hydrologists, and risk specialists hold this information and knowledge (Met., Hyd., CDEM NZ; Gov. Int. A, Met. Int. A, D). Furthermore, the agencies that possess impact, vulnerability, or exposure information may not have the operational remit to issue warnings. Maintaining the alignment of communications to designated roles and responsibilities is a valid concern, and has been discussed widely, such as following the 2009 L’Aquila earthquake (Cocco et al. 2015; Scolobig et al. 2014). Legislative restrictions have also been identified as a challenge for some NMHSs to implement IBFW systems by other researchers (e.g., Kaltenberger et al. 2020; Uccellini and Ten Hoeve 2019).
Hydrologists and CDEM participants in NZ indicated confusion over their role in an IBFW system. Hydrology workshop participants questioned why the NZ MetService would need to issue IBFWs since they see themselves as already doing this for floods. Furthermore, participants showed confusion over whether councils would be expected to supply the MetService with data before or after MetService issues a warning, or whether a premade template showing areas of higher risk would be used.
Some organizations also bring flood forecasters from the hydrological service into meteorological service offices to work alongside each other (Met. Int. D). However, some countries may face challenges when organizations do not want to share or work together, or are unable to due to resource limitations (Gov. Int. A; Met. Int. A; Hyd. Int. A).We have a challenge . . . because we’re not a [hydrometeorological] service. So the hydrology aspect of impact-based warnings for rain comes from our partners . . . for looking at flood risk. It’s been a very successful partnership, but internally that brings its challenges. Different cultures, different approaches. [They] have their own flood warning service, which alerts local people to the fact that their property could get flooded, which is threshold based, but the overarching guidance is impact-based. So there’s a bit of a mismatch there, which sometimes causes communication complexities (Met. Int. B).
While it is important that agencies retain their roles and responsibilities, they can work together to understand the likely impacts and produce impact-based warnings (Gov. Int. A, Met. Int. A) and decide on consistent messages. The notion of interagency cooperation to issue impact-based warnings in NZ was strongly supported (CDEM NZ).
5) Potential for conflicting messages
Participants indicated that consistency between agencies for warning messages is important to reduce confusion and maintain perceived trust and credibility (Met. Int. A, B; Met., Hyd. NZ), and that IBFWs may introduce potential for conflicting messages between agencies (Hyd. NZ), such as if one agency forecasts a likely impact, but another does not. Participants indicated that the shifting roles and responsibilities between agencies may confuse audiences as to who the point of contact is (Hyd. NZ). The need for consistency within and across warning messages has been identified in the literature as a key element of effective warnings (e.g., Mileti and Sorensen 1990). Care should be taken to ensure IBFW messages are consistent across and within agencies that have existing warning systems (Met. Int. A, B).
6) Increased burden on responding agencies
The vast majority of our NZ participants supported the use of IBFW; the few who indicated that they did not support an IBFW approach stated they would like to make the impact assessments themselves. If they could not do the assessments themselves, they indicated that it would become a circular process, as the flood managers would provide the impact information to the meteorological service, which would increase their workload, and then they would receive the warnings back, “so there wouldn’t be any surprises/changes” (Hyd. NZ). This highlights uncertainty around the usefulness of IBFWs for decision-makers who are providing the impact information in the first place. Perhaps this finding is an additional reason that the stakeholder participants in Terti et al.’s (2019) research used impact information less than traditional hydrometeorological information. Specialized users in Germany, such as utilities sectors, saw less use for generalized impact information than hazard information as they use their own models or past experiences to derive expected impacts (Kox et al. 2018b). However, fire brigades and other authorities did see that their responsive operational planning would benefit from information on expected impacts more so than just hazard information (Kox et al. 2018b).
Our NZ CDEM participants indicated that impact-based warning information would be useful for constructing their warning messages by providing extra information. Authorities in Germany also “presumed” the need to include impact information for public communications to help with individual threat assessments (Kox et al. 2018b, p. 5). Further research is recommended to investigate specific stakeholder uses of IBFWs compared to hazard-based warning systems, and their experiences during real events.
5. Conclusions, limitations, and implications
We investigated the challenges and benefits of impact-based forecast and warning systems for severe weather from an institutional or organization perspective, to help inform the development of such a system in NZ. We found that internationally, when determining thresholds, IBFW systems seem to be designed with responding stakeholders in mind, as opposed to individual members of the public. Yet NZ participants advocate for a system designed for the public, where warnings should be triggered even if only one person is present in an area, to give them an opportunity to prepare and respond. IBFWs could be improved by increasing our knowledge of vulnerability and coping capacities through further interagency coordination and interdisciplinary research. This will also assist with ensuring the delivery of consistent messages by multiple agencies. Harmonizing IBFWs across geographical regions (especially within each country) and over time based on common levels of impact would be beneficial in helping the public understand the severity of the potential impacts of an impending event. Vast improvements are needed for collecting and storing impact information as events occur, to inform forecasters and decision-makers about likely impacts in future events (e.g., Kaltenberger et al. 2020). In addition to recording mitigation actions taken, this impact information could also assist with verification of IBFWs.
Our research was an exploratory investigation; as such, it lent itself to a qualitative research method. The strengths of this type of method are the ability to investigate what the perceived challenges and benefits were by participants from a range of sectors and why. However, quantitative research is needed to understand the generalizability of our findings across the wider population. Because of budget limitations, we were unable to include participants from emergency responding agencies internationally. Thus, we could not triangulate the comments made by our meteorological and forecaster participants. We hope this is an area of more detailed research in future. Our NZ-based CDEM participants were restricted to one region; further investigation of the perspectives of lifelines agencies, primary industries, private companies, central government, and emergency managers from other regions would be beneficial.
Acknowledgments
This research was funded by NZ’s Ministry of Business, Innovation and Employment via the Strategic Science Investment Fund and Resilience to Nature’s Challenges. The authors thank the participants of the research, Chris Noble from NZ’s MetService, and those who helped to coordinate the interviews (especially Joanne Robbins) and the NZ workshops (especially Martin Doyle and Sarah Gauden-Ing). The authors declare no conflict of interest.
Data availability statement
Anonymized data are available by contacting the authors.
REFERENCES
Baker, E. J., 1991: Hurricane evacuation behavior. Int. J. Mass Emerg. Disasters, 9, 287–310.
Basher, R., 2006: Global early warning systems for natural hazards: Systematic and people-centred. Philos. Trans. Roy. Soc., 364, 2167–2182, https://doi.org/10.1098/rsta.2006.1819.
Becker, J. S., S. H. Potter, S. K. McBride, A. Wein, E. E. H. Doyle, and D. Paton, 2019: When the earth doesn’t stop shaking: How experiences over time influenced information needs, communication, and interpretation of aftershock information during the Canterbury Earthquake Sequence, New Zealand. Int. J. Disaster Risk Reduct., 34, 397–411, https://doi.org/10.1016/j.ijdrr.2018.12.009.
Blumer, H., 1969: Symbolic Interactionism: Perspective and Method. Prentice Hall, 208 pp.
Braun, V., and V. Clarke, 2006: Using thematic analysis in psychology. Qual. Res. Psychol., 3, 77–101, https://doi.org/10.1191/1478088706qp063oa.
Casteel, M., 2016: Communicating increased risk: An empirical investigation of the National Weather Service’s impact-based warnings. Wea. Climate Soc., 8, 219–232, https://doi.org/10.1175/WCAS-D-15-0044.1.
Casteel, M., 2018: An empirical assessment of impact based tornado warnings on shelter in place decisions. Int. J. Disaster Risk Reduct., 30, 25–33, https://doi.org/10.1016/j.ijdrr.2018.01.036.
Changnon, S. A., R. A. J. Pielke, D. Changnon, R. T. Sylves, and R. Pulwarty, 2000: Human factors explain the increased losses from weather and climate extremes. Bull. Amer. Meteor. Soc., 81, 437–442, https://doi.org/10.1175/1520-0477(2000)081<0437:HFETIL>2.3.CO;2.
Cocco, M., and Coauthors, 2015: The L’Aquila trial. Geoethics: The Role and Responsibility of Geoscientists, Geological Society Special Publ. 419, 43–56, https://doi.org/10.1144/SP419.13.
Cole, S. J., R. J. Moore, S. C. Wells, and P. S. Mattingley, 2016: Real-time forecasts of flood hazard and impact: Some UK experiences. Third European Conf. on Flood Risk Management, Centre des Congrès, France, EDP Science, 18015, https://doi.org/10.1051/E3SCONF/20160718015.
Golnaraghi, M., 2012: An overview: Building a global knowledge base of lessons learned from good practices in multi-hazard early warning systems. Institutional Partnerships in Multi-Hazard Early Warning Systems: A Compilation of Seven National Good Practices and Guiding Principles, M. Golnaraghi, Ed., Springer, 1–8.
Harrison, J., C. McCoy, K. Bunting-Howarth, H. Sorensen, K. Williams, and C. Ellis, 2014: Evaluation of the National Weather Service Impact-Based Warning Tool. NWS Central Region Headquarters, 37 pp., https://publications.aqua.wisc.edu/download/impactbasedwarningtool-pdf/.
Hemingway, R., and J. Robbins, 2019: Developing a hazard-impact model to support impact-based forecasts and warnings: The Vehicle OverTurning (VOT) Model. Meteor. Appl., 27, e1819, https://doi.org/10.1002/MET.1819.
Hosterman, H. R., J. K. Lazo, J. M. Sprague-Hilderbrand, and J. E. Adkins, 2019: Using the National Weather Service’s impact-based decision support services to prepare for extreme winter storms. J. Emerg. Manage., 17, 455–467, https://doi.org/10.5055/jem.2019.0439.
Kaltenberger, R., A. Schaffhauser, and M. Staudinger, 2020: “What the weather will do”—Results of a survey on impact-oriented and impact-based warnings in European NMHSs. Adv. Sci. Res., 17, 29–38, https://doi.org/10.5194/asr-17-29-2020.
Komendantova, N., R. Mrzyglocki, A. Mignan, B. Khazai, F. Wenzel, A. F. Patt, and K. Fleming, 2014: Multi-hazard and multi-risk decision-support tools as a part of participatory risk governance: Feedback from civil protection stakeholders. Int. J. Disaster Risk Reduct., 8, 50–67, https://doi.org/10.1016/j.ijdrr.2013.12.006.
Kox, T., C. Lüder, and L. Gerhold, 2018a: Anticipation and response: Emergency services in severe weather situations in Germany. Int. J. Disaster Risk Sci., 9, 116–128, https://doi.org/10.1007/s13753-018-0163-z.
Kox, T., H. Kempf, C. Lüder, R. Hagedorn, and L. Gerhold, 2018b: Towards user-orientated weather warnings. Int. J. Disaster Risk Reduct., 30, 74–80, https://doi.org/10.1016/j.ijdrr.2018.02.033.
LeClerc, J., and S. Joslyn, 2015: The cry wolf effect and weather-related decision making. Risk Anal., 35, 385–395, https://doi.org/10.1111/risa.12336.
MCDEM, 2015: The Guide to the National Civil Defence Emergency Management Plan 2015. Ministry of Civil Defence and Emergency Management Department of Internal Affairs Doc., 321 pp., https://www.civildefence.govt.nz/assets/guide-to-the-national-cdem-plan/Guide-to-the-National-CDEM-Plan-2015.pdf.
MCDEM, 2019: National Disaster Resilience Strategy = Rautaki ā-Motu Manawaroa Aituā. New Zealand Government Te Kawanatanga o Aotearoa Ministry of Civil Defence and Emergency Management,, 49 pp., https://www.civildefence.govt.nz/assets/Uploads/publications/National-Disaster-Resilience-Strategy/National-Disaster-Resilience-Strategy-10-April-2019.pdf.
MetService, 2020: Weather warning criteria. Meteorological Service of New Zealand, accessed 14 December 2020, https://www.metservice.com/warnings/home.
Miles, M. B., and A. M. Huberman, 1994: Qualitative Data Analysis: An Expanded Sourcebook. 2nd ed. Sage, 338 pp.
Mileti, D. S., and J. H. Sorensen, 1990: Communication of emergency public warnings—A social science perspective and state-of-the-art assessment. Oak Ridge National Laboratory Rep., 166 pp.
Ministry for the Environment, 1991: Resource Management Act 1991. New Zealand Government Doc., 836 pp., https://www.legislation.govt.nz/act/public/1991/0069/latest/096be8ed81a37f92.pdf.
Morss, R. E., and M. H. Hayden, 2010: Storm surge and “certain death”: Interviews with Texas coastal residents following Hurricane Ike. Wea. Climate Soc., 2, 174–189, https://doi.org/10.1175/2010WCAS1041.1.
Morss, R. E., J. L. Demuth, J. K. Lazo, K. Dickinson, H. Lazrus, and B. H. Morrow, 2016: Understanding public hurricane evacuation decisions and responses to forecast and warning messages. Wea. Forecasting, 31, 395–417, https://doi.org/10.1175/WAF-D-15-0066.1.
Morss, R. E., C. L. Cuite, J. L. Demuth, W. K. Hallman, and R. L. Shwom, 2018: Is storm surge scary? The influence of hazard, impact, and fear-based messages and individual differences on responses to hurricane risks in the U.S. Int. J. Disaster Risk Reduct., 30, 44–58, https://doi.org/10.1016/j.ijdrr.2018.01.023.
Obermeier, H., and M. R. Anderson, 2014: Verification and analysis of impact-based tornado warnings in the central region of the National Weather Service. Electron. J. Severe Storms Meteor., 10 (1), https://ejssm.org/ojs/index.php/ejssm/article/download/143/102.
Pascal, M., and Coauthors, 2006: France’s heat health watch warning system. Int. J. Biometeor., 50, 144–153, https://doi.org/10.1007/s00484-005-0003-x.
Paton, D., D. M. Johnston, and B. Houghton, 1998: Organisational response to a volcanic eruption. Disaster Prev. Manage., 7, 5–13, https://doi.org/10.1108/09653569810206226.
Paton, D., D. M. Johnston, B. Houghton, R. Flin, K. R. Ronan, and B. J. Scott, 1999: Managing natural hazard consequences: Information management and decision making. J. Amer. Soc. Prof. Emerg. Manage., 6, 37–48.
Patton, M. Q., 2002: Qualitative Research and Evaluation Methods. 3rd ed. Sage, 688 pp.
Perreault, M. F., J. B. Houston, and L. Wilkins, 2014: Does scary matter?: Testing the effectiveness of new National Weather Service tornado warning messages. Commun. Stud., 65, 484–499, https://doi.org/10.1080/10510974.2014.956942.
Potter, S. H., G. E. Jolly, V. E. Neall, D. M. Johnston, and B. J. Scott, 2014: Communicating the status of volcanic activity: Revising New Zealand’s volcanic alert level system. J. Appl. Volcanol., 3, 13, https://doi.org/10.1186/s13617-014-0013-7.
Potter, S. H., B. J. Scott, C. J. Fearnley, G. S. Leonard, and C. E. Gregg, 2018a: Challenges and benefits of standardising early warning systems: A case study of New Zealand’s Volcanic Alert Level System. Observing the Volcano World: Volcanic Crisis Communication, C. J. Fearnley et al., Eds., Springer, 601–620.
Potter, S. H., and Coauthors, 2018b: The influence of impact-based severe weather warnings on risk perceptions and intended protective actions. Int. J. Disaster Risk Reduct., 30, 34–43, https://doi.org/10.1016/j.ijdrr.2018.03.031.
Ripberger, J. T., C. L. Silva, H. C. Jenkins-Smith, and M. James, 2015: The influence of consequence-based messages on public responses to tornado warnings. Bull. Amer. Meteor. Soc., 96, 577–590, https://doi.org/10.1175/BAMS-D-13-00213.1.
Robbins, J. C., and H. A. Titley, 2018: Evaluating high-impact precipitation forecasts from the Met Office Global Hazard Map using a global impact database. Meteor. Appl., 25, 548–560, https://doi.org/10.1002/met.1720.
Rogers, D. P., and V. V. Tsirkunov, 2013: Weather and Climate Resilience: Effective Preparedness through National Meteorological and Hydrological Services. World Bank Publications, 163 pp., https://doi.org/10.1596/978-1-4648-0026-9.
Rothfusz, L. P., C. Karstens, and D. Hilderband, 2014: Next-generation severe weather forecasting and communication. Eos, Trans. Amer. Geophys. Union, 95, 325–326, https://doi.org/10.1002/2014EO360001.
Rouse, H., 2011: Flood risk management research in New Zealand: Where are we, and where are we going? GNS Science Rep. 2012/04, 77 pp., https://www.gns.cri.nz/static/pubs/2012/SR%202012-004.pdf.
Scolobig, A., R. Mechler, N. Komendantova, W. Liu, D. Schröter, and A. Patt, 2014: The co-production of scientific advice and decision making under uncertainty: Lessons from the 2009 L’Aquila earthquake, Italy. Planet@ Risk, 2, 71–76.
Scolobig, A., T. Prior, D. Schröter, J. Jörin, and A. Patt, 2015: Towards people-centred approaches for effective disaster risk management: Balancing rhetoric with reality. Int. J. Disaster Risk Reduct., 12, 202–212, https://doi.org/10.1016/j.ijdrr.2015.01.006.
Sharpe, M., 2010: Verification of weather warnings. Met Office Internal Rep., 9 pp.
Spruce, M., R. Arthur, and H. T. P. T. P. Williams, 2020: Using social media to measure impacts of named storm events in the United Kingdom and Ireland. Meteor. Appl., 27, e1887, https://doi.org/10.1002/met.1887.
Taylor, A. L., T. Kox, and D. Johnston, 2018: Communicating high impact weather: Improving warnings and decision making processes. Int. J. Disaster Risk Reduct., 30, 1–4, https://doi.org/10.1016/j.ijdrr.2018.04.002.
Terti, G., I. Ruin, M. Kalas, I. Láng, A. Cangròs, and V. Lorini, 2019: ANYCaRE: A role-playing game to investigate crisis decision-making and communication challenges in weather-related hazards. Nat. Hazards Earth Syst. Sci., 19, 507–533, https://doi.org/10.5194/nhess-19-507-2019.
Uccellini, L. W., and J. E. Ten Hoeve, 2019: Evolving the National Weather Service to build a weather-ready nation: Connecting observations, forecasts, and warnings to decision-makers through impact-based decision support services. Bull. Amer. Meteor. Soc., 100, 1923–1942, https://doi.org/10.1175/BAMS-D-18-0159.1.
UNISDR, 2005: Hyogo Framework for Action 2005-2015: Building the resilience of nations and communities to disasters. United Nations Office for Disaster Risk Reduction Doc., 25 pp., https://www.unisdr.org/2005/wcdr/intergover/official-doc/L-docs/Hyogo-framework-for-action-english.pdf.
UNISDR, 2015: Sendai Framework for Disaster Risk Reduction 2015-2030. United Nations Office for Disaster Risk Reduction Publ., 32 pp., https://www.undrr.org/publication/sendai-framework-disaster-risk-reduction-2015-2030.
Wei, L., J. Li, and X. Yang, 2018: Experiments on impact-based forecasting and risk-based warning of typhoon in China. Trop. Cyclone Res. Rev., 7, 31–36, https://doi.org/10.6057/2018TCRR01.04.
Wein, A., S. Potter, S. Johal, E. E. Doyle, and J. Becker, 2016: Communicating with the public during an earthquake sequence: Improving communication of geoscience by coordinating roles. Seismol. Res. Lett., 87, 112–118, https://doi.org/10.1785/0220150113.
Weyrich, P., A. Scolobig, D. N. Bresch, and A. Patt, 2018: Effects of impact-based warnings and behavioral recommendations for extreme weather events. Wea. Climate Soc., 10, 781–796, https://doi.org/10.1175/WCAS-D-18-0038.1.
Wilhelmi, O. V., and R. E. Morss, 2013: Integrated analysis of societal vulnerability in an extreme precipitation event: A Fort Collins case study. Environ. Sci. Policy, 26, 49–62, https://doi.org/10.1016/j.envsci.2012.07.005.
Williamson, M., 1998: Meteorological service contract. New Zealand Government Ministry of Transport Doc., https://www.beehive.govt.nz/feature/meteorological-service-contract.
Wilson, L. J., and A. Giles, 2013: A new index for the verification of accuracy and timeliness of weather warnings. Meteor. Appl., 20, 206–216, https://doi.org/10.1002/met.1404.
WMO, 2015: WMO guidelines on multi-hazard impact-based forecast and warning services. WMO Doc. 1150, 34 pp., https://library.wmo.int/doc_num.php?explnum_id=7901.
WMO, 2018: Multi-hazard early warning systems: A checklist. World Meteorological Organization Doc., 20 pp., https://library.wmo.int/doc_num.php?explnum_id=4463.
Zhang, Q., and Coauthors, 2019: Increasing the value of weather-related warnings. Sci. Bull., 64, 647–649, https://doi.org/10.1016/j.scib.2019.04.003.
