1. Motivation
As National Weather Service (NWS) forecasts continue to improve, the agency is increasingly using impact-based decision support services (IDSS) to connect those forecasts to actionable information that assists their audiences in protecting life and property (Uccellini and Ten Hoeve 2019). Emergency managers (EMs) are vital recipients of IDSS as decision-makers that trigger community action. Effective communication to members of the EM community requires both accurate and timely information about possible hazards (in the form of forecasts and warnings) and knowledge about the EMs who receive and use this information when making decisions. NWS forecasters have access to a wide variety of data and tools that facilitate the delivery of accurate and timely information but have access to few data and almost no tools that facilitate knowledge about the diverse populations of EMs that receive, process, and use the information that they provide. As a result, it is difficult to answer basic questions about EM communities: 1) What types of information do different members of the EM community require? 2) When do EMs require different types of information? 3) How do different EMs interpret information from NWS forecasters? 4) What, if anything, do different EMs do with the information they receive from the NWS—does it affect the actions they take, and, if so, how? Absent reliable answers to these questions, it is challenging to develop IDSS materials and strategies that fit the diverse needs of the EM community.
In 2014, Weaver et al. (2014) surveyed EMs across the country and collected valuable demographic data about the EM community. However, there have been no consistent nationwide panel studies of this group. This lack of systematic data makes it difficult to effectively evaluate IDSS programs and provide feedback to NWS forecasters about the information and services they provide to the EM community. The Extreme Weather and Emergency Management Survey (the WxEM survey) was created to address these challenges by routinely surveying a panel of nationwide EMs on various weather topics. This article briefly introduces the project and the resulting data. It describes the research methodology and provides a short case study to illustrate the value of the data collection effort.
2. Data and methods
a. Building an EM panel
The first step in the project was to create a sampling frame, a comprehensive database of publicly identifiable EMs across the country (the target population). This was challenging for a variety of reasons. First and foremost, definitions of “EM” vary from place to place. For this project, the NWS definition of an EM was used to determine the target population. This definition is from NWS Policy Directive 10–24 and is as follows: “Public safety officials who serve as employees or contract agents of a government agency at the federal, state, local, tribal, or territorial level” (NWS 2019). While not reflected in the definition, the research team decided to prioritize local EMs over state and federal EMs, although both are included in the panel (see below for more detail). The justification for this decision was that local EMs are more diverse and less connected to the NWS, and therefore less is known about their needs than the needs of state and national EMs. The next challenge was that there is not a current or standardized list of local EMs across the country. Some states provide relatively current lists with updated contact information; others do not provide these lists, or their lists are outdated. In addition to missing and outdated information, the structure of local emergency management varies from state to state. In many states, county EMs serve a primary role (e.g., Florida and California); in others, EM functions are organized by cities/towns (e.g., Massachusetts and Delaware) or districts/regions (e.g., Texas).
Because of these challenges, the research team had to build the database from scratch. This process began with the identification of each state’s EM structure. From there, the research team visited Federal Emergency Management Agency (FEMA), state emergency management associations (EMA), and jurisdiction websites to identify contact information for as many jurisdictions as possible. This process resulted in a database with contact information for over 4000 EMs across the United States. This database was used to populate an interactive web platform called EMdb that the research team uses to manage contact information for EMs across the country. EMdb stores information on each EM, such as: their name, title, email address, phone number(s), jurisdiction name, jurisdiction type, and website. In addition to indexing this information, it provides the ability to verify and update each EM’s information if/when it changes. The research team intends to check and update EMdb on a rolling basis, such that each EM’s information is verified at least once a year. This is a critically important step given the frequent turnover and shifting responsibilities and roles of emergency managers (Bureau of Labor Statistics 2023).
To ensure the protection of survey respondents, the WxEM survey protocol was approved by the Institutional Review Board of the University of Oklahoma. We obtain and monitor demographic information for members of our panel, but we do not publish any personally identifiable information. To access the panel, researchers must enter a unique username and password, ensuring the security of our panel members. In addition, survey respondents are assigned a personal identifier that grants participant anonymity while allowing us to study longitudinal data.
Once the sampling frame was created, the research team sent an introductory email to every local EM in the database. The email explained the project and invited the EMs to participate by filling out a short enrollment survey [see Wanless et al. (2023a) for more information on this survey]. Approximately 570 local EMs from the database enrolled in the project.
In addition to the process described above, the research team asked state EM associations and the International Association of Emergency Managers to share the enrollment survey with their members. Approximately 120 EMs enrolled in the survey using this mechanism of recruitment; many of them were state, federal, or private EMs that were not included in the initial database of local EMs. Last, to increase diversity and better represent the types of EMs with which NWS forecasters often interact, the research team identified and invited EMs from National Collegiate Athletic Association Division I universities and hospitals that are classified at Level-I Trauma Centers by the American Trauma Society. Approximately 30 EMs from these lists completed the enrollment survey.
b. Enrollment survey Wave 1
The enrollment survey (“Wave 1”) was distributed to the EM panel in August of 2022. It was designed to recruit and enroll participants in the WxEM project. The survey gathered demographic information on enrollees, including location, jurisdiction type, and experience. It also asked participants how much time they were willing to spend on surveys and how often they would be willing to complete the surveys.
Figure 1 shows the geographic distribution of the 720 EMs who completed the enrollment survey by zip code centroid. Note that some states, especially states in the eastern half of the United States, were more heavily represented in the enrollment survey. This was driven by a variety of factors, such as the way their emergency management jurisdictions are structured (e.g., Massachusetts has an EM for every township) or because an enthusiastic and well-connected state association encouraged their members to complete the enrollment survey (e.g., Virginia). As the project moves forward, the research team is working to recruit more EMs, particularly in the western United States. To facilitate this, the enrollment survey will remain open for the duration of the project.1 In the current sample, participants are responsible for a variety of jurisdictions, including, county, state, city/town, and other types (like university and hospital EMs). Most (60%) of the current participants represent county-level jurisdictions, followed by city/town jurisdictions (18%). This reflects the prioritization of local EMs in the sampling frame.
The location and jurisdiction type of EMs who agreed to participate in the WxEM project.
Citation: Weather, Climate, and Society 15, 4; 10.1175/WCAS-D-23-0085.1
In addition to questions about location and jurisdiction, the recruitment survey included questions to measure participant experience, office (unit) characteristics, and familiarity with extreme weather. Most of the participants said that they have been an EM for more than 5 years; 48% reported 10–20 years of experience in the field. About 80% of the participants work as full-time EMs; the remaining are part-time (15%) or volunteer (5%). Most of the participants work in relatively small offices; ∼60% reported three or fewer employees. Furthermore, most participants indicated high levels of familiarity with extreme weather; nearly 70% said that most (50%+) of the incidents their office respondents to relate to hazardous weather, and close to 85% said they have been through one or more weather trainings/courses (e.g., a college course or NWS Storm Spotter Training).
The enrollment survey also asked participants how much time and how often they would be willing to complete surveys. Most respondents said that they would be willing to spend 10–15 min filling out a survey 3–4 times per year. From those results, the goal is to send three–four 10–15 min surveys each year, although that frequency may vary depending on research and EM needs. Along these lines, the enrollment survey also asked participants if they would be willing to participate in other research projects, such as experiments and one-on-one or small-group discussions about recent emergencies in their jurisdictions; nearly all of them said yes. These responses provide an important avenue for qualitative work to complement the largely quantitative survey.
c. Wave 2
The 720 EMs who completed the enrollment survey were invited to complete the second survey in the series (“Wave 2”) in November of 2022; 444 of them completed it (Wanless et al. 2023b). Wave 2 focused on how EMs use severe weather forecast information. For example, the survey asked EMs what forecast attributes (chance, location, severity, timing, impacts, or protective actions) are most important and how those priorities change over time. In addition, Wave 2 measured EM perceptions about a variety of NWS severe weather products and tested different kinds of messaging and visualizations of the Storm Prediction Center’s Convective Outlook. The research team is using that information to directly inform future iterations of these products.
d. Wave 3
The EMs were invited to complete the third survey in the series (“Wave 3”) in July of 2023; 322 of them completed it (Wanless et al. 2023c). Wave 3 was codesigned with researchers from Texas Tech University. The questions focused on how EMs deal with hazardous weather events that occur at the same time and in the same place within their jurisdiction(s), such as compound tornado and flash flood (TORFF) events and compound wind and water events from tropical cyclones like Hurricane Ida [for more information on compound hazards, see Henderson et al. (2020)]. The survey also asked questions about vulnerable populations and the impact of health hazards like COVID-19 on emergency management. The collaboration on the survey demonstrates one of the primary goals of this project: to create and maintain an infrastructure that reduces research costs and the burden placed on EMs.
e. Future survey design and EMdb maintenance
The WxEM project is intended to be a living, breathing project that can be adapted for questions covering various weather hazards. As Wave 3 demonstrates, the project has built-in infrastructure to foster collaboration across institutions and disciplines. The recruitment indicated that participants prefer shorter surveys fielded more often, so researchers plan to continue fielding short surveys up to 4 times per year. To keep new survey questions rich and diverse, the research team will continue to invite outside research entities to collaborate in the design of survey instruments, answering research questions from a variety of sources and covering a variety of weather events. In addition to enriching research, these collaborations will reduce the number of participation requests that EMs receive, reducing potential participant fatigue.
It is also of upmost importance to maintain the database of EMs. So far, there has been a decline in the number of respondents in each survey. To remedy this and prevent further decline, the researchers will update the database at least once a year, making sure contact information for each participant is up to date. The researchers will also recruit additional participants at conferences and through other professional networks such as LinkedIn.
f. Data availability
All data and metadata from each wave of the survey will be publicly available in Harvard’s Dataverse (https://dataverse.harvard.edu/dataverse/emsurvey). In addition, the research team will disseminate results in NWS regional science talks, meetings with NWS offices, paper publications, and conference presentations.
3. Case study
This section will present the results of a set of survey questions from Wave 2 to serve as an example of the information that this project can provide. The motivation for this experiment was to better understand the types of information (also called information attributes) that are important to EMs as they face an upcoming hazardous weather event and whether those attributes change over time. This experiment and a similar analysis were also conducted for members of the public via the Institute for Public Policy Research and Analysis’s Extreme Weather and Society Surveys in 2021 and 2022 (Krocak et al. 2023). See the 2021 Severe Weather and Society Survey (WX21; Ripberger et al. 2021), the 2021 Tropical Cyclone and Society Survey (TC21; Krocak et al. 2022), and the 2022 Winter Weather and Society Survey (WW22; Bitterman et al. 2022) for more information about those surveys.
In part 1 of this experiment, the EM participants were asked to rank six severe weather information attributes from most important to least important. They ranked the attributes by clicking and dragging them in order from top (most important) to bottom (least important). They ranked location (what area the storm will affect), timing (when the storm will happen), chance (the likelihood that the storm will occur), severity (how intense the storm will be), impacts (how the storm will affect the area), and protective action (how people can stay safe during the storm). The mean ranking was taken for each attribute, with the most important attributes assigned the number 1, and the least important attributes assigned the number 6. So, attributes with a lower mean ranking were overall rated as more important than those with higher mean rankings (Fig. 2). On average, location was the most important information type, with a mean ranking of 2.01 (Fig. 2). This is unsurprising; more than anything, EMs want to know if their jurisdiction is at risk. Below location, the participants ranked severity (2.64), timing (2.91), and chance (3.36) (Fig. 2). Once EMs know their area is at risk, these rankings suggest that they want to know how bad the event will be, when it will happen, and the likelihood of it happening. The EM participants ranked impacts (4.45) and, especially, protective actions (5.62) as less important than the other information attributes (Fig. 2). As a group, these results are very similar to the rankings by members of the public (Krocak et al. 2023).
Mean rankings of importance for different information attributes. A ranking of 1 indicates the most important piece of information, whereas a ranking of 6 indicates the least important piece of information. Error bars represent the 95% confidence interval. Note that these are overall rankings and are not specific to any time frame.
Citation: Weather, Climate, and Society 15, 4; 10.1175/WCAS-D-23-0085.1
Part 2 of the experiment asked the EM participants to indicate which information type is most important at five different points in the event timeline: at 3 days, 1 day, 4 h, 60 min, and 15 min before a severe weather event. Instead of a ranking question, these questions were multiple choice, forcing the participants to pick the single most important attribute at each point in time. Findings are shown in Fig. 3. Three days before the event, 35% of the EMs said that chance was the most important information attribute, closely followed by location (33%). These findings suggest that at this point in the timeline, EMs are looking for two types of information from forecasters: 1) is the event going to happen and, 2) if so, is it going to affect their jurisdiction? At day 1 (the day before the event), significantly fewer participants selected the chance attribute; instead, they were evenly split in emphasizing the importance of timing (27%), location (26%), and severity (24%). At this point in the timeline, EMs seem to prioritize information that will assist in preparatory action: 1) when is the event going to happen, 2) what areas are going to be most affected, and 3) how bad will it be?
The percentage of survey respondents that chose each information attribute as the “most important” at different points in time for severe weather events.
Citation: Weather, Climate, and Society 15, 4; 10.1175/WCAS-D-23-0085.1
As we move from the outlook to the watch phase (4 h before the event), severity (36%) was clearly seen as the most important information attribute by the participants; timing (23%) and location (20%) began to decline in importance. The apparent importance of severity continued into the next point on the timeline (60 min before the event), where 37% of the participants selected that attribute in response to the survey question. On the warning time scale (15 min before the event), the participants continued to emphasize the importance of information about severity (31%), impacts (18%), and protective actions (27%). At this point in the timeline, it appears that EMs are beginning to think about how to prioritize resources for emergency response and recovery: 1) how much damage will there be, 2) what type of infrastructure will incur the damage, and 3) is there anything they can tell people to minimize the damage?
This very short case study demonstrates the value of this type of project. It gives EMs an opportunity to provide direct feedback to forecasters and researchers on the kinds of information they are looking for when responding to extreme weather events. This feedback is invaluable as forecasters strive to improve risk communication and IDSS by ensuring that the information they provide is relevant, understandable, and actionable to core partners.
4. Conclusions and call for collaboration
The WxEM project is the first of its kind, providing a nationwide longitudinal study of how EMs use weather forecast information. This is a living, constantly evolving project, with EMdb being consistently updated as new EMs enroll in the projects and with short, focused surveys that, together, will cover a broad variety of subjects.
This variety of subjects will come through collaboration with other groups, with the goal of not only helping other groups to address their research needs but also lessening the research burden on EMs. The authors invite fellow researchers and practitioners to reach out and explore opportunities for collaboration. This can include discussions of existing data as well as the creation of new survey items or topics of interest. If interested, please contact corresponding author Wanless (awanless@ou.edu) to initiate a conversation and explore collaboration options.
Acknowledgments.
The WxEM Survey was established with funding from the NOAA Weather Program Office under Grant NA21OAR4590213.
Data availability statement.
Survey instruments, reference reports, and data can be found online (https://dataverse.harvard.edu/dataverse/emsurvey).
If the reader is an emergency manager and is interested in participating in the project, please reach out to the corresponding author Anna Wanless at awanless@ou.edu.
REFERENCES
Bitterman, A., and Coauthors, 2022: WW22. Harvard Dataverse, accessed 27 July 2023, https://doi.org/10.7910/DVN/OXT3K8.
Bureau of Labor Statistics, 2023: Occupational outlook handbook, emergency management directors. U.S. Bureau of Labor Statistics, accessed 14 February 2023, https://www.bls.gov/ooh/management/emergency-management-directors.htm.
Henderson, J., E. R. Nielsen, G. R. Herman, and R. S. Schumacher, 2020: A hazard multiple: Overlapping tornado and flash flood warnings in a National Weather Service forecast office in the southeastern United States. Wea. Forecasting, 35, 1459–1481, https://doi.org/10.1175/WAF-D-19-0216.1.
Krocak, M., J. Ripberger, C. Silva, and H. Jenkins-Smith, 2022: TC21. Harvard Dataverse, accessed 27 July 2023, https://doi.org/10.7910/DVN/Q2S7OJ.
Krocak, M., J. Ripberger, S. Ernst, C. Silva, H. Jenkins-Smith, and A. Bitterman, 2023: Public information priorities across weather hazards and time scales. Bull. Amer. Meteor. Soc., 104, E768–E780, https://doi.org/10.1175/BAMS-D-22-0190.1.
NWS, 2019: National Weather Service Policy Directive 10-24: Operations and services: Impact-based decision support services. NOAA Tech. Rep. NWSPD 10-24, 19 pp., https://www.nws.noaa.gov/directives/sym/pd01024curr.pdf.
Ripberger, J., M. Krocak, C. Silva, and H. Jenkins-Smith, 2021: WX21. Harvard Dataverse, accessed 27 July 2023, https://doi.org/10.7910/DVN/QYZLSO.
Uccellini, L. W., and J. E. Ten Hoeve, 2019: Evolving the National Weather Service to build a weather-ready nation: Connecting observations, forecasts, and warnings to decision-makers through impact-based decision support services. Bull. Amer. Meteor. Soc., 100, 1923–1942, https://doi.org/10.1175/BAMS-D-18-0159.1.
Wanless, A., S. Stormer, A. Fox, J. Ripberger, M. Krocak, A. Bitterman, C. Silva, and H. Jenkins-Smith, 2023a: WxEM Wave 1. Harvard Dataverse, accessed 5 July 2023, https://doi.org/10.7910/DVN/8VUUDA.
Wanless, A., S. Stormer, A. Fox, J. Ripberger, M. Krocak, A. Bitterman, C. Silva, and H. Jenkins-Smith, 2023b: WxEM Wave 2. Harvard Dataverse, accessed 5 July 2023, https://doi.org/10.7910/DVN/RUUQRP.
Wanless, A., S. Stormer, A. Fox, J. Ripberger, M. Krocak, A. Bitterman, C. Silva, and H. Jenkins-Smith, 2023c: WxEM Wave 3. Harvard Dataverse, accessed 5 July 2023, https://doi.org/10.7910/DVN/QQNUPX.
Weaver, J., L. C. Harkabus, J. Braun, S. Miller, R. Cox, J. Griffith, and R. J. Mazur, 2014: An overview of a demographic study of United States emergency managers. Bull. Amer. Meteor. Soc., 95, 199–203, https://doi.org/10.1175/BAMS-D-12-00183.1.
