1. Introduction
The months of April and May 2011 produced a series of severe storm events that spawned many destructive tornadoes. Over 200 tornadoes occurred during 25–28 April 2011, impacting five southeastern states and resulting in 316 deaths (NOAA 2011b). About 1 month later, a single EF5 tornado hit Joplin, Missouri, resulting in 58 casualties (NOAA 2011a). Beyond the significant and tragic loss of life, what made these two events especially noteworthy is that both were well-warned events, with significant lead times and critical and timely information distributed (Levitan et al. 2013).
As is standard procedure, extensive NWS service assessments were conducted after the two outbreaks. Notably, both assessments identified problems with some of the generic warning terminology used in the standard NWS products (NOAA 2011b). The Joplin service assessment also explicitly suggested that a more credible warning would be one that was “impact-based more than phenomenon-based for clarity on risk assessment” and would be “easily understood and calibrated by the public to facilitate decision making” (NOAA 2011a, p. 11).
The conclusions of the two 2011 service assessments, particularly the Joplin service assessment, along with the recommendations provided by the National Institute of Standards and Technology (NIST) final report of the Joplin tornado (Levitan et al. 2013) provided the major impetus for an experimental warning product that was first piloted by the NWS Central Region in 2012. The NIST report, in particular, highlighted the need for a warning process that significantly enhances public perception of personal risk and promotes a rapid and effective public response. Given this “call to action,” a new, enhanced warning product known as an impact-based warning (IBW) was designed with the specific intent to improve risk communication by using expanded and more specific wording concerning the hazard, source, and impact of the forecast storm that clearly identified potential threats (Hudson et al. 2013). Threat tags at the bottom of each warning were also included that succinctly identified the reason for the warning (e.g., ground spotter or radar indicated) and potential damage. Both of these enhancements were designed to make the warning information easier to use. The initial IBW pilot project began in five Weather Forecast Offices (WFOs) in the NWS Central Region and has since expanded to 46 offices in spring 2014. The goal is to have all WFOs using IBWs by 2016 (Hudson et al. 2015).
As a new weather product from the NWS, the evaluation of the implementation of IBWs and their success has been from a qualitative perspective, focusing primarily on end users’ satisfaction. This research has generally found that users believe IBWs to be an improvement over the non-IBW warnings, with some caveats to be discussed in the next section (Losego et al. 2013; Harrison et al. 2014). To date, however, no experimental work examining the effectiveness of actual IBWs exists, and this void provided the impetus for the research reported here. A nice complement to the existing qualitative research would be provided by more experimentally driven, quantitative research. The results of such experimental research would also either provide important validation and support for the continued use of IBWs or provide motivation to potentially reevaluate and reconsider their use.
2. Literature review
a. Previous impact-based research
Evaluations of the IBW experimental product have been ongoing since its original inception in 2012. The scope of the evaluation research has assessed two issues: the physical science supporting the generation of threat tags and the social science addressing end users’ satisfaction with the product. Considering the physical science research, a central question has been whether meteorological data support the use of tornado damage threat tags. The inclusion of a damage threat tag in an IBW was specifically designed to provide the forecaster’s best estimate of the magnitude of severe damage. By design, most tornado warnings would not include a damage tag. The damage tag “TORNADO DAMAGE THREAT…CONSIDERABLE” would be used rarely and only when a forecaster had high confidence based on credible radar and ground spotter evidence that a tornado of a stronger variety was imminent or ongoing and the duration of the tornado was expected to be longer lived. Use of the threat tag “TORNADO DAMAGE THREAT…CATASTROPHIC” would be exceedingly rare and only when the forecaster had absolute certainty, based on direct observational evidence, that a tornado will strike (or is striking) a densely populated area (Hudson et al. 2015).
The main impetus of the physical science research has therefore been to see if there are particular environmental and radar attributes that are associated with greater tornado intensity that could then be used by forecasters to signal potentially greater risk. Recent work by Smith et al. (2015) suggests that this is indeed possible. Based on their analysis of over 4700 tornadoes from 2009 to 2013, Smith et al. found that conditional probabilities for EF2+ tornadoes are 55%–60% for storms with a peak rotational velocity of 60–69.9 knots (kt; 1 kt = 0.51 m s−1) and are 65%–70% for storms with a peak rotational velocity of 80–89.9 kt. Smith et al. (2014) suggested that forecasters could use the “considerable” threat tag for storms with a peak rotational velocity of ≥60 kt. Research by Entremont and Lamb (2015), using updated radar data, also found that strong or significant tornadoes (which could again prompt the use of the considerable threat tag) became increasingly likely when tornadic debris is lofted to 10 kft (1 kft = 304.8 m) or greater. Other recent research has shown credible relationships between radar signatures and tornado intensity (Kingfield et al. 2012; LaDue et al. 2012; Toth et al. 2013). Collectively, these recent findings are encouraging and suggest that the physical science can indeed assist NWS forecasters in more accurately predicting and warning tornado intensity.
Hudson and Perry (2014) have also shown that when a NWS forecaster does release a damage threat tag, it is correlated with a higher intensity tornado. Hudson and Perry examined the hit rate for tornadoes that were warned over a 2-yr span from the WFOs participating in the IBW project. As a useful baseline, Hudson and Perry report a 28% tornado warning hit rate since 2008. Of the 823 tornado warnings that were issued, 57 were released with damage threat tags. Of those 57 warnings, 37 resulted in an EF1 or greater tornado for a hit rate of 65%. Additionally, 74% of the warnings with damage threat tags were verified with a tornado, indicating that using a damage threat tag can be used as an indicator of forecaster confidence that a tornado will in fact occur.
The main focus of the research to be presented here, however, is social science based, and, as previously mentioned, the research conducted to date on end users’ satisfaction with IBWs is encouraging. After the 2012 storm season, Losego et al. (2013) conducted an early evaluation of the IBW pilot project by conducting interviews and focus groups with NWS forecasters (WFs), emergency managers (EMs), and broadcast meteorologists (BMs). Their overall finding was that the EMs (and to a lesser extent, WFs) thought the IBWs were helpful, especially for members of the public. A follow-up evaluation of the IBW project was conducted by Harrison et al. (2014). They also distributed surveys and conducted interviews and focus groups among WFs, EMs, and BMs. Their results showed that all three groups thought that the IBW threat tags were 1) helpful in suggesting appropriate action and 2) added new, helpful information above and beyond the older tornado warning messages. Interestingly, when asked about limitations of IBWs, the WFs were more critical than were the EMs and BMs. WFs were more likely to mention inconsistency in the decision-making framework used to issue threat tags as a concern. WFs were also more concerned about the accuracy of the tags and whether or not the tags conveyed helpful information about potential impacts.
One recent experimental study is also informative concerning the potential beneficial effects of enhanced tornado warnings. Ripberger et al. (2015) examined the influence of consequence-based tornado messages on the likelihood of taking protective action. Ripberger et al. presented a survey to two large samples in two consecutive years, with each sample numbering over 3950 respondents. The experimental manipulation involved randomly presenting different tornado impacts to participants, with the damage roughly corresponding to damage associated with EF0 through EF5 tornadoes. Ripberger et al. found that as impact severity increased, so did the likelihood of engaging in some sort of protective action, at least up to an impact severity of “significant.” Once the impact was rated as severe or higher, however, sheltering in place decisions actually decreased, while decisions to leave one’s residence continued to increase. Although Ripberger et al.’s results suggest that there may be a level at which the increasing severity of impacts produces diminishing returns, their results do show that differing levels of consequences can cause different behavioral intentions to shelter in place.
b. Important risk communication variables
At this juncture, it is important to note that the IBW experimental product was implemented based on the perceived need to do something immediately that would improve severe weather risk communication (Hudson et al. 2013). The traditional route of social scientists would have been to review the risk communication literature, identify potential improvements or enhancements, and then methodically test those alternatives to assess those that had the most promise. Of course, this is a more time-consuming approach, and immediate improvements were deemed necessary. It is important to note that some social scientists were instrumental in providing insight and guidance into the choice of words used in the enhanced warning text (Hudson et al. 2013). It should therefore not be surprising that the additional text added to IBWs that specifies particular hazard, source, and impact information is consistent with the risk communication literature. Hazard information provides specifics about the nature of the storm and predicted outcomes. An example would be “HAZARD…DEVELOPING TORNADO AND QUARTER SIZE HAIL.” Source information specifies whether the tornado was radar indicated or personally observed by a ground spotter, such as “SOURCE…RADAR INDICATED ROTATION.” Finally, impact information provides details about possible negative outcomes. An example would be “IMPACT…MOBILE HOMES WILL BE HEAVILY DAMAGED OR DESTROYED. SIGNIFICANT DAMAGE TO ROOFS…WINDOWS AND VEHICLES WILL OCCUR. FLYING DEBRIS WILL BE DEADLY TO PEOPLE AND ANIMALS. EXTENSIVE TREE DAMAGE IS LIKELY.”
Although a thorough review of the risk communication literature is beyond the scope of this paper, it is instructive to briefly consider how the additional use of hazard, source, and impact text in IBWs, along with the use of threat tags, corresponds with the literature on risk communication and protective action taking. Research has shown that effective warning messages both must be as viewed as personally relevant and spur one to take protective action (Mileti 1999). An effective warning is also one that specifies the exact nature of the threat (Corvello 1998; Mileti and Sorensen 1990), increasing the likelihood that the receiver responds to it (Lindell et al. 1983; Lindell and Perry 1987) and personalizes it (Perry 1979). The increased specificity provided by the “hazard” portion of the IBW should therefore enhance personal relevance and potentially increase the likelihood the message recipient takes protective action. Receiving the message by itself, however, is often not enough to prompt protective action; confirming the message is also important (Lindell and Perry 2004, 2012), especially for tornado warnings (Sherman-Morris and Brown 2012). Additionally, observing and correctly interpreting environmental cues increases the likelihood of an appropriate response (Perry et al. 1981; Ketteridge and Fordham 1998). The IBW “source” information, which specifies whether the tornado is radar indicated or has been observed by a ground spotter, should therefore help provide confirmation of the weather risk and provide an important environmental cue.
In addition to confirming the risk, many individuals also want to know the storm’s severity (Riad et al. 1999) and the likelihood of personal risk. In fact, personalization of risk plays a crucial role as it has been linked to an increased likelihood of taking protective action (Mileti and Peek 2000; Perry et al. 1981) and it is also a critical component of Lindell and Perry’s (2012) Protective Action Decision Model. In terms of IBWs, the “impact” information and the damage threat tags (if used) can be viewed as increasing personalization of risk due to the inclusion of very specific and likely damage outcomes. Providing location-specific information is also effective to help one personalize his or her risk (Nigg 1987). This sort of geographic information is already included in the current NWS warning with no changes made in the new IBWs.
The physical and social science research conducted to date on the effectiveness of IBWs paints a consistent picture—IBWs, at least as evaluated by the immediate end users and those issuing the warning, appear to successfully communicate increased risk. The physical science research suggests that environmental and storm attributes do exist that can help NWS forecasters distinguish more intense from less intense storms. The social science research also suggests that EMs and BMs, and to a lesser extent WFs, feel that IBWs are successful at effectively communicating increased risk. As mentioned previously, however, no experimental work examining the effectiveness of actual IBWs exists, and the research presented here was designed to address this issue.
3. Study rationale and specific hypotheses
The research reported here empirically investigates the effectiveness of the IBW approach. In three experiments, the present research examined the influence of the extra IBW text (e.g., the hazard, source, and impact information) on the ability of individuals to make protective action decisions. Given the laboratory-based nature of the experiments, behavioral intentions were used as an analog of actual behavior in a fashion similar to previous work in this area (Mason and Senkbeil 2015; Schultz et al. 2010). The use of damage threat tags was specifically not investigated because the intent of the present research was to examine the effectiveness of the IBW text, without the influence of damage threat tags. As stated previously, damage threat tags are only included for those tornadoes that are predicted to be especially damaging or long lasting. The most impartial assessment of the effectiveness of the IBWs would therefore be research that investigated the influence of the additional text without threat tags, as these types of IBWs are the most commonly released warnings (Hudson and Perry 2014).
Shelter in place decisions will not differ between the IBWs and non-IBWs at decision point 1 because at that point the messages do not differ in content (H1). At decision point 2 (H2) and decision point 3 (H3), however, IBWs will produce higher likelihood ratings of sheltering in place, compared to the non-IBWs because the IBWs provide more specific and useful information.
Participants in experiment 1 were traditional university undergraduates. The participant pool in experiments 2a and 2b was expanded to both replicate the results of experiment 1 and to expand their generalizability. Experiment 2a used a larger and more diverse sample of university undergraduates, while experiment 2b used a sample of graduate students in an M.S. program in emergency management. The methodology of experiments 2a and 2b was identical to that of experiment 1 except that a web platform was utilized to collect the data.
4. Methods
a. Participants
The participants in experiment 1 were undergraduates at a Pennsylvania university enrolled in psychology or communication courses who received extra credit for their participation; 21 females (mean age 19.00 yr) and 19 males (mean age 20.24 yr) participated. The participants in experiment 2a were undergraduates at two campuses of a Pennsylvania university enrolled in psychology or communication courses who received extra credit for their participation. Because of the web-based nature of experiment 2a, in order to ensure that the participants were taking the task seriously and completing it without interruptions, the data were limited to only those participants who completed the task between 10 and 60 min to exclude potentially frivolous data. Data from five participants were discarded because they took over 60 min to complete the experiment. Of the remaining 88 undergraduates, 56 were female (mean age 21.16 yr) and 32 were male (mean age 21.22 yr). The participants in experiment 2b were graduate students at a Pennsylvania university enrolled in an M.S. program in emergency management who received course extra credit. Data from four participants were discarded because they took over 60 min to complete the experiment. Of the remaining 37 participants, 20 were female (mean age 29.85 yr) and 17 were male (mean age 30.94 yr).
b. Materials
The materials for all three experiments were identical. Four 2013 impact-based warnings were taken from archived Central Region tornado warnings. Control versions of these warnings were created by using actual county and town names from a different state using the same number of characters as the original message. For instance, an Illinois IBW became a Michigan non-IBW and “WESTERN FRANKLIN COUNTY IN SOUTH CENTRAL ILLINOIS” became “WESTERN KALKASKA COUNTY IN SOUTH CENTRAL MICHIGAN.” Time stamps were also increased or decreased by 1 h. The materials therefore consisted of eight tornado warnings, four original IBWs (with no damage threat tags), and four control warnings with the hazard, source, and impact information removed. The full text of the Illinois/Michigan warning is shown in Table 1.
Full text of Illinois tornado warning taken from archived Central Region tornado warnings and decision points are described in the text. Note that the enhanced IBW text and decision points are marked in bold. EAS stands for Emergency Alert System.
In this study, you will read some weather warnings from the National Weather Service (NWS) and be asked to make some decisions. You will play the role of a plant manager of a manufacturing company. The plant stays open 24 hours a day and is always staffed. The safety of the employees is your responsibility, but it’s also your responsibility to keep the production line up and running if at all possible. One of the decisions plant managers have to make is whether to shut down production in a case of severe weather. Every once in a while, you will be interrupted and asked to make a decision about having your employees shelter in place (in other words, have them go to an interior safe location in the building). You will be asked to rate, on a 0 to 100 scale using whole numbers, your likelihood of shutting down the plant and having the employees shelter in place. A rating of 0 would be “I definitely would not shut down the line” while a rating of 100 would be “I would definitely shut down the line.”
c. Procedure
Participants in experiment 1 were tested individually in sessions that lasted about 30 min. A computer running E-Prime 2.0 software controlled the presentation of the warnings and decision prompts, scoring of the responses, and recording of response times. After providing informed consent, an overview of the study was provided and participant questions were answered. Participants then read the eight warnings in a randomized order unique for each participant. Randomizing the story order for every participant negated any potential primacy or recency effects. Each warning was read line by line, with participants pressing the space bar to advance the text. At three different locations in each message, the participants were interrupted and asked to make a shelter in place decision for the plant employees. Decision 1 appeared after the first two lines; at this point, the two warning versions were identical. Decision 2 occurred immediately after the impacted locations were mentioned. For the IBWs, this decision occurred after the hazard, source, and impact information was presented; this extra information was simply omitted in the non-IBWs. Decision 3 occurred at the end of each warning. It is important to note that the “PRECAUTIONARY/PREPAREDNESS ACTIONS…” information appeared between decision 2 and decision 3, for both versions, as this information is already present in non-IBW messages. For every decision, participants entered a number between 0 and 100 on the keyboard and the computer logged the data. At the end of every warning, participants answered a four-option multiple choice question about information they had just read to ensure careful reading. Finally, after reading and responding to all eight warnings, participants answered a 14 question multiple-choice quiz to assess their knowledge of tornadoes and severe weather (hereafter referred to as “tornado knowledge”). The majority of the questions and their answers were derived from the website Severe Weather 101 (http://www.nssl.noaa.gov/education/svrwx101/tornadoes/). The tornado knowledge questions are shown in Table 2.
Tornado knowledge quiz.
For experiments 2a and 2b, the experimental procedure, timing, and data collection were all controlled using Qualtrics software. The study flow was identical to that used in experiment 1, and the order of the NWS warnings was also randomized for each participant. Additionally, the EM graduate students provided the number of years they have been employed as a professional emergency manager (if any).
5. Results and discussion
For all three experiments, participants were divided into either low or high tornado knowledge based on a median split. The data for the three sheltering in place decisions were analyzed using a 2 (tornado knowledge: low vs high) × 2 (warning type: IBW vs non-IBW) × 3 (decision: 1, 2, or 3) mixed measures analysis of variance with warning type and decision repeated measures. There were no statistically significant main effects or interactions involving tornado knowledge. All tests for nonhomogeneity of variance were nonsignificant as well (ps > 0.12). As done by others assessing hazard perceptions (Perreault et al. 2014), in order to avoid potential problems associated with violations of compound symmetry and sphericity, Wilks’ multivariate tests were used on the repeated dependent measures. Multiple comparison tests with Bonferroni adjustment were used to compare differences in estimated means for variables with significant results. Multivariate analysis of variance (MANOVA) outputs for all three experiments are shown in Table 3.
MANOVA outputs for all three experiments.
a. Experiment 1
The main effect of decision was statistically significant: F(2, 37) = 120.166, p < 0.0001, and ηp2 = 0.84. Multiple comparisons revealed that the pattern of shelter in place ratings was decision 1 (26.42) < decision 2 (74.14) < decision 3 (87.02). The main effect of warning type was also statistically significant: F(1, 38) = 43.71, p < 0.0001, and ηp2 = 0.53. Averaging over all three decision locations, decisions to IBWs (66.88) were greater than decisions to non-IBWs (58.17). For the purposes of the present study, however, the most interesting analysis is the warning type X decision analysis. This interaction was also statistically significant: F(2, 37) = 25.46, p < 0.0001, and ηp2 = 0.40. A series of three dependent samples Student’s t tests with Bonferroni correction were therefore performed comparing the shelter in place decisions for IBW versus non-IBW at each decision point. For decision 1, decisions to the IBWs (26.73) and non-IBWs (26.21) did not differ: t(39) = 0.47 and p = 0.64. For decision 2, shelter in place decisions to IBWs (83.06) were significantly greater than decisions to non-IBWs (65.21): t(39) = 7.82 and p < 0.0001. For decision 3, shelter in place decisions to IBWs (90.86) were also significantly greater than decisions to non-IBWs (83.18): t(39) = 3.68 and p = 0.0007. These data are shown in Fig. 1 and Table 4.
Mean responses to shelter in place decisions at three different decision points for each warning type.
The results of experiment 1 showed that, for college undergraduates adopting the role of a plant manager faced with a tornado warning, the additional information in the IBWs produced higher likelihood decisions about sheltering in place compared to the non-IBWs. It is important to note that the ratings at decision 1 to the IBWs and non-IBWs were virtually identical and quite low. At decision 1, only two lines of information had been presented, and any decision could only be based on the fact that the NWS had issued a tornado warning. These low ratings suggest that the participants were taking the task seriously and support the first hypothesis that shelter in place decisions will not differ between the IBWs and non-IBWs at decision point 1. Hypotheses 2 and 3, which stated that the IBWs would produce higher shelter in place ratings at decisions 2 and 3, were supported as well. At decision 2, the IBW versions included the additional hazard, source, and impact information that was not present in the non-IBWs. This additional information clearly influenced the study participants because the average IBW rating was more than 18 points higher than the non-IBWs. This IBW advantage remained at decision 3, although at that point, only a small amount of additional information was in the IBW versions. Even with IBW decisions approaching ceiling, the average IBW rating was still a statistically significant 7 points higher than the non-IBW rating.
Mean correct answers for tornado knowledge were 7.05/14, indicating that the undergraduates used in this experiment were not particularly severe weather savvy.
b. Experiments 2a and 2b
In both experiments, the significant main effects and interactions exactly mirrored those found in experiment 1. The main effect of decision was statistically significant: for experiment 2a, F(2, 85) = 151.84, p < 0.0001, and ηp2 = 0.76; for experiment 2b, F(2, 34) = 50.85, p < 0.0001, and ηp2 = 0.72. Multiple comparisons revealed that the pattern of shelter in place ratings was decision 1 < decision 2 < decision 3. The main effect of warning type was also statistically significant, with decisions to IBWs greater than those to non-IBWs: for experiment 2a, F(1, 86) = 33.73, p < 0.0001, and ηp2 = 0.28; for experiment 2b, F(1, 35) = 28.46, p < 0.0001, and ηp2 = 0.45. More importantly, the warning type X decision interaction was also statistically significant: for experiment 2a, F(2, 85) = 29.15, p < 0.0001, and ηp2 = 0.29; for experiment 2b, F(2, 34) = 13.64, p < 0.0001, and ηp2 = 0.31. For experiment 2a, dependent samples Student’s t tests revealed that for decision 1, decisions to the IBWs (30.99) and non-IBWs (31.97) did not differ: t(88) = 0.96 and p = 0.34. For decision 2, decisions to IBWs (82.00) were significantly greater than decisions to non-IBWs (68.68): t(88) = 13.31 and p < 0.0001. For decision 3, decisions to IBWs (87.61) were also significantly greater than decisions to non-IBWs (82.12): t(88) = 5.49 and p < 0.0001. For experiment 2b, the pattern was the same. Decisions to the IBWs (40.39) and non-IBWs (39.52) did not differ: t(37) = 0.34 and p = 0.74. For decision 2, decisions to IBWs (82.45) were significantly greater than decisions to non-IBWs (71.06): t(37) = 6.30 and p < 0.0001. For decision 3, decisions to IBWs (87.64) were also significantly greater than decisions to non-IBWs (82.91): t(37) = 3.41 and p < 0.002. These data are shown in Fig. 1 and Table 4.
For experiment 2a, mean correct answers to the weather quiz were 4.42/14, indicating that the undergraduates in this experiment were even less weather savvy than those in experiment 1. For experiment 2b, mean correct answers to the weather quiz were 10.41/14, indicating that the EM graduate students in this sample were much more severe weather wise than the previous two samples.
To see whether the pattern of results differed as a function of participant background, two additional supplemental analyses were conducted. In the first analysis, data from experiments 2a and 2b were combined and reanalyzed treating experiment as a categorical variable to see if the responses from the undergraduates differed from those of the EM graduate students. This analysis showed that experiment did not interact with any other variable; the data revealed the same patterns as those shown in experiments 2a and 2b. The second analysis focused on the data from experiment 2b and included a categorical variable coding whether the EM graduate students had any years of professional emergency manager experience. Any response over zero was coded as a “yes” (19 out of 37; mean = 11.2). This analysis was done to see if those EM graduate students who had professional EM experience provided ratings that differed from the inexperienced graduate students. The only significant effect involving experience was a significant decision X experience interaction: F(2, 34) = 6.56, p = 0.004, and ηp2 = 0.17. Simple effects revealed an experience effect only for decision 1; those students who had professional experience had higher shelter in place ratings (49.76) than those students without experience (30.05). Decisions 2 and 3 did not differ between the two groups.
The results from experiments 2a and 2b show a remarkably similar pattern to that found in experiment 1. Shelter in place responses again did not differ for decision 1, but for both decisions 2 and 3, the IBWs prompted higher shelter in place ratings than the non-IBWs. Interestingly, although the EM graduate students possessed significantly more knowledge about severe weather than the undergraduates and many also possessed a number of years of professional EM experience, the pattern did not differ between the undergraduates and the EM graduate students. Clearly, for both groups, the additional information in the IBWs prompted higher decisions about sheltering in place. Although the participants were adopting a hypothetical role of a plant manager tasked with making a shelter in place decision, the additional information in the IBWs clearly influenced that decision.
6. General discussion
The research reported here was designed to answer a straightforward question: can it be experimentally demonstrated that the enhanced IBW product promotes greater behavioral intentions of taking protective action? Based on the methodology used here, the answer appears to be yes. Across three experiments, using a hypothetical plant manager scenario, the additional information found in the IBWs produced greater behavioral intentions of sheltering in place, but only when the decisions were made after the enhanced text was presented (i.e., decisions 2 and 3). Prior to decision 1, the IBW and non-IBWs were identical, so the lack of difference in the shelter in place ratings, coupled with their low average rating, suggests that participants were sensitive to the lack of information in the warnings. These results support hypothesis 1. Hypotheses 2 and 3 were also supported by the finding of higher shelter in place decisions made to the IBWs for decisions 2 and 3. These findings also suggest that the participants were sensitive to the additional hazard, source, and impact information provided by the IBWs. The significant differences between IBWs and non-IBWs for decision 3 is even more impressive given that overall ratings for decision 3 were approaching ceiling.
Although IBWs were originally designed to be used with NWS partners, it is therefore interesting to note that both experiments demonstrated that even undergraduates were sensitive to the extra information in the IBWs. Additionally, tornado knowledge had no effect. Collectively, these results suggest that the added hazard, source, and impact information present in IBWs is not only understandable to a young student population (and one not particularly knowledgeable about severe weather) but also compelling, as least as measured by the shelter in place ratings.
Another interesting finding is that the pattern of responses of the EM graduate students did not differ from those of the undergraduates. One might argue that the superior knowledge of severe weather of the EM graduate students and/or their heightened familiarity with the various competing demands facing an EM when making a decision about whether or not to temporarily close a business might have influenced their shelter in place decisions. Although the supplemental analyses did find an effect comparing those graduate students with professional emergency management experience to those without such experience, the finding was limited to higher ratings at decision 1. No other effects of experience were found. That background knowledge and experience had little effect is actually encouraging and suggests that the enhanced text used in the IBWs is clear and understandable, regardless of target audience.
The results reported here complement those of Ripberger et al. (2015), who found that as tornado impact severity increased, sheltering in place decisions also increased, at least up to a “significant” impact. Ripberger et al. argue that their results support the concept of IBWs by showing that language that emphasizes increasingly strong tornado impacts has the potential to increase sheltering in place and saving lives. The present results also extend those of Ripberger et al. Using a more demanding task requiring three decisions to eight different warnings, and by using actual IBWs, the present results showed that the enhanced wording causally affected sheltering in place decisions.
It might be argued that the number of IBWs and non-IBWs used in the current experiments was small (four of each). Note, however, that the non-IBWs were, in fact, the exact same warnings as the original IBWs, with just a simple change in location and town/county names, providing an ideal control condition. Additionally, although the results are based on a much smaller sample than that used by Ripberger et al. (2015), the partial eta squared (ηp2) effect sizes are quite robust (0.29–0.39).
One interesting question for future research to address is whether the results would be the same if a sample of professional EMs were tested, having a large number of years of experience making critical severe weather decisions. The current results, however, suggest any influence of background knowledge might be small. While the results of experiment 2b suggest that background knowledge could play a role, it may be limited to decisions made at early points in a warning message. Given their background, it may be that professional EMs would tend to view any tornado warning issued by the NWS as valid and therefore show a tendency to make higher shelter in place ratings based on the mere release of a tornado warning. Only additional research can answer this question.
7. Limitations
One potential limitation of the research design used in this study is that it only assessed behavioral intentions. Although measuring behavioral intentions is commonly used by social scientists when they study reactions to weather warnings (e.g., Schultz et al. 2010), it is valid to ask if intentions are related to actual behavior. Recall that the results of Ripberger et al. (2015) do suggest that behavioral intentions can be a useful proxy for actual behavior. Ripberger et al. assessed whether previous factors that have been shown to influence actual responses, such as having a plan (Nagele and Trainor 2012) and perceptions of risk (Kalkstein and Sheridan 2007), moderated their behavioral intentions. In two validation models, Ripberger et al. found that behavioral intentions did vary in the predicted way. Additionally, psychological research has shown statistically significant correlations between intended and actual behavior, as shown by Armitage and Conner’s (2001) meta-analysis finding of r = 0.47. This body of work therefore suggests that the use of behavioral intentions is reasonable.
Another potential limitation concerns the use of the hypothetical scenario involving role playing a plant manager. It is reasonable to ask whether demand characteristics were created, such that the participants figured out the experimental manipulation and then acted as “good subjects” by providing ratings consistent with the hypotheses (Orne 1962; Nichols and Edlund 2015). Some might also argue that the task is somewhat artificial, especially for individuals with little or no experience in making decisions about the safety of others. The plant manager scenario was specifically used, however, in an attempt to increase the realism of the exercise, given the participant’s responsibility for the safety of multiple individuals. Additionally, informal interviews with experiment 1 participants, asking if they could articulate the study’s true intent, were done to inoculate against a potential charge of demand characteristics. Not a single participant was able to articulate that some of the warnings had additional information. This lack of awareness was not surprising given how immersed in the task they appeared to be—each warning was quite long (a minimum of 20 lines) and filled with unfamiliar county and town names that were in the path of the warned tornado. If the participants were indeed unaware of the IBW manipulation yet trying to be good subjects, then the decisions made at decision 2 and decision 3 should not have differed by warning type, yet they did. It therefore appears unlikely that the results are due to demand characteristics.
Yet another potential limitation concerns the use of the plant manager scenario and the level of risk one might be willing to assume as a function of employee oversight. On the one hand, it could be argued that a plant manager would be relatively risk averse while at work, given that he/she is responsible for the plant’s employees. That same manager might be much less risk averse, however, while at home. The current study attempted to maximize risk aversion by intentionally placing each participant in responsibility over others. Nonetheless, future research could fruitfully investigate the role that responsibility for others plays (if any) on the relative level of risk one is willing to assume.1
Two final points deserve mention. First, the methodology used in the present study is not able to pinpoint which specific aspect of the IBWs contributed to their higher shelter in place ratings. It may be that all components of the expanded IBWs (hazard, source, and impact) are necessary and have a cumulative effect on prompting higher ratings. Alternatively, it could be that simply adding more information, regardless of content, is the mechanism behind the higher ratings. Yet another possibility is that the impact information heightened fear in the message recipient, with the other information playing little or no role. Each of these interpretations is plausible, and the current methodology cannot distinguish between them (or others). Nonetheless, regardless of the specific mechanism for the effect, the present study was able to demonstrate that some aspect of the IBWs did cause higher shelter in place ratings. Teasing apart the exact nature of this effect would be a fruitful avenue for future research.
A second issue concerns how best to evaluate the success of IBWs. The original impetus behind the implementation of IBWs was to provide a warning that promotes a rapid and effective public response. A legitimate question to ask, however, is what constitutes an effective response? It is possible, given the inherent uncertainty in severe weather events, that a tornado that was predicted to be an EF1 increases in strength to be one that is much stronger. In such a scenario, the protective action that was originally recommended in the IBW may turn out to not be the best advice. How would such an IBW be evaluated? Clearly, given the predictive nature of the entire weather warning process, any suggested protective action is only the forecaster’s best guess given the information at hand. Equally clear is that IBWs cannot be faulted based on this criterion. This issue simply reveals the somewhat subjective nature of evaluating IBWs.2
8. Additional warning improvements
The present study’s findings that IBWs produced greater behavior intentions to shelter in place, compared to non-IBWs, dovetail nicely with two other recent initiatives actively underway to improve the weather warning enterprise. One initiative, orchestrated by NOAA, is Weather Ready Nation (WRN; NOAA 2013). The emphasis of WRN is to unite the various entities involved in the weather warning process—the NWS, private sector entities, academia, state and local officials, the media, and EMs—to improve the entire disaster readiness, warning, and response process. The effort focuses particularly on building strong connections between operational and research communities with the ultimate goal of increasing community response and resilience to weather disasters. The effectiveness of IBWs, as demonstrated by the findings reported here, demonstrates the importance of such collaborations. The present study’s findings are valuable because they independently validate the operational community’s decision to implement IBWs.
The other initiative, known as Forecasting a Continuum of Environmental Threats (FACETS), is under the guidance of the National Severe Storms Laboratory (Rothfusz et al. 2014). FACETS focuses on the science side of the warning enterprise and aims to be the next generation severe weather watch and warning system. FACETS utilizes “threat grids” to communicate relative severe weather danger based on one’s location and is aided by advances in satellite, radar, and ground observation technologies. Such advances clearly have implications for IBWs; improved storm path predictions should allow for better location and impact information in IBWs. The implementation of FACETS, along with the WRN initiative and IBWs, reveals the amount of emphasis placed by NOAA on improving the weather warning process.
9. Conclusions
Based on the results reported here, the decision by the NWS Central Region to begin immediate implementation of the IBW experimental product in 2013 was indeed wise. There was an urgent perceived need to upgrade the standard NWS warnings to provide more information about impact and potential consequences in the hopes that it would increase protective action-seeking behavior. The results reported here show that enhanced warnings can indeed increase the likelihood that a message recipient will take the suggested protective action. The ultimate goal, of course, is that the suggested protective action saves lives.
Acknowledgments
I sincerely thank Sepi Yalda and Duane Hagelgans for providing access to their EM graduate students. I also thank three anonymous reviewers for their many helpful suggestions on earlier drafts of this manuscript.
REFERENCES
Armitage, C., and Conner M. , 2001: Efficacy of the theory of planned behaviour: A meta-analytic review. Br. J. Soc. Psychol., 40, 471–499, doi:10.1348/014466601164939.
Corvello, V. T., 1998: Risk communication. Handbook of Environmental Risk Assessment and Management, P. Callow, Ed., Blackwell Science, 520–541.
Entremont, C., and Lamb J. D. , 2015: The relationship between tornadic debris signature height and tornado intensity. Third Symp. on Building a Weather-Ready Nation: Enhancing Our Nation’s Readiness, Responsiveness, and Resilience to High Impact Weather Events, Phoenix, AZ, Amer. Meteor. Soc., 452. [Available online at https://ams.confex.com/ams/95Annual/webprogram/Paper268874.html.]
Harrison, J., Bunting-Howarth K. , Ellis C. , McCoy C. , Sorensen H. , and Williams K. , 2014: Evaluation of the National Weather Service impact based warning tool. Ninth Symp. on Policy and Socio-Economic Research, Atlanta, GA, Amer. Meteor. Soc., J5.3 [Available online at https://ams.confex.com/ams/94Annual/webprogram/Paper241631.html.].
Hudson, M. J., and Perry B. , 2014: An update on the Central Region impact-based warning demonstration. Ninth Symp. on Policy and Socio-Economic Research, Atlanta, GA, Amer. Meteor. Soc., J5.2. [Available online at https://ams.confex.com/ams/94Annual/webprogram/Paper239767.html.]
Hudson, M. J., Browning P. , Runk K. , Harding K. , Galluppi K. , Losego J. , and Montz B. , 2013: Central Region impact-based warnings demonstration: Helping to build a weather-ready nation. Eighth Symp. on Policy and Socio-Economic Research, Austin, TX, Amer. Meteor. Soc., JPD1.1. [Available online at https://ams.confex.com/ams/93Annual/webprogram/Paper222556.html.]
Hudson, M. J., Wagenmaker R. , Mann G. , Smith B. , Thompson R. , Ferree J. , and Entremont C. , 2015: Science that drives service: The NWS impact-based warning demonstration. Third Symp. on Building a Weather-Ready Nation: Enhancing Our Nation’s Readiness, Responsiveness, and Resilience to High Impact Weather Events, Phoenix, AZ, Amer. Meteor. Soc., TJ4.1. [Available online at https://ams.confex.com/ams/95Annual/webprogram/Paper260627.html.]
Kalkstein, A. J., and Sheridan S. C. , 2007: The social impacts of the heat–health watch/warning system in Phoenix, Arizona: Assessing the perceived risk and response of the public. Int. J. Biometeor., 52, 43–55, doi:10.1007/s00484-006-0073-4.
Ketteridge, A.-M., and Fordham M. , 1998: Flood evacuation in two communities in Scotland: Lessons from European research. Int. J. Mass Emerg. Disasters, 16, 119–143.
Kingfield, D. M., LaDue J. G. , and Ortega K. L. , 2012: An evaluation of tornado intensity using velocity and strength attributes from the WSR-88D mesocyclone detection algorithm. 26th Conf. on Severe Local Storms, Nashville, TN, Amer. Meteor. Soc., 3.2. [Available online at https://ams.confex.com/ams/26SLS/webprogram/Manuscript/Paper211488/Kingfield-etal-SLS2012-Peak_MDA_and_Tornado_Intensity.pdf.]
LaDue, J. G., Ortega K. L. , Smith B. R. , Stumpf G. J. , and Kingfield D. M. , 2012: A comparison of high resolution tornado surveys to Doppler radar observed mesocyclone parameters: 2011–2012 case studies. 26th Conf. on Severe Local Storms, Nashville, TN, Amer. Meteor. Soc., 6.3. [Available online at https://ams.confex.com/ams/26SLS/webprogram/Manuscript/Paper212627/Hi-res-tornado-nowcasting2-20121205hi-qual.pdf.]
Levitan, M. L., Jorgensen D. P. , Kuligowski E. D. , Lombardo F. T. , and Phan L. T. , 2013: Technical investigation of the May 22, 2011, tornado in Joplin, Missouri. National Institute of Standards and Technology Final Rep., 428 pp., doi:10.6028/NIST.NCSTAR.3.
Lindell, M. K., and Perry R. W. , 1987: Warning mechanisms in emergency response systems. Int. J. Mass Emerg. Disasters, 5, 137–153.
Lindell, M. K., and Perry R. W. , 2004: Communicating Environmental Risk in Multiethnic Communities. Sage, 262 pp.
Lindell, M. K., and Perry R. W. , 2012: The Protective Action Decision Model: Theoretical modifications and additional evidence. Risk Anal., 32, 616–632, doi:10.1111/j.1539-6924.2011.01647.x.
Lindell, M. K., Perry R. W. , and Greene M. , 1983: Individual response to emergency preparedness planning near Mt. St. Helens. Disaster Manage., 3 (January/March), 5–11.
Losego, J., Montz B. , Galluppi K. , Hudson M. , Browning P. , Runk K. , and Harding K. , 2013: Evaluating the effectiveness of IBW. Eighth Symp. on Policy and Socio-Economic Research, Austin, TX, Amer. Meteor. Soc., 226066. [Available online at https://ams.confex.com/ams/93Annual/webprogram/Paper226066.html.]
Mason, J. B., and Senkbeil J. C. , 2015: A tornado watch scale to improve public response. Wea. Climate Soc., 7, 146–158, doi:10.1175/WCAS-D-14-00035.1.
Mileti, D. S., 1999: Disasters by Design. Joseph Henry Press, 351 pp.
Mileti, D. S., and Sorensen J. H. , 1990: Communication of Emergency Public Warnings: A Social Science Perspective and State-of-the-Art Assessment. Oak Ridge National Laboratory Rep. ORNL-6609, 166 pp.
Mileti, D. S., and Peek L. , 2000: The social psychology of public response to warnings of a nuclear power plant accident. J. Hazard. Mater., 75, 181–194, doi:10.1016/S0304-3894(00)00179-5.
Nagele, D. E., and Trainor J. E. , 2012: Geographic specificity, tornadoes, and protective action. Wea. Climate Soc., 4, 145–155, doi:10.1175/WCAS-D-11-00047.1.
Nichols, A. L., and Edlund J. E. , 2015: Practicing what we preach (and sometimes study): Methodological issues in experimental laboratory research. Rev. Gen. Psychol., 19, 191–202, doi:10.1037/gpr0000027.
Nigg, J. M., 1987: Communication and behavior: Organizational and individual response to warnings. Sociology of Disasters: Contribution of Sociology to Disaster Research, R. R. Dynes, B. de Marchi, and C. Pelanda, Eds., F. Angeli, 103–117.
NOAA, 2011a: NWS Central Region service assessment: Joplin, Missouri, tornado—May 22, 2011. DOC/NOAA/NWS Rep., 41 pp. [Available online at www.nws.noaa.gov/om/assessments/pdfs/Joplin_tornado.pdf.]
NOAA, 2011b: The historic tornadoes of April 2011. DOC/NOAA/NWS Rep., 76 pp. [Available online at www.nws.noaa.gov/om/assessments/pdfs/historic_tornadoes.pdf.]
NOAA, 2013: National Weather Service weather-ready nation roadmap. NWS Rep., 81 pp. [Available online at http://www.nws.noaa.gov/com/weatherreadynation/files/nws_wrn_roadmap_final_april17.pdf.]
Orne, M. T., 1962: On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. Amer. Psychol., 17, 776–783, doi:10.1037/h0043424.
Perreault, M. F., Houston J. B. , and Wilkins L. , 2014: Does scary matter?: Testing the effectiveness of new National Weather Service tornado warning messages. Commun. Stud., 65, 484–499, doi:10.1080/10510974.2014.956942.
Perry, R. W., 1979: Evacuation decision-making in natural disasters. Mass Emerg., 4, 25–38.
Perry, R. W., Lindell M. K. , and Greene M. R. , 1981: Evacuation Planning in Emergency Management. Lexington Books, 199 pp.
Riad, J. K., Norris F. H. , and Ruback B. R. , 1999: Predicting evacuation in two major disasters: Risk perception, social influence, and access to resources. J. Appl. Soc. Psychol., 29, 918–934, doi:10.1111/j.1559-1816.1999.tb00132.x.
Ripberger, J. T., Silva C. L. , Jenkins-Smith H. C. , and James M. , 2015: The influence of consequence-based messages on public responses to tornado warnings. Bull. Amer. Meteor. Soc., 96, 577–590, doi:10.1175/BAMS-D-13-00213.1.
Rothfusz, L. P., Schlatter P. T. , Jacks E. , and Smith T. M. , 2014: A future warning concept: Forecasting a Continuum of Environmental Threats (FACETs). Second Symp. on Building a Weather-Ready Nation: Enhancing Our Nation’s Readiness, Responsiveness, and Resilience to High Impact Weather Events, Atlanta, GA, Amer. Meteor. Soc., 2.1. [Available online at http://www.nssl.noaa.gov/projects/facets/FACETs_AMS_2014.pptx.]
Schultz, D. M., Gruntfest E. C. , Hayden M. H. , Benight C. C. , Drobot S. , and Barnes L. R. , 2010: Decision making by Austin, Texas, residents in hypothetical tornado scenarios. Wea. Climate Soc., 2, 249–254, doi:10.1175/2010WCAS1067.1.
Sherman-Morris, K., and Brown M. E. , 2012: Experiences of Smithville, Mississippi residents with the 27 April 2011 tornado. Natl. Wea. Dig., 36, 93–101. [Available online at http://www.nwas.org/digest/papers/2012/Vol36No2/Pg093-Sherman-Brown.pdf.]
Smith, B. T., Thompson R. L. , Dean A. R. , Marsh P. T. , Wagenmaker R. , Mann G. , Hudson M. J. , and Ferree J. , 2014: Demonstrating the utility of conditional probabilities of tornado damage rating in the impact-based warning era. 27th Conf. on Severe Local Storms, Madison, WI, Amer. Meteor. Soc., 89. [Available online at http://www.spc.noaa.gov/publications/smith/tdmgprob.pdf.]
Smith, B. T., Thompson R. L. , Dean A. R. , and Marsh P. T. , 2015: Diagnosing the conditional probability of tornado damage rating using environmental and radar attributes. Wea. Forecasting, 30, 914–932, doi:10.1175/WAF-D-14-00122.1.
Toth, M., Trapp R. J. , Wurman J. , and Kosiba K. A. , 2013: Comparison of mobile-radar measurements of tornado intensity with corresponding WSR-88D measurements. Wea. Forecasting, 28, 418–426, doi:10.1175/WAF-D-12-00019.1.