1. Introduction
People live their lives believing that they are not at risk (Mileti 2018). However, when a hazardous situation emerges, at-risk publics need to be informed and prompted to take actions to protect themselves. Recognizing the challenge of prompting individuals to take action, risk communicators have emphasized characteristics of the hazard (Potter et al. 2018) and the potential impacts on populations (Ripberger et al. 2015) through the use of intense language to describe particularly dangerous conditions. Indeed, the inclusion of a dire warning such as describing conditions as “life threatening” has been recognized as the impetus motivating some to act (Cappucci 2019).
Capturing attention and prompting a change in risk perception by alerting people to danger is one element of a warning message. However, a second element is the provision of guidance that can instruct individuals to take action quickly to prevent loss of life or injury. Communicating a protective action guidance or instructional message (Frisby et al. 2014) has been shown to increase behavioral intent to take action when faced with a severe threat and, in some cases, has emerged as the primary driver of actions taken (Milne et al. 2000). Researchers have suggested that the inclusion of actionable and instructive information that guides people about the actions they should take to protect their safety both increases self-efficacy and motivates protective action response (Coombs 2009; Frisby et al. 2014; Sellnow et al. 2017). Despite evidence documenting the importance of including specific guidance, it is often missing from automated alert and warning messages, especially those delivered by short messaging channels such as tweet (on Twitter) or text (on cellular telephones), which limit content via restricted character counts.
One type of automated warning message distributed via the Twitter social media application is the tornado warning message (NWS 2020). These tornado warnings, produced by the National Weather Service (NWS) “bot” account @NWStornado, are automatically posted on the Twitter account of the issuing weather forecast office and include text copy naming the cities at risk along with a graphic containing a map, potential impacts, and populations at risk. These messages are sent in the event of a short-fuse warning to increase exposure to audiences who may be at risk of a tornado threat. Thus, Twitter provides a means by which the NWS can disseminate hazard warnings without first requiring users to “opt in” and can include graphics as part of the warning messages.
Unlike other warning channels, Twitter offers a mixture of modalities for delivering content. The Emergency Alert System (EAS) notifies members of the public who are watching television by breaking into broadcasts with a rolling text scroll. The National Oceanic Atmospheric Administration weather radio notifies those tuned in via an audio version of the EAS message. Sirens notify those at risk within a community by blasting a tone, and, in some cases, a spoken message. In contrast, Twitter messages can be constructed to deliver content in multiple forms including text, visual, and audiovisual. These are described as technological affordances (Sundar 2008), or features, that can facilitate specific tasks by users (Conole and Dyke 2016). The ability to deliver warning messages across multiple modalities bears additional considerations for message construction; in this case, the structure or organization of the message must be considered, in addition to its content and style.
Recent research investigated public perceptions of automated NWS tornado warning messages using think-aloud interviews and eye-tracking methods (Sutton and Fischer 2021). When study participants were shown an NWS Twitter message that included both textual and visual content about the tornado threat, the participants reported that, although the warning message delivered useful information about the location of the threat, potential impacts, and populations at risk, it failed to provide content that would contribute to successful protective actions (Sutton and Fischer 2021). Results from the think-aloud interviews suggested that, although knowing what the threat is, where it is located, and its potential for harm provided sufficient information to help message recipients make sense of the event, the lack of instruction about how to reduce risk impeded their ability to select and take appropriate protective actions. This problem is likely to be exacerbated among individuals facing an unfamiliar threat.
While the inclusion of protective action guidance is notably missing from these automated warning messages, how to make use of the technological affordances of Twitter to structure the message and include guidance has not yet been investigated. In this study, we investigate how automated messages can be constructed to improve message perceptions (Bean et al. 2015) and increase perceived self-efficacy (Rogers and Prentice-Dunn 1997) and perceived response efficacy (Bandura 2010), important factors that influence behavioral intent. We do so by altering the content of an automated tornado warning message to include protective-action information and then examining what effect the placement/location of such information within the message structure—that is, as part of the message text, the attached graphic, or both—has on message outcomes.
2. Literature review
Prior research on warning messages has emphasized the importance of message content, that is, what is said, as well as style, or how it is said (Mileti and Sorensen 1990). Indeed, focusing on these two message components has predominated efforts to improve warning design (Mileti and Sorensen 1990). Message structure, identified by persuasive communication scholars (Shen and Bigsby 2013), is another important feature of message design that has been shown to have a persuasive effect on individuals. Researchers define message structure as the presentation of the message’s arguments, which includes the number of arguments and their order of presentation. While little difference in message persuasiveness has been found by presenting an argument at the beginning versus the end of a message, when message processing time is limited, it may be advantageous to present key information up front rather than at the end (O’Keefe 2002; Shen and Bigsby 2013), suggesting that the location of information in the structure of the message matters.
a. Message content
Message content, that is, what is said, consists of the information included in a warning message via text, as well as any information included via image. Mileti and Sorensen (1990) identified five primary areas of message content that, when included, predict an increased likelihood that message receivers will take protective action. These content areas include the following: hazard, guidance, location, time, and message source.
1) Hazard
Effective warning messages describe the threat of the hazardous agent, the likelihood, or probability of risk, and the potential impacts and their severity on exposed populations (Wood et al. 2018). Phenomenon-based warnings (Potter et al. 2018) are triggered by and emphasize the characteristics of the hazard itself, such as size and intensity, while impact-based warnings emphasize the impact of the hazard on populations at risk (Potter et al. 2018; Ripberger et al. 2015). As noted by Ripberger et al. (2015), tornado warnings issued by the NWS have emphasized the probability of the threat occurring in addition to characteristics of the threat, and thus can be considered hazard based. Describing potential impacts, in addition to providing a clear description of the hazard and its characteristics, may be especially important during rare events and for visitors to the area at risk, in other words, when the hazard is unfamiliar (Morss et al. 2018; Schumacher et al. 2010).
2) Guidance
Effective warning messages should tell people what they should do, how they should do it, and how to maximize their health and safety (Janssen et al. 2006). Guidance essentially delivers two types of information: 1) information that instructs people about actions to take in response to a threat, including detailed information about how to accomplish the recommended protective action (designed to increase self-efficacy), and 2) information about how and why these recommended protective actions will reduce harm (designed to increase response efficacy) (Bandura 2010).
3) Location
Effective warning messages will specify the location of the threat/event (to increase personalization), who is and who is not at risk, who should take the protective action, and the geographical boundaries of the affected location and populations (Wood et al. 2018). Location information can be communicated via text or images, including maps, which provide a visual means of determining whether one is susceptible to the threat (Liu et al. 2017).
4) Time
Effective warning messages also include content about when protective actions should begin, when actions should be completed, and when a message will expire or will be updated (Mileti and Peek 2000). This is information designed to communicate an appropriate sense of urgency for the given event and timing for warning response.
5) Source
The message source, that is, the name of the individual or organization sending the message, should also be included in warning messages. The research record has shown sources that are familiar and recognizable, and represent official response organizations, will result in better message response outcomes than messages from unfamiliar and unofficial sources (Cox and Wogalter 2006). Message receivers assess the credibility of messages based upon perceptions of the message source (Anthony and Sellnow 2011).
b. Message style
How a message is written, that is, the message style, also influences the way in which people respond to a message (Mileti and Peek 2000). Warnings should be clearly worded and absent of any jargon and concepts that require specific knowledge for understanding. Information contained in warnings should be specific, rather than general, and should be as complete and stated with as much certainty as possible (Wood et al. 2018). In addition, messages should have internal consistency within a given message, that is, presenting information that is not contradicted within the message itself (Williams and Eosco 2021). There should also be external consistency across messages, that is, presenting information that is not contradicted by information contained in different messages sent at different times, possibly by different sources (Williams and Eosco 2021).
c. Message structure
Message structure, that is, the presentation of the data or the claim in a warning message (Shen and Bigsby 2013), also plays a role in communication aimed to persuade individuals to take protective action. Much of the research on message structure has investigated the ordering of contents, such as presenting the most important arguments at the beginning of a message or the conclusion (Shen and Bigsby 2013). Warning researchers have studied the ordering of contents for 90-character Wireless Emergency Alerts and 140-character and 280-character tweets that are text based (Sutton and Kuligowski 2019). Recent research on visual risk communication of tornado messages found that message receivers attend to both visual and textual content (Sutton and Fischer 2021). However, how the placement of content affects actual warning outcomes, that is, what protective actions are taken when exposed to a message, has not yet been investigated.
d. Measuring message outcomes
Empirical research has identified perceptions and behaviors that occur after message receipt and before a behavioral warning response. These outcomes can be measured to assess the effectiveness of warnings. These outcomes are understanding, believing, personalizing, deciding, and milling or response delay (Mileti and Peek 2000; Mileti and Sorensen 1990).
First, the message receiver must understand and attach meaning to the information being presented, such as the type of threat and the actions being communicated (i.e., Dash and Gladwin 2007; Mileti and Beck 1975; Mileti et al. 1975; Mileti and O’Brien 1992). They must understand what the threat is, what has happened or is happening, the potential impacts, what locations and populations are at risk, what they must do to protect themselves, who is sending the message, by when they must take protective action, and for how long they must continue doing so.
The individual must then believe the threat is real and that the recommended actions will decrease the likelihood of harm (Dash and Gladwin 2007; Nigg 1987; Schumacher et al. 2010). “Believing” assesses perceptions of message truthfulness and accuracy related to the threat as well as to the effectiveness of the recommended protective actions. Furthermore, message belief suggests the thought of taking a protective action has been established (Wood et al. 2018).
In response to the warning message, individuals then personalize, or appraise the likelihood that a threat will affect them personally. Evidence suggests personalization is a precursor to taking protective action. If people do not think they are the intended audience for a message, they are unlikely to attend to or act upon the guidance contained in the message (Wood et al. 2018). When people personalize a message, they conclude they are at risk and recognize that the threat may affect them, personally.
Message receivers must then decide what action to take, if any, to protect themselves. To do so, message receivers must determine that a behavioral response to the threat is warranted (Wood et al. 2018). Deciding to act occurs as people process warning information and is a cognitive precursor to a behavioral warning response (Wood et al. 2018).
When processing a message, an individual routinely engages in milling behavior, that is, seeking out additional information to confirm the threat and the actions recommended in the message (Casteel 2016; Mileti and Peek 2000; Perry 1979; Perry et al.1981). Milling is described as a social process, where individuals form a new normative understanding, in response to the message and interaction with others (Wood et al. 2018).
Although these five message outcomes assess prebehavioral cognitions (understanding, believing, personalizing, and deciding) and the timing of behavioral response (milling), they do not directly measure self-efficacy, that is, the degree to which people believe they can perform the recommended protective actions (Bandura 2010; Witte 1996), nor do they measure response efficacy, that is, the degree to which they believe that taking the recommended actions will protect their safety (Bandura 2010). Self-efficacy and response efficacy are included in the protection motivation theory (PMT) as key constructs that motivate the initiation of protective action (Rogers and Prentice-Dunn 1997). While threat appraisal in response to a warning message results in a shift in individual threat perception (akin to personalizing above), perceptions of self-efficacy and response efficacy are formed in response to the presence or absence of protective action guidance contained within the message. The present study was designed to help fill in this gap by including self-efficacy and response efficacy along with the five more traditional message outcomes.
3. Research questions
This research addresses the following research questions (RQs):
RQ1: How does the manipulation of the message text (content), the message graphic (structure), and the interaction of the two, affect warning response outcomes (understanding, believing, personalizing, deciding, and milling)?
RQ2: How does manipulation of the message text (content), the message graphic (structure), and the interaction of the two, affect perceptions of self-efficacy and response efficacy?
4. Method
a. Study context
In the spring of 2019, the National Weather Service developed a set of experimental products to warn the public about tornadoes for dissemination via social media channels. Described as an experimental product “depicting the watch or warning area as well as GIS-informed exposure information” (NWS 2020, p. 1) they include images, text, and icons to communicate imminent threat. They also include two maps, one showing a regional view and a second containing a polygon overlayed on a map of the local warning area. The product also includes a black sidebar (on the left side) that provides information about GIS-informed population exposure and the potential impacts (see Fig. 1). As an automated product, it also populates the text of the tweet, which provides a signal word (“warning”), plus the names of the three cities (locations) contained within the polygon with the highest population or the county name with the largest geographic coverage of the polygon (D. Deroche, NWS Central Region Headquarters, 2020, personal communication). The experimental products were created to increase the “accessibility of life-saving products via interfaces [such as social media] that are not conducive to the display of long fuse products” (NWS 2020, p. 1) allowing for a wide reach to external users by redistribution through social media.
Stimulus: standard (control).
Citation: Weather, Climate, and Society 13, 4; 10.1175/WCAS-D-21-0097.1
b. Study design
This study was a 2 × 2 factorial experiment. The first independent variable was “message text,” and the second independent variable was “message graphic.” Message text refers to the content of the text in the tweet copy, and message graphic refers to the graphical portion found in the automated experimental product described above. Each of the independent variables included two conditions: 1) a real-world message (“standard practice”), and 2) additional protective action content (“enhanced”). We examined the effects of these two independent variables on the participants’ perceptions of the message (i.e., understanding, believing, personalizing, deciding, and milling) and efficacy (i.e., self-efficacy and response efficacy).
c. Participants
Data were collected from 279 participants at a large southeastern university; however, 6 people failed to complete all portions of the protocol, resulting in a final sample of 273 participants. Just over half (number n = 148; 54.25%) of the participants were men, and 125 (45.8%) were women. The majority indicated that they were Caucasian (n = 207; 75.8%), followed by Black or African American (n = 37; 13.6%), Hispanic/Latino(a) (n = 10; 3.7%), Asian (n = 8; 2.9%), Other (n = 7; 2.6%), and prefer not to answer (n = 3; 1.1%). Participants included first-year college students (n = 86; 31.5%), second-year students (n = 73; 26.7%), third-year students (n = 59; 21.6%), and fourth-year students (n = 55; 20.1%).
The majority of participants indicated that they lacked formal training in map reading (n = 215; 78.8%) and meteorology (n = 259; 94.9%). To gauge prior experience to tornado events, participants were asked a series of questions about their prior experience with tornadoes. Participants were asked the following: “People can have multiple experiences with tornadoes over the course of their lifetime. Please think about all of your experiences with tornadoes and indicate how much experience you have with each of the statements listed below.” They were then asked to indicate their level of experience using a scale that ranged from 1 = strongly disagree, 2 = somewhat disagree, 3 = neither agree nor disagree, 4 = somewhat agree, to 5 = strongly agree (Demuth 2019). The majority reported prior experience with tornadoes. (See Table 1).
Tornado experience.
Because these data were collected at a time when campus was closed because of the COVID-19 pandemic, we also asked participants to indicate their current residence. The majority indicated that they resided in the state where the research was conducted (n = 176; 65.9%); the remainder indicated a contiguous (n = 45; 16.9%) or noncontiguous (n = 46; 17.2%) U.S. state, and a few chose not to answer (n = 6).
d. Stimuli
We used a tweet that included a tornado warning graphic distributed by the Storm Prediction Center for a previous tornado event in Lexington, Kentucky (Fig. 1), as the basis for our stimuli. This real-world message, including both the text and graphic, was considered the standard of practice or “control” for this study. It was enhanced to create three text and graphic treatment conditions. Thus, participants in the control condition viewed the standard, real-world text and graphic, and those in the three treatment conditions viewed variations of the enhanced text and graphics.
1) “Standard” (control)
The structure of the control message included text copy stating, “Tornado Warning including Lexington, KY until 8:45 PM EDT” and included an attached graphic. The graphic layout included a red banner with a headline displaying the warning name (Tornado Warning) and a single large main panel depicting the geographical area of the warning with a basic map background and a red polygon identifying areas at risk. A smaller panel, in the bottom left of the graphic, provided a regional view of the warning area. On the left of the graphic was a rectangular box with a black background, red and white text, and white icons depicting a tornado and hail. The text in the box included information about timing, the threat, and population, schools, and hospitals that may have potential exposure.
2) Enhanced text (treatment 1)
For our first manipulation, we worked with the Louisville Weather Forecast Office (WFO) to alter the tweet copy of the message while retaining the graphic from the original message (Fig. 2). We included instructional content that used an imperative sentence style, directing message receivers to take action. To draw attention to key words and to emphasize the call to action, we used all caps (TORNADO WARNING; TAKE COVER NOW) and an exclamation point. We also added a visual element, the warning emoji, which is a filled triangle with a black exclamation point on a yellow background.
Stimulus: enhanced text (treatment 1).
Citation: Weather, Climate, and Society 13, 4; 10.1175/WCAS-D-21-0097.1
3) Enhanced graphic (treatment 2)
For the second manipulation (Fig. 3), we worked with the Louisville WFO to alter the graphic from the control message while retaining the control text. To manipulate the graphic, we removed information about “Potential Exposure” from the black box on the left of the image and replaced it with “Safety Precautions” that included a white icon of a house and instructions for message receivers to “Move to an interior room on the lowest floor of a sturdy building. Stay away from windows.”
Stimulus: enhanced graphic (treatment 2).
Citation: Weather, Climate, and Society 13, 4; 10.1175/WCAS-D-21-0097.1
4) Enhanced text and graphic (treatment 3)
Our third manipulation included the enhanced revisions made to both the text and graphic (Fig. 4).
Stimulus: enhanced text and graphic (treatment 3).
Citation: Weather, Climate, and Society 13, 4; 10.1175/WCAS-D-21-0097.1
e. Dependent variables
Following presentation of the randomly assigned stimulus, message outcomes were measured via five primary dependent variables (understanding, belief, personalization, deciding, and milling) based on the prior literature surrounding emergency warning messaging (i.e., Sutton et al. 2018, 2020; Wood et al. 2018). We also measured self-efficacy and response efficacy. Scales were adapted from prior research to measure the dependent variables (Sutton et al. 2018, 2020; Witte 1996; Wood et al. 2018); internal consistency ranged from “acceptable” (0.77) to “good” (0.89).
1) Understanding
Understanding was measured using seven items: “After viewing this message, I understood: 1) What is happening, 2) The risks (impacts), 3) What to do to protect myself, 4) What location is affected, 5) Who the message is from, 6) When I am supposed to take action to protect myself, and 7) How long I am supposed to continue taking actions to protect myself.” Respondents indicated their agreement with each statement using a standard 5-point Likert scale (1 = strongly disagree, 2 = somewhat disagree, 3 = neither agree nor disagree, 4 = somewhat agree, and 5 = strongly agree). Cronbach’s alpha was 0.77 for the seven items.
2) Belief
Belief was measured using three items: “After viewing this message, I would believe that: 1) Severe weather is heading my way, 2) I should take action to protect myself and, 3) Taking protective action will make me safer.” Respondents indicated their agreement with each statement using a standard 5-point Likert scale (1 = strongly disagree, 2 = somewhat disagree, 3 = neither agree nor disagree, 4 = somewhat agree, and 5 = strongly agree). Cronbach’s alpha was 0.87 for the three items.
3) Personalization
Personalization was measured using seven items (Wood et al. 2018): “After viewing this message, I think that: 1) I might become injured, 2) People I know might become injured, 3) People I do not know might become injured, 4) I might die, 5) People I know might die, 6) People I do not know might die, and 7) The message was meant for me.” Respondents indicated their agreement with each statement using a standard 5-point Likert scale (1 = strongly disagree, 2 = somewhat disagree, 3 = neither agree nor disagree, 4 = somewhat agree, and 5 = strongly agree). Cronbach’s alpha was 0.88 for the seven items.
4) Deciding
Deciding was measured with three items: “After viewing this message, I believed: 1) It will be easy to decide what to do, 2) I will be able to decide what to do quickly, and 3) I can decide what to do with confidence.” Respondents indicated their agreement with each statement using a standard 5-point Likert scale (1 = strongly disagree, 2 = somewhat disagree, 3 = neither agree nor disagree, 4 = somewhat agree, and 5 = strongly agree). Cronbach’s alpha was 0.89 for the three items.
5) Milling
Milling was measured by two questions that tapped into participants’ intent to search for more information prior to taking action. Specifically, participants were asked to respond to the following statements: “After viewing this message, how likely would you be to look for additional information about what is happening before taking action?” and “After viewing this message, how likely would you be to look for additional information about what to do before taking action?” Participants responded using a 5-point semantic differential scale where 1 = extremely unlikely, 2 = unlikely, 3 = neutral, 4 = somewhat likely, and 5 = extremely likely. Cronbach’s alpha was 0.82 for the two questions.
6) Self-efficacy
Self-efficacy was measured with two items. The self-efficacy items were, “I know what actions I should take after reading this warning,” and “I can do the actions described in the warning.” Cronbach’s alpha was 0.82 for the self-efficacy. Items were rated using a standard 5-point Likert scale (1 = strongly disagree, 2 = somewhat disagree, 3 = neither agree nor disagree, 4 = somewhat agree, 5 = strongly agree).
7) Response efficacy
To measure response efficacy, participants answered one statement, “ “The actions in the warning will keep me safe.” Items were rated using a standard 5-point Likert scale (1 = strongly disagree, 2 = somewhat disagree, 3 = neither agree nor disagree, 4 = somewhat agree, 5 = strongly agree).
f. Procedure
University students were invited to participate in an online Qualtrics questionnaire through a subject participant pool (“SONA”) at a large southeastern university in the United States. They were first asked to read and electronically provide informed consent. Next, participants were randomly assigned to one of four conditions (control, enhanced text, enhanced graphic, and enhanced text and graphic). Participants were then asked to take a moment to read and review the assigned message. After viewing the message, to increase ecological validity, participants were then asked to watch a short distraction video about cats (Wimmer and Dominick 2014). Following the video, participants answered a series of questions about their message perceptions and their background/demographics. The questionnaire took approximately 15 min to complete. Participants received course credit through SONA to thank them for their time.
Data were collected using IBM SPSS Statistics, version 25, and were reviewed and cleaned prior to analysis. Composite variables were created for each construct. Descriptive analysis included frequency, mean, median, and standard deviation. Inferential analysis included one-way analysis of variance (ANOVA). Post hoc tests (Bonferroni) were conducted to identify statistically significant differences between the specific conditions.
5. Results
One-way ANOVA was used to determine the effect of the message condition on the participants’ perceptions (understanding, believing, personalizing, milling, deciding, self-efficacy, and response efficacy). Overall, we found main effects of message condition on understanding, personalizing, deciding, self-efficacy, and response efficacy. Table 2 provides descriptive statistics for the message perception variables by condition, and Table 3 provides a summary of the ANOVAs.
Mean and standard deviation message outcomes by condition.
Effects of condition on message perception. Here, SS indicates sum of squares and MS indicates mean square for the ANOVAs.
a. Understanding
We first tested whether message condition (standard, enhanced text, enhanced graphic, enhanced text and graphic) elicited differences in message understanding. We found a main effect of message condition, F(3, 272) = 4.43, η2 = 0.047, and p = 0.005, with a small to medium effect size (Cohen 1988). Post hoc tests revealed that the standard condition (mean M = 3.84) resulted in statistically lower message understanding than the enhanced text (M = 4.19; p = 0.18), enhanced graphic (M = 4.19; p = 0.18), and enhanced text and graphic (M = 4.19; p = 0.17) conditions. There were no statistically significant differences in message understanding among treatment conditions, however.
b. Believing
Second, we tested whether message condition was associated with differences in message belief. There were no main effects, F(3, 272) = 1.50, η2 = 0.00, and p = 0.215; thus, post hoc comparisons were not conducted.
c. Personalizing
Third, we tested whether message condition elicited differences in message personalization. We found a main effect of message condition, F(3, 272) = 5.75, η2 = 0.061, and p = 0.001, with a medium effect size. We did not find statistically significant differences between the standard and enhanced conditions (enhanced text, enhanced graphic, enhanced text and graphic) or between the enhanced graphic and the enhanced text and graphic conditions. However, personalization was significantly lower in the enhanced text condition (M = 3.16) relative to the enhanced graphic (M = 3.58; p = 0.015) and the enhanced text and graphic (M = 3.30; p = 0.001) conditions.
d. Deciding
Fourth, we tested whether message condition elicited differences in the ability to decide what to do in response to the warning message. There was a main effect of message condition, F(3, 272) = 5.46, η2 = 0.059, and p = 0.001, with a medium effect size. Post hoc tests revealed statistically significant differences between standard and enhanced conditions. Specifically, the standard condition (M = 3.61) resulted in significantly lower ability to decide how to respond relative to the enhanced text (M = 4.10; p = 0.03), enhanced graphic (M = 4.15; p = 0.013), and enhanced text and graphic (M = 4.14; p = 0.005) conditions. There were no differences between the enhanced conditions, however.
e. Milling
Fifth, we tested whether message condition elicited differences in milling intention following message receipt. There was no main effect of message condition, F(3, 272) = 0.58, η2 = 0.006, and p = 0.63.
f. Self-efficacy
Next, we tested whether message condition was associated with self-efficacy. We found a main effect of message condition, F(3, 267) = 20.33, η2 = 0.18, and p < 0.001, with a large effect size. Post hoc tests revealed statistically significant differences between standard and enhanced conditions. Specifically, the standard condition (M = 3.10) resulted in significantly lower self-efficacy relative to the enhanced text (M = 4.05; p < 0.001), enhanced graphic (M = 4.28; p < 0.001), and enhanced text and graphic (M = 4.23; p < 0.001) conditions. We did not find statistically significant differences in self-efficacy between the three enhanced conditions.
g. Response efficacy
Last, we tested whether message condition was associated with response efficacy. We found a main effect of message condition, F(3, 267) = 16.95, η2 = 0.14, and p < 0.001, with a large effect size. Post hoc tests revealed statistically significant differences between standard and enhanced conditions. Specifically, the standard condition (M = 3.09) resulted in significantly lower response efficacy relative to the enhanced text (M = 3.95; p < 0.001), enhanced graphic (M = 4.12; p < 0.001), and enhanced text and graphic (M = 4.18; p < 0.001) conditions. We did not find statistically significant differences in response efficacy between the three enhanced conditions.
6. Discussion
Our results indicate that the inclusion of enhanced protective action guidance in the text, graphic, or both elicited an increase in understanding of the message. We also observe that the enhanced text message (treatment 1) resulted in decreased personalization in comparison with the standard (“control”), enhanced graphic (treatment 2), and the enhanced text and graphic (treatment 3) messages. The inclusion of protective action content in messages, whether via text, infographic, or a combination, resulted in increased ability to make decisions about the message, as well as increased self-efficacy and response efficacy among participants. Moreover, the size of these effects is telling. Based on the nomenclature of Cohen (1988), the effect of message content and graphic on understanding, personalization, and deciding outcomes can be considered “small” to “medium”; however, the effects on self-efficacy and response efficacy can be considered “large.” This difference in magnitude is important given that we found significant, large effects of exposure to protective action guidance information (in the three enhanced messages) on participants’ belief that they could perform the recommended actions. Furthermore, it is worth noting that in the case of warning messages, which typically are delivered to populations, even small “effects” can make a substantive difference for large numbers of individuals in terms of reduced numbers of deaths and injuries, reduced economic losses, and improved quality of life following events such as tornadoes.
Warning message research has tended to focus on the textual content, identifying how to improve message perceptions and reduce milling behavior under conditions of imminent threat (Wood et al. 2018). Importantly, the inclusion of protective action information has been identified as a key content area that will increase behavioral intent among message receivers (Frisby et al. 2014). Message effects, however, have not traditionally included measurements of self-efficacy or response efficacy when measuring the outcomes associated with communicating about the threat and the associated recommended protective actions.
In this research, we find support for including measurements of efficacy in response to message design. While manipulations of message content, style, and structure explain limited amounts of variance among research participants’ understanding, personalizing, and deciding, and little to no effect on believing and milling, the inclusion of these actions showed a significant change in perceptions of efficacy—both self-efficacy and response efficacy. It is the inclusion of these efficacy measures that provide the greatest insight into our findings about the importance of providing protective action guidance as part of a tornado warning message. Its absence, in this case, would offer limited insight into the effects of adding a call to action and instructive information in a tornado warning, raising the question, How do we know when we are measuring the right things?
Research using the protection motivation theory has found that efficacy is, in many cases, a more significant predictor of behavior than risk appraisal, that is, perceptions about the characteristics of the threat itself (Milne et al. 2000). Knowing how to take action to protect oneself, and believing that the actions will make oneself safe, are key motivators to taking action when faced with a significant threat. If message designers are interested in measuring behavioral intent, they may consider including measures of self-efficacy and response efficacy in future research.
Importantly, message manipulations resulted in no effects on believing or milling outcomes. Believing outcomes had the highest means across all of the messages, suggesting that our participants believed that severe weather was heading their way and that they should take action to make themselves safer in response to the threat. Milling response was also moderately high across all four message conditions, suggesting that regardless of the information contained in the message, some individuals would actively seek out additional confirmatory information before acting, a behavioral response that is likely to delay protective action.
This research also investigated the effects of message structure for automated tornado warning messages. The affordances of Twitter allow risk communicators to present information in text, graphic, or both, suggesting new ways to consider how to best construct an effective message. Here, we manipulated the placement of protective action information in the structure of the message by adding a call to action in the text copy and instructive content in the graphic. In comparison with the original control message, outcomes in response to the enhanced messages were consistently improved for understanding, deciding, self-efficacy, and response efficacy, suggesting that the inclusion of protective action information placed somewhere in the message increases message effects. Further, we find limited differences between the three enhanced messages (enhanced text, enhanced graphic, and enhanced text and graphic). This finding suggests that the message structure may have less of an effect on outcomes than the decision to include efficacy content it in general.
One caveat should be included, however, for the effect of message manipulation on personalization. The mean response to personalization was the lowest among all of the message effects. Two possible explanations come to mind. First, due to coronavirus disease 2019 (COVID-19) most of our research participants had moved away from the location that was placed under threat for the tornado warning scenario; it is possible that they interpreted the relevance of the threat in relation to their location while participating in the online experiment. Second, the manipulation of the graphic removed information about populations exposed to the threat. This eliminated content was replaced with protective action instruction. However, it is possible that the information about populations exposed has an effect on how individuals personalize the threat and its potential impact on human life.
a. Theoretical implications
This research contributes to warning response theory by including efficacy—both self-efficacy and response efficacy—along with commonly measured warning response constructs. The work also extends prior research by investigating message structure in addition to content and style. We found that where protection action guidance information is placed in a message is not as important as that it is included.
b. Practical implications
The National Weather Service has continued to develop experimental products for other hazards, suggesting additional opportunities to communicate risk via social media under conditions of imminent threat. If these products are designed to include similar content and structured in a similar format as tornado warnings, they too may benefit from research to investigate the message receiver perceptions and efficacy outcomes. This is likely to be especially important for hazards that are less familiar to populations that may face them in the future. NWS tornado warning text products include precautionary/preparedness actions, which can serve as a guideline for the type of information that local WFOs should include in future short messages. In addition, we recommend that the NWS Storm Prediction Center modify the experimental products to include protective actions in the graphic or adapt the text of the tweet with content that has been demonstrated to increase message perception outcomes and perceived efficacy, regardless of hazard type. If one hopes to increase behavioral intent, one must tell people what to do.
c. Limitations
Because our sample is limited to university undergraduate students, we are cautious to generalize beyond this group; however, the experimental conditions and the clear effects of the inclusion of instructional information suggests that such findings are likely to be replicated in future studies. Additionally, given the online research setting and scenario context, there is potential response bias. And, notably, while this study addresses the motivators leading to important behavioral outcomes, message perceptions and efficacy do not equate to behavioral intent. Finally, we were unable to assess the impact of hazard experience on warning response across message conditions given that the majority of participants had tornado experience.
d. Future research
Future warning research should draw from other persuasive messaging and health behavior theories, especially PMT, and should include self-efficacy and response efficacy as important causal factors driving warning response. This will expand knowledge of how additional variables are affected by enhanced message content. It should also look across additional hazards to determine if these outcomes differ by the length of forewarning and hazard type. Future research should include larger, more diverse samples and should develop and test messages in languages beyond English among varied populations. Multivariate analysis incorporating location and hazard experience as a covariate can help to account for differences in hazard familiarity. In contexts in which participants are likely to have similar levels of hazard experience, recruitment procedures should be adjusted to increase variability within the sample so that the effect of experience can be examined. Additionally, future research should explore the public’s understanding of protective action guidance. For example, researchers could examine the extent to which participants are able to correctly perform recommended actions included in warnings, and they could investigate what people think common protective action guidance language, such as “take cover now,” means. To understand how those with varying levels of prior hazard experience may respond to a message, researchers should examine how different populations, with low and high hazard experience, perceive the warning message. In addition, further research on message style should be conducted, including the use of all caps, icons, and other message style characteristics; eye tracking can help determine what aspect of the message draws the first and greatest amount of attention for different message types. Experimental products designed for other hazards should be tested, and findings should be integrated to provide a more complete understanding of human response to hazard warnings. Researchers should examine the relative and combined effect of providing message content focusing on the hazard, its impact, and recommended protective action guidance. Future research may also consider postevent surveys to identify how automated tornado messages affected actual behaviors and whether individuals sought out instructive information before taking action.
Acknowledgments
This material is based upon work supported by the National Oceanic and Atmospheric Administration under Grant NA210OAR4590124. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of NOAA.
Data availability statement
Anonymized data may be made available upon request.
REFERENCES
Anthony, K. E., and T. L. Sellnow, 2011: Information acquisition, perception, preference, and convergence by Gulf Coast residents in the aftermath of the Hurricane Katrina crisis. Argum. Advocacy, 48, 81–96, https://doi.org/10.1080/00028533.2011.11821756.
Bandura, A., 2010: Self-efficacy. The Corsini Encyclopedia of Psychology, I.B. Weiner and W.E. Craighead, Eds., 1–3, http://onlinelibrary.wiley.com/doi/10.1002/9780470479216.corpsy0836/abstract.
Bean, H., J. Sutton, B. F. Liu, S. Madden, M. Wood, and D. Mileti, 2015: The study of mobile public warning messages: A research review and agenda. Rev. Comm., 15, 60–80, https://doi.org/10.1080/15358593.2015.1014402.
Cappucci, M., 2019: ‘Something incredible happened last week’: Forecasters credited with saving lives in outbreak with 434 tornado reports. Washington Post, accessed 21 June 2021, https://www.washingtonpost.com/weather/2019/06/03/forecasters-credited-with-saving-lives-outbreak-reported-tornadoes/.
Casteel, M. A., 2016: Communicating increased risk: An empirical investigation of the National Weather Service’s impact-based warnings. Wea. Climate Soc., 8, 219–232, https://doi.org/10.1175/WCAS-D-15-0044.1.
Cohen, J., 1988: Statistical Power Analysis for the Behavioral Sciences. Routledge Academic, 567 pp.
Conole, G., and M. Dyke, 2016: Understanding and using technological affordances: A response to Boyle and Cook. Res. Learn. Technol., 12, 301–308, https://doi.org/10.3402/rlt.v12i3.11261.
Coombs, W. T., 2009: Crisis, crisis communication, reputation, and rhetoric. Rhetorical and Critical Approaches to Public Relations II, R. L. Heath, E. L. Toth, D. Waymer, Eds., Routledge, 249–264.
Cox, E. P., III, and M. S. Wogalter, 2006: Warning source. Handbook of Warnings, M. S. Wogalter, Ed., CRC Press, 111–122.
Dash, N., and H. Gladwin, 2007: Evacuation decision making and behavioral responses: Individual and household. Nat. Hazards Rev., 8, 69–77, https://doi.org/10.1061/(ASCE)1527-6988(2007)8:3(69).
Demuth, J. L., and Coauthors, 2019: Risk, information, and vulnerability for evolving tornado threats in the Southeast. 29th Conf. on Severe Local Storms, Stowe, VT, Amer. Meteor. Soc., 12.1, https://ams.confex.com/ams/29SLS/webprogram/Paper348786.html.
Frisby, B. N., S. R. Veil, and T. L. Sellnow, 2014: Instructional messages during health-related crises: Essential content for self-protection. Health Commun., 29, 347–354, https://doi.org/10.1080/10410236.2012.755604.
Janssen, A. P., R. R. Tardif, S. R. Landry, and J. E. Warner, 2006: “Why tell me now?” The public and healthcare providers weigh in on pandemic influenza messages. J. Public Health Manage. Pract., 12, 388–394, https://doi.org/10.1097/00124784-200607000-00013.
Liu, B. F., M. M. Wood, M. Egnoto, H. Bean, J. Sutton, D. Mileti, and S. Madden, 2017: Is a picture worth a thousand words? The effects of maps and warning messages on how publics respond to disaster information. Public Relat. Rev., 43, 493–506, https://doi.org/10.1016/j.pubrev.2017.04.004.
Mileti, D. S., 2018: PrepTalks: Dr. Dennis Mileti “Modernizing public warning messaging.” FEMA, accessed 21 June 2021, https://www.fema.gov/blog/preptalks-dr-dennis-mileti-modernizing-public-warning-messaging.
Mileti, D. S., and E. M. Beck, 1975: Communication in crisis: Explaining evacuation symbolically. Commun. Res., 2, 24–49, https://doi.org/10.1177/009365027500200102.
Mileti, D. S., and J. H. Sorensen, 1990: Communication of emergency public warnings: A social science perspective and state-of-the art assessment. Oak Ridge National Laboratory Rep. ORNL-6609, 162 pp., https://www.osti.gov/servlets/purl/6137387.
Mileti, D. S., and P. W. O’Brien, 1992: Warnings during disaster: Normalizing communicated risk. Soc. Probl., 39, 40–57, https://doi.org/10.2307/3096912.
Mileti, D. S., and L. Peek, 2000: The social psychology of public response to warnings of a nuclear power plant accident. J. Hazard. Mater., 75, 181–194, https://doi.org/10.1016/S0304-3894(00)00179-5.
Mileti, D. S., T. E. Drabek, and J. E. Haas, 1975: Human Systems in Extreme Environments: A Sociological Perspective. University of Colorado Boulder Institute of Behavioral Science, 165 pp.
Milne, S., P. Sheeran, and S. Orbell, 2000: Prediction and intervention in health-related behavior: A meta-analytic review of protection motivation theory. J. Appl. Soc. Psychol., 30, 106–143, https://doi.org/10.1111/j.1559-1816.2000.tb02308.x.
Morss, R. E., C. L. Cuite, J. L. Demuth, W. K. Hallman, and R. L. Shwom, 2018: Is storm surge scary? The influence of hazard, impact, and fear-based messages and individual differences on responses to hurricane risks in the USA. Int. J. Risk Reduct., 30, 44–58, https://doi.org/10.1016/j.ijdrr.2018.01.023.
Nigg, J. M., 1987: Communication and behavior: Organizational and individual response to warnings. Sociology of Disasters, R. R. Dynes et al., Eds., Franco Angeli Libri, 103–117.
NWS, 2020: Experimental severe weather impact graphics. NOAA Product Description Doc., 7 pp., https://nws.weather.gov/products/PDD/PDD_ExpSevereWeatherImpactGraphics_2020.pdf.
O’Keefe, D. J., 2002: Persuasion: Theory and Research. 2nd ed. Sage, 384 pp.
Perry, R. W., 1979: Evacuation decision-making in natural disasters. Mass Emerg., 4, 25–38.
Perry, R. W., M. K. Lindell, and M. R. Greene, 1981: Evacuation Planning in Emergency Management. Lexington Books, 199 pp.
Potter, S. H., P. V. Kreft, P. Milojev, C. Noble, B. Montz, A. Dhellemmes, R. J. Woods, and S. Gauden-Ing, 2018: The influence of impact-based severe weather warnings on risk perceptions and intended protective actions. Int. J. Risk Reduct., 30, 34–43, https://doi.org/10.1016/j.ijdrr.2018.03.031.
Ripberger, J. T., C. L. Silva, H. C. Jenkins-Smith, and M. James, 2015: The influence of consequence-based messages on public responses to tornado warnings. Bull. Amer. Meteor. Soc., 96, 577–590, https://doi.org/10.1175/BAMS-D-13-00213.1.
Rogers, R. W., and S. Prentice-Dunn, 1997: Protection motivation theory. Handbook of Health Behavior Research 1: Personal and Social Determinants, D. S. Gochman, Ed., Plenum Press, 113–132.
Schumacher, R., D. Lindsey, A. Schumacher, J. Braun, S. D. Miller, and J. L. Demuth, 2010: Multidisciplinary analysis of an unusual tornado: Meteorology, climatology, and the communication and interpretation of warnings. Wea. Forecasting, 25, 1412–1429, https://doi.org/10.1175/2010WAF2222396.1.
Sellnow, D. D., D. R. Lane, T. L. Sellnow, and R. S. Littlefield, 2017: The IDEA model as a best practice for effective instructional risk and crisis communication. Commun. Stud., 68, 552–567, https://doi.org/10.1080/10510974.2017.1375535.
Shen, L., and E. Bigsby, 2013: The effects of message features: Content, structure, and style. The Sage Handbook of Persuasion: Developments in Theory and Practice, J. P. Dillard and L. Shen Eds., Sage, 20–35.
Sundar, S. S., 2008: The MAIN model: A heuristic approach to understanding technology effects on credibility. Digital Media, Youth, and Credibility, M. J. Metzger and A. J. Flanagin, Eds., MIT Press, 73–100, https://www.issuelab.org/resources/875/875.pdf.
Sutton, J., and E. D. Kuligowski, 2019: Alerts and warnings on short messaging channels: Guidance from an expert panel process. Nat. Hazards Rev., 20, 2, https://doi.org/10.1061/(ASCE)NH.1527-6996.0000324.
Sutton, J., and L. M. Fischer, 2021: Understanding visual risk communication messages: An analysis of visual attention allocation and think-aloud responses to tornado graphics. Wea. Climate Soc., 13, 173–188, https://doi.org/10.1175/WCAS-D-20-0042.1.
Sutton, J., S. C. Vos, M. Wood, and M. Turner, 2018: Designing effective tsunami messages: The role of short messages and fear in warning response. Wea. Climate Soc., 10, 75–87, https://doi.org/10.1175/WCAS-D-17-0032.1.
Sutton, J., L. M. Fischer, L. James, and S. Sheff, 2020: Earthquake early warning message testing: Visual attention, behavioral responses, and message perceptions. Int. J. Risk Reduct., 49, 101664, https://doi.org/10.1016/j.ijdrr.2020.101664.
Williams, C. A., and G. M. Eosco, 2021: Is a consistent message achievable? Defining “message consistency” for weather enterprise researchers and practitioners. Bull. Amer. Meteor. Soc., 102, E279–E295, https://doi.org/10.1175/BAMS-D-18-0250.1.
Wimmer, R. D., and J. R. Dominick, 2014: Mass Media Research: An Introduction. 10th ed. Cengage Learning, 496 pp.
Witte, K., 1996: Fear as motivator, fear as inhibitor: Using the extended parallel process model to explain fear appeal successes and failures. Handbook of Communication and Emotion, Elsevier, 423–450.
Wood, M. M., D. S. Mileti, H. Bean, B. F. Liu, J. Sutton, and S. Madden, 2018: Milling and public warnings. Environ. Behav., 50, 535–566, https://doi.org/10.1177/0013916517709561.
