1. Introduction
A long-term goal for warning-message designers is to determine the most effective type of warning message that can instruct individuals to take an action to quickly prevent loss of life and/or injury when faced with an imminent threat. The content included in warning messages must inform the public about an approaching or potential hazard and should also provide protective action guidance to inform individuals of the actions necessary to protect themselves (Mileti and Peek 2000).
Prior research has indicated that the inclusion of protective action guidance, or instructions, is likely to increase an individual’s behavioral intent to act when they are at risk (Frisby et al. 2014; Milne et al. 2000). Further, the guidance included in a warning message has been shown to be a primary motivator that drives which actions a person takes when faced with a threat and their confidence to be able to perform such actions (Coombs 2009; Frisby et al. 2014; Milne et al. 2000; Sellnow et al. 2017). Although the inclusion of these protective actions is important in warning-message design, messages often exclude them, especially in short-form messages such as those sent via Twitter (Sutton et al. 2021).
To inform the public of an imminent threat, the National Weather Service (NWS) developed a suite of experimental warning products that are automatically disseminated to the public through Twitter “bot” accounts such as @NWSTornado, @NWSSnowSquall, and @NWSDustStorm (NWS 2016, 2020). These automated messages include tweet copy, or text, naming the type of hazard and the locations at risk, and a graphic containing a map, potential impacts, and populations exposed (see Fig. 1). These Twitter messages are designed to deliver and disseminate warning information for fast-moving hazards, such as tornadoes, snow squalls, and dust storms, to members of the public and core partners in broadcast/electronic media, emergency managers, and other government agencies. However, several of these experimental messages do not include the protective action guidance that may inform the public about what to do during hazardous conditions (Sutton et al. 2021). When a person lacks prior experience or familiarity with a threat and its corresponding actions, the exclusion of protective action information may result in their inability to take protective actions (Sutton et al. 2018).
An example of an experimental warning product distributed by the NWS for tornado.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-22-0092.1
While Twitter-based tornado products were initially issued in 2016 for hazards like tornado, in 2018 the National Weather Service introduced two new experimental warning products: dust storm and snow squall. The NWS distributes these experimental warning products from their official Twitter accounts to warn the public about the approaching hazard. Although these threats existed in the past, because of the more recent development of messages for these hazards, recipients are less likely to be familiar with the appropriate actions to take when exposed to threat. Notably, dust storm and snow squall both have potential to result in significant harm. For example, in 2022, a snow squall in Pennsylvania resulted in a 50-vehicle pileup, extended closures of a 7-mi. (∼11 km) stretch of highway, and 8 deaths (Cappucci et al. 2022), and in 2021, a dust storm in Utah resulted in a 20-vehicle crash leading to injuries for at least 10 people and 8 deaths (Firozi and Cappucci 2021).
A critical challenge for warning-message designers and risk communicators is determining how best to design messages that promote understanding of the severity of the situation and motivate individuals to take action to protect themselves (Perreault et al. 2014). Limited research has been directed to identify how prior warning experience affects cognitive perceptions of warning messages, including individuals’ understanding of, belief in, and decisions to take the actions prescribed in the message. This research addresses that gap by exploring the effect of prior hazard warning experience on warning-message perceptions for three hazards.
2. Literature review
a. Warning-message design
Prior research on warning messages has focused on how best to construct messages that will lead individuals to take protective actions. Persuasive message design theories specify how message content, message style, and message structure can be manipulated to produce the most effective outcomes (Shen and Bigsby 2013; Sutton et al. 2021).
Message content has been defined as the “what is said” or represented in the text and graphic portions of a persuasive message (Shen and Bigsby 2013). In warning messages, there are five types of content that should be included in an effective warning message: 1) information about the hazard itself (i.e., what it is and information about the potential severity, impact, and consequences), 2) protective action guidance (i.e., what people should do to protect themselves from the threat), 3) the location of the threat and who it might impact, 4) the time to take action in response to the potential threat and the time the message expires, and 5) the message source (i.e., the organization that distributed the message) (Mileti and Sorensen 1990). Effective warning messages must convey content that tells people what to do and how to do it while maximizing their health and safety (Janssen et al. 2006; Sutton et al. 2021).
Message style refers to how message designers use linguistic styles to present information (Shen and Bigsby 2013), such as through hyperboles, phonetic symbolism, powerful versus powerless language, and message framing (Shen and Bigsby 2013; O’Keefe and Jensen 2006). For warnings, message content should be presented in a style that relays certainty, is consistent, is specific, and uses language that is unambiguous (Mileti and Sorensen 1990). Scholars have indicated that warning messages should be clear in wording with minimal references to jargon to explain their concepts (Wood et al. 2018). Further, internal consistency should be achieved so that information does not contradict itself within the message (Williams and Eosco 2021).
Message structure refers to how content is presented; this includes the order of contents and the format, such as if information is presented in the graphic or text of a message (Shen and Bigsby 2013; Sutton et al. 2021). Scholars have previously examined how the ordering of contents, that is, presenting the most important information at the beginning of the message versus at the end of the message, influences perceptions (Shen and Bigsby 2013). Others have examined message structure in the information presented in short versus long-form Twitter warning alerts, and they found message perceptions were higher for the longer messages (Sutton et al. 2018). Although research has previously explored how the placement of the message contents in different types of messages, limited research has explored how the placement of protective action guidance influences perceptions of the message. One study by Sutton et al. (2021), however, examined how the placement of the protective action guidance, whether in the text or the graphic, of the message influences outcomes. Their results indicated the inclusion of protective action guidance, whether in the main Tweet copy, the graphic, or both, increased the message perception outcomes (Sutton et al. 2021). However, there has not been research to examine how those with varying levels of prior experience with a hazard and the inclusion of protective action guidance in varying message structure influences the participants perceptions of the message.
b. The inclusion of protective action guidance in warning messages
The role of a warning message is to provide message receivers with information about a threat and the actions they should take to protect themselves (Mileti and Sorensen 1990; Potter et al. 2018; Ripberger et al. 2015; Cappucci 2019). However, recent research investigating the experimental Twitter product for tornado has shown that although these warning messages deliver useful information about the threat, they failed to include information about protective actions. This absence of guidance information has the potential to diminish message receivers’ knowledge about and ability to undertake protective actions (Sutton and Fischer 2021).
Subsequent recent research explored the @NWSTornado experimental product via experimental design, where participants were exposed to an original message, or a message manipulated to include protective action guidance (Sutton et al. 2021). Results indicated that the inclusion of protective action guidance content elicited increased understanding, increased ability to make decisions, increased self-efficacy, and increased response efficacy.
c. Familiarity and prior hazard warning experience
Scholars have provided empirical evidence demonstrating the importance of protective action information; however, less attention has been directed to how prior hazard warning experience affected message perception outcomes. To put it simply, prior experience has been operationalized as the idea that humans are shaped by their own previous experiences, and it impacts their ability to understand information and to make judgments and decisions. However, prior experience has been measured in a variety of ways, ranging from simplistic approaches (e.g., “have you experienced a <hazard>?” to multi-item Likert scales resulting in conflicting findings on the influence of prior hazard experience on outcomes and perceptions of risk (Demuth 2018). These differences in measurements and definitions of prior hazard experience may have influenced whether and to what extent it relates to individuals’ assessments of and responses to future risks. Thus, there have been calls for more systematic and attention to the treatment of past hazard experience (Demuth 2018; Weinstein 1989; Lindell and Perry 2012; Kellens et al. 2013).
Empirical evidence does demonstrate prior experience has a powerful impact on an individuals’ ability to recognize risk (Weinstein 1989), including the characteristics of the threat, the level of value or importance the person places on the risk, the emotional response to the threat, and the judgments and decisions they make when faced with the threat (Demuth 2018; Greening et al. 1996; Tversky and Kahneman 1974). When exposed to a warning message about a threat, prior experience may result in increased message salience (Brown et al. 2018; Becker et al. 2017), as well as increased understanding, ability to make decisions, and motivation to take action (Demuth 2018; Lindell and Perry 2012). Furthermore, prior experience with a hazard affects how individuals become aware of, assess, and respond to risk (Demuth 2018), and personal experience with an event influences how people react to a message and the included protective action guidance (Perreault et al. 2014; Atwood and Major 1998). For example, for some, prior tornado experience leads to an increased likelihood to take protective action (Weinstein et al. 2000), to be attune to communication channels, and to seek out further information about the oncoming hazard (Perreault et al. 2014).
Scholars have also emphasized that geographic location and proximity to hazardous locations (i.e., living in a coastal region and experiencing hurricanes) will also shape risk perceptions and behaviors (Lindell and Perry 2012). For example, those who live in higher risk areas tend to be more familiar with local hazards and how to protect themselves during hazardous events (Lindell and Perry 2012). Additionally, the amount of personal experience with official threat information (i.e., being under a tornado warning, hearing tornado sirens firsthand) and news information about a threat (i.e., hearing or watching news coverage when a threat is happening) also shapes risk perceptions and judgements when individuals are confronted with new information (Demuth 2018). In the case where an individual has not directly experienced a hazard, they may have learned about the threat and its protective action behaviors through indirect channels of communication (Demuth 2018).
d. Measuring warning-message outcomes
Prior empirical research has identified a series of perceptions, or message outcomes, that occur after individuals are exposed to a warning message and prior to their behavioral response. These include message understanding, believing, personalizing, deciding, milling, self-efficacy, and response efficacy (Mileti and Peek 2000; Mileti and Sorensen 1990; Sutton et al. 2021).
After exposure to a warning message, the message receiver must first understand the message and attach meaning to the information presented. To understand the message, the receiver comprehends what the threat is, what is happening, what the potential impacts are, what population is at risk, where the location of the threat is, what they must do to protect themselves, who the sender of the message is, and the time at which and duration of protective actions that should be taken (i.e., Dash and Gladwin 2007; Mileti and Beck 1975; Mileti and O’Brien 1992).
Message recipients must then believe the threat to be real and that a threat is coming to harm a specific area to harm the individual (Dash and Gladwin 2007; Nigg 1987; Schumacher et al. 2010). Within belief, message receivers must also perceive or believe that the message and its protective action guidance is truthful and accurate. After the receiver believes the threat to be real, individuals must personalize the threat—that is, the receiver must assess whether the threat will affect them personally (Wood et al. 2018), prompting action. Next, message receivers must be able to decide or to make a judgment of what actions to take, if any, to protect themselves from the threat. The decision to take protective action includes determining if a behavioral response is warranted in the situation and serves as a precursor to behavioral actions (Wood et al. 2018).
Throughout these message perception processes, message receivers engage in milling, or information seeking, to confirm that the threat is real or protective action guidance is accurate (Casteel 2016; Mileti and Peek 2000; Perry 1979; Perry et al. 1981). The process of milling may spark individuals to seek out other information sources or interact with other people to find more information about the threat and its recommended actions (Wood et al. 2018).
More recently, researchers have included measures of efficacy in response to a warning message (i.e., self-efficacy and response efficacy). Self-efficacy and response efficacy are key to identifying whether message receivers will act, and efficacy has referred to the individual’s level of confidence that the message’s protective action guidance will keep them safe. Through efficacy items, individuals must believe they themselves could take the recommended protective actions and that those actions will keep them safe (Sutton et al. 2021). Self-efficacy refers to the receiver’s belief that they could perform the protective action (Bandura 2010; Witte 1996), and response efficacy refers to the belief that taking the recommended actions would protect life safety (Bandura 2010).
3. Research questions
The current study investigates perceptions of NWS experimental warning product Twitter messages for three hazard types: tornado, snow squall, and dust storm. We vary these messages by manipulating their content and structure through 1) the inclusion and 2) the presentation of protective action guidance, located either in the message graphic or in the text. We also examine the role of prior hazard warning experience on participants’ message perceptions. This study was guided by four research questions (RQs):
RQ1: How does the type of hazard (tornado, snow squall, or dust storm) affect message perception outcomes (understanding believing, personalizing, deciding, milling, self-efficacy, and response efficacy)?
RQ2: How does the type of message (control, enhanced graphic, enhanced text, and enhanced graphic and text) affect message perception outcomes (understanding believing, personalizing, deciding, milling, self-efficacy, and response efficacy)?
RQ3: How does prior hazard warning experience (low vs high) affect message perception outcomes (understanding believing, personalizing, deciding, milling, self-efficacy, and response efficacy)?
RQ4: Does the type of hazard or the level of the participant’s prior hazard warning experience modify the relationship between message type and message outcomes?
4. Methods
a. Study design
To address the research questions, this study uses a 3 × 4 × 2 between-subjects factorial experiment. The first independent variable was the hazard type referring to the type of hazard event presented in the message: dust storm, snow squall, or tornado. The second independent variable was the message type. Message type refers to how protective action guidance was included in the structure of the message. Four message types were included: 1) a “standard practice” or control message that is not enhanced with protective action guidance, 2) an enhanced graphic message, where protective action guidance was added in the graphic portion of the message, 3) an enhanced text message, where protective action guidance was added to the text portion of the message, and 4) an enhanced graphic and text, where protective action guidance was added to both the text and graphic portions of the message. The third independent variable was the participant’s level of prior hazard warning experience with their assigned hazard, and it was categorized as low versus high using a median split. We examined the effects of the three independent variables on the participants’ perceptions of the message (i.e., understanding, believing, personalizing, deciding, milling, self-efficacy, and response efficacy). To examine the effects, the experiment used a between-subjects design in which participants were randomly assigned to one hazard type and received one of the four warning messages. Afterward, the participants answered questions about their perceptions of the message, their perceptions of self-efficacy and response efficacy, prior hazard warning experience, and demographic information.
b. Participants and sampling procedures
Data were collected from 1050 adult participants through an online, third-party company (Qualtrics research panels), to obtain a nonprobability, opt-in sample of residents in three locations: Atlanta, Georgia; Buffalo, New York; and Phoenix, Arizona. These three locations were selected because they had each recently received an NWS hazard warning message for the three hazards of interest (Atlanta, tornado; Buffalo, snow squall; and Phoenix, dust storm) and had similar population characteristics and sizes.
Participants were recruited through Qualtrics research panels. Qualtrics, a third-party recruitment firm, obtains participants through actively managed market research panels and social media platforms. To verify unique responses and ensure validity, Qualtrics employed digital fingerprinting technology and internet protocol (IP) address checks. Power analysis (power = 0.80; α = 0.05) was conducted using G*Power software assuming small-to-medium effect sizes (f = 0.15), which determined a minimum sample size of N = 990.
Our sample encompassed approximately one-third of the participants in each location for a final sample size of N = 1050 (i.e., n = 363, or 35%, in Atlanta; n = 326, or 31%, in Buffalo; and n = 361, or 34%, in Phoenix), which was verified through ZIP (postal) code identifiers. To ensure variability in prior hazard warning experience, the participants in each location were randomly assigned to receive a type of message hazard type: Atlanta sample (n = 141, 40%, tornado; n = 95, 27%, snow squall; n = 116, 33%, dust storm); Buffalo sample (n = 112, 32%, snow squall; n = 119, 34%, tornado; n = 121, 34%, dust storm); Phoenix sample (n = 124, 36%, dust storm; n = 119, 34%, snow squall; n = 103, 30%, tornado). In addition, we developed quotas for the sample to match census demographics for age (ages 18–34: 30%, ages 35–54: 32%, and ages 55+: 38%) and gender (approximately 50% who identified as a woman, approximately 50% who identified as a man, and nonbinary natural fallout). Respondents who did not match these quotas were disqualified from participation and omitted from the study. Table 1 displays the demographic data on all study participants.
Demographic characteristics of the respondents.
c. Independent variables
Three independent variables were included in this study: hazard type, message type, and prior hazard warning experience. Below, we discuss each of these independent variables.
1) Hazard type
Hazard type refers to the type of hazard presented in the message: tornado, snow squall, or dust storm. These three hazards were selected by the NWS because they were recently adopted into the suite of experimental products and lacked empirical testing with populations that may be alerted or warned in future events.
2) Control
Currently, NWS Weather Forecast Offices (WFOs) distribute experimental warning products to the public through Twitter and Facebook. These messages were chosen due to the study’s design to test the visual aspect of a warning, and how the structure of the message and the contents located in this structure influence message outcomes. The control or standard practice messages used for this study were direct replicas of NWS experimental warning products that had been recently distributed by NWS in Atlanta, Buffalo, and Phoenix. Each of the standard messages included a tweet with text copy that stated the location of the hazard and the duration for which the warning is in effect [i.e., tornado—tornado warning including Atlanta, North Atlanta, and Decatur, Georgia, until 1115 eastern daylight time (EDT)] and an attached graphic (Fig. 2). The attached graphic included the type of warning, a large map with a polygon depicting the areas at risk, and a sidebar that provided details about the threat information and population exposed. At the bottom-left corner of the message was a smaller, inset map, which oriented the viewer to the broader geographical context. To replicate the messages, the researchers chose messages from each of the three locations and manipulated the time and date to match the study parameters.
Stimuli used in experiment for tornado, snow squall, and dust storm for each message type.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-22-0092.1
3) Enhanced graphic message
The first manipulation is to the structure of the graphic portion of the message. We focused on manipulating the structure of the message to include protective action guidance in the graphic. The “enhanced graphic” messages include protective action content within the graphic while retaining the text copy from the standard practice message (highlighted in yellow in Fig. 2). Members of the research team consulted with practitioners from the NWS Storm Prediction Center and NWS regional offices on content manipulations. Information about the potential exposure to populations in the black sidebar area was removed and replaced with a warning icon and protective action information under the heading “Safety Precautions.” The descriptions about the safety precautions for each type of message are as follows: Tornado—Move to an interior room of the lowest floor of a sturdy building. Stay away from windows. Snow Squall—Avoid or delay travel. If on the road, turn on your headlights and hazard lights. There’s no safe place on a highway in a snow squall. Dust Storm—Pull aside and stay alive. Park your vehicle with all lights off until storm passes.
4) Enhanced text message
The second manipulation is to the tweet copy portion of the message. To manipulate the structure of this message, the “enhanced text” messages include protective action content within the text copy while retaining the graphic from the standard practice message (highlighted in blue in Fig. 2). Members of the research team consulted with practitioners from the NWS Storm Prediction Center and NWS regional offices on content manipulations in the tweet copy. In addition some text was presented in capital letters or imperative tense (TAKE COVER NOW!, AVOID or DELAY TRAVEL!, and PULL ASIDE. STAY ALIVE!) to draw the message receiver’s visual attention.
5) Combined format message
The third manipulation is to both the graphic portion and the tweet copy portion of the message by including protective action guidance content as detailed above. The combined format message is presented in Fig. 2.
d. Prior HW experience
Prior hazard warning (HW) experience was measured using four items that indicated the extent to which participants had warning experience with their assigned hazard [adapted from Demuth (2018)]. The items were as follows: 1) I have been under a (tornado, snow squall, dust storm) warning; 2) I have received (tornado, snow squall, dust storm) warnings (not as a test) firsthand; 3) I have heard or watched live news coverage on radio, TV, or online of (tornado, snow squall, dust storm) as it was happening; and 4) I have seen news coverage about the aftermath of a (tornado, snow squall, dust storm). Respondents indicated their agreement with each statement using a standard 5-point Likert scale (1 = strongly disagree, 2 = somewhat disagree, 3 = neither agree nor disagree, 4 = somewhat agree, and 5 = strongly agree). Table 2 reports the Cronbach’s α, means, standard deviations, and medians for experience by each hazard type.
Means, medians, and SDs prior hazard warning experience by hazard type.
e. Dependent variables
Message perceptions were measured via five primary dependent variables (understanding, belief, personalization, deciding, and milling) (Sutton et al. 2018, 2021; Wood et al. 2018). Perceptions of self-efficacy and response efficacy were also measured to determine participants beliefs about their ability to complete and confidence in the recommended protective actions (Sutton et al. 2021). These measures were drawn from protective motivation theory (Rogers and Prentice-Dunn 1997) and the prior research from Sutton et al. (2021). For all of the dependent variables, participants indicated their agreement with corresponding statements using a standard 5-point Likert scale (1 = strongly disagree, 2 = somewhat disagree, 3 = neither agree nor disagree, 4 = somewhat agree, and 5 = strongly agree).
1) Understanding
Understanding (Cronbach’s α = 0.88) was measured using seven items: “After viewing this message, I understood: 1) What is happening, 2) The risks (impacts), 3) What to do to protect myself, 4) What location is affected, 5) Who the message is from, 6) When I am supposed to take action to protect myself, and 7) How long I am supposed to continue taking actions to protect myself.”
2) Belief
Belief (Cronbach’s α = 0.83) was measured using five items: “After viewing this message, I would believe that: 1) The threat is heading my way, 2) The message is reliable, 3) I know when I will be in danger, 4) I should take action to protect myself and, 5) Taking protective action will make me safer.”
3) Personalization
Personalization (Cronbach’s α = 0.91) was measured using seven items from Wood et al. (2018): “After viewing this message, I think that: 1) I might become injured, 2) People I know might become injured, 3) People I do not know might become injured, 4) I might die, 5) People I know might die, 6) People I do not know might die, and 7) The message was meant for me.”
4) Deciding
Deciding (Cronbach’s α = 0.91) was measured with three items: “After viewing this message, I believed: 1) It will be easy to decide what to do, 2) I will be able to decide what to do quickly, and 3) I can decide what to do with confidence.”
5) Milling
Milling (Cronbach’s α = 0.90) was measured with three items: “Before following the information in the message to protect myself, I would look for additional information about . . . 1) What is happening, 2) What to do, 3) How to perform the actions.”
6) Self-efficacy
Self-efficacy (Cronbach’s α = 0.90) was measured with three items: “1) I know what actions I should take after reading this warning, 2) I am confident I can follow the information described in this message, and 3) I am capable of following the information advised in this warning.”
7) Response efficacy
Response efficacy (Cronbach’s α = 0.88) was measured with three items: “After viewing this message, I feel: 1) The information in this message will keep people safe, 2) Following the information in this message will be successful for reducing harm, and 3) Following the information in this message will be effective in keeping me safe.”
f. Procedure
To complete the study, the invited participants were first asked to read and electronically provide informed consent. Next, the participants were randomly assigned to a type of hazard (i.e., tornado, snow squall, or dust storm) and one of four message types (i.e., control, enhanced text, enhanced graphic, or enhanced text and graphic). After exposure to the message, participants answered a series of questions about their perceptions of the message, their perceptions of self-efficacy and response efficacy, their prior HW experience, and their background/demographics. The data reported in this study are part of a larger study; however, the data reported in this paper were analyzed independently from the other variables. The questionnaire took approximately 15–20 min to complete. Participants received incentives through Qualtrics to thank them for their time.
Data were analyzed using the IBM SPSS Statistics, version 28, software. Data were reviewed and cleaned prior to analysis. Composite variables were created for each construct (mean). In addition, we recoded prior HW experience using a median split where 1 = low prior HW experience and 2 = high prior HW experience. Descriptive analysis included frequency, mean, median, and standard deviation. Inferential analysis included analysis of variance (ANOVA) to examine main and interaction effects with significance where p < 0.05. Post hoc tests (Bonferroni) were conducted to identify statistically significant differences (main effects, interaction effects, and Bonferroni post hoc comparisons) between the specific message types.
5. Results
A series of ANOVAs were used to determine the effects of the hazard type (tornado, snow squall, or dust storm), message type (control, enhanced graphic, enhanced text, and enhanced graphic and text), and prior HW experience level on the participants’ perceptions of the given message (understanding, believing, personalizing, deciding, milling, self-efficacy, and response efficacy). First, we discuss the main effects for the ANOVA by hazard type. Second, we discuss the main effects for the ANOVA by message type. Third, we discuss the main effects for the ANOVA by prior HW experience. Last, we describe the interaction effects for all two-way and three-way interactions. The two tables below provide the results for the estimated marginal means (EMM) and standard errors (SE) (Table 3) between-subjects effects (Table 4) and for message outcomes by message type and HW experience. The raw means M and standard deviations (SD) for message outcomes by hazard type, message type, and prior HW experience are also included in Table A1 of appendix A. Using thresholds by Cohen (1988), we interpret magnitudes for effect sizes (partial eta-squared) as 0.01 = small, 0.06 = medium, and 0.14 = large.
Estimated marginal means (EMM) and standard errors (SE) for message outcomes by message type and prior HW experience.
Effects of hazard type, message format, and hazard warning experience on warning-message outcomes. Here, SS = sum of squares; df = degrees of freedom, MS = mean squares, and effect size is given by η2 or partial η2. An asterisk indicates significance at p < 0.05, as indicated in the second-to-last column.
a. The main effects of hazard type on message perception outcomes
RQ1 investigates how the type of hazard presented in the message affects message perception outcomes. As seen in Table 4, we found no significant main effects for hazard type except for personalization, suggesting participants’ levels of understanding, believing, deciding, milling, self-efficacy, and response efficacy are not significantly different depending on the type of hazard the messages reflected.
We did find a significant main effect for personalization: F(2, 1026) = 6.95, p = 0.001, and η2 = 0.013 (small effect size). Bonferroni post hoc comparisons found differences in the levels of personalization based on the hazard type portrayed in the message (i.e., tornado, snow squall, or dust storm). Figure 3 demonstrates that the tornado group (M = 3.40) has significantly higher levels of personalization as compared with the dust storm group (M = 3.10; p < 0.001).
Main effects of personalization on hazard type.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-22-0092.1
b. The main effects of message type on message perception outcomes
RQ2 investigates whether the type of message (control, enhanced graphic, enhanced text, and the combined format) affects the message perception outcomes. As shown in Table 4, we found significant main effects between the message type in levels of all outcome variables except personalization. This includes significant main effects for message type on understanding [F(3, 1026) = 14.27, p = 0.001, and η2 = 0.04], believing [F(3, 1026) = 5.99, p = 0.001, and η2 = 0.017], deciding [F(3, 1026) = 9.64, p = 0.001, and η2 = 0.027], milling [F(3, 1026) = 2.83, p = 0.04, and η2 = 0.008]; self-efficacy [F(3, 1026) = 18.04, p < 0.001, and η2 = 0.050], and response efficacy [F(3, 1026) = 7.42, p = 0.001, and η2 = 0.021]. We present the estimated marginal means of the Bonferroni post hoc comparisons for message type on message perceptions in Fig. 4.
Estimated marginal means for message type on message perception outcomes; outcomes with significant main effects for message type are indicated by three asterisks (p < 0.001), two asterisks (p < 0.01), or one asterisk (p < 0.05) on the x axis.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-22-0092.1
1) Message type and understanding
The Bonferroni post hoc comparisons (Table 3; Fig. 4) show the respondents who received the message with both enhanced graphic and text (combined format) had significantly higher levels of understanding (M = 4.41) relative to those who received the standard practice or control (M = 4.01; p < 0.001) and the enhanced graphic (graphic) message (M = 4.18; p < 0.01). Those who received the enhanced text (text) message (M = 4.32; p < 0.001) and the enhanced graphic (graphic) message (M = 4.18; p < 0.05) also showed significantly higher levels of understanding relative to those who received the standard (control) message. While the mean for the message containing both enhanced graphic and text (combined format) is higher than the enhanced text message (by 0.9), the difference was nonsignificant.
2) Message type and belief
The Bonferroni post hoc comparisons (Table 3; Fig. 4) show the combined format message (M = 4.29; p < 0.01) and the text message (M = 4.29; p < 0.01) resulted in significantly higher levels of belief as compared with the control message (M = 4.04).
3) Message type and deciding
The Bonferroni post hoc comparisons (Table 3; Fig. 4) show the combined format message (M = 4.35; p < 0.001), text message (M = 4.23; p < 0.001), and the graphic message (M = 4.22; p < 0.01), resulted in significantly higher levels of deciding as compared with the control (M = 3.93).
4) Message type and milling
The Bonferroni post hoc comparisons (Table 3; Fig. 4) show the text message (M = 3.50) showed significantly lower levels of milling relative to control message (M = 3.75; p < 0.05) and the graphic message (M = 3.74; p < 0.05).
5) Message type and self-efficacy
The Bonferroni post hoc comparisons (Table 3; Fig. 4) show the combined format message (M = 4.48; p < 0.001), the text message (M = 4.44; p < 0.001), and the graphic message (M = 4.31; p < 0.001) had significantly higher levels of self-efficacy relative to the control message (M = 3.98).
6) Message type and response efficacy
The Bonferroni post hoc comparisons (Table 3; Fig. 4) show the combined format message (M = 4.33; p < 0.001), the text message (M = 4.28; p < 0.01), and graphic message (M = 4.20; p < 0.05) result in significantly higher levels of response efficacy as compared with the standard, or control message (M = 3.99).
c. The main effects of prior HW experience on message perception outcomes
RQ3 investigates if the level of prior HW experience affects the message perception outcomes. We found significant main effects for prior HW experience levels on each of the message perception outcomes except personalization. Specifically, we found significant differences in participants’ levels of understanding [F(1, 1026) = 61.00, p = < 0.001, and η2 = 0.056], believing [F(1, 1026) = 28.13, p = 0.001, and η2 = 0.038], deciding [F(1, 1026) = 67.006, p = 0.001, and η2 = 0.061], self-efficacy [F(1, 1026) = 60.00, p < 0.001, and η2 = 0.055], and response efficacy [F(1, 1026) = 46.60, p = 0.001, and η2 = 0.043] based on their prior HW experience.
Bonferroni post hoc comparisons (Fig. 5; Table 3) demonstrated understanding to be higher for those with high HW experience (M = 4.41) as compared with those with low HW experience (M = 4.05; p < 0.001). Perceptions of belief were also higher for those with high HW experience (M = 4.35) as compared with those with low HW experience (M = 4.03; p < 0.001). Deciding was higher for those with high HW experience (M = 4.41) as compared with those with low HW experience (M = 3.95; p < 0.001). Similarly, among both efficacy outcomes, we found that self-efficacy was higher for those with high HW experience (M = 4.50) as compared with those with low HW experience (M = 4.10; p < 0.001), and response efficacy was higher for those with high HW experience (M = 4.38) as compared with those with low HW experience (M = 4.02; p < 0.001).
Estimated marginal means for prior hazard experience level on message perception outcomes; outcomes with significant main effects for prior HW experience are indicated by three asterisks (p < 0.001), two asterisks (p < 0.01), or one asterisk (p < 0.05) on the x axis.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-22-0092.1
d. Message type, hazard type, and prior HW experience
RQ4 investigates whether either the type of hazard or the participants’ level of prior hazard warning experience modifies the relationship between message type and the message outcomes.
1) Does hazard type modify the relationship between message type and message outcome?
As shown in Table 3, there were no significant two-way interactions for hazard type × message type (understanding, p = 0.32; believing, p = 0.56; personalization, p = 0.95; deciding, p = 0.51; milling p = 0.28; self-efficacy, p = 0.63; response efficacy, p = 0.30) or hazard type × prior HW experience (understanding, p = 0.38; believing, p = 0.35; personalization, p = 0.11; deciding, p = 0.37; milling, p = 0.16; self-efficacy, p = 0.37; response efficacy, p = 0.66) for any of the outcome variables.
2) Does prior HW experience modify the relationship between message type and message outcome?
Next, we explore potential interaction effects among message type and prior HW experience to understand whether the former’s effect on message perceptions varies across levels of prior HW experience (see Table 3).
As shown in Table 3, we found significant interaction effects for message type × prior HW experience on deciding [F(3, 1026) = 2.80, p = 0.04, and η2 = 0.008] and self-efficacy [F(3, 1026) = 2.96, p = 0.03, and η2 = 0.009]. However, we found nonsignificant interaction effects for message type × prior HW experience on understanding [F(3, 1026) = 2.46, p = 0.06, and η2 = 0.007], believing [F(3, 1026) = 1.56, p = 0.20, and η2 = 0.005], personalization [F(3, 1026) = 0.58, p = 0.63, and η2 = 0.002], milling [F(3, 1026) = 1.01, p = 0.39, and η2 = 0.003], and response efficacy [F(3, 1026) = 1.14, p = 0.33, and η2 = 0.003].
The results from a series of Bonferroni comparisons and simple slopes plots show for deciding and self-efficacy, prior HW experience modified the effect of message type on message perception outcomes. The effect of message type, for each message type except the combined format message, was found to be different depending on the level of prior HW experience participants possessed. Specifically, when receiving the control, graphic, or text message, the participants with lower prior HW experience showed significantly lower levels of deciding and self-efficacy than those with higher prior HW experience. However, no differences were seen when those with a lower level of prior HW experience received the combined format message. As Fig. 6 demonstrates (dark red flatter line), the combined format message produced the highest levels in deciding and self-efficacy for recipients regardless of their prior experience level.
Interaction effects of message type × prior HW experience on significant message perception outcomes; note that multiple pairwise comparisons were conducted on significant outcomes only.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-22-0092.1
In summary, prior HW experience significantly modified the relationship between message type and deciding, self-efficacy, understanding, belief, and response efficacy. The results suggest the combined format message elicits the highest levels in these perceptual outcomes for the most people, despite their experience with the hazard or whether the messages are warning for tornadoes, snow squalls, or dust storms. We also report three-way interaction information in appendix B, which contains an expanded write up of our results to provide interested readers with more nuance and detail about our results.
6. Discussion and conclusions
The results of our study provide insights relating to the type of hazard warned in a message, the inclusion of protective action guidance and its placement in a message, and the influence of prior HW experience on message perceptions. We first determined that hazard type did not have meaningful effects on participant message perceptions, suggesting that participants rate outcomes similarly regardless of the type of hazard these messages are designed for. Thus, NWS experimental message products in their existing form may serve as an effective tool to inform the public about a hazard during an imminent threat event. This is particularly important for communicating about hazards that have more recently designed experimental products, such as dust storm and snow squall (NWS 2020), or when a population is exposed to unfamiliar hazards. Our research may suggest these NWS experimental products may help to provide content that informs participants, regardless of prior hazard experience, about the hazard and its impacts.
Second, we determined that message type, or the structure of the message, affects message perception outcomes. Our findings suggested the messages that were enhanced to include protective action guidance in either the graphic portion, text portion, or in both portions of a message, elicited higher levels of participant understanding, believing, ability to decide, perceived self-efficacy, and perceived response efficacy, and decreased milling in comparison with the standard or control message. Importantly, we found that message perception outcomes were higher for messages that included the manipulated structure of the enhanced text and messages that include both enhanced text and graphic in comparison with the message that included only the enhanced graphic or control. This finding differs from Sutton et al.’s (2021) work, who indicated the inclusion of the protective action guidance, whether in the text, graphic, or both, influenced message perceptions. Our findings suggested minimal differences between the control and the enhanced graphic. Perhaps, more details are needed in the enhanced graphic or more icons and visuals to help explain about the threat.
Prior warning research has not taken into account the effect that participant prior HW experience has on warning-message perceptions (Sutton et al. 2021; Wood et al. 2018). In this study we found that for all message types, higher levels of prior HW experience leads to higher message perception outcomes. That is, participants with high levels of prior HW experience had significantly higher message understanding, believing, deciding, self-efficacy, and response efficacy, and less milling than those with low HW experience. Prior experiences with a hazard have a powerful impact on how people respond to risk and risk information (Demuth 2018; Greening et al. 1996; Lindell and Perry 2012; Weinstein 1989). Scholars have provided evidence that those with higher levels of experience tend to have stronger reactions to risk-based messages, and tend to have an increased understanding, ability to make decisions, and motivations to take action (Demuth 2018; Lindell and Perry 2012).
Our results indicated the message including protective action content in the “both” message elicited the greatest understanding, believing, deciding, self-efficacy, and response efficacy, for the participants with lower HW experience. The text, graphic, and control messages instead tend to better serve those with high HW experience. It is possible that the repetition of protective action guidance in both the structure of the text and the graphic of the message reinforces information to unfamiliar audiences. Thus, we recommend that messages include protective action content in both the text and the graphic portion to inform and motivate individuals who have both high hazard warning experience and low hazard warning experience.
The effect sizes for message type and most of the message perception outcomes were small to medium. It is important to note that small effects can make large differences in the numbers of people who may be able to act when exposed to a warning message in response to an imminent threat such as a tornado, dust storm, or snow squall, a finding that is consistent with those identified in previous research (Sutton et al. 2021). The largest effect size was found for self-efficacy, and it suggests that exposure to protective action guidance information leads to greater confidence in one’s ability to protect themselves during these hazard events. Larger effect sizes were found for main effects of prior HW experience (largest for deciding), which our findings also suggest modifies the relationship between message type and numerous outcomes, and thus may serve as a potential confounder in message manipulation studies.
a. Theoretical implications
This research contributes to warning response theory by including prior HW experience and varying hazard types as independent variables. Additionally, this work extends prior research by investigating how manipulations of message structure affect message perceptions based on hazard type and prior HW experience levels. We found that the inclusion of protective action guidance information elicits higher message perceptions for understanding, deciding, believing, self-efficacy, and response efficacy. However, for those who lack prior HW experience, it is critical to deliver information in both the text and graphic format. Through this study, we found that messages that include informative protective action guidance, lead to the highest understanding, deciding, believing, self-efficacy, and response efficacy—demonstrating that a well written message will work well, regardless of hazard type or prior HW experience.
b. Practical implications
The National Weather Service has continued to develop experimental products to communicate to the public about imminent threats disseminated through Twitter (the most recent additions including dust storm, snow squall, and high wind). Social media channels include technological affordances that allow for the inclusion of graphical information. With this, there is an opportunity to both inform the public about severe weather conditions as well as to motivate appropriate protective actions. From this research, we recommend that the NWS Storm Prediction Center modify existing and future experimental warning products to include protective actions in both the graphic and the text portions of a tweet.
c. Limitations
Nonprobability samples have bias and limitations (e.g., potential exclusion, selection, and participation bias), and readers should be cautioned when attempting to generalize the findings of this study to larger populations. Although the sample for the study was intended to match census characteristics for age, gender, and race, members of the population may be excluded, which is a limitation of nonprobability sampling and online surveys. However, through the use of experimental design, multiple hazard types, varying levels of prior HW experience, and clear effects of the inclusion of protective action guidance suggest these findings are likely to be replicated in future studies. It is also important to address that we did not measure for behavioral responses, nor do we measure for behavioral intent; however, our study addresses key motivators leading to important behavioral outcomes, message perceptions, and efficacy. Another limitation of this study was the design and use of Twitter messages to disseminate risk information. While online survey respondents tend to be more communication savvy, this study was not about how these respondents interact with Twitter. Instead, we focused on how the respondents perceive these messages and if they believed the messages provided enough information to take protective actions. Last, we recognize ecological validity of the messages themselves as a limitation for this study. For example, if people are driving at the time where there is a tornado, snow squall, or dust storm, they should not be reading Twitter. However, our results have less to do with the timing of the delivery of the message via Twitter and more to do with how the design of the message affects message receiver perceptions.
d. Future research
Warning response research should continue to examine how messages persuade the public to take action in response to message exposure in real life conditions. Thus, future postevent survey and field research should include accounts of messages received by warned populations. Future research may also explore why the message with enhanced content in both the text and the graphic was the most effective with those with low prior HW experience. Although our population was matched to census demographics for the three cities (Atlanta, Buffalo, and Phoenix), we recommend expanding the population to those who might not be included in these areas. Perhaps future research could attempt to collect survey data through mail in surveys and/or by phone. These different survey techniques may expand the scope of the population to noncommunication savvy groups and older generations. To investigate this further, we suggest using eye tracking methods that will capture what facets of a message affects visual attention for populations with both high and low HW experience. Additionally, the enhancement to these messages focused on the inclusion of content; future research should manipulate other design elements such as color, use of icons, types of maps, and other placement options. Last, it will be important to understand the extent to which experimental products are utilized by the public versus those who are within NWS partner organizations and may make use of population exposure information contained in the graphic portion of the standard message. While a single warning product cannot be all things to all people, a tweet has the potential to serve the purposes of many audiences and motivate protective actions that can save lives.
Acknowledgments.
This project is supported by the NOAA VORTEX-SE Grant NA21OAR4590124.
Data availability statement.
Anonymized data may be made available upon request.
APPENDIX A
Additional Statistics
Table A1 presents the raw means M and standard deviations (SD) for message outcomes by hazard type, message type, and prior HW experience, allowing examination of interaction effects among them.
Raw means M and standard deviations (SD) for message outcomes by hazard, condition, and prior hazard warning experience.
APPENDIX B
Interaction Effects: Message Type and Prior HW Experience
Below, we provide a full, detailed write-up describing “How does the type of hazard or the level of the participant’s prior hazard warning experience modify the relationship between message type and message outcomes?”
We explore potential interaction effects among message type and prior HW experience to understand whether the former’s effect on message perceptions varies across levels of prior HW experience (see Table 3, Fig. B1, and Table A1). Below, we report the interactions and Bonferroni adjusted comparisons for each outcome separately, excluding personalization (p = 0.63) and milling (p = 0.39).
Interaction effects: message type × prior HW experience on message outcomes. The asterisk indicates a significant interaction effect at p < 0.05.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-22-0092.1
a. Message type × prior HW experience on understanding
We found interaction effects between message type × prior HW experience to be nonsignificant, with F(3, 1026)= 2.46, p = 0.061, and η2 = 0.007. However, Bonferroni post hoc comparisons revealed that at low levels of prior HW experience (red bars) the combined format message (M = 4.34) results in significantly higher levels of understanding as compared with the control (M = 3.76; p < 0.001) and the graphic message (M = 3.97; p < 0.001). At low HW experience, the combined format message also shows higher levels of understanding (M = 0.23) than the text, but the difference was only approaching significance (M = 4.11; p = 0.074). At high HW experience (blue bars), we see only the text message (M = 4.54) resulting in significantly higher means of understanding than the control message (M = 4.24; p = 0.013).
In addition, the Bonferroni post hoc comparisons demonstrated that prior levels of HW experience (i.e., high/low) drove differences in participants ratings of understanding within each message type (except for the combined format message), where the control message showed significantly lower levels of understanding for those with low HW experience (M = 3.76) as compared with those with high HW experience (M = 4.24; p < 0.001). Those who received the graphic and had low HW experience (M = 3.97) showed lower understanding in comparison with those with high HW experience (M = 4.38; p < 0.001). Similarly, the text message showed significantly lower levels of understanding for the low HW experience group (M = 4.11) relative to the high HW experience group (M = 4.53; p < 0.001). We found no significant differences for those who received the combined format message across low/high HW experience levels, where the entire HW experience group rated understanding collectively high.
b. Message type × prior HW experience on belief
The interaction effect of message type × prior HW experience on belief was nonsignificant (p = 0.198). However, we further investigated the Bonferroni post hoc comparisons of message type at each level of HW experience, which showed clear evidence for an effect of message type on belief at low levels of HW experience, with F(3, 1026) = 6.43, p = 0.001, and η2 = 0.018, but not at high levels of HW experience [F(3, 1026) =1.33, p = 0.265, and η2 = 0.004].
As seen in Fig. B1, Table 3, and Table A1, for the low HW experience group, the combined format message (M = 4.22) and the text message (M = 4.12; p < 0.01) resulted in a significantly higher level of belief as compared with the control (M = 3.81; p < 0.001). Moreover, in comparing belief scores within each message type, we again found those with low HW experience and who received the control (M = 3.81); the graphic (M = 3.99), and text messages (M = 4.12) rated belief significantly lower than those receiving the same respective message but instead with high HW experience (control and high HW experience: M = 4.26; p < 0.001) (graphic and high HW experience: M = 4.32; p < 0.001) (text and high HW experience: M = 4.45; p < 0.001). However, we again see no significant difference between the low (M = 4.22) and high HW experience groups (M = 4.36) for the combined format message (p = 0.17) on levels of belief, which are similarly high for both HW experience groups (0.14 difference).
c. Message type × prior HW experience on deciding
Significant interaction effects for message type × prior HW experience were found for deciding, with F(3, 1026) = 2.80, p = 0.039, and η2 = 0.008. As seen in Table 3, Fig. B1, and Table A1, at low levels of HW experience, we found that the combined format (M = 4.23) message results in significantly higher levels of ability to decide as compared with the control (M = 3.60; p < 0.001) and the graphic message (M = 3.94; p < 0.05). The text (M = 4.05; p < 0.001) and graphic (M = 3.94; p < 0.05) messages also result in significantly higher ability to decide when compared with the control (M = 3.60) at low HW experience. At high HW experience, we found no significant differences in ability to decide across the message types.
When comparing the average levels of deciding within each of the message types at high/low HW experience, our results showed a series of significant interaction effects. Similar to the previous outcomes, we found the control message showed significantly lower levels of deciding for those with low HW experience (low HW experience: M = 3.60) as compared with those with high HW experience (high HW experience: M = 4.26; p < 0.001). Those who received the graphic and had low HW experience (low HW experience: M = 3.94) also had significantly lower levels of deciding relative to those with high HW experience (M = 4.50; p < 0.001). While those who received the text or combined format message showed higher scores in deciding combined (across HW experience groups), those who received these enhanced messages and had low HW experience (text and low HW experience: M = 4.05) (both and low experience: M = 4.23) also showed significantly lower levels in deciding relative to their counterparts who received the same message and had high HW experience (text and high HW experience: M = 4.41; p < 0.001) (both and high HW experience: M = 4.47; p = 0.04). Again, the perceived ability to decide are highest for both HW experience groups for the combined format message (low HW experience: M = 4.23; high HW experience: M = 4.47), suggesting that the combined format message elicits the highest levels of deciding for the most people, despite their experience with the hazard warning.
d. Message type × prior HW experience on self-efficacy
As shown in Table 3, Fig. B1, and Table A1, we found significant interaction effects for message type × prior HW experience on differences in self-efficacy, with F(3, 1026) = 2.96, p = 0.03, and η2 = 0.009. At low levels of prior HW experience, the combined format (M = 4.36) message produces significantly higher levels of self-efficacy as compared with the control (M = 3.66; p < 0.001), and the graphic (approaching significance: M = 4.12; p = 0.09). The text message (M = 4.26) also has significantly higher perceived self-efficacy when compared with the control (p < 0.001). At high HW experience, we found significant differences in self-efficacy between the text message (M = 4.61) as compared with the control (M = 4.31; p = 0.029) and approaching significant differences between the combined format message (M = 4.59) as compared with the control (M = 4.31; p = 0.07).
When comparing the average levels of self-efficacy within each message type at high/low HW experience, our results again showed a series of significant differences. At low HW experience, the control message (low HW experience: M = 3.66) shows significantly lower levels of self-efficacy as compared with the high HW experience group that received the same message (high HW experience: M = 4.31; p < 0.001). A similar trend is found for all the enhanced messages, with scores of self-efficacy being lower for those with low HW experience and who received the graphic message (M = 4.12), text message (M = 4.26), and combined format message (M = 4.36), relative to those with higher HW experience receiving the same respective messages (graphic and high HW experience: M = 4.51; p < 0.001) (text and high HW experience: M = 4.61; p < 0.001) (both and high HW experience: M = 4.59; p = 0.04). While these differences are significant, overall, perceived self-efficacy is highest for both experience groups for the combined format message (low experience: M = 4.36; high experience: M = 4.59), suggesting that the combined format message elicits the highest levels of self-efficacy for the most people, despite their experience with the hazard or whether the messages are warning for tornadoes, snow squalls, or dust storms. The control, graphic, and text, messages have higher perceived self-efficacy for those with high levels of experience.
e. Message type × prior HW experience on response efficacy
Although the interaction effects for message type × prior HW experience were nonsignificant (p = 0.33) for response efficacy, we again consider that there is not a global effect of message type at all levels of prior HW experience. Thus, we further investigated the Bonferroni post hoc comparisons, of message type within each level of HW experience, which showed clear evidence for an effect of message type at low HW experience, with F(3, 1026) = 6.96, p = 0.001, and η2 = 0.020, but not at high HW experience, with F(3, 1026) = 1.81, p = 0.143, and η2 = 0.005), on response efficacy. To further illustrate these differences, Fig. B1 presents the marginal means for the low HW experience groups next to the high HW experience groups by message type. As shown, for the low HW experience group, the combined format message (M = 4.22) produced a significantly higher level of response efficacy as compared with the control message (M = 3.75; p < 0.001). The text message (M = 4.05; p = 0.019) and the graphic (M = 4.03; p = 0.036) message also generate higher response efficacy than the control (M = 3.75) message.
When comparing the average levels of response efficacy within each of the message types, our results again showed a series of significant differences: Specifically, those with lower levels of prior HW experience and who received the control message (M = 3.75) showed significantly lower levels of response efficacy as compared with those with higher levels of HW experience (M = 4.23; p < 0.001). For the enhanced messages, those with low HW experience and who received the graphic (M = 4.03) and the text (M = 4.05) messages showed significantly lower levels of response efficacy as compared with their counterparts with high HW experience (graphic and high HW experience: M = 4.37; p < 0.001) (text and high HW experience: M = 4.46; p < 0.001). The difference within the combined format message is approaching significance, with the low HW experience group (M = 4.22) showing slightly lower average in response efficacy than the high HW experience group (M = 4.44; p = 0.056). While the differences between the means within the control, graphic, and text messages are much larger for the high HW experience group than the low HW experience group (on average M = 0.40 difference), the difference within the combined format message is about half the size as compared with the other message types. These findings suggest that for the combined format message, perceptions among both low and high HW experience groups are most similar and highest in ranking response efficacy. Thus, the combined format message generates the highest amount of response efficacy for the most people, regardless of the hazard. The text, graphic, and control messages instead tend to better serve those with high HW experience.
f. Three-way interaction
As shown in Table 3, we find the three-way interaction to be significant for perceptions of milling, with F(6, 1026) = 3.06, p = 0.01, and η2 = 0.018. For interpretation, we isolated the three-way interaction and found that it is only significant under high levels of prior HW experience and for those in the tornado hazard type/group [F(3, 495) = 7.216, p < 0.001, and η2 = 0.042]. As seen in Fig. B2, the results show that participants with high levels of prior tornado warning experience who received the standard or control message had significantly higher milling intention (M = 4.03) than those who instead received the enhanced text message (M = 3.07; p < 0.001) and the message with both enhanced text and graphic (M = 3.04; p < 0.01).
Interaction effects: message type × prior HW experience × hazard type on milling.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-22-0092.1
REFERENCES
Atwood, L. E., and A. M. Major, 1998: Exploring the ‘cry wolf’ hypothesis. Int. J. Mass Emerg. Disasters, 16, 279–302, https://doi.org/10.1177/028072709801600303.
Bandura, A., 2010: Self-efficacy. The Corsini Encyclopedia of Psychology, I. B. Weiner and W. E. Craighead, Eds., John Wiley and Sons, 1–3, https://doi.org/10.1002/9780470479216.corpsy0836.
Becker, J. S., D. Paton, D. M. Johnston, K. R. Ronan, and J. McClure, 2017: The role of prior experience in informing and motivating earthquake preparedness. Int. J. Disaster Risk Reduct., 22, 179–193, https://doi.org/10.1016/j.ijdrr.2017.03.006.
Brown, N. A., C. Orchiston, J. E. Rovins, S. Feldmann-Jensen, and D. Johnston, 2018: An integrative framework for investigating disaster resilience within the hotel sector. J. Hosp. Tourism Manage., 36, 67–75, https://doi.org/10.1016/j.jhtm.2018.07.004.
Cappucci, M., 2019: ‘Something incredible happened last week’: Forecasters credited with saving lives in outbreak with 434 tornado reports. Washington Post, 3 June, https://www.washingtonpost.com/weather/2019/06/03/forecasters-credited-with-saving-lives-outbreak-reported-tornadoes/.
Cappucci, M., J. Feuerstein, and L. Beachum, 2022: Blinding snow squalls lead to deadly pileup on I-81 in Pennsylvania. Washington Post, 30 March, https://www.washingtonpost.com/weather/2022/03/28/snow-squall-pileup-i81-pennsylvania/.
Casteel, M. A., 2016: Communicating increased risk: An empirical investigation of the National Weather Service’s impact-based warnings. Wea. Climate Soc., 8, 219–232, https://doi.org/10.1175/WCAS-D-15-0044.1.
Cohen, J., 1988: Statistical Power Analysis for the Behavioral Sciences. Routledge Academic, 567 pp.
Coombs, W. T., 2009: Crisis, crisis communication, reputation, and rhetoric. Rhetorical and Critical Approaches to Public Relations II, 2nd ed. R. L. Heath, E. L. Toth, and D. Waymer, Eds., Routledge, 249–264.
Dash, N., and H. Gladwin, 2007: Evacuation decision making and behavioral responses: Individual and household. Nat. Hazards Rev., 8, 69–77, https://doi.org/10.1061/(ASCE)1527-6988(2007)8:3(69).
Demuth, J. L., 2018: Explicating experience: Development of a valid scale of past hazard experience for tornadoes. Risk Anal., 38, 1921–1943, https://doi.org/10.1111/risa.12983.
Firozi, P., and M. Cappucci, 2021: At least eight killed, several injured after Utah sandstorm. Washington Post, 26 July, https://www.washingtonpost.com/nation/2021/07/26/sandstorm-utah/.
Frisby, B. N., S. R. Veil, and T. L. Sellnow, 2014: Instructional messages during health-related crises: Essential content for self-protection. Health Commun., 29, 347–354, https://doi.org/10.1080/10410236.2012.755604.
Greening, L., S. J. Dollinger, and G. Pitz, 1996: Adolescents’ perceived risk and personal experience with natural disasters: An evaluation of cognitive heuristics. Acta Psychol., 91, 27–38, https://doi.org/10.1016/0001-6918(94)00040-9.
Janssen, A. P., R. R. Tardif, S. R. Landry, and J. E. Warner, 2006: “Why tell me now?” The public and healthcare providers weigh in on pandemic influenza messages. J. Public Health Manage. Pract., 12, 388–394, https://doi.org/10.1097/00124784-200607000-00013.
Kellens, W., T. Terpstra, and P. D. Maeyer, 2013: Perception and communication of flood risks: A systematic review of empirical research. Risk Anal., 33, 24–49, https://doi.org/10.1111/j.1539-6924.2012.01844.x.
Lindell, M. K., and R. W. Perry, 2012: The protective action decision model: Theoretical modifications and additional evidence. Risk Anal., 32, 616–632, https://doi.org/10.1111/j.1539-6924.2011.01647.x.
Mileti, D. S., and E. M. Beck, 1975: Communication in crisis: Explaining evacuation symbolically. Commun. Res., 2, 24–49, https://doi.org/10.1177/009365027500200102.
Mileti, D. S., and J. H. Sorensen, 1990: Communication of emergency public warnings: A social science perspective and state-of-the art assessment. Oak Ridge National Laboratory Tech. Rep. ORNL-6609, 162 pp., https://doi.org/10.2172/6137387.
Mileti, D. S., and P. W. O’Brien, 1992: Warnings during disaster: Normalizing communicated risk. Soc. Probl., 39, 40–57, https://doi.org/10.2307/3096912.
Mileti, D. S., and L. Peek, 2000: The social psychology of public response to warnings of a nuclear power plant accident. J. Hazard. Mater., 75, 181–194, https://doi.org/10.1016/S0304-3894(00)00179-5.
Milne, S., P. Sheeran, and S. Orbell, 2000: Prediction and intervention in health-related behavior: A meta-analytic review of protection motivation theory. J. Appl. Soc. Psychol., 30, 106–143, https://doi.org/10.1111/j.1559-1816.2000.tb02308.x.
Nigg, J. M., 1987: Communication and behavior: Organizational and individual response to warnings. Sociology of Disasters, R. R. Dynes, B. DeMarchi, and C. Pelanda, Eds., Franco Angeli Libri, 103–117.
NWS, 2016: Experimental severe weather impact graphics. NOAA Product Description Doc., 10 pp., https://nws.weather.gov/products/PDD/ExperimentalSevereWeatherImpactGraphics-PDD.docx.pdf.
NWS, 2020: Experimental severe weather impact graphics. NOAA Product Description Doc., 7 pp., https://nws.weather.gov/products/PDD/PDD_ExpSevereWeatherImpactGraphics_2020.pdf.
O’Keefe, D. J., and J. D. Jensen, 2006: The advantages of compliance or the disadvantages of noncompliance? A meta-analytic review of the relative persuasive effectiveness of gain-framed and loss-framed messages. Communication Yearbook, C. S. Beck, Ed., Vol. 30, Lawrence Erlbaum Associates Publishers, 1–43.
Perreault, M. F., B. J. Houston, and L. Wilkins, 2014: Does scary matter? Testing the effectiveness of new national weather service tornado warning messages. Commun. Stud., 65, 484–499, https://doi.org/10.1080/10510974.2014.956942.
Perry, R. W., 1979: Evacuation decision-making in natural disasters. Mass Emerg., 4, 25–38.
Perry, R. W., M. K. Lindell, and M. R. Greene, 1981: Evacuation Planning in Emergency Management. Lexington Books, 199 pp.
Potter, S. H., P. V. Kreft, P. Milojev, C. Noble, B. Montz, A. Dhellemmes, R. J. Woods, and S. Gauden-Ing, 2018: The influence of impact based severe weather warnings on risk perceptions and intended protective actions. Int. J. Risk Reduct., 30, 34–43, https://doi.org/10.1016/j.ijdrr.2018.03.031.
Ripberger, J. T., C. L. Silva, H. C. Jenkins-Smith, and M. James, 2015: The influence of consequence-based messages on public responses to tornado warnings. Bull. Amer. Meteor. Soc., 96, 577–590, https://doi.org/10.1175/BAMS-D-13-00213.1.
Rogers, R. W., and S. Prentice-Dunn, 1997: Protection motivation theory. Handbook of Health Behavior Research 1: Personal and Social Determinants, D. S. Gochman, Ed., Plenum Press, 113–132.
Schumacher, R. S., D. T. Lindsey, A. B. Schumacher, J. Braun, S. D. Miller, and J. L. Demuth, 2010: Multidisciplinary analysis of an unusual tornado: Meteorology, climatology, and the communication and interpretation of warnings. Wea. Forecasting, 25, 1412–1429, https://doi.org/10.1175/2010WAF2222396.1.
Sellnow, D. D., D. R. Lane, T. L. Sellnow, and R. S. Littlefield, 2017: The IDEA model as a best practice for effective instructional risk and crisis communication. Commun. Stud., 68, 552–567, https://doi.org/10.1080/10510974.2017.1375535.
Shen, L., and E. Bigsby, 2013: The effects of message features: Content, structure, and style. The Sage Handbook of Persuasion: Developments in Theory and Practice, J. P. Dillard and L. Shen, Eds., SAGE Publications, 20–35.
Sutton, J., and L. M. Fischer, 2021: Understanding visual risk communication messages: An analysis of visual attention allocation and think-aloud responses to tornado graphics. Wea. Climate Soc., 13, 173–188, https://doi.org/10.1175/WCAS-D-20-0042.1.
Sutton, J., S. C. Vos, M. M. Wood, and M. Turner, 2018: Designing effective tsunami messages: The role of short messages and fear in warning response. Wea. Climate Soc., 10, 75–87, https://doi.org/10.1175/WCAS-D-17-0032.1.
Sutton, J., L. Fischer, and M. M. Wood, 2021: Tornado warning guidance and graphics: Implications of the inclusion of protective action information on perceptions and efficacy. Wea. Climate Soc., 13, 1003–1014, https://doi.org/10.1175/WCAS-D-21-0097.1.
Tversky, A., and D. Kahneman, 1974: Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty. Science, 185, 1124–1131, https://doi.org/10.1126/science.185.4157.1124.
Weinstein, N. D., 1989: Effects of personal experience on self-protective behavior. Psychol. Bull., 105, 31–50, https://doi.org/10.1037/0033-2909.105.1.31.
Weinstein, N. D., J. E. Lyon, A. J. Rothman, and C. L. Cuite, 2000: Changes in perceived vulnerability following natural disaster. J. Soc. Clin. Psychol., 19, 372–395, https://doi.org/10.1521/jscp.2000.19.3.372.
Williams, C. A., and G. M. Eosco, 2021: Is a consistent message achievable?: Defining “message consistency” for weather enterprise researchers and practitioners. Bull. Amer. Meteor. Soc., 102, E279–E295, https://doi.org/10.1175/BAMS-D-18-0250.1.
Witte, K., 1996: Fear as motivator, fear as inhibitor: Using the extended parallel process model to explain fear appeal successes and failures. Handbook of Communication and Emotion, P. A. Andersen and L. K. Guerrero, Eds., Elsevier, 423–450.
Wood, M. M., D. S. Mileti, H. Bean, B. F. Liu, J. Sutton, and S. Madden, 2018: Milling and public warnings. Environ. Behav., 50, 535–566, https://doi.org/10.1177/0013916517709561.
