Abstract

Weather hazards in the United States inflict both personal and economic tolls on the public. Communicating warnings about weather hazards is an important duty of TV weathercasters. Televised weather warnings are typically conveyed through live radar, live coverage, and warning scrolls. However, these traditional approaches may not be entirely effective given the limited attention some members of the public pay to these warnings. A study comparing individual responses to a traditional warning, an animated warning, and an audio warning was undertaken to evaluate the impact of delivery methods on viewer attention, retention, and preferences during viewing of severe weather warnings. A Tobii T60 eye tracker was used to document visual interactions with on-screen warnings and surveys were used to collect evidence of warning retention and preference. Demographic variables were also collected to describe the study population. Results indicate that viewers of the animated warning retained more pertinent information about the tornado warning than viewers of the traditional warning, and retention during the traditional warning was equivalent to that of the audio warning. In addition, gaze patterns for the traditional warning were much more diffuse than for the animated warning, suggesting that attention was more focused on the animation than the live video. In addition, modifications to reduce visual complexity of traditional warnings may positively impact viewer attention to individual warning elements. Future studies will consider the effectiveness of a hybrid warning containing both traditional and animated components. The current research study can be used to advance current severe weather warning communication techniques and increase public awareness during severe weather events.

1. Introduction

Severe weather warnings have traditionally been disseminated through the radio or television, with recent innovations capitalizing on the high rate of cell phone ownership. In live televised warnings, traditional elements such as weathercaster reporting, radar imagery, and warning scrolls appearing at the bottom of the television screen are typically included (WMO 2005). One recent development designed to enhance the effectiveness of severe weather warnings is the addition of text that contains enhanced “storm impact” information (NWS 2012). Early research on the effectiveness of this extra impact information is quite promising (Hudson et al. 2015). Wireless emergency alerts (WEAs) are another recent development used to warn of severe weather, especially tornadoes. WEAs are brief text messages sent by the wireless carriers to smartphones in the affected area (National Research Council 2011). In this way, WEAs are very geographically targeted. Two drawbacks to the use of WEAs, however, limit their potential effectiveness. First, only some cell phones are capable of receiving WEAs; older phones typically do not have the software necessary to receive the messages. Second, as noted by Brotzge and Donner (2013), WEAs will not reach everyone in an affected area simply due to a variety of demographic factors that influence who is likely to own a WEA-capable phone. Given these limitations, severe weather warnings distributed through the more traditional channels of televised broadcasts and radio continue to play a very important role (Schmidlin et al. 2009; Sherman-Morris 2010).

Televised severe weather warning broadcasts typically utilize a number of visual elements to communicate information. Television producers assume that viewers have the ability to attend to both visual elements and verbal information being presented by the weathercaster simultaneously (Bergen et al. 2005). While accessing verbal and visual information simultaneously is quite feasible (Baddeley and Hitch 1974), attending to multiple visual elements at once represents a challenging visual task (VanRullen et al. 2007). As warning broadcasts become more visually complex through the addition of multiple visual elements, (e.g., radar, live footage, or warning scrolls), a pressing question is how this extra information influences the receiver. Evaluating whether the additional content provides added benefit to communicating warning information or is instead an unintended distraction from effective communication becomes essential (Josephson and Holmes 2006). In some instances, less information or fewer methods of conveying information may actually be more effective for enhancing viewer understanding (Josephson and Holmes 2006). Reduction of screen clutter represents a potential for improved viewer attention since the inclusion of multiple visual elements has been shown to negatively impact the effectiveness of the main broadcast message (IPG Media Lab 2011) and reduces viewer comprehension of the presented information (Bergen et al. 2005). While attempts to incorporate multiple modes of communication into severe weather warnings are common, complex warnings have not been entirely successful at encouraging public action (NOAA 2009; Lazo et al. 2009). A better understanding of how the public engages and interacts with traditional broadcasts, and the elements that both contribute to and detract from warning message comprehension, is necessary in order to increase the effectiveness of current severe weather warnings.

Although multiple media are used to disseminate weather warnings, televised warnings continue to be the most utilized source of information during severe weather outbreaks (Wolf 2009). Despite rapidly developing advancements made to severe weather warning delivery and communication, significant impact to property and lives continues to be a major consequence associated with severe weather throughout the United States (Folger 2013). Improvements to current warning mechanisms may lead to additional avoidance of these consequences, and result in a reduction of the lives and property lost during severe weather events, or from natural hazards in general.

Natural hazards exist in many forms throughout the United States (USGS 2007); severe weather in the form of blizzards, hurricanes, and thunderstorms poses the most frequently experienced risk (Smith 2013). Of these, thunderstorms occur most frequently throughout the United States (NOAA 2010). Thunderstorms produce lightning, tornadoes, straight-line winds, floods, and hail; together, these average about three billion dollars in insured losses and over 65 fatalities each year (NOAA 2010). In recent years, these losses have grown considerably, and tornadoes have taken a lead role in damage associated with severe thunderstorms (Munich Re 2012). The 2011 outbreak in Joplin, Missouri, which claimed 161 lives, and the 2013 event in Moore, Oklahoma, that killed 24 (Kuligowski et al. 2013) illustrate recent events that have resulted in record property losses and fatalities associated with tornadoes (NCDC 2013).

The ability to reduce the impact of severe thunderstorms and tornadoes typically relies on efforts that focus on coupling storm detection with providing effective warnings. Detection and warning systems developed by the National Oceanic and Atmospheric Administration (NOAA) and the National Weather Service (NWS), in conjunction with institutions focusing on severe weather detection and prediction, are examples of these efforts (Folger 2013; NOAA 2013). Indeed, a primary NOAA mission is to deliver daily weather information and storm warnings in conjunction with the NWS. The NWS supplies a wide range of data and related weather products that are then used by both the government and private sector (NWS 2013). Media outlets in turn utilize these data to communicate weather information to the public through a variety of sources, including television, radio, smartphone applications, and text alerts. Regardless of the information source, however, in extreme weather events a concise and timely forecast is crucial for decision-making to reduce losses to property and loss of life (Mileti and Sorensen 1990).

The research reported here investigated the effectiveness of warning elements and delivery methods associated with traditional televised severe weather warning broadcasts. More specifically, we examined how a tornado warning, presented in one of three formats, influenced both attention to the warning and memory of the warning, and whether or not those outcomes were influenced by knowledge of severe weather or previous experience with severe weather. One aim of our study was to investigate whether the use of animation might facilitate memory and understanding of a severe weather warning. We used a tornado warning due to the relatively rapid nature of a tornadic event, as compared to other forms of severe weather such as a hurricane or blizzard (Erickson and Brooks 2006). We also focused on television broadcasts given ample research showing that most people become aware of a tornado through local television or a siren (Schmidlin et al. 2009; Sherman-Morris 2010).

It is important to note that in our study we focused on the retention of warning information (a measure of comprehension) rather than exploring possible next steps, such as taking appropriate protective action. This is an important distinction, as Mileti’s (1999) well-regarded model of warning response makes clear. According to Mileti’s model, there are seven steps in the warning process: receiving the warning, comprehending the warning, believing the warning, confirming the threat, personalizing the threat, and then determining whether protective action is needed and, if so, if it is feasible. Receiving and comprehending a warning are important first steps, of course, toward the ultimate goal of taking protective action, but as Mileti’s model makes clear, simply comprehending a warning does not guarantee that one will take protective action. Given the paucity of research examining the influence of animation on how message recipients process, understand, and ultimately act on a warning, however, we thought it reasonable to limit the scope of our explanatory study to warning message comprehension.

Our participants viewed and/or listened to a tornado warning in one of three formats: a traditional televised warning, an animated warning, and an audio warning. All three versions of the warning contained the identical warning information because the warning narrative that was read in all three versions was the same. The televised and audio warnings were chosen given their continuing prominent role in the weather warning enterprise (Hammer and Schmidlin 2002). The animated warning was included to examine whether the relevant visual aspects of the severe weather warning are enough, by themselves, to focus attention and promote satisfactory warning comprehension or whether the presence of a weathercaster is necessary to help direct and focus viewer attention. As mentioned earlier, televised warning broadcasts are visually complex, and this increased complexity may hinder comprehension (Josephson and Holmes 2006), especially if some of the presented information is irrelevant (Canham and Hegarty 2010). The presence of an on-screen weathercaster may, therefore, help the viewer to focus his or her attention on the most relevant aspect of the warning message. Conversely, although certainly counterintuitive, it is possible that the weathercaster her- or himself may simply add to the visual complexity, therefore providing another potentially distracting visual input. An animated warning condition was therefore used that presented the same visual information as the televised version but that did not require the use of an onscreen weathercaster.

The use of animation to enhance the presentation of weather information is not new. Bech et al. (2010), for instance, discuss the use of an animated character to present weather forecasts on mobile media in Catalonia, Spain. The animation used in the present study, however, did not employ animated characters. Instead, the present study utilized animation in a manner similar to that used by Wagenaar et al. (1985). In their first two experiments, Wagenaar et al. found that memory for Dutch Weather Bureau radio presentations was quite low, with few differences as a function of type of television presentation: displaying elements of the message on a map, having an on-screen presenter delivering the information, or having a silent presenter placing symbols on a map in correspondence to the spoken message. In their third experiment, however, Wagenaar et al. included an animated condition consisting of pictograms for simple weather phenomena (e.g., clouds, wind, rain, snow) that moved across the map, representing motion. Interestingly, the animated pictograms significantly improved recall, compared to the silent presenter condition. Although overall recall of the weather reports was quite low, their results suggest that the use of animation can improve the retention of weather information.

In addition to the empirical evidence showing that animation may improve memory for weather information, individuals also show a distinct preference for animation. Hegarty et al. (2009), for instance, found in two studies that both undergraduates and more experienced U.S. Navy forecasters expressed a preference for both realism and animations in visual displays depicting both weather and dynamic processes (e.g., how a pump works). Interestingly, however, a third study showed that increased realism increased reaction time on a weather inference task for both the undergraduates and the Navy meteorologists and also decreased the accuracy of the undergraduates. Although Hegarty et al. did not specifically study the influence of animation, their results do suggest caution in reading too much into participant preferences. In other words, what people think they want might not necessarily produce better performance.

Indeed, other research suggests that animations do not always provide the added benefit that some believe should exist. Much of the empirical work on the effectiveness of animation comes from using animated displays for training purposes. This literature suggests that, if an animation condition and a condition involving a series of static diagrams are equated in terms of the amount of information they both depict, an animation provides little advantage (Hegarty and Kriz 2008; Tversky et al. 2002). In other words, as long as the same information is presented in both, little advantage seems to exist for animations. Mayer et al. (2005) even found that animations hurt performance, relative to a static diagram condition. One limitation of this work, however, is that it has largely been focused on teaching people about how things work. It is an empirical question whether or not weather displays that include animation, but that otherwise are informationally equivalent to the nonanimation displays, show the same findings. As noted by Hegarty et al. (2009), an important goal is to assess whether or not these findings are also applicable to geospatial displays, such as understanding the weather.

To assess the impact of our three warning conditions on warning comprehension, we used a variety of dependent measures. We hope that the results of our study can inform best practices for constructing a severe weather warning that positively impacts participant retention. Therefore, in addition to measuring participant preferences, retention of information from the tornado warning, and responses to open-ended questions, we also measured eye tracking for those participants who viewed a video warning. Eye tracking gaze data were used to identify participants’ gaze fixations during viewing (Tobii Eye Tracking 2010). A participant’s fixation time spent on a particular detail is often indicative of the attention that is being paid to that area or item being viewed (Rayner 1998). Indeed, Duchowski (2002) provides an excellent overview of the use of eye tracking to measure attention involved in a variety of complex cognitive tasks including scene perception, perception of film, and even complex human tasks such as driving and flying simulations. Our eye tracking data allow for the comparison of the gaze patterns of multiple participants for each video.

2. Method

a. Participants

The participants were recruited through listserv and e-mail solicitation at a large Midwestern institution. Ninety members of the community (53 females; 37 males), including undergraduate and graduate students, faculty, and staff, were paid $20 each for their participation. Eighty-three percent of participants were students; ages of the participants ranged from 18 to 46 years, with a median of 23 years. Study participants were 76% Caucasian, 18% Asian, 4% Black, and 2% self-coded as unspecified. The majority of our participants (74%) were not science majors and reported having taken no Earth science–related coursework.

b. Materials

All three warnings were based on a portion of an actual severe weather warning broadcast delivered live as an interruption to a normal television broadcast in the Colorado Springs, Colorado, viewing area. The ”Traditional” warning was the actual 95-s video broadcast. The ”Animated” warning was based entirely on the traditional broadcast video and was created specifically for this study by a coauthor (Fig. 1). The Animated warning included the original narration from the Traditional warning, plus animated representations of the same imagery used in the broadcast. The animations were created using a Panasonic PV-GS300 digital camcorder and Apple’s Final Cut Pro. The Animated warning also included images that provided precautionary recommendations, such as avoiding standing near a window and seeking shelter in a lower level. This precautionary information was presented verbally in the Traditional warning, but the informational content of both warnings was the same. Finally, the Audio warning contained the original narration from the Traditional warning without any accompanying visual component.

Fig. 1.

Example of traditional and animated media clips. The audio only clip contains no images and is not illustrated here.

Fig. 1.

Example of traditional and animated media clips. The audio only clip contains no images and is not illustrated here.

c. Procedure

Data were collected in three successive steps during this study; pre–eye tracking, during eye tracking, and post–eye tracking. Table 1 shows the flow of each step of the experiment. First, participants were asked to complete a weather knowledge (Knowledge) questionnaire and weather experience (Experience) questionnaire. Both measures were previously validated in Drost (2013). The Knowledge questionnaire consisted of 11 questions pertaining to tornado development, a presentation of severe weather facts, and recommended precautionary measures. It measures weather knowledge among study participants to determine their understanding of common severe weather event communication. The Experience questionnaire consisted of 12 questions divided into three categories intended to measure an individual’s prior exposure to severe weather events. Two questions asked for participant reaction to the terms “severe weather warning” and “tornado,” four inventoried the participants’ exposure to severe weather, specifically tornadoes, and the remaining six questions evaluated attitudes and emotions toward severe weather and community response to warnings. These final six questions were used to determine the level of trust in weather warnings by the participants. A factor analysis was performed on the questions and as a result the first four of the six questions were retained, yielding a “Trust” scale calculated from the average of these four items (Table 2).

Table 1.

Data source descriptions and timing during the study.

Data source descriptions and timing during the study.
Data source descriptions and timing during the study.
Table 2.

Likert scale statements and average participant (n = 90) responses (1 = strongly agree, 5 = strongly disagree).

Likert scale statements and average participant (n = 90) responses (1 = strongly agree, 5 = strongly disagree).
Likert scale statements and average participant (n = 90) responses (1 = strongly agree, 5 = strongly disagree).

Next, participants randomly received one of the three tornado warnings. Participants in the Audio condition simply heard the tornado warning over speakers, as spoken by the broadcast meteorologist. Participants in both the Traditional and Animated condition were given a brief explanation of the Tobii T60 eye tracker and the tracker was calibrated for each participant before data collection began. The eye tracker was used to capture participant fixations and gaze information on the video monitor. Participants then watched and listened to their respective tornado warning.

After hearing or watching the tornado warning, each participant then completed three additional questionnaires. The tornado warning retention questionnaire (Retention) was made up of 10 multiple choice questions specific to the tornado warning to which each participant was exposed. Participant response to this questionnaire was evaluated by conducting an analysis of variance (ANOVA) to compare Retention scores for the Animated, Traditional, and Audio groups.

Regression was undertaken to evaluate the relationship between the outcome retention variable and other variables. All continuous variables were converted to z scores to ensure normality, and other basic assumptions of regression were evaluated and satisfied. A multiple linear regression was carried out in three steps (Table 3). In the first step, the effect of the treatment was evaluated. Variables over which the participant had some control, such as experience, were included in the second step; variables over which the participant had no control, such as gender, were evaluated in the final step (Table 3).

Table 3.

Summary of hierarchical regression analysis for tornado warning retention (asterisk indicates p ≤ 0.05). B is the unstandardized regression coefficient, SE B is the standard error of B, and β is the standardized regression coefficient. The “Z” beginning a variable denotes standardized values for that variable.

Summary of hierarchical regression analysis for tornado warning retention (asterisk indicates p ≤ 0.05). B is the unstandardized regression coefficient, SE B is the standard error of B, and β is the standardized regression coefficient. The “Z” beginning a variable denotes standardized values for that variable.
Summary of hierarchical regression analysis for tornado warning retention (asterisk indicates p ≤ 0.05). B is the unstandardized regression coefficient, SE B is the standard error of B, and β is the standardized regression coefficient. The “Z” beginning a variable denotes standardized values for that variable.

The Retention questionnaire was used to determine what specific information participants remembered while viewing and/or listening to each warning. The tornado warning preference survey (Preference) asked participants to list in rank order the most helpful resources, aspects, and information contained in the warning model they viewed. Resources included a range of four to seven elements, such as radar imagery, location information, and weathercaster delivery, depending on the warning model viewed. In addition, an open-ended question asking about the most useful elements of the warnings was included. The open-ended question was intended to compare alternative warning elements to those currently in place, and to determine what participants considered most useful, or preferred, during severe weather warnings. The domain experience questionnaire (DEQ) collected information on participants’ educational major, gender, age, ethnicity and family educational background, as well as exposure to college-level Earth science courses.

The study concluded with a series of think-aloud questions. These open-ended questions allowed participants to express their views and opinions about the tornado warning they experienced and their preferences for severe weather communication, if any. The participants were then debriefed and thanked for their assistance.

In addition to the information gathered from participant questionnaires, eye tracking data were also collected to determine the influence of tornado warning types on gaze patterns. Eye tracking gaze data from the Traditional and Animated treatments were used to generate bee swarm data, depicting the location of each participant’s gaze during video viewing. These bee swarm videos were analyzed to determine the extent of attention or distraction experienced by participants during the experiment.

Individual screenshots were extracted from the Tobii T60 eye tracker bee swarm data for both the Traditional and Animated warnings. For each warning, individual scene clips were captured every 5 s to quantify the subject’s gaze, resulting in 15 unique clips for each of the Traditional and Animated videos (Fig. 2). These screenshots were then compared and paired according to the similarity of the content and audio elements. These screenshots displayed station and weathercaster identification, warning level, current storm track and location, and predicted future storm movements, representative of questions asked on the Retention and Preference questionnaires.

Fig. 2.

Series of images showing participant gaze for both the Traditional (Trad) and Animated (Anim) warning. An individual participant’s gaze is indicated by the black dots. Clip Trad1 is representative of a “diffuse” gaze pattern; Trad2 is representative of a “concentrated” gaze pattern. Clip Anim1 is representative of a gaze focusing on multiple elements; Anim2 is representative of a concentrated gaze pattern. For each warning individual clips were taken at 5-s intervals resulting in 15 clips and paired according to the similarity of content and audio.

Fig. 2.

Series of images showing participant gaze for both the Traditional (Trad) and Animated (Anim) warning. An individual participant’s gaze is indicated by the black dots. Clip Trad1 is representative of a “diffuse” gaze pattern; Trad2 is representative of a “concentrated” gaze pattern. Clip Anim1 is representative of a gaze focusing on multiple elements; Anim2 is representative of a concentrated gaze pattern. For each warning individual clips were taken at 5-s intervals resulting in 15 clips and paired according to the similarity of content and audio.

A rubric was developed that allowed researchers to code the viewer’s gaze as either concentrated, where the majority of individuals focused on common elements, or diffuse, where the majority of individuals focused on different elements. Two authors coded all 15 clips for each video. Each set of clips was coded separately, and a reliability check yielded an initial agreement of 84%. Inconsistencies discovered during the coding process were discussed between the two authors and worked through until a mutual coding agreement was reached.

3. Results

a. Knowledge, Experience, and Preference

The results of the Knowledge questionnaire indicated that our sample exhibited moderate levels of understanding of weather [M = 6.5; standard deviation (SD) = 1.5]. Several questions stood out in participant responses. When asked the typical speed of a tornado and how much advance warning is given before a tornado strikes, only 20% of the respondents answered correctly. Somewhat more disturbing is the fact that only 50% of the respondents understood that a “warning” is the most urgent National Weather Service Statement.

In terms of weather experience, a majority of the participants reported personally experiencing some form of severe weather: Results indicated that 86% of the participants experienced a severe weather warning at least once a year while 50% had experienced an actual severe weather event. Of those that experienced a severe weather event within the last two years, 40% experienced a severe thunderstorm, 36% experienced heavy or blowing snow, 23% experienced torrential rain or flooding, 22% experienced damaging winds, and 13% experienced a tornado. Additionally, the participants that reported physically experiencing a tornado estimated that these tornadoes ranged in strength from an EF0 to EF4 (from the enhanced Fujita scale) based on reported damage.

Questions derived from the Experience category used to determine the Trust scale analysis yielded a Cronbach’s alpha of 0.69, acceptable for use as an aggregate sample-level measure. A trust mean score of 2.44 (SD = 0.72) indicated that the groups neither trusted nor distrusted severe weather warnings.

An analysis of variance indicated that no significant differences existed among the three warning groups for knowledge [F(2,87) = 0.455, p = 0.64], experience [F(2,87) = 1.03, p = 0.36], or trust [F(2,87) = 0.872, p = 0.42], suggesting little impact of these variables on the Retention data.

b. Influence of tornado warning type on retention

Results of the ANOVA conducted to compare Retention scores for the Animated, Traditional, and Audio groups indicated a statistically significant difference between groups, F(2,87) = 3.38, p = 0.04. Post hoc tests using the Tukey correction revealed no statistical differences between the Traditional and the Audio groups (p = 0.96). Scores for the Animated group (M = 4.66, SD = 1.10) were significantly higher than the scores for the Traditional group (M = 3.5, SD = 1.66; p = 0.05) and approached statistical significance when compared with the Audio group (M = 3.6, SD = 1.48; p = 0.09). These results suggest that the animated tornado warning had a greater effect on participant retention of information than the traditional warning, and may have had a greater effect than the audio tornado warning. To determine if knowledge, experience, and/or trust impacted Retention results, a multiple linear regression was performed. This analysis revealed that knowledge (beta = −0.052, p = 0.33), experience (beta = 0.178, p = 0.11), and trust (beta = −0.067, p = 0.58) were not significant predictors for retention, indicating that knowledge, experience, and trust had no main effects within the groups. Retention scores for participants were therefore directly related to the individual warning viewed during the study such that the Animated group’s performance was better than that of the Traditional and Audio participants.

As expected from the significant t test, treatment played a statistically significant yet small (ΔR2 = 0.071) role in predicting retention. The addition of experience, knowledge, and trust to the model added a very small amount of explanatory power (ΔR2 = 0.044). Demographic variables had negligible impact on the model (ΔR2 = 0.013), indicating that explained variance was likely not attributable to these individual differences. Overall, only ~12% of the variance was explained by the regression, indicating that other variables—such as working memory capacity—likely impacted retention (Mayer and Moreno 1998).

c. Influence of tornado warning type on preference

Results from the Preference questionnaire revealed that participants who viewed the Traditional and Animated warnings indicated that the radar image was the most helpful resource. Since the Audio participants viewed no video, the option to choose that response was not included in their questionnaire. These participants selected the weathercaster’s live in-studio report of storm location and movement as the most helpful resource.

In respect to what were the most helpful aspects in determining storm movement and location the Traditional and Animated group ranked names of towns and cities first, while the Audio group ranked the weathercaster’s description of storm location and movement first and highways and roads second. This difference may be due to the lack of visual content associated with the audio warning or unfamiliarity with one’s geographical surrounding.

When determining the preferred information when faced with a tornado participants in all three warning types selected the radar image, NWS warning message, and the forecaster’s recommendation as the top three informational sources. Responses also showed that all three groups chose the radar image as the most helpful type of information on which to base an informed response decision.

When asked what was most useful when making decisions in the event of a tornado those participants who viewed the Traditional model most often mentioned radar imagery, storm direction information, and the timing and location of the event as most useful. Participants who viewed the Animated model also mentioned radar imagery and storm direction information, but differed by then mentioning precautionary measures as most useful. The Audio group mentioned timing and location and precautionary measures as most useful and made no mention of radar imagery.

At the conclusion of the study participants were asked to comment on the tornado warning they had previously viewed. This was meant to provide information not easily captured as part of a questionnaire and allow the three groups to express their opinions about the warning model they viewed. The Traditional group frequently expressed feelings of confusion and cited the abundance of information being provided during the warning as “a lot to consider.” The Animated group overall found the animated warning enjoyable, but mentioned that an animation would be “hard to take seriously”. Last, the Audio group mentioned that the lack of visuals contributed to the difficulty in remembering warning information and that they needed to “concentrate on the message” to remember what was being communicated.

d. Influence of tornado warning type on gaze patterns

This comparison of the Traditional and Animated bee swarm data indicated that participants exhibited a diffuse pattern of gaze during 33% (N = 5) of the clips for the Traditional warning. In comparison, participants viewing the Animated warning clips exhibited a diffuse pattern in less than 6% of the selected clips. This indicates that the Animated warning initiated more focused attention on salient elements by participants. This focus on salient elements during the weather warning could be expected to translate to greater retention post viewing.

4. Discussion

Effective severe weather warnings are essential for providing the population with the information needed to take action when threatening weather approaches. Although our study did not explicitly investigate the influence of animation on the likelihood of taking protective action, creating severe weather warnings that are more readily comprehended is a necessary first step in crafting an effective earning (Mileti 1999). Warnings that contain salient information and are delivered to the public quickly provide critical information likely influencing the decision to take precautionary measures to protect lives and property. Traditional severe weather warnings contain multiple elements such as radar, live reports, warning scrolls, and weathercaster recommendations that convey storm information to the population. Our findings show that animation is another element that can convey important information.

Overall, participant knowledge of basic severe weather concepts was moderate. Participants demonstrated a lack of knowledge and understanding of weather statements, tornado speed, and lead time available to take precautionary measures. Understanding the NWS weather statement information and knowing when to take precautionary measures is an important first step to ensuring personal safety during threatening weather. Knowledge of a tornado’s speed and direction figures prominently in determining an individual’s course of action and heightens the need for immediate action. This lack of understanding makes for a potentially disastrous situation.

While the majority of participants had experienced a severe weather warning, only a small percentage had recent exposure to a tornado. As a whole, participant trust in the information contained in the warnings and potential for serious harm was neutral. The neutral attitude toward warning information and potential for threats to their lives and property exposes them to additional danger when combined with the lack of knowledge about severe weather location and movement. Perhaps the high frequency in which participants had experienced warnings and subsequent lack of understanding of the warning messages themselves contribute to the neutral attitude toward the perceived potential for harm.

Our data suggest that the Animated treatment was better than the Traditional one at communicating information, and that the Traditional treatment had surprisingly similar effects to an Audio treatment alone. The animated severe weather warning was shown to statistically impact participant retention of severe weather information when compared to the groups who viewed the traditional or audio warning. Not only did the Animated group retain more of the severe weather warning information contained in the clip, but bee swarm data also indicated increased attention on visually salient warning elements contained within the Animated media clip. Participants viewing the Animated model focused on specific emerging aspects of the warning in a highly concentrated fashion while participants viewing the Traditional model exhibited a more diffuse focus. The amount of concentrated engagement with the warning may ultimately influence knowledge retention.

These findings support a need to further investigate the benefits and risks associated with supplementing traditional warnings with animated elements. Novel elements may initially provide a positive impact in viewer attention or retention of warning information. As a new source of communication, viewer’s attention may be drawn to the novel element, resulting in more effective delivery of weather warnings. However, once the novel element has become commonplace in warning messages, its effectiveness may diminish as the “newness” fades (Mendelson 2001).

Preferences in severe weather warning elements were relatively consistent among all groups. Participants selected radar imagery, weathercaster information, and NWS warning scroll as useful sources of storm intensity, location, and future track information. Although the Animated and Audio groups’ warning models differed slightly with the inclusion of weathercaster recommendations for precautionary measures, all groups consistently rated this element among the most helpful for making decisions during a severe weather event. The Traditional group selected the “recap of weather information” in place of “weathercaster recommendations” since recommendations were not included in the Preference for the traditional media clip. This indicates that updated weather information was of primary consideration to all groups.

Additional elements rated useful among all groups were information associated with names of towns and cities, highways and roads, and county boundary lines. Although this information was ranked highly by all groups, the Audio group differed by selecting the “weathercaster description of location and movement” first. This may be explained by the context of where the viewer would be listening to an audio warning, such as an automobile, and/or to the lack of visual imagery associated with the depiction of these elements.

Preference data suggest that traditional warnings may contain information unhelpful to the viewer, and that excess information may lead to viewer distraction as evidenced by the scattered pattern of the bee swarm data in the Traditional warning clips. In addition, our study indicates that traditional warning messages could be improved by reducing the number of components shown simultaneously, potentially reducing viewer distraction and increasing attention to salient elements. This combination, along with the fact that participant knowledge, experience, and trust did not impact retention during the viewing of the three models, supports the inclusion of novel animated elements within traditional severe weather warnings. Care should be taken however, when including animated elements since participants often commented that a warning that was completely animated would be taken less seriously than a traditional warning.

5. Limitations

One limitation of our current findings is that they speak to the influence of animations on severe weather memory and comprehension. As mentioned earlier, however, a number of steps intervene between a warning message’s reception and a decision to take action based on that decision (Mileti 1999). Clearly, there is a vast difference between a research participant sitting in a room and viewing a weather warning and a person looking outside and seeing dark swirling clouds or hearing the clap of thunder. As previous research has shown (Mileti and Peek 2000; Perry et al. 1981), message confirmation and personalization are important variables that influence how individuals process and respond to warning messages. Research has also shown that other variables, such as the perceived severity of the threat (Riad et al. 1999), proximity to the warned event (Quarantelli 1980; Perry and Lindell 1991), influence of false alarms (Simmons and Sutter 2009), and source credibility (Fiedrich and Burghardt 2007; Perry and Lindell 1991) are also important. Although they are important variables to consider, their lack of inclusion in the present research does nothing to negate our findings: the animated warning more effectively communicated information than did either the Traditional or Audio warning. Clearly, a fruitful avenue of future research would be to explore whether an animated forecast also increases one’s likelihood of taking protective action.

One might also question the relevance of the eye-tracking data, given the rather artificial nature of the experiment. In the real world, recipients of severe weather warnings are likely focused on many activities when a severe weather warning is broadcast on television. These other activities, such as making dinner, doing laundry, or any of a variety of other mundane life events, are likely distracters from paying attention to the television. Other variables such as hearing one’s town name (or not) as being in the path of the tornado also likely influence attention and would impact whether one attends closely to the warning. Keep in mind, however, that such an argument is applicable to any laboratory-based study of weather warnings. We argue that our gaze data are relevant because they show that, in an ideal situation, an animated weather warning does promote more focused gaze and attention and better retention. This situation would likely occur if one were attending to a weather warning and heard one’s town’s name as being directly in the path of the tornado. Such an event likely would prompt very focused attention.

One final point concerns our lack of an experimental condition comparing the presence of a weathercaster and the normal televised visuals to that of a condition with the same visuals but with only a weathercaster voiceover, to see whether the weathercaster him- or herself influenced message comprehension. Although such a weathercaster versus nonweathercaster comparison would have been ideal, we were unable to include such a condition given our use of a real, on-air severe weather broadcast. It is also unclear whether such a comparison would have produced significant differences. Recall that the research by Wagenaar et al. (1985) found no differences for memory of a radio presentation between a condition using an on-screen presenter versus one using a voiceover. Only future research can address this issue.

6. Conclusions

These results support the development of a hybrid severe weather warning model that includes traditional and animated (or novel) elements. To accomplish the development of a hybrid warning, we propose further studies that will incorporate data fielded from a larger population, including heads of households, in order to more aptly represent the general population. These studies should utilize eye tracking gaze data to identify the most salient aspects of traditional and animated warning models and viewer attention information. These data could be combined with retention and preference data in order to validate the effectiveness of a hybrid severe weather warning to positively impact viewer retention of warning information and attention to warning elements. Positive impacts associated with elevated retention and attention may provide a sounder basis for viewer decision-making during severe weather events. These developments should build upon previous work in order to advance warning and communication techniques, leading to more effective public communication during severe weather events.

Acknowledgments

This project was partially funded through a grant to Drost from the Environmental Science and Policy Program at MSU. We thank members of the Geocognition Research Lab who reviewed an earlier draft of this paper.

REFERENCES

REFERENCES
Baddeley
,
A. D.
, and
G.
Hitch
,
1974
: Working memory. The Psychology of Learning and Motivation, G. A. Bower, Ed., Academic Press, 48–79.
Bech
,
J.
,
T.
Molina
,
E.
Vilaclara
, and
J.
Lorente
,
2010
:
Improving TV weather broadcasts with technological advancements: Two cases from a 20 year perspective
.
Meteor. Appl.
,
17
,
142
148
, doi:.
Bergen
,
L.
,
T.
Grimes
, and
D.
Potter
,
2005
:
How attention partitions itself during simultaneous message presentations
.
Hum. Commun. Res.
,
31
,
311
336
, doi:.
Brotzge
,
J.
, and
W. R.
Donner
,
2013
:
The tornado warning process: A review of current research, challenges, and opportunities
.
Bull. Amer. Meteor. Soc.
,
94
,
1715
1733
, doi:.
Canham
,
M.
, and
M.
Hegarty
,
2010
:
Effects of knowledge and display design on comprehension of complex graphics
.
Learn. Instr.
,
20
,
155
166
, doi:.
Drost
,
R.
,
2013
:
Memory and decision-making: Determining action when the sirens sound
.
Wea. Climate Soc.
,
5
,
43
54
, doi:.
Duchowski
,
A. T.
,
2002
:
A breadth-first survey of eye-tracking applications
.
Behav. Res. Methods Instrum. Comput.
,
34
,
455
470
, doi:.
Erickson
,
S. A.
, and
H. E.
Brooks
,
2006
: Lead time and time under tornado warnings: 1986–2004. 23rd Conf. on Severe Local Storms, St. Louis, MO, Amer. Meteor. Soc., 11.5. [Available online at https://ams.confex.com/ams/23SLS/techprogram/paper_115194.htm.]
Fiedrich
,
F.
, and
P.
Burghardt
,
2007
:
Agent-based systems for disaster management
.
Commun. ACM
,
50
,
41
42
, doi:.
Folger
,
P.
,
2013
: Severe thunderstorms and tornadoes in the United States. Congressional Research Service 7-5700, R40097, 23 pp. [Available online at https://www.fas.org/sgp/crs/misc/R40097.pdf.]
Hammer
,
B.
, and
T.
Schmidlin
,
2002
:
Response to warnings during the 3 May 1999 Oklahoma City tornado: Reasons and relative injury rates
.
Wea. Forecasting
,
17
,
577
581
, doi:.
Hegarty
,
M.
, and
S.
Kriz
,
2008
: Effects of knowledge and spatial ability on learning from animation. Learning with Animation: Research Implications for Design, R. Lowe and W. Schnotz, Eds., Cambridge University Press, 3–29.
Hegarty
,
M.
,
H. S.
Smallman
,
A. T.
Stull
, and
M. S.
Canham
,
2009
:
Naïve cartography: How intuitions about display configuration can hurt performance
.
Cartographica
,
44
,
171
186
, doi:.
Hudson
,
M. J.
,
R.
Wagenmaker
,
G.
Mann
,
B.
Smith
,
R.
Thompson
,
J.
Ferree
, and
C.
Entremont
,
2015
: Science that drives service: The NWS impact-based warning demonstration. Third Symp. on Building a Weather-Ready Nation: Enhancing Our Nation’s Readiness, Responsiveness, and Resilience to High Impact Weather Events, Phoenix, AZ, Amer. Meteor. Soc., TJ4.1. [Available online at https://ams.confex.com/ams/95Annual/webprogram/Paper260627.html.]
IPG Media Lab
,
2011
: Are all screens created equal? A research study by the IPG Media Lab. [Available online at https://www.ipglab.com/2013/08/22/are-all-screens-created-equal-a-research-study-by-the-ipg-media-lab/.]
Josephson
,
S.
, and
M.
Holmes
,
2006
: Clutter or content? How on-screen enhancements affect how TV viewers scan and what they learn. Proceedings ETRA 2006: Symposium on Eye Tracking Research and Applications, Stephen N. Spencer, Ed., Association for Computing Machinery, 155–162.
Kuligowski
,
E. D.
,
L. T.
Phan
,
M. L.
Levitan
, and
D. P.
Jorgensen
,
2013
: Preliminary reconnaissance of the May 20, 2013, Newcastle-Moore tornado in Oklahoma. National Institute of Standards and Technology, NIST SP 1164, 59 pp. [Available online at http://www.nist.gov/manuscript-publication-search.cfm?pub_id=914721.]
Lazo
,
J. K.
,
R. E.
Morss
, and
J. L.
deMuth
,
2009
:
300 billion served: Sources, perceptions, uses, and values of weather forecasts
.
Bull. Amer. Meteor. Soc.
,
90
,
785
798
, doi:.
Mayer
,
R. E.
, and
R.
Moreno
,
1998
:
A split-attention effect in multimedia learning: Evidence for dual processing systems in working memory
.
J. Educ. Psychol.
,
90
,
312
320
, doi:.
Mayer
,
R. E.
,
M.
Hegarty
,
S.
Mayer
, and
J.
Campbell
,
2005
:
When static media promote active learning: Annotated illustrations versus narrated animations in multimedia instruction
.
J. Exp. Psychol. Appl.
,
11
,
256
265
, doi:.
Mendelson
,
A.
,
2001
:
Effects of novelty in news photographs on attention and memory
.
Media Psychol.
,
3
,
119
157
, doi:.
Mileti
,
D. S.
,
1999
:
Disasters by Design: A Reassessment of Natural Hazards in the United States.
Joseph Henry Press, 371 pp.
Mileti
,
D. S.
, and
J. H.
Sorensen
,
1990
: Communication of emergency public warnings. A social science perspective and state-of-the-art assessment. Oak Ridge National Laboratory Rep. ORNL-6609, 166 pp.
Mileti
,
D. S.
, and
L.
Peek
,
2000
:
The social psychology of public response to warnings of a nuclear power plant accident
.
J. Hazard. Mater.
,
75
,
181
194
, doi:.
Munich Re
,
2012
: Severe weather in North America: Perils, risks, insurance. Munich Re, 8 pp. [Available online at https://www.munichreamerica.com/site/mram/get/documents_E1449378742/mram/assetpool.mr_america/PDFs/3_Publications/ks_severe_weather_na_exec_summary.pdf.]
National Research Council
,
2011
: Public response to alerts and warnings on mobile devices: Summary of a workshop on current knowledge and research gaps. National Academies Press, 78 pp. [Available online at http://www.nap.edu/catalog/15853/public-response-to-alerts-and-warnings-using-social-media-report.]
NCDC
,
2013
: Extreme events. National Climatic Data Center (now National Centers for Environmental Information), accessed December 2013. [Available online at www.ncdc.noaa.gov/climate-information/extreme-events.]
NOAA
,
2009
: Service Assessment of the Super Tuesday tornado outbreak of February 5–6, 2008. 48 pp. [Available online at http://www.nws.noaa.gov/om/assessments/pdfs/super_tuesday.pdf.]
NOAA
,
2010
: Thunderstorms, tornadoes, lightning … Nature’s most violent storms. 18 pp. [Available online at http://www.nws.noaa.gov/om/severeweather/resources/ttl6-10.pdf.]
NOAA
,
2013
: Weather. Accessed December 2013. [Available online at http://www.noaa.gov/wx.html.]
NWS
,
2012
: Impact based warnings. National Weather Service. Accessed December 2013. [Available online at http://www.weather.gov/impacts/#.Vdda6_lViko.]
NWS
,
2013
: National Weather Service mission statement. National Weather Service. Accessed December 2013. [Available online at http://www.nws.noaa.gov/mission.php.]
Perry
,
R. W.
, and
M.
Lindell
,
1991
:
The effects of ethnicity on evacuation decision-making
.
Int. J. Mass Emerg. Disasters
,
9
,
47
68
.
Perry
,
R. W.
,
M.
Lindell
, and
M. R.
Greene
,
1981
:
Evacuation Planning in Emergency Management.
Lexington Books, 199 pp.
Quarantelli
,
E. L.
,
1980
: Evacuation behavior and problems: Findings and implications from the research literature. Disaster Research Center, Ohio State University, 214 pp.
Rayner
,
K.
,
1998
:
Eye movements in reading and information processing: 20 years of research
.
Psychol. Bull.
,
124
,
372
422
, doi:.
Riad
,
J. K.
,
F. H.
Norris
, and
R. B.
Ruback
,
1999
:
Predicting evacuation in two major disasters: Risk perception, social influence, and access to resources
.
J. Appl. Soc. Psychol.
,
29
,
918
934
, doi:.
Schmidlin
,
T. W.
,
B. O.
Hammer
,
Y.
Ono
, and
P. S.
King
,
2009
:
Tornado shelter-seeking behavior and tornado shelter options among mobile home residents in the United States
.
Nat. Hazards
,
48
,
191
201
, doi:.
Sherman-Morris
,
K.
,
2010
:
Tornado warning dissemination and response at a university campus
.
Nat. Hazards
,
52
,
623
638
, doi:.
Simmons
,
K.
, and
D.
Sutter
,
2009
:
False alarms, tornado warnings, and tornado casualties
.
Wea. Climate Soc.
,
1
,
38
53
, doi:.
Smith
,
D. F.
,
2013
: The frequency of severe weather events. EQECAT, Casualty Actuarial Society, 36 pp. [Available online at https://www.casact.org/education/rpm/2013/handouts%5CPaper_1667_handout_877_0.pdf.]
Tobii Eye Tracking
,
2010
: An introduction to eye tracking and Tobii eye trackers. Tobii Technology White Paper, 12 pp.
Tversky
,
B.
,
J.
Morrision
, and
M.
Betrancourt
,
2002
:
Animation: Can it facilitate?
Int. J. Hum. Comput. Stud.
,
57
,
247
262
, doi:.
USGS
,
2007
: Natural hazards—A national threat. United States Geological Survey. [Available online at http://pubs.usgs.gov/fs/2007/3009/2007-3009.pdf.]
VanRullen
,
R.
,
T.
Carlson
, and
P.
Cavanagh
,
2007
:
The blinking spotlight of attention
.
Proc. Natl. Acad. Sci. USA
,
104
,
19 204
19 209
, doi:.
Wagenaar
,
W.
,
R.
Schreuder
, and
A.
van der Heijden
,
1985
:
Do TV pictures help people to remember the weather forecast?
Ergonomics
,
28
,
765
772
, doi:.
Wolf
,
P. L.
,
2009
:
Warning success rate: Increasing the convective warning’s role in protecting life and property
.
Electron. J. Oper. Meteor.
,
10
,
1
17
. [Available online at www.nwas.org/ej/pdf/2009-EJ7.pdf.]
WMO
,
2005
: Guidelines on weather broadcasting and the use of radio for the delivery of weather information. WMO/TD No. 1278, 54 pp. [Available online at https://www.wmo.int/pages/prog/amp/pwsp/pdf/TD-1278.pdf.]

Footnotes

a

Current affiliation: Southern New Hampshire University, Manchester, New Hampshire.