1. Introduction
Online channels for communicating risk frequently include features and technological capabilities to support sharing images of risk. In particular, the affordances found in social media, such as Twitter, include the ability to attach maps, photos, videos, and other graphical information. Since approximately 2012, the National Weather Service has increasingly made use of social media to relay risk messages (Fritz 2014). Currently, NWS includes over 130 Twitter accounts across the United States, reaching more than 2.8 million followers (@NWS), where they engage their audiences by providing daily forecast information, education, and communicating threats and their potential impacts (Olson et al. 2019). Prior research on NWS use of Twitter has found that including visual imagery, such as weather graphics, maps, and warning products, increases message amplification (Sutton et al. 2019) suggesting that visual risk communication is highly salient to online audiences.
The inclusion of visual cues such as colors, shapes, and their different sizes are important for making sense of approaching threats, populations at risk, the potential impacts, and ranges of associated uncertainty (Eosco 2011). On social media platforms, where individuals must quickly assess whether they will attend to stimuli as they scroll through messages, visuals serve to capture attention while also relaying key information (Sutton et al. 2015). The reception of information, that is, noticing an environmental or social cue about a potential threat, is the necessary first stage to making a decision about protective actions (Lindell and Perry 2012). Individuals must first be exposed and attend to a cue, then process that information and the recommended protective actions (Lindell and Perry 2012) while managing their individual perceptions of risk. Further understanding what visual features capture individual attention and how attention is directed to visual images of risk on social media has the potential to affect the design of risk communication messages and the protective actions that follow.
To date, much research has investigated how people perceive and respond behaviorally to warning messages sent over short messaging channels (for a review, see Sutton and Kuligowski 2019). However, limited research has addressed how people allocate attention to messages, that is, what they actively look at, and how the attributes of warning messages influence attention. Prior eye-tracking research within operational meteorology has investigated forecaster behavior and decision processes (Wilson et al. 2016) and attention distribution across visual interfaces (Wilson et al. 2018). Further, scholars and practitioners have called for additional research to address the visual components of tornado warning messages, including their design and impact on behavioral intent (Ash et al. 2014) and the potential effect of maps and other visuals on public decision-making for all hazards (Liu et al. 2017).
Scholars and methodologists have suggested that using a combination of eye-tracking techniques with qualitative “think aloud” methods will provide insight into both what people attend to as well as the stated reasons why people fixate on specific parts of presented information (Bol et al. 2016). Message designers incorporate a variety of visual cues into their designs so as to elicit the public’s attention. However, no research to our knowledge has been conducted on how these elements garner the attention of viewers to imminent threat messages.
Eye-tracking technology allows researchers to identify where people allocate attention to stimulus on images, to determine what people attend to in their visual field, and to identify what aspects of the visual are salient (Lang 2017). While they are not direct measures of comprehension or information processing, eye tracking highlights where people pay attention. When eye-tracking research is paired with verbal elicitations (thinking “out loud” about what is being viewed at the time of the viewing) researchers can begin to identify the most salient aspects of a warning message that receive attention. The current research showcases research from two studies; one that uses eye-tracking methods and the second that employs qualitative think alouds (Fonteyn et al. 1993), both in response to the same set of images, to gain novel insights into what aspects of visual risk information lead to attention allocation. The findings from this research have the potential to inform the practice of visual communication by weather practitioners as they design their weather messages to inform the public of emerging threats.
2. Theoretical background
In this section, we first discuss theories of attention and how visual attention is affected by visual and motivational saliency. Then, we explore principles of design, which visually convey uncertainty information for risks and hazards.
a. Allocation of attention
Theories of information processing have shown that individuals have limited amounts of mental resources to allocate to a stimulus in noisy environments, and they will choose selectively to attend to a portion of information (i.e., Duchowski 2007; Lang 2017). As described above, attending to environmental cues, including risk messages, is the first step to making decisions about protective actions (Lindell and Perry 2012), suggesting that successfully drawing attention to alerts and warnings and their visual cues is a primary goal for risk communicators. The process of understanding how individuals allocate attention to visual stimuli in informational messages is known as visual attention. Visual attention refers to a combination of eye movement across a stimulus and the cognitive and mental resources directed to specific visual objects and their spatial locations so that individuals can process one item contained within a message at a time (Duchowski 2007).
In physiological terms, eye movement is directed through a sequence of both parafoveal and foveal vision. When considering what aspects of the message will elicit attention, the individual will first perceive the stimulus with parafoveal vision (Duchowski 2007), that is, uncontrolled eye movements. After the initial parafoveal vision, the individual will begin to focus on more specified locations through foveal vision (Duchowski 2007). Using controlled eye movement, an individual will focus more critically, that is, for longer lengths of time, on specified areas of interest that are high in salience (Duchowski 2007). Those items of the message that are sought through foveal vision are tied to more fixation counts, demonstrating the number of times eyes fixate on an area of attention, and longer durations of visual attention allocation linked to higher levels of cognitive and mental resources (Gong and Cummins 2016). Just and Carpenter (1972) proposed that where the eye fixates is where the individual processes information. This process of controlled eye movements has been used as a key to decipher what elements of a message stimulus elicit directed attention, that is, what elements of a message may be more salient to the viewer.
b. Message saliency
Message saliency can be understood in two ways: as the visual features that capture the initial attention of a message receiver (Pieters and Wedel 2007) becoming important in the moment that they are observed, and features of a message that capture more fixations and longer durations of attention due to intrinsic or personal relevance. In visual design, initial message saliency concerns “the prominence of sensory information in the message, especially in the form of the attention-grabbing features” (Gong and Cummins 2016, p. 7). These attention-grabbing features include visual aspects of the stimulus that “pop out” to the receiver and elicit automatic, unintentional attention (Zhang and Lin 2013).
Attention-grabbing components of imagery in a message may be salient for a variety of reasons. First, elements such as high levels of contrast, new additions of color, an object’s edge, or motion, draw visual attention (Bruce and Tsotsos 2009; Yantis 2005; Zhang and Lin 2013). Visual salience also occurs when a new item or color, such as an element that adds novelty or surprise, is added to routine imagery or regular scenes (Zhang and Lin 2013).
Textual design and content can also be visually salient. Research on typography has found that font selection and layout have an effect on readability and legibility. For example, the use of uppercase and lowercase letters helps to visually discriminate words; removing this contrast, such as in the use of all capital letters (or, “all caps”), removes differences in word profile and deteriorates reading performance, but it also “screams” at message readers (Frascara 2006) such as by calling attention to the signal word, for example, “WARNING” (Hellier and Edworthy 2006). The use of punctuation in a sentence, such as the use of an exclamation point, may also call attention to words and phrases. Message designers may use punctuation to help emphasize imperative, important sentences such as “go now!” (Vos et al. 2018).
Visual representations of risk have largely focused in three areas: statistical graphics, cartography, and symbols (Bostrom et al. 2008). Statistical graphics include visuals such as risk ladders, confidence intervals, pie charts, and the like to provide quantitative representations of exposures and probability to various threats in a visual manner (Lipkus and Hollands 1999). Cartography focuses on how to spatially communicate risk information by linking risk to geographical locations and populations (Roth 2017). Symbols, such as emojis in text-based messages, shapes on maps, and as stand-alone images, are another way to communicate risk information visually. Because symbols attract attention and tend to be more legible than text to some viewers (Wogalter et al. 2006), they frequently provide information about the hazard, deliver general safety information, or prescribe specific actions to take (Hancock et al. 2006) in a single image.
In all three types of risk visualizations, a range of visual attributes provide cues about risk information, guide visual attention, and influence risk perception (Nelson et al. 2009; Pang et al. 1997).Visual features, such as size, shape, color hue, and color value, convey additional visual information to the message receiver (Roth 2017).
Color, in particular, affects visual attention, by drawing the eye to specific portions of an image, as well as informing risk perceptions. In the United States, color informs a “risk hierarchy” where red is perceived to be riskier than yellow, and yellow is riskier than green (Braun and Silver 1995; Wogalter et al. 2002). Frequently, such risk hierarchies are represented by colors on maps. For example, ordinal information (such as information presented from light to dark) on a map allows encoding of ordered information (Roth 2017) such as low, medium, and high. This ordered information will often include a legend for interpreting the colors present. In other cases, risk information may be represented by a specific color, such as unordered nominal information (MacEachren 2004). For example, the use of yellow or red on a map may represent a particular type of hazard, such as the threat of a thunderstorm or tornado in a specific location. An important aspect is that the appropriate use of color can greatly increase the effectiveness of visual communication, whereas poor color use can create confusion (Bostrom et al. 2008; Keller et al. 1994).
From this review of the literature, we pose two research questions (RQs) about viewer attention to the placement of information in a visual field and the features of that information in a communicated risk message:
RQ1: Where do participants allocate attention when viewing visual risk messages about a tornado threat?
RQ2: To what visual features do participants allocate attention when viewing visual risk messages about a tornado threat?
3. Methods
To answer the research questions, we completed two studies to investigate where participants allocate visual attention and what features elicit visual attention when presented with visual risk communication messages. In this design, we collect and independently analyze both quantitative and qualitative data then triangulate the findings in the discussion and conclusions of the paper. Because two types of data (eye tracking and verbal think aloud, described next) are collected for the same scenario, their integration allows for a deeper interpretation of the data and more satisfying conclusions (Creswell 2014; Creswell and Clark 2017). To be more specific, we conducted two studies with two different populations using the same scenario and images.
In the first study we used quantitative, eye tracking to explore visual attention to a series of tweets about an emerging tornado scenario. In the second study, we used qualitative, verbal elicitations (think aloud) methods to explore what features in the visual tornado risk messages are visually and motivationally salient to the viewer. We present these two studies to show where attention was placed on the visual risk message (study 1; eye tracking) before reading about what details and features in the visual field drew the gaze of our participants (study 2; qualitative think alouds).
Tornado scenario
An emerging tornado scenario communicated via Twitter was presented to participants. To do so, we used graphics and tweet copy that were previously developed and distributed by the National Weather Service. The scenario included five messages that escalated by threat level (Day 1 Outlook, Watch A, Watch B, Warning, and Emergency). In addition, we used two different versions of a “watch” message (Watch A and Watch B) that were included at the request of our Weather Forecast Office (WFO) partner. The five messages with visual imagery contained visual elements such as maps, a risk scale or legend, hazard icons, and risk polygons (see Figs. 1–5). To categorize our visual data into specific aspects on each tweet, we divided tweets into areas of interest. Nine areas of interest (AOIs) were identified; not every AOI was present on each tweet. In Table 1 we define and describe each AOI and its associated color that can be found highlighted on the representative tweets in Figs. 1–5 below.
(left) Outlook included in emerging tornado scenario and (right) the AOIs.
Citation: Weather, Climate, and Society 13, 1; 10.1175/WCAS-D-20-0042.1
(left) Watch A included in emerging tornado scenario and (right) the AOIs.
Citation: Weather, Climate, and Society 13, 1; 10.1175/WCAS-D-20-0042.1
(left) Watch B included in emerging tornado scenario and (right) the AOIs.
Citation: Weather, Climate, and Society 13, 1; 10.1175/WCAS-D-20-0042.1
(left) Warning included in emerging tornado scenario and (right) the AOIs.
Citation: Weather, Climate, and Society 13, 1; 10.1175/WCAS-D-20-0042.1
(left) Emergency included in emerging tornado scenario and (right) the AOIs.
Citation: Weather, Climate, and Society 13, 1; 10.1175/WCAS-D-20-0042.1
Descriptive statistics (means and standard deviations) of the participants’ prior experience to tornadoes in the eye-tracking study (n = 171).
1) Outlook
The Outlook tweet includes an image of a map with ordered information using colors (yellow, green, and red) to represent the potential risk of thunderstorms in a specified geographical area (Fig. 1). It is accompanied by a legend and textual content in the message of the tweet. The header of the image is presented on a black background.
2) Watch
Two watch messages are included in this study. As seen in Fig. 2, Watch A includes an image of a map with nominal information about tornado and thunderstorm represented by colors on the map (red and yellow, respectively). It also includes a rainbow-color risk slider bar that presents hazard icons on an ordinal scale. There is no legend corresponding to the colors on the map. Textual content is included in the message of the tweet. The header of the image is presented in black. Watch B (Fig. 3) is identical to the Watch A except, in place of the rainbow-color risk slider bar, hazards are described by using text in a risk table.
3) Warning
The Warning message (Fig. 4) includes an image of a map with a red polygon placed over the area of threat. The header of the image is presented in red. On the left side of the image, a set of potential impacts and population exposure are illustrated with icons and described by text. There is no legend provided and textual content is included in the message of the tweet.
4) Emergency
The Emergency message (Fig. 5) includes an image of a map with a purple polygon placed over the area of the threat and the header of the image is presented in purple. On the left side of the image, a set of potential impacts and population exposure are illustrated with icons and described by text. There is no legend provided and textual content is included in both the copy of the tweet and the attached image. The variation of color used in the headers of the warning (red) and emergency (purple) represents ordinal information of increasing threat.
Additional features found in all five visual messages include the agency logo in the top-left corner of the tweet and the public impressions of the tweet (displayed as comments, likes, and retweets) found at the bottom of the image. The tweet copy that was used in the study was identical to previously distributed NWS Twitter messages. This copy included text, hashtags, and links.
4. Study 1: Eye tracking
a. Study overview
In study 1, we investigated where participants allocated attention to messages describing an emerging tornado event. Due to the many visual differences found across the six NWS graphics, and the fact that tweets were presented sequentially from low threat to high threat, we limit our results to measures of visual attention in response to each separate image; we do not compare between them. Instead, we identify the length of time (fixation duration) and the number of times (fixation count) participants viewed each area of interest in each presented message.
b. Participants
Eye-tracking data were collected from a large sample of participants. A total n of 171 undergraduates were recruited from courses within a communications college at a large southeastern university. Participants earned course credit for their participation in the study. Fifty-four percent (n = 93) of the participants identified as male, whereas 45.6% (n = 78) identified as female. The majority of the participants described themselves as white (77.8%; n = 133) followed by African American (14%; n = 24), Hispanic (3.5%; n = 6), Asian (2.3%; n = 4), and other (2.3%; n = 4). Participants selecting other described themselves as Middle Eastern or African. All of the participants were undergraduate students: 33.1% (n = 57) were sophomores, 28.7% (n = 49) were juniors, 19.3% (n = 33) were seniors, and 18.7% (n = 32) were freshmen. As an indicator of familiarity with weather graphics, we asked participants if they had any formal training in meteorology. We found that 5.8% (n = 10) indicated that they did have formal training in meteorology, whereas 94.2% (n = 161) reported that they did not have any formal training in meteorology.
To determine the participants’ prior experience with tornadoes, we asked them to respond to three statements with regard to multiple tornado experience (Demuth 2018). Participants were asked, “People can have multiple experiences with tornadoes over the course of their lifetime. Please think about all of your experiences with tornadoes, and indicate how much experience you have with each of the statements listed below.” The statements include the following: 1) I have feared for my life due to a tornado, 2) I have feared for my loved ones due to a tornado, and 3) I have worried about my home due to a tornado. Each of the statements was rated on a 5-point scale where 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, and 5 = strongly agree (Chronbach α = 0.873). As seen in Table 1, participants on average were below the midpoint of the scale (neither agree nor disagree) as the aggregated sum of these measures was 2.68 [standard deviation (SD) = 1.22].
Eye-tracking studies typically have relatively low numbers of participants (King et al. 2019). For example, some studies focusing on usability and descriptive information only have included fewer than 20 participants (Jacob and Karn 2003), whereas experimental design studies typically chose participant numbers by power analysis to examine appropriate effect sizes (Sutton et al. 2020; Bol et al. 2016). Because this study sought only to explore descriptive trends in participant viewing, the number of participants exceeds the typical mean for eye-tracking research.
c. Procedure
To complete the study, participants were invited into a computer laboratory to view a series of tweets about a tornado scenario and complete a posttest questionnaire. Because the procedure included eye tracking, a laboratory setting was necessary to increase control and internal validity, and for the researchers to “uncover similarities of viewing patterns of large groups of viewers” (Duchowski 2007, p. 160).
In individual sessions, participants completed informed consent and then moved to the eye-tracking station. Participants were asked to sit approximately 24 in. (~60 cm) from a computer screen. A Tobii TX300 eye-tracker control unit was located just below the screen to monitor the participant’s pupil gaze during the experiment. To capture eye-tracking data, the Tobii device uses a camera and diodes that emit infrared light into the participant’s eye. The light reflected from the cornea is measured in relationship to the center of the pupil to identify viewing pattern and fixation duration. This type of data collection is noninvasive and appears no different than viewing a computer screen. Tobii studio software was used to collect the gaze data at a sampling rate of 60 Hz s−1. For this study, all participants had a weighted gaze sample 30% or higher. Before presentation of the tornado scenario stimuli, the researcher used the above software and apparatus to calibrate the participant’s gaze using a nine-point calibration procedure. Participants were asked to fixate on each of the nine moving objects on the screen.
After the calibration procedure, participants were verbally instructed with the following information: “As you saw in the study description, you’re going to see a series of tweets about weather. After that, we’ll ask you some questions about those tweets, the maps, and about you. Do you have any questions before we begin?”
Next, the research assistant read aloud the following statement to the participants as they followed along with the text on the computer screen in front of them: “In this scenario, you are a member of the public. You will be given information regarding a scenario, and you will be given a few images to view. Please press the space bar to move to each page of the scenario. Do you have any questions before we begin?” After discussing the procedures, the participants moved through the scenario at their own pace, pressing the space bar to forward each slide.
Participants were monitored by a laboratory assistant throughout the procedure and took approximately 5 min to complete the eye-tracking portion of the study. Afterward, participants were asked to complete a quantitative survey on a separate computer (approximately 13.38 min).
d. Data analysis
To identify the locations where a participant focused within each message (Duchowski 2007), we measured fixation duration and fixation count. Total fixation duration was operationalized as the total amount of time (in seconds) of fixations to each of the tweets. Fixation duration was also measured for each of the AOIs on each visual image (see Table 2 and Figs. 1–5). The average total fixation duration to each of the five visual risk messages and the AOIs on each message were assessed for this study. We report means and standard deviations for overall fixation duration and fixation counts and each AOI in the five stimulus tweets that includes visual imagery (Tables 3 and 4 and Figs. 6 and 7). To identify how many times someone fixated on the tweet, we assessed fixation frequency, or “count,” for the entire message and each of the respective AOIs. Josephson and Holmes (2008) operationalized fixation frequency as the number of times that the eye was stationary within a 62-pixel radius around a central point. At the time of the fixation, the participant’s pupil is focused on one particular point and indicates what location on the image is being cognitively processed at the particular moment in time (Duchowski 2007). In our analysis, we report the average amount of fixation counts (means and standard deviations) for the overall message and each of the areas of interest (Table 4).
Descriptions of areas of interest in the emerging tornado scenario messages.
Descriptive statistics [means, standard deviations, and percent of total time allocated (%)] for fixation duration in AOIs in seconds (n = 171).
Descriptive statistics (means and standard deviations) for the number of fixation counts in AOIs (n = 171).
Percentage of total fixation duration per AOI on each message (n = 171).
Citation: Weather, Climate, and Society 13, 1; 10.1175/WCAS-D-20-0042.1
Percentage of fixation counts per AOI on each message (n = 171).
Citation: Weather, Climate, and Society 13, 1; 10.1175/WCAS-D-20-0042.1
e. Results for study 1
1) Visual attention to the Outlook message
As seen in Table 3, the map in the Outlook (Fig. 1) received the longest fixation duration (48.58%; mean M = 6.48 and SD = 4.35), followed by the tweet copy (26.01%; M = 3.47 and SD = 2.49), and the legend (19.19%; M = 2.56 and SD = 1.87). Aspects of the message receiving less visual attention included the impressions (5.10%; M = 0.68 and SD = 0.84), the header (5.02%; M = 0.67 and SD = 0.95), the logo (2.62%; M = 0.34 and SD = 0.52), and the social media bar (1.12%; M = 0.15 and SD = 0.15). The overall mean time spent on the Outlook was 13.34 s (SD = 7.48 s).
We also identified the average fixation counts for the overall message and each of the areas of interest. For the Outlook message, the map received the most fixation counts (42.57%; M = 28.03 and SD = 17.04), followed by the tweet copy (29.43%; M = 19.39 and SD = 12.00) and the legend (16.43%; M = 10.82 and SD = 7.09). Fixation counts for the remaining AOIs can be found in Table 4.
2) Visual attention to the Watch A message
In this version of the watch message, (Fig. 2) participants allocated the most attention to the map (43.08%; M = 4.45 and SD = 4.87), followed by the tweet copy (42.01%; M = 4.34 and SD = 3.15) and the slider bar/legend (22.94%; M = 2.37 and SD = 2.67). The social media bar (0.77%; M = 0.075 and SD = 0.19), header (2.52%; M = 0.26 and SD = 0.52), logo (2.03%; M = 0.21 and SD = 0.47), and impressions (1.65%; M = 0.17 and SD = 0.33) received less attention. We found that the mean overall time spent on this version of the watch tweet was 10.33 s (SD = 7.52 s). Fixation counts for Watch A were highest for the tweet copy (73.73%; M = 24.45 and SD = 14.84), the map (57.06%; M = 18.92 and SD = 18.57), and the slider bar (30.64%; M = 10.16 and SD = 10.05).
3) Visual attention the Watch B message
In the second version of the watch message, (Fig. 3), participants allocated the most attention to the map (65.92%; M = 4.70 and SD = 4.57), followed by the risk table/legend (51.75%; M = 3.69 and SD = 2.45), and tweet copy (23.42%; M = 1.67 and SD = 2.09). Similar to Watch A, less attention was placed on the header (3.51%; M = 0.25 and SD = 0.40), impressions (1.12%; M = 0.08 and SD = 0.19), logo (0.98%; M = 0.07 and SD = 15), and social media bar (0.28%; M = 0.022 and SD = 0.062). The second version of the watch message elicited 7.13 s (SD = 5.52 s) of visual attention. Similar to Watch A above, in response to Watch B we found the highest number of fixation counts directed to the map (56.36%; M = 18.67 and SD = 17.57) the risk table/legend (42.58%; M = 14.12 and SD = 12.95), and the tweet copy (30.34%; M = 10.06 and SD = 11.61).
4) Visual attention to the Warning message
As for the Warning message (Fig. 4), the impacts and population exposure elicited the most amount of visual attention (35.71%; M = 2.45 and SD = 2.99), followed by the map (26.09%; M = 1.79 and SD = 2.02), and tweet copy (16.18%; M = 1.11 and SD = 1.19). We found that the header (4.66%; M = 0.32 and SD = 0.43), inset map (2.48%; M = 0.17 and SD = 0.45), logo (2.33%; M = 0.16 and SD = 0.30), and impressions (2.04%; M = 0.14 and SD = 0.30) received relatively little visual attention. Within the Warning message, participants allocated 6.86 s (SD = 5.93 s) to the overall message. Fixation counts on the Warning message were highest on the area containing information about impacts and population exposure (30.53%; M = 9.93 and SD = 11.09), followed by the map (23.67%; M = 7.70 and SD = 7.68) and the tweet copy (20.50%; M = 6.67 and SD = 6.40).
5) Visual attention to the Emergency message
The final tweet of the scenario was the Emergency tweet (Fig. 5). As seen in Table 3, the tweet copy received the most amount of visual attention (47.28%; M = 3.73 and SD = 2.22), followed by the impacts and population exposure (20.30%; M = 1.58 and SD = 2.22) and map (12.80%; M = 1.01 and SD = 1.17). The least amount of attention was directed toward the header (4.69%; M = 0.38 and SD = 0.54), logo (2.41%; M = 0.19 and SD = 0.30), impressions (0.89%; M = 0.07 and SD = 0.21), inset map (0.89%; M = 0.07 and SD = 0.19), and protective actions (1.90%; M = 0.15 and SD = 0.40). For the Emergency tweet, participants allocated a total of 7.89 s (SD = 4.93 s) to the overall message. Fixation counts on the Emergency message were highest on the area of the tweet copy (51.77%; M = 21.67 and SD = 11.07), followed by impacts and exposure (15.43%; M = 6.46 and SD = 8.02) and the map (11.75%; M = 4.92 and SD = 4.92).
5. Study 2: Think alouds
a. Study overview
For the second study, we sought to understand what features in each visual risk message elicited attention. To do so, we asked participants to “think aloud” as they viewed each of the tornado messages, describing what they saw and sharing their thoughts and reactions.
b. Participants
Verbal, qualitative think-aloud data were collected from 20 participants who were recruited from the graduate program in communication at a large southeastern university. Of the 20 participants, the majority described themselves as white (70%; n = 14) followed by African American (15%; n = 3), Asian (10%; n = 2), or other (5%; n = 1). Participants mainly identified as female (80%; n = 16), while 20% (n = 4) identified as male. Think-aloud data collection ranged from 8 to 17 min, with an average of 12.75 min. This was followed by the completion of the quantitative survey, lasting approximately 10 min. Participants were incentivized with a $25 gift card for their participation in the study. Similar to our undergraduate population, we found that the majority of participants did not have formal training in meteorology (n = 20).
Similar to study 1, to determine the participants’ prior experience with tornadoes, we asked them to respond to three statements with regard to multiple tornado experience (Demuth 2018). Participants were asked, “People can have multiple experiences with tornadoes over the course of their lifetime. Please think about all of your experiences with tornadoes, and indicate how much experience you have with each of the statements listed below.” The statements include the following: 1) I have feared for my life due to a tornado, 2) I have feared for my loved ones due to a tornado, and 3) I have worried about my home due to a tornado. Each of the statements was rated on a 5-point scale where 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, and 5 = strongly agree (Chronbach α = 0.844). As seen in Table 5, we found that participants’ reported prior experiences were above the midpoint (neither agree nor disagree) as the aggregated sum of these measures was 3.533 (SD = 1.22).
Descriptive statistics (means and standard deviations) of the participants’ prior experience to tornadoes in the think-aloud study (n = 20).
c. Procedure
The think-aloud portion of the study explored what features of the tornado messages were visually and motivationally salient, using verbal elicitation, or think-aloud, methods (Fonteyn et al. 1993). To complete the think-aloud portion of the study participants were audio recorded while they verbally described the features of the message they attended to.
You are going to see a series of tweets about weather. I am also going to be asking you a few questions throughout about your reactions to the messages. We want you to talk out loud as you are viewing the message. In this scenario, imagine you are a member of the public. You are scrolling through Twitter and you see these messages on your newsfeed. We want to understand your thoughts and feelings as you receive these tweets; talk about them as you look at them.
As the participant viewed each message, to encourage talking out loud, additional prompts included the following questions: 1) what is your initial reaction to the message? 2) What feelings do you have in response to the message? 3) What are your ongoing thoughts about the message? To assess participant understanding of the risk information relayed in the two watch messages, we included one additional message-specific prompt: What do you think about the levels of risk presented on the side and their relation to the map? When participants were finished talking out loud about the image, they pressed the space bar to move to the next slide.
d. Data analysis
Think alouds were transcribed verbatim by a student research assistant, and thematic analysis was conducted using word-processing tools. Qualitative findings were analyzed using a grounded-theory approach (Strauss and Corbin 1997). The qualitative thematic analysis was guided by research questions focusing on what people were attending to and the characteristics of those features.
Themes were identified using the constant comparative method of qualitative analysis via open and axial coding for specific parent and child codes (Glaser and Strauss 1967). Similar themes were constructed from different phrases, patterns, and words presented in the data (Glaser and Strauss 1967). Parent codes included expressions about features in the message that were visually salient, such as colors and icons, and stylistic aspects, such as the presentation of text (i.e., in capital letters), of the messages. Child codes were generated when explanatory statements were made about message comprehension or confusion and how the visual display of contents of the message affected message perceptions. For example, a participant might express noticing the color purple on the Emergency message (a parent code) and describe their reaction, such as it being unexpected, in response to the color purple (a child code). Credibility and accuracy were ensured in the data analysis, as two researchers analyzed the data individually. Afterward, using the methods of peer debriefing, the two researchers compared and contrasted their results (Huberman and Miles 1994). The two researchers approved and confirmed the themes that emerged from the data analysis process (Erlandson et al. 1993). Only the most commonly occurring emergent themes are reported in the findings below.
e. Results for study 2
1) Think alouds in response to the Outlook message
Participants frequently commented on two visual aspects of the Outlook: color and locations. First, they noted the color, in particular the red and its location on the map. For example, one participant stated, “Visually speaking, my eyes directly go to the red area.” Another said, “obviously the red stands out the most, even though it says it’s moderate right there [in reference to the legend]. It seemed as if that would be the highest chance.”
Second, in correspondence to the location of the color, they also noted the locations of areas that were at risk of strong storms. For example, one participant said, “I’m just looking at it, trying to figure out like where [city name] is and the zones of colors.” In their discussion of message features, participants frequently commented about what caused their focused attention on these two aspects: they were attending to their personal location relative to the named locations under moderate threat, indicated by the color red. For example, one participant stated that she was focused on determining “is it gonna hit me? Am I in the area of where the severe weather is expected?” and another noted that there is a “lot of severe weather coming my way.”
2) Think alouds in response to the Watch A message
In response to Watch A, one primary theme emerged related to the visual display of the emerging threats and their location on the map: the inconsistency of colors. Participants were immediately drawn to, and commented on, the bright colors on the map and on the risk color slider bar. In this tweet, the colors on the map depicted two geographical areas under two threats: red, labeled “tornado,” and yellow, labeled “thunderstorm.” The image also included a risk slider bar with rainbow colors to represent a range of severity for four different hazards (hail, wind, tornado, and flooding) that could accompany the approaching storm. Every participant who viewed this image described using the rainbow color slider bar to try to interpret the colors on the map. For example, one participant stated, “I feel like there’s like conflicting messages here. Cause there’s like . . . it shows there’s a tornado watch [on the map] but then it says tornado medium [on the slider bar] . . . I don’t really like that it’s like tornado watch is red on the map and then on the little side bars it’s showing it as yellow. I don’t know, that sort of . . . that like is confusing me I think.”
Another participant rationalized the apparent inconsistency in colors by saying the map is “not quite as accurate because it doesn’t really blend the colors, it just shows like one area is all one color and another is a different color. I’m sure it’s kind of a mix of, this makes it seem like it’s a mix of everything and then the picture doesn’t seem like that.”
3) Think alouds in response to the Watch B message
In contrast with Watch A, the second Watch message, B, displayed emerging threats in a tabular format using text that described threat ranges for each of the four risks (tornado, hail, wind, and flood). Consistency of information emerged as a primary theme in response to Watch B. Here, participants described how the text presentation of threat information helped to reduce their confusion about the potential severity associated with each risk, eliminating the need to “figure it out” because the text is “more descriptive” in contrast with the risk slider bar. For example, one participant said, “while it’s not as colorful, it kinda helps me understand it a little better. So, I can see that we’re red [on the map] and then I have to look over here [at the table] and see ok the risk for tornado is medium and the hail describes the size.”
Another said, “for some reason I correlated it [the rainbow color sidebar in Watch A] with the map . . . anywhere in red . . . but I couldn’t figure out why it wasn’t marked better. So this one [the table presented in Watch B] is more effective for me.”
While participants commented that the table in Watch B was generally clearer than Watch A, the lack of color was noted by many participants who said that using text only was not as eye-catching and was found to be less appealing. For example, one participant explained that the text-only display seems “distant” and “doesn’t really have any sort of impact . . . it just seems like ‘here are some facts’ and it’s hard for me to become more concerned.” Another explained that, “it doesn’t look as scary as the other message with bright red.” Others said they preferred the rainbow color sidebar to explain the map colors (albeit incorrectly).
4) Think alouds in response to the Warning message
Themes about the use of color and text also emerged in response to the Warning tweet. For example, participants consistently mentioned the color red on the image heading “Tornado Warning” and on the polygon, which showed the area of risk on the map. Participants described assessing their personal location in relation to the polygon and the increased need for awareness of the threat. One participant stated that the color, combined with the word “Warning” led her to “thinking this is urgent” in contrast with the previously viewed Watch message that included both yellow and red.
Participants also described the text copy of the tweet, noting that the word “Warning” (a signal word), is “stronger than a Watch.” However, they also commented on the style and content of the Warning text. For example, one person noted that a different style of text presentation, such as putting text in all caps would increase perceptions of urgency. Another participant stated that the lack of information about protective actions reduced the sense of urgency because it “doesn’t really talk much about like actual risks or like anything that you should do. It’s just like ‘here it is’.” This sentiment was also repeated by others who discussed the section of the image that presented the number of persons exposed to the threat with limited interest. One person asked, “why do I need to know how many schools and hospitals are in this area? I’m getting a severe thunderstorm, right? I wanna know the actual threat and not what is exposed to it.”
5) Think alouds in response to the Emergency message
In response to the Emergency tweet, participant comments again suggested themes about the use of color, the textual content, and the style in the message copy. Notably, nearly every participant discussed the purple color used for the Emergency message. One person explained that “none of the past colors have been purple, so since this is a new color to me, I’d probably pay more attention to it ’cause it would catch my attention.” However, the use of purple, in contrast with red, was incongruous with what they expected for this level of threat. For example, one person said, “I wouldn’t expect an emergency to be purple, I would expect it to be like red or orange . . .” Another stated a dislike of the use of the color because of her mental model of color representing severity. She said, “I’m not really lovin’ the purple. It doesn’t feel as emergency, life threatening as the other one.” Another participant summed it up, “I think the purple makes it seem like it’s not a big deal.”
In contrast, participants discussed the style and content of the message copy that captured their attention and relayed the seriousness of the threat. For example, one participant said, “If I was scrolling on Twitter, all caps catches my eye just because it’s something different.” Furthermore, the words “TORNADO EMERGENCY” in the text copy called attention to the extreme threat. One participant explained that if he were scrolling on Twitter, he would likely notice “tornado emergency in capital letters before you might even see the picture.” Several participants also read aloud the impacts: “this is a life threatening situation” and “to protect your life” and the protective actions “seek shelter now” and “take cover now.” One person described how this phrasing “elicits a more fearful emotion and . . . makes it extremely serious.” For example, one person said, “this is scary . . . like, ‘to protect your life,’ it’s life or death basically. It gives me very specific things to do. So yeah, I’d be acting on this pretty quickly.” Another explained it, “this makes me want to take cover right now.”
6. Discussion
In this study, we used eye-tracking methods to investigate attention allocated within each tweet with visual imagery and qualitative methods to collect think-aloud data from a separate group of participants who represent members of the public. Here, we triangulate the results from our two data sources and provide a discussion linking results to visual design for communicating risk.
a. Attention allocation to visual imagery
When considering the amount of time participants spent attending to each tweet with visual imagery, we see evidence of a decrease in overall time allocated to each message as the tornado scenario progressed, starting with the Outlook. While testing for differences of attention allocation between images is beyond the scope of this project, we do offer some suggestions about why time spent on each image decreased as participants progressed through the scenario. It is possible that at the start of the study, participants spent additional time becoming familiar with threat potential, its location, the organization that was posting messages, and how to ingest the messages. While attention continued to be directed to individual AOIs on every visual, the overall amount of time decreased over the course of viewing the scenario. This finding is consistent with prior research that suggests that when introduced to a scenario, attention will be the greatest at the start (i.e., Duchowski 2007). As participants become familiar with the scenario and consistency of the emerging narrative, they can focus more on visual differences; these visual cues capture their attention and direct them to key emergent information.
Over the course of the scenario, we found participants generally directed the longest durations of visual attention and the highest number of fixations toward the map and the text, copy. Think-aloud findings suggest that attention to the map served as a way for participants to become acquainted with their location, while the text copy provided information relating to more specifics of the event at hand.
However, this pattern of visual attention described above differed for the Emergency message. In response to this message, the text copy received the greatest proportion of fixation duration (47.28%) and fixation count (51.77%), whereas the map received 11.75% of the fixation count and 12.80% of fixation duration. This corresponds with findings from participants’ think alouds as well. Participants described being initially drawn to the copy displayed in all capital letters and then focused on the text expressing severity of the threat and the protective actions.
When considering attention allocation to the two presentations of the Watch messages, we find differences primarily in relation to the time spent on the risk slider bar versus the risk table. In response to Watch A, participants divided their attention fairly evenly between the text copy (fixation duration, 42.01%; fixation count, 73.3%), and the map (fixation duration, 43.08%; fixation count, 57.06%), with much less attention to the risk slider bar (fixation duration, 22.94%; fixation count, 30.64%). In response to Watch B participants allocated the greatest amount of time to the map (fixation duration, 65.92%; fixation count, 56.36%) and the risk table (fixation duration, 51.75%; fixation count, 42.55%) with limited time attention to the text copy of the message (fixation duration, 23.42%; fixation count, 30.34%). One possible explanation for this is that participants recognized the similarities between the images in the Watch A and Watch B messages and focused on the differences instead—the characterization of risk and its relationship to the map. Think-aloud findings suggest that when participants viewed Watch A, they intuitively tried to connect the rainbow-colored slider bar to the colors on the map, as though it were a legend, and quickly made the connections before moving on. In contrast, when viewing Watch B, participants took longer to ingest the words on the risk table, eliciting longer durations of attention. The think-aloud data suggested participants were able to decipher the risk information in Watch B more accurately.
b. Features that attract visual attention
1) Colors
Across the five visual messages, participants consistently made statements about the colors used on the maps, the image headers, and the presentation of risk as a rainbow-colored slider bar. Relative to the maps, participants described the use of the color red as visually salient in the Outlook, Watch, and Warning messages saying that it stood out from other aspects and captured their initial attention. The participants also highlighted the color of the Emergency message (a purple color), and described it in contrast with red, which participants expected to represent the worst case. Participants indicated that while the use of red was generally described as signifying something “bad,” the use of purple in the Emergency message was described as being unfamiliar and leading to some confusion about the urgency and severity of the threat. However, participants did indicate that the use of purple suggested something was different, as they had not seen it before. Although participants were not expecting the color purple for the Emergency message, we found that it did direct attention to the message, perhaps due to its inconsistency with the previous messages that followed an anticipated color progression.
The color of the message headers also affected understanding that the threat level was changing. In particular, the use of red and purple header for the Warning and Emergency messages received attention from the participants, but color also served as a signal that the threat was changing over time. In the eye-tracking data, we saw a continual increase in time allocated to the header area over the course of the emerging scenario. In contrast with the Outlook and Watch messages that did not include brightly colored headings, we found the headers for the Warning and Emergency tweets received more visual attention.
One additional use of color that attracted visual attention was the use of multiple color schemes to represent different types of hazards and risks, which clearly created confusion for participants. We saw this primarily in response to Watch A, where participants assumed that the colors on the map (using a nominal color scheme) corresponded to the colors on the rainbow-colored slider bar (using an ordinal color scheme) and referenced it as a legend to read the map. While we found that participants allocated longer durations of attention to the descriptive risk table in Watch B, participants demonstrated a clearer understanding of the relationship between the risks and the map and also stated that it was easier to understand.
2) Text
Some textual features also drew visual attention during the eye-tracking experiment and received comment as part of the think alouds. For example, many participants spoke about the importance of the inclusion of a “signal word,” such as watch, warning, or emergency. This suggests that these indicators helped participants to orient themselves to understanding the severity of the threat.
Participants allocated the most attention to the text copy presented in the Emergency tweet relative to the overall time spent on each visual message. Part of this may be due to the length of the text copy (i.e., it was longer than the text in the Warning message). However, the results of our think-aloud procedure found that the use of all caps to emphasize specific words in text copy (e.g., TORNADO EMERGENCY), and the inclusion of an exclamation point to punctuate key words in a sentence (“now!”) were also key to attracting visual attention from participants. This information suggests that implementation of distinctive text features can stand in contrast with less emphatic content and draw attention when it is most necessary.
We also note that that participants consistently directed their gaze toward and discussed the features and text in the legend/risk slider bar/impacts and exposure areas of interest. While each of these three are depicted by different visual features (color, text, icon), they represent a similar concept, that of threat and impact information. This is an important visual element that is used for sense making, resulting in both confusion and comprehension, depending upon the ease of translation.
7. Limitations and future research
This study investigated visual attention allocation and message saliency in response to a tornado threat. While the mixed methods design provides insights into where people attend to in visual risk messages and what they are drawn to in those areas of interest, several limitations have emerged that can be addressed in future research.
In this research, participants were exposed to scenario stimuli that showed a threat progressing over time, and we used images and text that were previously distributed by our partners from National Weather Service offices, which offers a level of external validity but also presents limitations. First, it is likely that attention allocation to areas of interest were influenced by the fact that the visual risk communication messages were viewed in sequence from Outlook to Emergency. We expect that participants became familiar with the threat, the location of impact, and the format of the messages over the course of the scenario, perhaps influencing the attention they gave to each of the various sections of subsequent messages. To reduce the familiarization of each image, future research should use an experimental design to investigate visual allocation to messages that are presented in randomized orders and include control groups. Furthermore, because we used messages that were identical to those that they might see from a NWS Twitter account, including the spacing of image and text in the visual display, the areas of attention around contiguous features may not be precise. Future research might adapt images to include larger margins between message features in order to obtain more precise measurements for the eye-tracking portion of the study.
Furthermore, our decision to triangulate data from two different participant samples may be a limitation because of differences in perception. However, the richness of the think-aloud data provides insights that the eye-tracking data alone cannot offer. Eye tracking allows us to measure gaze unobtrusively, absent any spoken or cognitive processing of the presented images, whereas think-aloud methods influence viewer duration and fixation counts (Duchowski 2007). Therefore, the triangulation of the two studies allows us to consider both to where the eye was drawn and what features affected that visual attention. However, it is only a first step in identifying attention allocation to visual images. Future research might investigate attention allocation in cognitive processing, such as through the use of cognitive recall after viewing messages and analyze fixation patterns such as first fixations and gaze paths.
This research was conducted in a laboratory setting under a context in which participants were not truly exposed to an emerging threat. A real-life context may change information viewing and seeking under threatening conditions, especially when there are likely to be competing messages containing visual information. Future research might investigate how visual attention might differ under real-life conditions and in the context of a broader information ecology.
Last, this study was descriptive in nature. It does not account for other variables that may influence visual attention such as prior experience with tornado threats (Demuth 2018), weather salience (Stewart 2009), formal training in meteorology, or use of social media. Future experimental research might include these additional population measures to identify how viewer attention differs when facing an impending threat.
8. Conclusions and practical implications
This study adds to our knowledge about where people attend to when they view visual risk communication messages and what features attract this attention. This is an important first step to increase visual saliency of specific information known to affect message perceptions and comprehension leading to protective action decisions among the public.
From these findings a few practical conclusions emerge for designing visual risk communication messages. Colors, such as green, yellow, and red, represent a learned scheme of increasing levels of risk. While the use of an unexpected color, such as purple, may draw attention to the visual image, it may also result in some confusion, especially relative to the urgency and severity of a threating situation. If purple is to represent the most serious threats, public education is needed to inform people of the gravity of the situation and to emphasize the need to pay attention under those dire conditions. In addition, thoughtful design of legends that serve as a reference for the risk levels displayed on a map or to explain the representation of the threat as a set of nominal categories (i.e., the use of yellow to represent thunderstorm) is likely to increase viewer understanding of the risk image. Ideally, when colors are presented on a visual risk message, they will be clearly marked for ease of interpretation by members of the public who are expected to interpret them accurately. Careful decisions about the design and presentation of text copy can also help to draw attention to key words and emphasize phrases. For example, the use of signal words displayed in all capital letters can help to highlight differences so that they pop out from descriptive text presented in lowercase letters. Considering the serious nature of the threat, warning messages will benefit from including protective action information as part of the textual copy or in place of information about population exposure. These changes help message receivers to learn what to do in response to a warning in addition to understanding the message.
By taking into consideration the careful design and presentation of visual risk information, risk communicators have the opportunity to reduce confusion and increase understanding about an emerging threat among populations at risk. This has the potential to lead to better outcomes for risk messaging and protective action responses in the future.
Acknowledgments
The authors thank the anonymous reviewers for their insightful comments and suggestions that have helped to improve this paper. Research reported in this publication was supported by the National Oceanic and Atmospheric Administration under Award NA190AR4590211. The content is solely the responsibility of the authors and does not necessarily represent the official views of NOAA.
REFERENCES
Ash, K. D., R. L. Schumann III, and G. C. Bowser, 2014: Tornado warning trade-offs: Evaluating choices for visually communicating risk. Wea. Climate Soc., 6, 104–118, https://doi.org/10.1175/WCAS-D-13-00021.1.
Bol, N., J. C. van Weert, E. F. Loos, J. C. Romano Bergstrom, S. Bolle, and E. M. Smets, 2016: How are online health messages processed? Using eye tracking to predict recall of information in younger and older adults. J. Health Commun., 21, 387–396, https://doi.org/10.1080/10810730.2015.1080327.
Bostrom, A., L. Anselin, and J. Farris, 2008: Visualizing seismic risk and uncertainty. Ann. N. Y. Acad. Sci., 1128, 29–40, https://doi.org/10.1196/annals.1399.005.
Braun, C. C., and N. C. Silver, 1995: Interaction of signal word and colour on warning labels: Differences in perceived hazard and behavioural compliance. Ergonomics, 38, 2207–2220, https://doi.org/10.1080/00140139508925263.
Bruce, N. D., and J. K. Tsotsos, 2009: Saliency, attention, and visual search: An information theoretic approach. J. Vis., 9 (3), 1–24, https://doi.org/10.1167/9.3.5.
Creswell, J. W., 2014: Qualitative Inquiry and Research Design. SAGE Publications, 448 pp.
Creswell, J. W., and V. L. P. Clark, 2017: Designing and Conducting Mixed Methods Research. SAGE Publications, 520 pp.
Demuth, J., 2018: Explicating experience: Development of a valid scale of past hazard experience for tornadoes. Risk Anal., 38, 1921–1943, https://doi.org/10.1111/risa.12983.
Duchowski, A. T., 2007: Eye Tracking Methodology: Theory and Practice. Springer, 301 pp.
Eosco, G., 2011: It’s story time: Weather story guidelines for the National Weather Service Western Region. NWS Doc., 12 pp., https://www.weather.gov/media/aviation/afp/Weather%20Story%20Content%20Full%20Paper.pdf.
Erlandson, D., E. Harris, B. Skipper, and S. Allen, 1993: Doing Naturalistic Inquiry: A Guide to Methods. SAGE Publications, 198 pp.
Fonteyn, M. E., B. Kuipers, and S. J. Grobe, 1993: A description of think aloud method and protocol analysis. Qual. Health Res., 3, 430–441, https://doi.org/10.1177/104973239300300403.
Frascara, J., 2006: Typography and the visual design of warnings. Handbook of Warnings, M. S. Wogalter, Ed., Lawrence Erlbaum Associates, 385–406.
Fritz, A., 2014: National Weather Service makes Twitter operational. Washington Post, accessed 22 September 2020, https://www.washingtonpost.com/news/capital-weather-gang/wp/2014/09/25/national-weather-service-makes-twitter-accounts-operational/.
Glaser, B. G., and A. L. Strauss, 1967: The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine, 271 pp.
Gong, Z., and R. G. Cummins, 2016: Processing capacity in visual search: The impact of visual salience and involvement on attention. Annual Meeting of the Association for Education in Journalism and Mass Communication, Minneapolis, MN, AEJMC.
Hancock, H. E., T. C. Bowles, W. A. Rogers, and A. D. Fisk, 2006: Comprehension and retention of warning information. Handbook of Warnings, M. S. Wogalter, Ed., Lawrence Erlbaum Associates, 267–278.
Hellier, E., and J. Edworthy, 2006: Signal words. Handbook of Warnings, M. S. Wogalter, Ed., Lawrence Erlbaum Associates, 407–418.
Huberman, A. M., and M. B. Miles, 1994: Data management and analysis methods. Handbook of Qualitative Research, N. K. Denzin and Y. S. Lincoln, Eds., SAGE Publications, 428–444.
Jacob, R. J., and K. S. Karn, 2003: Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. The Mind’s Eye, Elsevier, 573–605.
Josephson, S., and M. E. Holmes, 2008: Eye monitoring studies on TV news and the web converge: A tale of two screens. Passive Eye Monitoring: Algorithms, Applications, and Experiments, R. J. Hammoud, Ed., Springer, 387–420.
Just, M. A., and P. A. Carpenter, 1972: Eye fixations and cognitive processes. Cognit. Psychol., 8, 331–480, https://doi.org/10.1016/0010-0285(76)90015-3.
Keller, P. R., M. M. Keller, S. Markel, A. J. Mallinckrodt, and S. McKay, 1994: Visual cues: Practical data visualization. Comput. Phys., 8, 297–298, https://doi.org/10.1063/1.4823299.
King, A. J., N. Bol, R. G. Cummins, and K. K. John, 2019: Improving visual behavior research in communication science: An overview, review, and reporting commendations for using eye-tracking methods. Commun. Methods Meas., 13, 149–177, https://doi.org/10.1080/19312458.2018.1558194.
Lang, A., 2017: Limited capacity model of motivated mediated message processing (LC4MP). The International Encyclopedia of Media Effects, John Wiley and Sons, https://doi.org/10.1002/9781118783764.wbieme0077.
Lindell, M. K., and R. W. Perry, 2012: The protective action decision model: Theoretical modifications and additional evidence. Risk Anal., 32, 616–632, https://doi.org/10.1111/j.1539-6924.2011.01647.x.
Lipkus, I. M., and J. G. Hollands, 1999: The visual communication of risk. JNCI Monogr., No. 25, Oxford Academic, 149–163, https://doi.org/10.1093/oxfordjournals.jncimonographs.a024191.
Liu, B. F., M. M. Wood, M. Egnoto, H. Bean, J. Sutton, D. Mileti, and S. Madden, 2017: Is a picture worth a thousand words? The effects of maps and warning messages on how publics respond to disaster information. Public Relat. Rev., 43, 493–506, https://doi.org/10.1016/j.pubrev.2017.04.004.
MacEachren, A. M., 2004: How Maps Work: Representation, Visualization, and Design. Guilford Press, 513 pp.
Nelson, D. E., B. W. Hesse, and R. T. Croyle, 2009: Making Data Talk: Communicating Public Health Data to the Public, Policy Makers, and the Press. Oxford University Press, 321 pp.
Olson, M. K., J. Sutton, S. C. Vos, R. Prestley, S. L. Renshaw, and C. T. Butts, 2019: Build community before the storm: The National Weather Service’s social media engagement. J. Contingencies Crisis Manage., 27, 359–373, https://doi.org/10.1111/1468-5973.12267.
Pang, A. T., C. M. Wittenbrink, and S. K. Lodha, 1997: Approaches to uncertainty visualization. Vis. Comput., 13, 370–390, https://doi.org/10.1007/s003710050111.
Pieters, R., and M. Wedel, 2007: Goal control of attention to advertising: The Yarbus implication. J. Consum. Res., 34, 224–233, https://doi.org/10.1086/519150.
Roth, R. E., 2017: Visual variables. International Encyclopedia of Geography: People, the Earth, Environment and Technology, John Wiley and Sons, https://doi.org/10.1002/9781118786352.wbieg0761.
Stewart, A. E., 2009: Minding the weather: The measurement of weather salience. Bull. Amer. Meteor. Soc., 90, 1833–1842, https://doi.org/10.1175/2009BAMS2794.1.
Strauss, A., and J. M. Corbin, 1997: Grounded Theory in Practice. SAGE Publications, 288 pp.
Sutton, J., and E. D. Kuligowski, 2019: Alerts and warnings on short messaging channels: Guidance from an expert panel process. Nat. Hazards Rev., 20, 04019002, https://doi.org/10.1061/(ASCE)NH.1527-6996.0000324.
Sutton, J., C. B. Gibson, N. E. Phillips, E. S. Spiro, C. League, B. Johnson, S. M. Fitzhugh, and C. T. Butts, 2015: A cross-hazard analysis of terse message retransmission on Twitter. Proc. Natl. Acad. Sci. USA, 112, 14 793–14 798, https://doi.org/10.1073/pnas.1508916112.
Sutton, J., S. L. Renshaw, S. C. Vos, M. K. Olson, R. Prestley, C. B. Gibson, and C. T. Butts, 2019: Getting the word out, rain or shine: The impact of message features and hazard context on message passing online. Wea. Climate Soc., 11, 763–776, https://doi.org/10.1175/WCAS-D-19-0021.1.
Sutton, J., L. M. Fischer, L. James, and S. Sheff, 2020: Earthquake early warning message testing: Visual attention, behavioral responses, and message perceptions. Int. J. Risk Reduct., 49, 101664, https://doi.org/10.1016/j.ijdrr.2020.101664.
Vos, S. C., J. Sutton, Y. Yu, S. L. Renshaw, M. K. Olson, C. B. Gibson, and C. T. Butts, 2018: Retweeting risk communication: The role of threat and efficacy. Risk Anal., 38, 2580–2598, https://doi.org/10.1111/risa.13140.
Wilson, K. A., P. L. Heinselman, and Z. Kang, 2016: Exploring applications of eye tracking in operational meteorology research. Bull. Amer. Meteor. Soc., 97, 2019–2025, https://doi.org/10.1175/BAMS-D-15-00148.1.
Wilson, K. A., P. L. Heinselman, and Z. Kang, 2018: Comparing forecaster eye movements during the warning decision process. Wea. Forecasting, 33, 501–521, https://doi.org/10.1175/WAF-D-17-0119.1.
Wogalter, M. S., V. C. Conzola, and T. L. Smith-Jackson, 2002: Research-based guidelines for warning design and evaluation. Appl. Ergon., 33, 219–230, https://doi.org/10.1016/S0003-6870(02)00009-1.
Wogalter, M. S., N. C. Silver, S. D. Leonard, and H. Zaikina, 2006: Warning symbols. Handbook of Warnings, M. S. Wogalter, Ed., Lawrence Erlbaum Associates, 159–176.
Yantis, S., 2005: How visual salience wins the battle for awareness. Nat. Neurosci., 8, 975–977, https://doi.org/10.1038/nn0805-975.
Zhang, L., and W. Lin, 2013: Selective Visual Attention: Computational Models and Applications. John Wiley and Sons, 352 pp.
