1. Introduction
In the United States from 1980 to 2021, tropical cyclones caused over $1 trillion in damage and cost 6697 lives (NOAA 2023). Moving forward, risks might be magnified even further by the combination of increasing population density along coastlines, threats from climate change that may influence the frequency and intensity of tropical storm systems, and land-use/land-cover changes (Emanuel 2017; Freeman and Ashley 2017; Weinkle et al. 2018). Accuracy of hurricane forecasts continues to improve (Alley et al. 2019; DeMaria et al. 2014; Emanuel 2017; Cangialosi et al. 2020). However, as any storm approaches, projections of the storm’s future path inherently become more uncertain with greater look-ahead time, and yet the general public needs to understand the risk in order to engage in preparations as early as possible (Matyas et al. 2011; Meyer et al. 2013; Wu et al. 2015; Millet et al. 2020b; Dormady et al. 2022).
A standard approach to convey information about the possible future path of a storm is the cone of uncertainty (COU) issued by the National Hurricane Center (NHC). The COU has several known limitations (discussed below), which raises the issue of whether a different kind of visualization might better serve the public. Drawing from our research expertise on the visual system and cognitive reasoning, we developed a new visualization to communicate hurricane forecasts to nonexperts (Witt and Clegg 2022). This current study empirically examines relative perceptions of risk from these two visualization formats.
a. Cone of uncertainty limitations
The COU graphic is based on a single most probable track of the center of a tropical cyclone derived from the official NHC forecast, which is shown as points that can be connected with or without a middle line. The radius of the cone represents the track forecast uncertainty by encompassing two-thirds of the previous 5 yr of NHC error tracks for each forecast period [approximately 26–205 n mi (1 n mi ≈ 1.9 km); National Hurricane Center 2022]. The COU primarily conveys spatial trajectory information, although it also includes limited information about storm intensity, such as maximum sustained winds.
Surveys of the media (Demuth et al. 2012) have found that some NHC forecast products, such as the COU, may provide too many technical details that are difficult to disseminate in public weather forecasts. There are also several fundamental problems with the nature of the COU that impact its value in effective decision-making about potential risk (Broad et al. 2007; Millet et al. 2020b). For example, even though the COU communicates information such as maximum anticipated wind speed, it does so through arbitrary representations through symbols such as “S,” “H,” and “M” rather than through sensory representations that are directly interpretable. Understanding the information in the arbitrary symbols requires either training or reading the legend, but research shows few eye movements toward this aspect of the display (Millet et al. 2023).
One issue with the COU visualization is that it represents a summary statistic; it depicts the projected storm path and a measure of uncertainty based on historical forecast track errors. Like all summary statistics, the summary might not accurately represent the underlying distribution. One illustration of this point is shown by Anscombe (1973) quartet. Anscombe generated four different sets of data with the same values for their summary statistics (such as mean, variability, and correlation), but visualizations of the data reveal clear differences across the sets (see Fig. 1).
Anscombe’s quartet, displaying four datasets that have identical statistics (mean, variance, correlation, linear regression) but have qualitatively different patterns from each other. The quartet was created to emphasize the importance of visualizing the underlying data.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-21-0173.1
An example of how a summary statistic fails to capture the underlying distribution can be seen with Hurricane Irma. The COU forecast on 6 September 2017 (NHC Intermediate Advisory Number 29A; see Fig. 2) indicated the most probable track over central Florida. However, the range in weather forecast model projections more clearly showed that Irma could pass up either the east or west coast of Florida. As the forecast then updated on 7 September 2017 (NHC Advisory Number 34), the most probable track shifted to a path up the east coast. At this time, several forecast models still suggested that a trajectory up the west coast of Florida was also possible. It was only later in the day, on 8 September 2017 (NHC Advisory Number 40), that the center path of the COU shifted to the west coast of Florida, where the eye of the storm would make landfall just 2 days later.
(a) NHC forecast for Hurricane Irma on 6 Sep 2017, which is displayed as the COU from Intermediate Advisory 29A (see https://www.nhc.noaa.gov/archive/2017/al11/al112017.public_a.029.shtml). (b) Hurricane Irma tracks on 7 Sep 2017 (image credit: Climate Forecast Applications Network; https://www.wunderground.com/cat6/hurricane-watch-se-florida-ahead-powerhouse-irma).
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-21-0173.1
This example illustrates an issue in how the forecasts were being conveyed using the COU visualization. While the weather models successfully predicted the range of the storm’s probable paths, and the NHC watches and warnings indicated elevated risks along both coasts, this information could not be fully communicated with only a single projected storm track presented with the COU. In such cases, a graphic that can be more easily understood by nonexperts as communicating more than just a single central tendency could better serve the public when making evacuation decisions and assessments of potential risks.
A second issue with the COU is that its design can lead to misinterpretations about the size of the storm. The cone shows an increase in area over time because forecasting a storm’s position at later times is more uncertain than for proximal times (Emanuel and Zhang 2016). However, some individuals misinterpret the cone as displaying an increase in the size of the storm rather than increasing uncertainty about the position (Ruginski et al. 2016; Padilla et al. 2017; Boone et al. 2018). The NHC has attempted to counter this issue by adding text to explicitly state that the cone does not show storm size (see the black box at the top of Fig. 2; National Weather Service and Eastern Research Group 2020). Ideally, visualizations should not be open to such misinterpretation, and not all consumers will necessarily read or understand the additional explanatory text. Moreover, the misinterpretation points to desired information that is not readily obtained in the current format of the COU, such as imprecise labeling of storm categories (Millet et al. 2020a). It would be useful to have a single, standard graphic that could more intuitively convey several dimensions, such as projected storm paths, uncertainty, and storm size, while requiring only a simple legend.
Another issue with the COU is that people overestimate the certainty in the graphic. In the case of the cone, this can include overconfidence in the center points (or connecting line) being the actual path (Broad et al. 2007). The edges of the COU also cause a problem due to the defined outer boundaries (Tversky 2011). The presence of such boundaries tends to imply a very different sense of potential risk for locations inside as compared with outside the cone (Padilla et al. 2018; Millet et al. 2020b; Ding and Millet 2020); this is also known as the containment effect (Padilla et al. 2018). This issue came to prominence recently in several media stories after Hurricane Ian (Dance and Ajasa 2022; Pulver and Rice 2022), highlighting real-world consequences from confusion around the meaning of the cone. There are no visual indications in the COU graphic that would inform the public that recent historical data suggest one-third of storm paths occur outside the cone’s boundaries, nor does the current version convey that the uncertainty being presented is based exclusively on historical error, and therefore, the variability in the predicted path for the current storm may be substantially higher or lower than is depicted.
Crucially, despite the presence of a text warning, the COU is misinterpreted as not indicating that more peripheral locations from the current most likely forecast path are also at risk because these locations are not contained within the visual boundaries of the cone (Broad et al. 2007; Cox et al. 2013). Attempts to modify the nature of the cone by incorporating fuzzy boundaries have been met with very limited success (Steed et al. 2009; Millet et al. 2021).
Despite several shortcomings, the COU remains an important forecast product for the general public and for government officials in making emergency management preparations and issuing evacuation orders (Broad et al. 2007; Wu et al. 2014). In fact, a recent study of residents living in Florida found that the COU was the only NHC product named by the public (Bostrom et al. 2018). Even with public and media miscommunications about the information provided by the COU (Broad et al. 2007; Wu et al. 2014; Ruginski et al. 2016; Senkbeil et al. 2020; Morss et al. 2016) and alternatively proposed visualization products (Cox et al. 2013; Liu et al. 2019; Millet et al. 2020b; Yang et al. 2019; Hamill et al. 2012), the essence of original COU design persists. To address communication of weather hazards and other risks, changes to how hurricane forecast information is communicated may help improve public understanding of hurricane track forecasts and aid decision-making for evacuations (Ding and Millet 2020; Dormady et al. 2022).
b. Track ensembles to visualize hurricane forecasts
An alternative approach to convey information and statistics about hurricane tracks to the public is the use of ensemble forecast data (Hamill et al. 2012; Cox et al. 2013). A display of ensemble forecasts usually consists of a group of individual objects, each of which conveys a prediction of a specific instance. One approach used is track ensembles, which present a set of lines that represent projected hurricane paths, often from different weather forecast models (e.g., Cox et al. 2013). Research on risk perception when viewing ensemble displays has found that track ensembles reduce one bias found with the COU, namely, that track ensembles do not lend themselves to interpretations about storm size as the COU does (Ruginski et al. 2016).
Track ensembles also do not lend themselves to a containment effect like that found with the COU, as demonstrated in Cox et al. (2013). In their study, participants viewed the COU or an animated version of the track ensemble for which tracks appeared then faded as more tracks appeared. Nonexpert participants were given a finite number of resources that they had to distribute among various sectors of the shown area. When the COU was largely contained within a single sector, participants distributed resources almost exclusively to this sector, whereas the resources were distributed more broadly when viewing the track ensembles. This suggests the COU created a bias to only see areas contained with the COU as being at risk, as shown by the behavior to localize distribution of resources to only a single sector.
One issue with track ensembles is that the lines can appear unorganized. This lack of organization is reflected in their colloquial name of “spaghetti plots.” To avoid visual clutter and confusion, one variation of the ensemble displays is to present parallel lines that have been sampled from the distribution of projected storm tracks (Liu et al. 2019). An ensemble display can also forgo visualizing the storm’s trajectory altogether and instead present icons at the locations where storm impacts are forecast to occur (Liu et al. 2017).
Another issue with track ensembles is that ensemble displays using lines can create boundaries that produce misinterpretations of the data. For example, upon viewing bar graphs, people rate data that would be contained within the bar (i.e., values lower than the height of the bar) as being more likely than data that would be beyond the bar (Newman and Scholl 2012). This “within the bar bias” may also apply to track ensemble displays. Nonexperts can interpret locations on the individual lines as being more likely to incur damage, so locations on a line but farther away from the mean location are estimated as being more likely to be damaged (Padilla et al. 2017). Thus, there may be some benefit to finding another visualization design that avoids some of these pitfalls.
c. Animated risk trajectories of hurricane forecasts
A recently proposed alternative to the COU is to show icons moving along potential hurricane trajectories as a way to communicate uncertainty about storm paths to the public (Witt and Clegg 2022; Witt et al. 2021, 2022a). We refer to this visualization as animated risk trajectories (ARTs). ARTs are a set of small icons that move across a map analogous to how hurricanes move across a region. ARTs are depicted ensembles but are not intended to be an animated version of track ensembles. As currently instantiated, they are not intended to show tracks of specific individual, different forecast models. Instead, they aim to provide similar statistical information as the COU but via a different visual format.
One core goal of the current research was to replicate the finding that ARTs lead to less of a containment effect than the COU and expand on the previous work by comparing ARTs with a version of the COU that includes a text description that was added in an attempt to prevent the containment effect [research objective (RO) 1; see section 1d]. As outlined below, we also sought to examine whether ARTs convey levels of risk via the number of icons, and whether ARTs can effectively capture bimodal distributions that may have been beneficial for storms like Hurricane Irma (see Fig. 2).
ARTs have several potential advantages over COUs. As a group, the icons can readily be interpreted by the visual system, which is highly tuned to this type of collective information and can extract it from a display quickly, effortlessly, and accurately (Whitney and Yamanashi Leib 2018). Moreover, the visual system can extract this information even without expertise in the specific area, making the display well suited for a nonexpert audience. In addition, ARTs also render the information as natural frequencies (e.g., 20 of 30 predicted paths come close to one’s town), rather than as probabilities (e.g., 67% of predicted hurricane paths are expected to be in the region shown by the cone). Presenting natural frequencies has been shown to greatly improve reasoning under uncertainty when compared with presenting probabilities (Gigerenzer and Hoffrage 1995).
ARTs are not subject to the same biases and misinterpretations as the COU. For example, ARTs feature no boundary lines and have been shown to eliminate the containment effect found with the COU for which areas beyond the cone’s boundaries are perceived as being at low risk (Witt and Clegg 2022; Witt et al. 2021). ARTs communicate uncertainty via the spread of their icons, whereas the increased spread of the COU is often misinterpreted as signaling an increase in storm strength (Ruginski et al. 2016). Thus, ARTs have good visual–conceptual compatibility, which is a desirable feature for a visualization (Wickens and Carswell 1995; Witt 2019). RO3 is that the distribution of icons in the ARTs will better communicate the specific locations at risk when compared with the COU. Specifically, a bimodal distribution of ARTs could communicate two more likely paths of the storm’s trajectory as compared with a unimodal distribution or the COU. In cases for which the track ensembles reveal a bimodal grouping of forecast tracks (such as with Irma; see Fig. 2), testing whether ARTs can successfully communicate bimodal predictions would be useful as it is clear the COU cannot successfully communicate this information (RO3).
In the current experiments, we compared how participants perceived risk after viewing two visualizations (ARTs and a simplified version of the COU) used to display the forecast of a hurricane. Our focus is on how the public might react to a graphic showing individual potential storm trajectories for hurricane communication. Participants took the perspective of emergency managers (whether to evacuate a town) rather than that of the individual (whether to evacuate oneself). We have recently conducted additional research that has used decisions that more closely mimic the decision that would be made by an individual (whether to evacuate, engage in additional preparations such as buying additional food or water, or do neither of these actions) and found similar patterns of results (Witt et al. 2022a).
Additionally, because both visualizations (ARTs and COU) are intended to communicate uncertainty, we also tested the extent to which people responded differently when the visualization communicated different levels of uncertainty. For the COU, the greater width that occurs with increased lead times has been misinterpreted that the storm will increase in magnitude and size rather than as a forecast with greater uncertainty (Ruginski et al. 2016). We tested whether ARTs are less biased in the sense that greater spread of the icons is interpreted instead as greater uncertainty (RO4).
To begin understanding the user interpretation of ARTs, an earlier proof-of-concept study explored ARTs in comparison with relatively unrealistic cones of uncertainty and did not include the explicit reminder that hazardous conditions can occur outside the cone (Witt and Clegg 2022). In addition, another earlier study using ARTs (Witt et al. 2020) found a slight tendency toward underrepresenting the risk at the very center of the distribution. This may have resulted from the relatively sparse number of icons used in those ARTS, which featured only 18 icons. While these two papers showed promise for ARTs, there are many open questions about how to effectively design ARTs. For example, little is known about the influence of the number of icons employed in an ART display and the resulting interpretation of risk (RO2). Perceived risk could be associated with proximity to one or more icons, proximity to the average projected path, or the density of the icons that are within proximity. If density is a cue people use to assess risk, thinning out or reducing the number of icons should substantially affect responses to the hurricane forecast. Such thinning would have less impact if proximity or the average projected path were the cues being used. These issues are examined in more detail in this current study.
d. ROs
Our research objectives are organized as follows:
-
For RO1, we tested whether ARTs would better convey risk as gradually decreasing over distance, rather than as an abrupt decrease indicative of a containment effect as found with the COU. In addition, we compared ARTs with a version of the COU that also included the text that was added to COUs to try to reduce the containment effect.
-
For RO2, we tested whether risk perception would align with the density of the icons in an ART visualization. If so, density could be used to communicate risk to nonexperts.
-
For RO3, we tested whether risk perception would align with the distribution of icons in an ART visualization. Specifically, we tested whether bimodal groupings of ARTs would lead to bimodal risk perceptions as compared with unimodal ARTs and the COU. If so, ARTs could communicate more precisely in situations for which weather forecast models grouped in two or more clusters.
-
RO4 relates to the fact that one issue with the COU is that nonexperts can misinterpret that the width of the COU relates to storm size (Ruginski et al. 2016) rather than to uncertainty. Thus, if the ARTs can better convey uncertainty, that would reveal an additional benefit of the ARTs. We tested this by manipulating the level of uncertainty and measuring subsequent responses across both visualizations.
2. Experiment 1
The main question of this experiment was whether people could better extract information about the path of a hurricane displayed using ARTs when compared with the COU. Although previous research has already shown a decreased or lack of containment effect with the ARTs when compared with the COU, the previous work used a COU that did not include the text that had been specifically added to try to reduce the containment effect. To correct for this, the current study compares ARTs with the COU with the added text intended to reduce the containment effect.
a. Method
1) Participants
Twenty-two participants enrolled in an introductory psychology course at Colorado State University (CSU) received partial, optional course credit in exchange for completing the experiment through Qualtrics. This sample leads to greater than 95% power based on published effect sizes (Witt et al. 2021). Participants had self-reported normal or corrected-to-normal vision. The protocol was deemed exempt, so informed consent was provided but written consent was not required. We did not collect information about whether participants ever lived in a geographical area where hurricanes pose a risk. Over 60% of students at CSU are from Colorado. In another experiment, we asked participants recruited from CSU whether they had ever lived in a hurricane-prone area, and 32 of 41 (78%) said “never” (J. K. Witt 2022, unpublished data). It is reasonable to infer that many of our participants had little to no direct personal experience with threats from hurricanes. However, other research has reached similar conclusions when comparing ARTs with the COU among students at CSU and those at a university in Florida (see Witt et al. 2021).
2) Stimuli
All stimuli were created using R software (R Core Team 2019) and presented via the internet using Qualtrics. The map stimuli were created using the maps R software package (Becker et al. 2022) and depicted the Gulf of Mexico coastline of the United States. For each display, a graphic showed the forecast for a storm and a single town visualized as a red circle.
To create a range of hypothetical stimuli, we manipulated three factors: storm angle, prediction uncertainty, and angular deviation between storm angle and town. The storm angle refers to the general direction of the storm and was set to one of four angles (50°, 70°, 90°, or 110°). These angles were chosen so the forecast impact of the storm was within the visible coastline. For the NHC COU, the width of the COU is defined by historical forecast errors averaged over the previous 5 yr. Therefore, as numerical weather prediction forecast skill improves (Alley et al. 2019), the historical error decreases, so the width of the COU decreases. In contrast, in our experiment, the COU width was a function of the manipulation of prediction uncertainty, which was set to be low, medium, or high. For the ARTs, these widths corresponded to the standard deviation (SD) used to sample the ART icon angles.
Each forecast was paired with 1 of 13 angular deviations; an angular deviation is the angular distance between the most likely storm path and the town. The 13 angular deviations were selected so 5 towns would be located within the COU’s boundaries, 2 towns located at the cone’s boundaries, and 6 towns located beyond the COU’s boundaries regardless of the width of the COU (see Fig. 3). This distribution of angular deviations of the icons was chosen to have more towns located at the edge of the COU. These areas are also critical for model fit and were particularly important to help calculate risk transitions (viz., the distance across which people go from largely evacuating to not evacuating) and decision thresholds (viz., the distance at which people evacuate at a 50% rate; see below in analysis section). The combination of 4 storm angles, 3 levels of projection uncertainty, and 13 angular deviations resulted in 156 unique scenarios. However, 12 scenarios had to be excluded because of the town not aligning with the intended placement, leaving a total of 144 scenarios.
Coastline used for stimuli. The dots show 13 possible town positions for a projected storm with (a) an angle of 110° and a low projection uncertainty level, (b) an angle of 70° and a medium projection uncertainty, and (c) an angle of 90° and a high projection uncertainty. The town was shown as a red dot, but in this figure, the color of the dot corresponds to the magnitude of angular deviation between the town and most likely storm path.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-21-0173.1
For each of the 144 scenarios, we created two visualizations: a simplified COU and ARTs. The COU was created by connecting two lines with an arc. The mean angle of the COU was set to the angle of the storm path (50°, 70°, 90°, or 110°). The width of the COU was set to the hypothetical forecast error (±10°, 20°, or 30°). Each COU contained three future timesteps along the hypothetical storm path (see Fig. 3).
The ARTs display was an animated graphics interchange format (GIF) image showing 50 small icons that moved toward the coastline at a constant rate of 10 pixels per second from the center of the bottom of the display. One example of an animated display can be viewed online (https://osf.io/kxg3z). Each icon followed a linear path with a smooth and continuous motion from the origin point. The angles at which the icons traveled were sampled from a normal distribution with the mean set to the storm’s most likely path and the standard deviation set to the historical forecast errors. Thus, the data underlying the development of the ARTs were the same as the data underlying the COU. On average, one-third of the icons were beyond the COU’s boundary, although there was variation in the exact number due to random sampling. The angle assigned to each icon was used to calculate the horizontal and vertical displacement that the icon should make at each rendering. A small amount of random deviation was added to reduce crowding and increase the visibility of the icons. The icons were shown at six consecutive positions, but the speed of the presentation led to the perceptual experience of smooth motion rather than to the visual impression of six unique time points. Given that the task involved decisions about an approaching hurricane at the coastline, both the COU and the ARTs were presented solely over water rather than extending over the land. The animation lasted 600 ms before resetting and then playing on infinite loop until the participant made their response. The animated GIFs can all be viewed online (https://osf.io/u6qh8/).
3) Procedure
Imagine it is hurricane season and you are in charge of deciding whether to evacuate a town based on the predicted hurricane path. The town will be marked with a red circle. If you choose not to evacuate the town and a hurricane hits, damage will be extensive and costly. If you choose to evacuate the town and the hurricane does not hit there, money will be spent on the evacuation for nothing. Thus, there are benefits and costs to evacuating the town. Towns must be evacuated 12 h in advance of when the hurricane will hit. For each decision, a hurricane is hovering and is approximately 12 h away, so it will be time to make your decision. You will see a cone that shows the predictions of the hurricane’s path. The cone shows the probable path of the storm center but does not show the size of the storm. Hazardous conditions can occur outside of the cone.
The description of the COU was created using the text found at the National Hurricane Center web page (National Hurricane Center 2022). The instructions were the same for the ARTs condition, except the italicized part was replaced with: “You will see several predicted hurricane paths, each presented as a black dot. These dots show the probable path of the storm center. Hazardous conditions can occur outside of these paths.” The text in the experiment was not italicized and is only italicized here to highlight the difference in the instructions between the conditions.
For each trial, participants viewed a map of the Gulf of Mexico coastline of the United States along with a projected hurricane path, shown either as a COU or as ARTs (see Fig. 4). Text across the screen prompted the question, “Should the town be evacuated?”, and participants clicked the box labeled “Yes” to evacuate or the box labeled “No” to not evacuate. The intent of the ARTs is to communicate information to the public, but we tasked participants to make a decision more akin to those made by emergency managers as an attempt to imbue the task with gravity despite its hypothetical nature. Follow-up research using different decisions for participants revealed near-identical results (Witt et al. 2022a). We used evacuation rates as a measure of relative risk perception: when towns are evacuated most of the time, this is an indicator that risk was perceived as high, whereas when towns are only evacuated infrequently, this is an indicator that risk was perceived as low. Using a binary choice in our experiment lends itself to several ways of characterizing risk perception based on traditional psychophysics (Fechner et al. 1966; Knoblauch and Maloney 2012), as will be discussed in the results section.
An illustration of (a) a COU trial and (b) an ART trial. For each trial, only one type of visualization was displayed. The ART display was an animated GIF that continued to loop until participants responded.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-21-0173.1
The forecast was displayed until the participant indicated their response; there was no time limit. Feedback was not provided, given that it can directly impact task performance (Nadler 1979). Participants completed one block of trials with forecasts shown using the COU and one block of trials with forecasts shown as ARTs. The starting visualization condition was counterbalanced across participants, and the order of trials within each block was randomized. Each block contained 144 trials for a total of 288 trials across the two blocks, which took approximately 10–15 min.
b. Results
Risk perception was analyzed by conducting a general linear mixed model (GLMM). The dependent measure was evacuation decision (coded as 1 for “yes” and 0 for “no”). The fixed effects were angular deviation (coded as the absolute deviation between the town and the most likely storm path), visualization condition, projection uncertainty (coded as −1, 0, and 1), and all two- and three-way interactions. Random effects included random intercepts and slopes for angular deviation for participants and intercepts for storm angle. Given the repeated measures design, it is not appropriate to assume the measurements are independent, in which case including random effects is the appropriate way to analyze the data (Magezi 2015). The results of the model are shown in Fig. 5.
Risk perception (shown as marginal mean evacuation rates) as a function of angular deviation and visualization condition from the model for experiment 1. Angular deviation is measured in degrees from the town to the most likely storm path. In other words, an angular deviation of 0° means the most likely path of the storm was predicted as heading directly toward the town. Shading represents 1 standard error of the mean (SEM) as calculated from the model.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-21-0173.1
1) Risk transitions
The model coefficients were used to quantify two aspects of risk perception. Risk transitions refer to how risk perception transitioned from being high risk to low risk as angular deviation increased. Risk transitions were quantified as the slopes from the model. A steeper slope indicates a stricter transition over which perception goes from high risk to low risk over a short distance. For example, the containment effect is the idea that all towns located within the COU are perceived to be at high risk and towns located outside of the COU are perceived to be at low risk (Padilla et al. 2018). The containment effect would lead to a steep slope. In contrast, the risk transition should be less strict if risk is perceived to gradually decrease as angular deviation increases. A gradual decrease in perceived risk would be revealed by a shallower slope.
Risk transitions (i.e., slopes) were steeper for the COU, with mean M = 0.10 and 95% confidence interval (CI) of [0.09, 0.11], than for the ARTs, with M = 0.07 and 95% CI of [0.06, 0.08]. Risk transitions were approximately 43% steeper for the COU than for the ARTs [(0.10–0.07)/0.07 × 100% = 43%]. The steeper slopes for the COU were consistent with the containment effect (Padilla et al. 2018; Witt et al. 2021; Witt and Clegg 2022). In contrast, for the ARTs, participants’ decisions to evacuate showed a more gradual decrease in risk perception as the angle to the town increased. As shown in Fig. 5, participants had a similar sense of risk for towns located in the center of the projected path for both visualization conditions, but they had an increased sense of risk in the more peripheral locations with the ARTs than for the COU. Whether this increased sense of risk is merited would depend on the forecast for each impending storm. Experiment 2 evaluated whether the design of the ARTs can be modified to communicate different levels of geospatial risk.
2) Decision threshold
The second aspect of risk perception is the decision threshold. The decision threshold refers to the angular deviation at which risk perception shifted from high to low. Borrowing from the literature on sensory thresholds (Fechner et al. 1966; Gescheider 2013), the theory of signal detection (Gescheider 2013; Macmillan and Creelman 2004; Wickens 2001; Tanner and Swets 1954), and previous work on ARTs (Witt and Clegg 2022), the decision threshold was quantified as the angular deviation at which towns were evacuated 50% of the time according to model fit. Towns located within this threshold were more likely to be evacuated (e.g., at a rate greater than 50%), and towns located beyond this threshold were less likely to be evacuated (e.g., at a rate less than 50%). If the COU led to a containment effect, we would expect decision thresholds to be just beyond the containment area, namely, at the edge of the cone. The GLMM coefficients were used to calculate decision thresholds and 95% confidence intervals (CIs) via the bootstrap method using the MixedPsy package (Moscatelli et al. 2012). Note this package provides CIs but not p values.
Decision thresholds were closer to the storm center for the COU, with M = 21.07° and 95% CI of [18.35, 24.54], than for the ARTs, with M = 30.86° and 95% CI of [26.83, 36.00]. These estimates of decision thresholds correspond to the medium level of historical forecast errors at which the boundary of the cone is set to 20° from the most likely storm path. The decision threshold for the COU was approximately at the edge of the cone, consistent with the idea that the COU leads to a containment effect. No containment effected was found with the ARTs.
3) Projection uncertainty
We questioned how risk perception varied across the two types of visualizations when the uncertainty of the storm projection was low, medium, or high. With greater uncertainty in the forecast, it might be less clear where to evacuate. This would be shown by shallower slopes with high uncertainty relative to low uncertainty. We calculated risk transitions (slopes) for each visualization condition for each level of projection uncertainty. For both visualization conditions, risk transitions became less strict as projection uncertainty increased (see Figs. 6 and 7), and participants showed more uncertainty in their decisions when the forecast was also more uncertain.
Risk perception (quantified as mean evacuation rates) as a function of angular deviation, prediction uncertainty, and visualization condition for experiment 1. Angular deviation is measured in degrees from the town to the most likely storm path. Shading represents 1 SEM as calculated from the model.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-21-0173.1
We hypothesized that the decision threshold would be farther away from the storm center as projection uncertainty increased. This was true for both visualization conditions (see Fig. 7). For the COU, the decision threshold was always just beyond the edge of the cone’s boundary. For the ARTs, it was farther out than for the COU.
(a) Mean risk transitions and (b) decision thresholds as a function of projection uncertainty and visualization condition for experiment 1. Higher risk transitions correspond to steeper slopes; lower risk transitions correspond to more gradual transitions from high to low perceived risk. Decision thresholds correspond to the angular distance at which evacuation rates were estimated to be 50% according to the statistical model. Higher mean decision thresholds occurred at a distance farther from the most likely storm path. Horizontal dotted lines correspond to the edge of the cone and 1 SD of the ARTs. Error bars represent 95% CIs as calculated from the model.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-21-0173.1
c. Discussion
Risk perception was considerably higher for towns within the COU boundaries than for towns outside the boundaries. This reflects a limitation of the COU, known as the containment effect (Padilla et al. 2018). The COU can lead to nonexperts misunderstanding risk beyond the cone’s edges, despite the inclusion in the current experiment of an explicit statement to indicate that hazardous conditions could occur outside the cone. In contrast, the ARTs led participants to perceive a gradual decrease in risk. The ARTs still conveyed high risk at the center of the distribution and conveyed risk beyond the center. With 50 icons used here, evacuation rates (which we used to estimate perceived risk) were just as high as with the COU, which fails to replicate a previous finding we had reported for which intended evacuation rates were lower for ARTs than for COUs, but in that experiment, there were only 18 icons, which could explain the lower rates (Witt et al. 2020). This raises the question about the potential influence of the number or density of items on the determination of likelihood of evacuation.
3. Experiment 2
Experiment 1 provided evidence that ARTs can be an effective visualization for conveying the probable paths of a storm. What remains unclear is whether risk perception is driven by the number of icons moving toward the coastline. Understanding the most effective design formats for ARTs requires developing a sense of which visual features strongly influence the interpretation of the visualization and how (e.g., whether the features affect the risk transitions vs the decision thresholds). With the COU, there is a clear signal of the summary statistics of the projection, namely, the most likely storm path and the magnitude of historical forecast errors. With ARTs, these summary statistics must be extracted from the distribution of icons. In experiment 2, we manipulated the number of icons to see whether the number influenced risk perception.
a. Method
1) Participants
Twenty-five participants who had not participated in experiment 1 were recruited from the introductory psychology participant pool at Colorado State University. Participants had self-reported normal or corrected-to-normal vision and received course credit in exchange for participating.
2) Stimuli
Stimuli were created in R and presented in Qualtrics. The ART displays were the same as in experiment 1 except that the number of icons presented on each trial was either 10 or 50 (see Fig. 8). There were 144 scenarios (i.e., combination of 4 storm paths and 3 levels of projection uncertainty and 13 town locations, but with 12 scenarios discarded). For each scenario, we created three visualizations (COU, ART-10, and ART-50), for a total of 144 COU stimuli and 288 ARTs stimuli.
An illustration of ARTs with (a) 10 icons and (b) 50 icons. These trials show a forecast with the most likely storm path set to an angle of 90° and medium projection uncertainty.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-21-0173.1
3) Procedure
Instructions were identical to experiment 1. For each trial, participants viewed a forecast shown on a map of the Gulf of Mexico coastline of the United States. To assess perceived risk, we asked participants to indicate whether the town should be evacuated. The visualization was displayed until they indicated their response. No feedback was provided about their response.
Participants completed one block for each visualization condition. The block with the COU consisted of 144 trials (so each participant saw all scenarios presented using COUs). To keep the number of trials consistent between the two visualization types, the block with ARTs consisted of 144 trials randomly selected from 288 possible stimuli for each participant. Because the selection was randomized, some participants saw more ARTs with only 10 icons, and some saw more ARTs with 50 icons. Starting visualization was counterbalanced across participants, and the order of trials within each block was randomized. Participants completed a total of 288 trials, which took approximately 15 min.
b. Results
We analyzed the data using similar GLMM specification as in experiment 1. Results from the model are shown in Fig. 9.
Risk perception (quantified as mean evacuation rates) as a function of angular deviation, prediction uncertainty, and visualization condition for experiment 2. Angular deviation is measured in degrees from the town to the most likely storm path. Shading represents 1 SEM as calculated from the model.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-21-0173.1
1) Risk perception across town location
Participants perceived risk to be higher when there were more icons than when there were fewer icons. A pairwise comparison averaged across all town angles between the two ART conditions showed lower evacuation rates with 10 icons than with 50 icons [odds ratio (OR) = 0.12, standard error (SE) = 0.03, z = −8.03, and p < 0.001].
2) Risk transitions
Participants showed a steeper slope in risk perception with the COU (estimate = 0.10, with 95% CI of [0.09, 0.11]) than both the ARTs 10 condition (estimate = 0.04, 95% CI of [0.03, 0.05], with p < 0.001) and the ARTs 50 condition (estimate = 0.06, 95% CI of [0.05, 0.08], with p < 0.001). This replicates the steeper decrease in risk perception when a forecast is shown as a COU when compared with ARTs. This is consistent with a containment effect occurring with the COU visualization.
3) Decision thresholds
Decision thresholds varied across the visualization conditions. The number of icons had a large effect on decision thresholds. The decision threshold for the ARTs with 10 icons condition was 22.44°, with 95% CI of [12.11, 35.69], the decision threshold for ARTs with 50 icons was 41.04°, with 95% CI of [33.24, 49.50], and the decision threshold for the COU was 25.97°, with 95% CI of [12.47, 31.01]. Decision thresholds were approximately 83% farther out for 50 icons than for 10 icons. Thus, risk can be effectively communicated using the number of icons as well as their distribution. This flexibility to communicate risk is not afforded by the visual information in the COU, which may need additional markers such as the category of hurricane being forecast, a corresponding legend, and additional text. The additional markers for the COU lead to a potential issue of information clutter, and despite their importance, are neglected (Millet et al. 2023).
4) Projection uncertainty
As in experiment 1, the COU led to a containment effect. Decision transitions were abrupt, and decision thresholds occurred just beyond the edge of the cone (see Fig. 10). In contrast, for the ARTs with 10 icons, the decision threshold was at approximately the same angle regardless of projection uncertainty (as shown by similar decision thresholds), but risk transitions increased dramatically (as shown by steeper risk transitions, i.e., slope of the curves, with fewer uncertainties; see Fig. 11). The fact that the curves are steeper for the lower-uncertainty condition means the risk transitions increased with increasing certainty. While we did not make this pattern as a prediction a priori, in this scenario, fewer icons were grouped more closely in the high-uncertainty condition. This may have otherwise helped indicate the areas at the greatest risk, which would have led to steeper risk transitions, which is what we found. While fewer icons can indicate lower risk, when projection uncertainty is high, too few icons made it challenging to see the distribution, thus leading to more variable responses and shallower slopes. For 50 icons, increasing projection uncertainty led to greater spread of the icons and increased the risk transitions, which led to the decision thresholds being farther out.
Risk perception as a function of angular deviation, visualization condition, and projection uncertainty for experiment 2. Shading represents 1 SEM calculated from the model. The angular deviation at which the horizontal, dashed black lines intersect each curve corresponds to the decision thresholds. The steepness of each curve corresponds to the risk transitions.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-21-0173.1
Estimated (a) risk transitions and (b) decision thresholds as a function of projection uncertainty and visualization condition for experiment 2. Higher risk transitions correspond to steeper slopes, whereas lower risk transitions correspond to more gradual decreases in risk perception as angular distance increases. Decision thresholds correspond to the angular distance at which evacuation rates were estimated to be 50% according to the model. Higher decision thresholds indicate the decision threshold occurred at a distance farther from the most likely storm path. Horizontal dotted lines correspond to the edge of the cone and 1 SD of the ARTs. Error bars represents 95% CIs.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-21-0173.1
c. Discussion
The number of icons clearly communicate threat and had a large impact on perceived risk. Perceived risk was substantially higher when more icons were shown than when fewer icons were shown. When fewer icons were shown, even areas at high risk were evacuated at low rates. From a practical standpoint, the number of icons could be used to help communicate the category of an impending hurricane or the potential risk factors associated with it. For example, fewer icons would communicate lower risk, whereas more icons would communicate higher risk. The number of icons could also be manipulated based on look-ahead time. At a longer look-ahead time to landfall, the use of fewer instances in the visualizations might promote more awareness of potential future risk without triggering premature evacuations. However, this suggestion must first be tested because showing fewer icons at one time point and more icons at another time point could also lead to distrust in the forecast. Additional research is necessary to measure the impact of ARTs and changes in storm projections as visualized by ARTs on public trust and behavior.
The current results again support the claim that the COU leads to a containment effect (Padilla et al. 2018). Regardless of projection uncertainty, participants generally evacuated towns within the cone and did not evacuate towns outside the cone. This was shown by the strict risk transitions and the decision thresholds that aligned just beyond the edge of the cone, thereby evacuating towns contained within the cone. If this type of behavior is desired, the COU is an excellent choice, although the density of the icons with ARTs could likely be selected to also give rise to a containment effect. In contrast, the ARTs can lead to risk perception that gradually decreases as the risk decreases. Participants were more aware of the risk beyond the towns closer to the center of the COU/ARTs. Furthermore, the level of risk was perceived relative to the number of icons, with higher risk perception when there were more icons. However, a caveat is that more icons are needed to visualize the distribution of projected storm paths. A sample of 10 icons will lead to variable responses when the distribution is wide. The correct number of icons will depend on the level of risk imposed by the impending storm and the spread of its predicted area. If the spread is wide, more icons will be needed to reveal the distribution.
4. Experiment 3
Sometimes, output from operational weather forecast models group into separate clusters of possible hurricane tracks. As discussed for one case study in the introduction (e.g., Fig. 2), the forecast for Hurricane Irma projected the storm to travel up the east coast of Florida or, with less probability, the west coast of Florida. Initially, the underlying knowledge of meteorologists indicated an increased risk along both the coasts rather than one uniformly distributed across the state’s interior, but no such sense would have been conveyed to the general public when the storm projection was presented using only the COU (Pittman 2017). For example, as the storm drew closer, the cone centered first on the east coast of the Florida peninsula and only later shifted to the west coast of Florida.
ARTs are not restricted to following a normal distribution, and research on vision and visual displays suggests that ensemble displays are effective in depicting nonnormal distributions, such as bimodal distributions (Szafir et al. 2016). To determine whether nonexperts are sensitive to the shape of the distribution of ARTs, we conducted a simplified experiment in which the distribution of icons was either unimodal or bimodal. Furthermore, the experiment poses a more direct test of whether the density of icons is a key visual feature in dictating evacuation decisions because areas at the edge of the central tendency will have greater densities of icons than in the center for the bimodal distributions.
a. Method
1) Participants
Thirty-one participants enrolled in an introductory psychology course at Colorado State University received course credit after completing the experiment. Participants had self-reported normal or corrected-to-normal vision and had not participated in experiments 1 or 2.
2) Stimuli
Stimuli were created in R and presented in Qualtrics. The most likely storm path was set to one of four angles (50°, 70°, 90°, or 110°). The projection uncertainty was either medium or high (20° or 30°). We eliminated the low projection uncertainty condition to reduce the number of stimuli, given that we added more angular deviations and the low uncertainty was not particularly realistic. Angular deviation of the town relative to the most likely storm path was set to 1 of 16 angles for a total of 128 unique scenarios (4 × 2 × 16 = 128). For each scenario, three visualizations were created. One was the COU, which had the central path along the storm angle and the width of the cone set to the projection uncertainty. Another visualization was the unimodal ARTs. For this visualization, 50 angles were sampled from a normal distribution, with a mean set to the storm angle and the standard deviation set to the projection uncertainty. For the bimodal ARTs, 25 angles were randomly sampled from a normal distribution, with the mean set to the storm angle plus the projection uncertainty (e.g., 90° + 20° = 110°). Another 25 angles were randomly sampled, with the mean of the distribution set to the storm angle minus the projection uncertainty (e.g., 90° − 20° = 70°). For both, the standard deviation of the distribution was the projection uncertainty divided by 2 (e.g., 20°/2 = 10°). Figure 12 shows an illustration of this example. For both ART displays, small random noise was added to each icon’s movement to increase visibility of the icons.
An illustration of ART trials with (a) a bimodal distribution and (b) a unimodal distribution. For both examples, the angle of the storm’s path is 90° and the projection uncertainty is 20°. For each trial, only one visualization was shown.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-21-0173.1
3) Procedure
The task was the same as in experiment 1 (see also Fig. 12). Participants completed one block of trials with COU images and one block with both kinds of ARTs intermixed. Initial visualization type was counterbalanced across participants. Due to a programming error, 16 cone images and 32 ART images from a prior experiment were included. The trials showing stimuli from prior experiments were excluded from the analyses (11% of the data). The COU block contained 96 trials randomly selected from 144 possible stimuli (128 scenarios plus 16 unintended stimuli), and the ART block contained 192 trials randomly selected from 288 possible stimuli (128 scenarios times 2 types of ARTs, plus 32 unintended stimuli). The order of presentation within the block was randomized for each participant. Because selection was random, some participants saw more stimuli that were bimodal ARTs, whereas other participants saw more stimuli that were unimodal ARTs. At extremes, one participant saw 46% unimodal ARTs (and 54% bimodal) and one participant saw 55% unimodal (and 45% bimodal) ARTs. On average, participants saw 50% of each.
b. Results
The critical result was whether participants would perceive risk for areas located in the center of the storm’s overall paths even when the storm projection was presented as a bimodal distribution, or whether their perception of risk would group based on how the icons were grouped. Mean evacuation rates across angular deviation are shown in Fig. 13. Because we did not fit linear models to the data, we only plotted the means [and corresponding standard errors of the mean (SEMs)]. For the high-uncertainty projections, evacuation rates were low for towns located in between the two distributions of icons (town position < 20°). This effect was reduced but still present for the medium-uncertainty projections.
Risk perception (quantified as mean evacuation rates) as a function of angular deviation, visualization condition, and projection uncertainty for experiment 3. Error bars are 1 SEM calculated within subjects.
Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-21-0173.1
To analyze the data, we ran a GLMM for each level of projection uncertainty. The dependent measure was evacuation decision (coded as 1 for evacuate and 0 for not evacuate). The fixed effect was visualization condition. We only included data for which the towns were centrally located (0°–10° for the medium-uncertainty condition, 945 trials included in analysis; 0°–20° for the high-uncertainty condition, 1199 trials included in the analysis). All data were from trials for which the town position was located between the two bimodal distributions. Evacuation rates for these central areas were lower for the bimodal ARTs, with M = 0.87, and 95% CI of [0.78, 0.93], than for the unimodal ARTs, with M = 1.00 and 95% CI of [0.996, 1.00], or the cone, with M = 0.97 and 95% CI of [0.94, 0.99], for the medium-uncertainty condition, both with p < 0.003 (p = 0.15 for comparison of COU and unimodal ARTs). The difference was even greater for the high-uncertainty condition, with lower evacuation rates for the bimodal ARTs, with M = 0.51 and 95% CI of [0.40, 0.63], than for the unimodal ARTs, with M = 0.97 and 95% CI of [0.93, 0.99], or the COU, with M = 0.98 and 95% CI of [0.96, 0.99], both with p < 0.001 (p > 0.99 for comparison of COU and unimodal ARTs).
c. Discussion
Participants were sensitive to the shape of the distribution of ARTs. Perceived risk, quantified as evacuation rates, closely aligned with the distribution of icons. This result sets up the case that ARTs have flexibility in their design for showing clusters of weather forecast model projections.
Risk perception for towns located in the middle of the bimodal distribution was considerably lower in the high-uncertainty condition than in the medium-uncertainty condition. One possible explanation is that there was a larger separation between the left and right ARTs when uncertainty was high than when it was medium. As a result, the town was closer to the nearest ARTs in the medium-uncertainty condition. If this explains the discrepancy between the medium- and high-uncertainty conditions, proximity of the town to the closest ARTs may be one visual cue used to signal risk. The results are consistent with the idea that the density of icons provides a signal to users about the amount of risk at each location: more icons are interpreted as more risk, whereas fewer icons are interpreted as less risk. Overall, this experiment’s findings highlight the viability of ARTs as a visualization to convey to the public a detailed and more nuanced understanding of potential storm trajectories where some paths may be more likely and others highly unlikely.
5. General discussion
The COU is one of the most recognized forms of communication about tropical weather events (Broad et al. 2007; Bostrom et al. 2018; Senkbeil et al. 2020), but it can also lead to systematic misinterpretations (Broad et al. 2007; Cox et al. 2013; Padilla et al. 2018). To better communicate hurricane forecasts to nonexperts, alternative visualizations should be considered. Here, we explored a new visualization called ARTs for communicating potential hurricane tracks. ARTs are not intended to be an animated version of track ensemble plots; rather, they are composed of a set of small icons that move across a map to convey information about the storm’s projected path. The development of ARTs is supported by work that demonstrates how the visual system can quickly and accurately detect summary statistics, like uncertainty quantification, from groups of objects (e.g., Whitney and Yamanashi Leib 2018). They are also supported by work that demonstrates that people are better able to make decisions under uncertainty when the information is presented as natural frequencies rather than as proportions (Gigerenzer and Hoffrage 1995). Whereas the COU visualizes the forecast as a probability (the boundaries are based on historical errors such that 33% of storms are anticipated to track outside the COU boundaries), ARTs visualize the forecast as a frequency (e.g., 10 icons of 50 move across or near the cued location).
The current experiments tested whether a visualization that portrayed hurricane projections as a group of animated icon trajectories would better convey risk information than the COU. In our experiments, we measured risk perception as mean evacuation rates for towns at varying angles from the central most probable forecast track. The ART visualization led to similar perceptions of risk as the COU for the areas closest to the most likely storm path. However, the ARTs were interpreted as portraying more risk beyond the central storm track when compared with the COU. As previously discussed, the COU has been shown to lead to a containment effect (e.g., Padilla et al. 2018), and we also found this issue with the COU in our experiments, as shown by steeper slopes and steeper risk transitions at the edge of the COU. In contrast, the ARTs led to a more gradual decline in risk perception, which is not consistent with a containment effect.
Furthermore, the ARTs allow considerable flexibility in how the information is presented. When the ARTs included more icons, the ARTs were interpreted as representing higher risk, and fewer icons led to lower perceived risk. Understanding how the number of icons impacts evacuation rates may allow for more flexibility in how information is conveyed. For example, with an approaching storm that puts the coast at high risk, a visualization could communicate this risk by showing more icons. Because of this flexibility, rules need to be set to determine how many icons should be shown for communication purposes. These rules do not yet exist and will need to be tested in future work before ARTs should be shown to the public. Future research could investigate whether people interpret more ARTs as a more severe storm or whether they perceive the threat of a hurricane as more likely, because we did not differentiate between these options.
Another aspect of flexibility afforded by the ARTs is the ability to show different distributions of projected storm paths. The COU looks the same regardless of whether the hurricane projections are unimodal, bimodal, uniform, or skewed (National Hurricane Center 2022). In contrast, the ARTs can be drawn from the same distribution as the numerical weather prediction models or allow meteorologists to tailor the ARTs display to communicate potential risk at areas outside the mean of the forecast tracks. The current research found that with the ARTs, people were sensitive to information about bimodality and could use this information to judge risk.
Although track ensembles seem to reduce some biases found with the COU, the track ensembles are prone to a different bias related to whether a given location happens to be on a specific track line. When asked how they made decisions, nonexperts indicated that they used proximity to one of the tracks in the display (Ruginski et al. 2016). Similarly, locations situated along one of the tracks were perceived as being at higher risk than locations that were closer to the most likely storm path but not directly on a track (Padilla et al. 2017). While we did not test whether ARTs are also prone to this bias in the current work, it is an open question for future research. Last, we also have not yet tested whether ARTs lead to other biases related to storm size. For example, if the icons are tightly clustered (i.e., the forecast track is fairly certain), then communicating that a large storm has far-reaching impacts would be a concern, especially if the icon distribution is randomly selected as was done here. This remains to be explored in future work.
The visual features of the ARTs—such as their color, size, shape, and speed—could also be manipulated to communicate additional impacts such as wind speed, flash flooding, storm surge, and storm timing. Recent work showed people were sensitive to the color of the icons (Witt and Clegg 2022). Subsequent work has also found people were sensitive to color, size, and whether the icons were flashing (Witt et al. 2022b). The current research tested the impact of the number of icons and their distribution but did not yet test potential interactions between density and distribution. Research on how ARTs can best communicate the timing of an incoming storm is also necessary to explore the usefulness of ARTs for communicating to the public.
Although not the focus of the current research, future work should also explore the impact on trust and decisions by the public when viewing ARTs, particularly when the projected path of the storm changes dramatically. In cases for which a storm’s projected trajectory changes dramatically, the public may question whether the current forecast (and any future forecast) is to be trusted at all. Research is needed to test the most effective way to present forecasts with the ARTs in these cases.
6. Limitations
The current research assessed risk perception by having participants make evacuation decisions about a town given a hypothetical forecast. We do not claim, nor would we consider it likely, that actual evacuation rates would align with the levels of hypothetical evacuation decisions found here. Rather, we conjecture that the relative differences in risk perception across the two visualizations would generalize to real-world scenarios and decisions.
That the decision in the current experiments mimicked decisions made by emergency managers (whether to evacuate a town), rather than decisions made by the public (whether to evacuate oneself), is another limitation. However, we find similar results when participants are asked to make different decisions (Witt et al. 2022a).
Another limitation is that the participants in the current studies consisted of students from Colorado State University, which is in an area where there is no immediate risk of hurricanes. While prior experience with hurricanes is likely to impact decision-making, previous work has shown similar differences in risk perception between naïve and experienced participants across the COU and ARTs (Witt et al. 2021). Future work should aim to explore interpretation of the visualizations across a broader range of demographic characteristics (including different ages, educational levels, and cultural backgrounds).
The COU can sometimes be presented as an animation in news media outlets rather than as a static image as presented in the current experiments. Future work could compare an animated version of the COU with the ARTs to examine risk perceptions when both visualizations are in an animated form. These animations could then extend the forecast track over land areas, since changes in evacuation decisions for inland locations were not formally tested here.
Also, we note that there are explicit guidelines from the NHC for how to construct the COU. These guidelines do not yet exist for ARTs. Such guidelines must be set for ARTs to be a useful tool for the public.
7. Conclusions
Animated risk trajectories (ARTs) are proposed as an alternative to the cone of uncertainty (COU) to communicate hurricane risks to the public. The COU, despite its widespread recognition, is marred by misinterpretations. This is not surprising: the visual impression of the COU is so strong in its communication of a contained area that even prior experience cannot override this impression. Specifically, when we tested people living in Florida who had experienced that hurricane tracks deviate outside the areas specified by the COU, they still showed a strong containment effect (Witt et al. 2021). The ARTs leverage powerful visual processes that can quickly and accurately summarize statistical information from a group of objects, and ARTs offer considerable flexibility in the way information can be communicated for a variety of scenarios. ARTs also have simple visual–conceptual compatibility: their movements across a map are analogous to the movements of hurricanes across Earth. This compatibility should make their adoption quick and seamless.
Acknowledgments.
This work was supported by the National Science Foundation (Grants BCS-1632222 and SES-2030059), by a fellowship from the CSU School of Global Environmental Sustainability to author Witt, and by Dr. Jeff Morrison and the Office of Naval Research (N00014-20-1-2518) to author Clegg.
Data availability statement.
Materials, data, and analysis scripts are freely available on the Open Science Framework web page (https://osf.io/u6qh8/).
REFERENCES
Alley, R. B., K. A. Emanuel, and F. Zhang, 2019: Advances in weather prediction. Science, 363, 342–344, https://doi.org/10.1126/science.aav7274.
Anscombe, F. J., 1973: Graphs in statistical analysis. Amer. Stat., 27, 17–21, https://doi.org/10.1080/00031305.1973.10478966.
Becker, R. A., A. R. Wilks, R. Brownrigg, T. P. Minka, and A. Deckmyn, 2022: Package ‘maps’: Draw geographical maps. R Project Doc., 37 pp., https://cran.r-project.org/web/packages/maps/maps.pdf.
Boone, A. P., P. Gunalp, and M. Hegarty, 2018: Explicit versus actionable knowledge: The influence of explaining graphical conventions on interpretation of hurricane forecast visualizations. J. Exp. Psychol. Appl., 24, 275–295, https://doi.org/10.1037/xap0000166.
Bostrom, A., R. Morss, J. K. Lazo, J. Demuth, and H. Lazrus, 2018: Eyeing the storm: How residents of coastal Florida see hurricane forecasts and warnings. Int. J. Disaster Risk Reduct., 30, 105–119, https://doi.org/10.1016/j.ijdrr.2018.02.027.
Broad, K., A. Leiserowitz, J. Weinkle, and M. Steketee, 2007: Misinterpretations of the “cone of uncertainty” in Florida during the 2004 hurricane season. Bull. Amer. Meteor. Soc., 88, 651–668, https://doi.org/10.1175/BAMS-88-5-651.
Cangialosi, J. P., E. Blake, M. DeMaria, A. Penny, A. Latto, E. Rappaport, and V. Tallapragada, 2020: Recent progress in tropical cyclone intensity forecasting at the National Hurricane Center. Wea. Forecasting, 35, 1913–1922, https://doi.org/10.1175/WAF-D-20-0059.1.
Cox, J., D. House, and M. Lindell, 2013: Visualizing uncertainty in predicted hurricane tracks. Int. J. Uncertainty Quantif., 3, 143–156.
Dance, S., and J. Ajasa, 2022: Cone of confusion: Why some say iconic hurricane map misled Floridians. Washington Post, 4 October, https://www.washingtonpost.com/climate-environment/2022/10/04/hurricane-cone-map-confusion/.
DeMaria, M., C. R. Sampson, J. A. Knaff, and K. D. Musgrave, 2014: Is tropical cyclone intensity guidance improving? Bull. Amer. Meteor. Soc., 95, 387–398, https://doi.org/10.1175/BAMS-D-12-00240.1.
Demuth, J. L., R. E. Morss, B. H. Morrow, and J. K. Lazo, 2012: Creation and communication of hurricane risk information. Bull. Amer. Meteor. Soc., 93, 1133–1145, https://doi.org/10.1175/BAMS-D-11-00150.1.
Ding, Q., and B. Millet, 2020: Visualizing uncertainty in weather forecasts. Proc. Human Factors and Ergonomics Society Annual Meeting, Los Angeles, CA, Human Factors and Ergonomics Society, 1064–1068, https://doi.org/10.1177/1071181320641255.
Dormady, N., A. Fasano, A. Roa-Henriquez, D. Flanagan, W. Welch, and D. Wood, 2022: Informational determinants of large-area hurricane evacuations. Bull. Amer. Meteor. Soc., 102, E642–E656, https://doi.org/10.1175/BAMS-D-21-0008.1.
Emanuel, K., 2017: Will global warming make hurricane forecasting more difficult? Bull. Amer. Meteor. Soc., 98, 495–501, https://doi.org/10.1175/BAMS-D-16-0134.1.
Emanuel, K., and F. Zhang, 2016: On the predictability and error sources of tropical cyclone intensity forecasts. J. Atmos. Sci., 73, 3739–3747, https://doi.org/10.1175/JAS-D-16-0100.1.
Fechner, G. T., 1966: Elements of Psychophysics. Vol. 1, D. H. Howes and E. G. Boring, Eds., Holt, Rinehart and Winston, 286 pp.
Freeman, A. C., and W. S. Ashley, 2017: Changes in the US hurricane disaster landscape: The relationship between risk and exposure. Nat. Hazards, 88, 659–682, https://doi.org/10.1007/s11069-017-2885-4.
Gescheider, G. A., 2013: Psychophysics: The Fundamentals. Psychology Press, 446 pp.
Gigerenzer, G., and U. Hoffrage, 1995: How to improve Bayesian reasoning without instruction: Frequency formats. Psychol. Rev., 102, 684–704, https://doi.org/10.1037/0033-295X.102.4.684.
Hamill, T. M., M. J. Brennan, B. Brown, M. DeMaria, E. N. Rappaport, and Z. Toth, 2012: NOAA’s future ensemble-based hurricane forecast products. Bull. Amer. Meteor. Soc., 93, 209–220, https://doi.org/10.1175/2011BAMS3106.1.
Knoblauch, K., and L. T. Maloney, 2012: Modeling Psychophysical Data in R. Springer, 365 pp.
Liu, L., and Coauthors, 2017: Uncertainty visualization by representative sampling from prediction ensembles. IEEE Trans. Visualization Comput. Graphics, 23, 2165–2178, https://doi.org/10.1109/TVCG.2016.2607204.
Liu, L., L. Padilla, S. H. Creem-Regehr, and D. H. House, 2019: Visualizing uncertain tropical cyclone predictions using representative samples from ensembles of forecast tracks. IEEE Trans. Visualization Comput. Graphics, 25, 882–891, https://doi.org/10.1109/TVCG.2018.2865193.
Macmillan, N. A., and C. D. Creelman, 2004: Detection Theory: A User’s Guide. Psychology Press, 512 pp.
Magezi, D. A., 2015: Linear mixed-effects models for within-participant psychology experiments: An introductory tutorial and free, graphical user interface (LMMgui). Front. Psychol., 6, 2. https://doi.org/10.3389/fpsyg.2015.00002.
Matyas, C., S. Srinivasan, I. Cahyanto, B. Thapa, L. Pennington-Gray, and J. Villegas, 2011: Risk perception and evacuation decisions of Florida tourists under hurricane threats: A stated preference analysis. Nat. Hazards, 59, 871–890, https://doi.org/10.1007/s11069-011-9801-0.
Meyer, R., K. Broad, B. Orlove, and N. Petrovic, 2013: Dynamic simulation as an approach to understanding hurricane risk response: Insights from the Stormview lab. Risk Anal., 33, 1532–1552, https://doi.org/10.1111/j.1539-6924.2012.01935.x.
Millet, B., A. Cairo, S. J. Majumdar, C. Diaz, S. D. Evans, and K. Broad, 2020a: Beautiful visualizations slain by ugly facts: Redesigning the National Hurricane Center’s ‘cone of uncertainty’ map. OSF Preprints, accessed 31 August 2020, https://doi.org/10.31219/osf.io/wzk8p.
Millet, B., A. P. Carter, K. Broad, A. Cairo, S. D. Evans, and S. J. Majumdar, 2020b: Hurricane risk communication: Visualization and behavioral science concepts. Wea. Climate Soc., 12, 193–211, https://doi.org/10.1175/WCAS-D-19-0011.1.
Millet, B., S. J. Majumdar, A. Cairo, C. Diaz, Q. Ding, S. D. Evans, and K. Broad, 2021: End-user preference for and understanding of hurricane forecast graphs. Proc. Human Factors and Ergonomics Society Annual Meeting, Los Angeles, CA, Human Factors and Ergonomics Society, 606–610, https://doi.org/10.1177/1071181321651.
Millet, B., S. J. Majumdar, A. Cairo, B. D. McNoldy, S. D. Evans, and K. Broad, 2023: Exploring the impact of visualization design on non-expert interpretation of hurricane forecast path. Int. J. Hum. Comput. Interact., https://doi.org/10.1080/10447318.2022.2121036, in press.
Morss, R. E., J. L. Demuth, J. K. Lazo, K. Dickinson, H. Lazrus, and B. H. Morrow, 2016: Understanding public hurricane evacuation decisions and responses to forecast and warning messages. Wea. Forecasting, 31, 395–417, https://doi.org/10.1175/WAF-D-15-0066.1.
Moscatelli, A., M. Mezzetti, and F. Lacquaniti, 2012: Modeling psychophysical data at the population-level: The generalized linear mixed model. J. Vision, 12, 26, https://doi.org/10.1167/12.11.26.
Nadler, D. A., 1979: The effects of feedback on task group behavior: A review of the experimental research. Organ. Behav. Hum. Perform., 23, 309–338, https://doi.org/10.1016/0030-5073(79)90001-1.
National Hurricane Center, 2022: Definition of the NHC Track Forecast Cone. National Hurricane Center, accessed 26 February 2022, https://www.nhc.noaa.gov/aboutcone.shtml.
National Weather Service, and Eastern Research Group, Eds., 2020: Support for the Cone of Uncertainty Social and Behavioral Science Research Project. NOAA Final Rep., 23 pp., https://repository.library.noaa.gov/view/noaa/29111.
Newman, G. E., and B. J. Scholl, 2012: Bar graphs depicting averages are perceptually misinterpreted: The within-the-bar bias. Psychon. Bull. Rev., 19, 601–607, https://doi.org/10.3758/s13423-012-0247-5.
NOAA, 2023: Hurricane costs. NOAA, accessed 27 March 2023, https://coast.noaa.gov/states/fast-facts/hurricane-costs.html.
Padilla, L. M., I. T. Ruginski, and S. H. Creem-Regehr, 2017: Effects of ensemble and summary displays on interpretations of geospatial uncertainty data. Cognit. Res., 2, 40, https://doi.org/10.1186/s41235-017-0076-1.
Padilla, L. M., S. H. Creem-Regehr, M. Hegarty, and J. K. Stefanucci, 2018: Decision making with visualizations: A cognitive framework across disciplines. Cognit. Res., 3, 29, https://doi.org/10.1186/s41235-018-0120-9.
Pittman, C., 2017: Irma forecast wasn’t as far off as you may think, experts say. Tampa Bay Times, 12 September, https://www.tampabay.com/news/weather/hurricanes/irma-forecast-wasnt-as-far-off-as-you-may-think-experts-say/2337133/.
Pulver, D. V., and D. Rice, 2022: Many people misunderstand this famous hurricane forecast graphic. It can be a deadly mistake. USA Today, 19 October, https://www.usatoday.com/story/news/2022/10/19/hurricane-ian-new-criticism-for-the-cone-of-uncertainty/10529838002/.
R Core Team, 2019: R: A language and environment for statistical computing. R Foundation for Statistical Computing, accessed, https://www.gbif.org/tool/81287/r-a-language-and-environment-for-statistical-computing.
Ruginski, I. T., and Coauthors, 2016: Non-expert interpretations of hurricane forecast uncertainty visualizations. Spat. Cognit. Comput., 16, 154–172, https://doi.org/10.1080/13875868.2015.1137577.
Senkbeil, J., and Coauthors, 2020: Perceptions of hurricane-track forecasts in the United States. Wea. Climate Soc., 12, 15–29, https://doi.org/10.1175/WCAS-D-19-0031.1.
Steed, C. A., T. Jankun-Kelly, and J. E. Swan, 2009: Illustrative visualization techniques for hurricane advisory information. OCEANS 2009, Biloxi, MS, Institute of Electrical and Electronics Engineers, 1–9, https://doi.org/10.23919/OCEANS.2009.5422400.
Szafir, D. A., S. Haroz, M. Gleicher, and S. Franconeri, 2016: Four types of ensemble coding in data visualizations. J. Vision, 16, 11, https://doi.org/10.1167/16.5.11.
Tanner, W. P., Jr., and J. A. Swets, 1954: A decision-making theory of visual detection. Psychol. Rev., 61, 401–409, https://doi.org/10.1037/h0058700.
Tversky, B., 2011: Visualizing thought. Top. Cognit. Sci., 3, 499–535, https://doi.org/10.1111/j.1756-8765.2010.01113.x.
Weinkle, J., C. Landsea, D. Collins, R. Musulin, R. P. Crompton, P. J. Klotzbach, and R. Pielke Jr., 2018: Normalized hurricane damage in the continental United States 1900–2017. Nat. Sustainability, 1, 808–813, https://doi.org/10.1038/s41893-018-0165-2.
Whitney, D., and A. Yamanashi Leib, 2018: Ensemble perception. Annu. Rev. Psychol., 69, 105–129, https://doi.org/10.1146/annurev-psych-010416-044232.
Wickens, C. D., and C. M. Carswell, 1995: The proximity compatibility principle: Its psychological foundation and relevance to display design. Hum. Factors, 37, 473–494, https://doi.org/10.1518/001872095779049408.
Wickens, T. D., 2001: Elementary Signal Detection Theory. Oxford University Press, 276 pp.
Witt, J. K., 2019: Graph construction: An empirical investigation on setting the range of the y-axis. Meta-Psychology, 3, https://doi.org/10.15626/MP.2018.895.
Witt, J. K., and B. A. Clegg, 2022: Dynamic ensemble visualizations to support understanding for uncertain trajectories. J. Exp. Psychol. Appl., 28, 451–467, https://doi.org/10.1037/xap0000370.
Witt, J. K., B. A. Clegg, C. D. Wickens, C. A. P. Smith, E. L. Laitin, and A. C. Warden, 2020: Dynamic ensembles versus cones of uncertainty: Visualizations to support understanding of uncertainty in hurricane forecasts. Proc. Human Factors and Ergonomics Society Annual Meeting, Los Angeles, CA, Human Factors and Ergonomics Society, 1644–1648, https://par.nsf.gov/servlets/purl/10304289.
Witt, J. K., B. A. Clegg, L. D. Blalock, and A. C. Warden, 2021: The impact of familiarity on visualizations of spatial uncertainty. Proc. Human Factors and Ergonomics Society Annual Meeting, Baltimore, MD, Human Factors and Ergonomics Society, https://doi.org/10.1177/1071181321651.
Witt, J. K., Z. M. Labe, and B. A. Clegg, 2022a: Comparisons of perceptions of risk for visualizations using animated risk trajectories versus cones of uncertainty. Proc. Human Factors and Ergonomics Society Annual Meeting, Atlanta, GA, Human Factors and Ergonomics Society, 1716–1720, https://doi.org/10.1177/1071181322661308.
Witt, J. K., Z. M. Labe, and B. A. Clegg, 2022b: An alternative to the “cone of uncertainty” that is flexible, intuitive, and desirable for communicating hurricane forecasts. 17th Symp. on Societal Applications: Policy, Research and Practice, Houston, TX, Amer. Meteor. Soc., 11A.3, https://ams.confex.com/ams/102ANNUAL/meetingapp.cgi/Paper/395666.
Wu, H.-C., M. K. Lindell, C. S. Prater, and C. D. Samuelson, 2014: Effects of track and threat information on judgments of hurricane strike probability. Risk Anal., 34, 1025–1039, https://doi.org/10.1111/risa.12128.
Wu, H.-C., M. K. Lindell, and C. S. Prater, 2015: Strike probability judgments and protective action recommendations in a dynamic hurricane tracking task. Nat. Hazards, 79, 355–380, https://doi.org/10.1007/s11069-015-1846-z.
Yang, K., and Coauthors, 2019: Hurricane evacuations in the face of uncertainty: Use of integrated models to support robust, adaptive, and repeated decision-making. Int. J. Disaster Risk Reduct., 36, 101093, https://doi.org/10.1016/j.ijdrr.2019.101093.
