Improving disaster preparation, evacuation, survival, and recovery outcomes for those vulnerable to natural hazards such as hurricanes is an ongoing challenge. A better understanding of hurricane risk by the public could potentially improve timely decisions and life-saving evacuations. With this motivation in mind, we formed a team to integrate aspects of weather prediction, environmental anthropology, data visualization and design principles, human factors and user experience, and community psychology and engagement with a goal of exploring and evaluating the graphical communication of hurricane risk. To identify priorities, we first reviewed research on visual communications and how individuals process, understand, and make decisions regarding them (see Millet et al. 2020). This review suggested that graphical communication strategies need to be revised to better support the different ways in which people understand forecast products, and that these strategies should be tested for ecological validity in real world settings. Graphical products must be understandable by an end user, and they must account for cognitive biases that can foster misinterpretation. This study, and the knowledge gained by each of our team members outside our respective disciplines, spurred our team to employ multiple research methods, working with a variety of partners, to better understand how people process graphical forecast information. The phases of our collaboration involve generative research, information and visualization design, evaluative research, and the provision of design guidelines.
During our generative research phase, we first conducted eight focus groups (five in English, three in Spanish) in 2018–19, totaling 54 adult residents in Miami, Florida. At the time that the focus groups were being planned, the “Cone of Uncertainty” (hereafter “Cone”) product was the most frequently accessed graphic on the National Hurricane Center (NHC) website, which influenced our decision to begin with this graphic. Participants—all of them from the Miami area, and with different degrees of experience facing tropical storms and hurricanes—were asked to discuss their interpretations of the NHC Cone. The focus group participants had difficulty understanding the graphic due to the sheer amount of information, and there was frequent and consistent misinterpretation of the intended messages. The participants explained that they were interested in receiving clear information that would help them make informed decisions about what to do and when. More detailed findings from these focus groups will be discussed in a separate paper.
To evaluate how generalizable the focus group results were for interpreting hurricane forecast graphics, we developed an online survey targeting a much larger sample of Florida residents in 2019, the findings of which are the focus of this short article. Insights from the focus groups described above informed a series of questions to determine how well they were able to interpret the specific information that was intended to be communicated in the Cone graphic. Unlike previous public surveys that focused more on evacuation decisions, forecast usage, and perception of hurricane risk, our approach focuses on sources of hurricane risk communication information and pays attention to specific design aspects and other factors that contribute to graphic misinterpretations. While the overall project deploys a mixed-methods approach, this paper is focused on findings from this state-wide survey.
Survey methodology and findings
A total of 2,847 participants were recruited through Amazon Mechanical Turk (MTurk) and directed to the electronic survey hosted by Qualtrics. All participants were at least 18 years of age and had lived in Florida for at least 18 months. Each participant was paid $1.05 upon completing the survey.
The survey comprised 55 mostly multiple-choice and true/false questions, including demographic information and a series of 42 questions aimed at addressing three main research questions. About 2,500 residents completed the demographic portion of the survey, and a summary is provided in Table 1. Black/African American and Hispanic or Latino residents together with those born outside the United States were underrepresented in the survey. A disproportionately high fraction of respondents were under 35 years of age, and a disproportionately low fraction of senior or disabled citizens participated. The survey was comprised of residents with higher degrees and median incomes than the Florida-wide values. The highlights from three primary research questions are provided below.
Demographics of Florida survey residents. Responses of “not sure” or no response were discarded from the sample before the percentages were computed. Florida-wide statistics from the 2020 Census, where available, are provided for comparison. Percentages are rounded to the nearest integer, except for the disability statistic. Source of Florida Census data: www.census.gov/quickfacts/FL.
What are the most frequent and trusted sources of information that Florida residents use when they learn that a hurricane is coming their way?
The first question of how frequently the respondents used each source of information revealed that there was not one dominant source: those that were stated as being used “very frequently” (several times per day) were web/Internet (56%), TV (47%), and weather apps (45%) followed by social media (32%). In contrast, neighbors and relatives, local radio, and newspapers were “sometimes” (less than once per day) or “never” used by a majority of respondents. These results were largely as expected. We note that the respondents were not restricted to using just one source “very frequently,” and it is encouraging that multiple sources are used more than once per day by many respondents. The second question asked respondents to choose their most trusted source of information. In contrast to the first question, the standout was TV, which 39% of respondents stated they trusted the most. Only 28% stated web/Internet sources the most, even though the largest fraction of respondents accessed them “very frequently.” Interestingly, the fraction of respondents who trusted weather apps (14%) or social media (6%) above all other sources was small, suggesting that many respondents frequently used these sources knowing that they were not the most trustworthy. This may be due to people having more frequent access to their devices than their TV. Third, the respondents were asked an open-ended question to identify what specific TV/radio station, website, newspaper, social media platform, or weather app they most rely on for hurricane information. The Weather Channel and other weather-related sources stood out, with 33% of all respondents identifying these sources. CNN (12%) was the most widely used TV news channel. As expected, there was a wide variety of other sources (Twitter, Facebook, local TV, etc.), none of which exceeded 6%. Our results suggest that it is important to design graphics that are accessible and usable across multiple platforms.
How accurately are Florida residents able to interpret risk based on the NHC Cone graphics?
The participants were provided with the Cone graphic (Fig. 1) and were asked a series of questions to determine how well they were able to interpret the information that was intended to be communicated. First, using the overall graphic, they were asked (true/false) if they could find 1) the size of the storm (false); 2) the type of storm (e.g., tropical storm, hurricane; true); 3) where the storm could go in the next few days (true); 4) regions where watches and warnings have been issued (true); and 5) areas where damage will occur (false). Only 18% of the respondents answered all five questions correctly, although 80% answered three or more questions correctly. Questions 1 and 5 had the most incorrect responses. A total of 44% responded incorrectly that they could determine the forecasted size of the storm, and 40% responded incorrectly that they could identify areas where damage will occur. In addition to demonstrating a misinterpretation of what the Cone graphic directly conveys, it also may further support our focus group findings that members of the public are seeking more direct information on the size of the storm and where damage may occur.
Example of a National Hurricane Center Cone graphic used in the survey. The name and date were removed to reduce the chances of association with a specific storm.
Citation: Bulletin of the American Meteorological Society 103, 10; 10.1175/BAMS-D-21-0250.1
Next, the participants were asked to focus specifically on the Cone, and to respond (true/false) to whether the Cone shows 1) that there is more uncertainty about the forecast storm as it moves further into the future (true), 2) that areas outside the Cone are not predicted to be damaged (false), 3) all possible paths of the center of the storm (false), and 4) the extent of the damage of the storm or hurricane getting larger with time (false). Similar to the responses to the full graphic, only 18% of respondents were able to answer all four Cone questions correctly. A total of 48% of the respondents believed incorrectly that areas outside the Cone are not predicted to be damaged, and 48% believed incorrectly that the Cone shows all possible paths of the center of the storm. The participants were also asked true/false questions about the dotted region of the Cone in Fig. 1. A total of 58% of respondents incorrectly responded that the dotted area shows the storm at its largest size, and 46% responded incorrectly that the dotted area in the graphic shows areas that will endure heavy rain because of the storm or hurricane.
Using the four-color watches and warnings legend, participants were asked to rank the level of worry they would have if they lived in the areas colored to indicate watches and warnings. Although 58% were correctly “most worried” or “somewhat worried” about a hurricane warning (red), 29% stated that they were “least worried” about a hurricane watch. Next, the respondents’ ability to interpret the letters that convey the different intensity categories (D = depression, S = storm, H = hurricane, M = major hurricane) was evaluated. Only 53% of the participants stated that they were “most” or “somewhat” worried about “M.” Surprisingly, more people were “most worried” about “D” (29%) than “M” (26%). These results suggest that alternative colors or depictions for the watches and warnings, and alternative graphical illustrations of the intensity (e.g., the category as is often shown on TV stations), are worth testing.
What is the relationship, if any, between the number of correct interpretations and income, age, education, housing location, housing type, or “most trusted” sources of information?
Correlations between the number of questions with correct responses and each individual variable were computed, and regression/analysis of variance (ANOVA) significance tests were performed at the p < 0.05 level. The most significant result is that the number of correct responses was positively correlated with both the age and the level of income. On the other hand, and surprisingly, the number of correct responses was significantly negatively correlated with the education level (for questions about the full graphic and the dotted area). These findings suggest that older and more affluent residents in our sample are better able to interpret the full graphic while more educated residents in our sample make more mistakes in interpretation. However, since the number of correct responses is also positively related to age, one might conclude that experience with hurricanes could be a more important factor while income and/or education may have a more spurious relationship. Furthermore, since income level and education level are typically positively correlated, this finding might be due to the limited sample of low income and low education respondents.
Interestingly, for questions about the full graphic only, the number of correct responses was significantly negatively correlated with the distance from the coast that the respondent lived, suggesting that coastal residents are more aware of how to interpret the full graphic. No significant correlations were found based on the type of housing. Due to some problematic construction of response categories, we were unable to adequately determine if the mean number of correct responses for true/false questions is significantly different depending on “most trusted” sources of information: neighbors and relatives, TV, local radio, websites/Internet, local newspaper, social media, weather apps, and other. However, after reviewing the mean scores across the three sets of Cone questions, those who selected social media as their “most trusted” source of storm information had consistently lower mean scores of correct responses.
Lessons learned
Online surveys are a useful, albeit limited, tool to engage the public. Reflecting on the demographics of our survey participants (Table 1), we acknowledge that a limitation of this survey approach is that it was entirely disseminated online. Senior citizens may be less comfortable using the Internet, which likely contributed to their underrepresentation in our sample. The overrepresentation of respondents with higher education and income levels may be an Internet access issue as well. While recognizing these design limitations, our analysis confirmed that many residents have difficulty interpreting several aspects, suggesting a rethink on how to graphically communicate aspects such as uncertainty, the size of the storm, areas of likely damage, watches and warnings, and wind intensity categories. The Cone’s hard boundary and the “dotted” part of the Cone provided further confusion. The hard boundary created confusion regarding the containment of the storm path, and the extended (day 4–5) “dotted” part of the Cone led to interpretations unrelated to the intended message (e.g., heavy rain). Graphic designers should be cognizant that users who are inexperienced with tropical cyclones may not pay close attention to the legends and may be expecting app-like customization and interactive functionality on their phone. We note also that while the Cone graphic was most widely used at the time of our survey, NHC now emphasizes “Key Messages” when a user clicks on the storm symbol on the front page of their website.
The difficulties in interpreting the Cone graphic can negatively affect one’s ability to determine their level of risk and take action. Given that the Cone forms the basis for most graphics that are regularly shown on TV, websites, apps, and social media, it is necessary to make these visualizations not only easily understandable but useful for informing decision-making. Since TV was ranked as the most trusted source, it highlights the benefits of collaborating with the broadcasting community regarding on-air graphics. In parallel, the provision of public feedback to the National Weather Service (NWS) will help in their roadmap toward creating clear graphics on platforms that the public use. An acute understanding of the ever-changing use of social media is also necessary. One interesting finding from our survey was that users who trusted social media the most gave the fewest correct responses in their interpretation of the Cone graphic. As social media usage is increasing in different forms, continuous awareness and a rapid, flexible strategy is necessary to enable the community to receive clear, easily understandable graphics through these channels. While surveys can inform us about how well participants understand aspects of the graphics, they do not provide sufficient insight into why certain graphical components are often misinterpreted. The survey can provide a large-scale view; however, it may not be truly generative unless the methodology is combined with other empirical approaches. Mixed method approaches such as focus groups and laboratory experiments are expected to reveal nuanced and detailed insights on graphical interpretation, via in-depth discussions, participant observations, and eye tracking technologies. The ability to triangulate survey findings with these new insights on the thinking behind these responses will strengthen the findings and implications for determining future avenues for utilizing these findings to ultimately improve communication.
Alternative methods of surveying also require exploration. In our survey, we provided the participants with the standard Cone graphic in the same format as provided by NHC. However, a more sequential approach that focuses on individual elements of graphical design followed by selected combinations of elements may be more useful in identifying specific barriers to graphical interpretation. As is always the challenge with survey construction, participants’ incorrect responses to some questions may be, in part, due to the selection and phrasing of the questions. Questions that rely on a deeper application of existing design principles and practices derived from what we know about how humans perceive and process visual information may yield more revealing results.
One of our team’s ongoing goals is to benefit those living in the most vulnerable conditions. However, as is evident from Table 1, non-White residents were underrepresented, as were senior citizens and low-income residents. Proactive approaches are therefore necessary to overcome barriers to participation such as language or the lack of an online presence. More targeted sampling and increasing incentives are possible strategies. The focus groups that we conducted through local community organizations, including those exclusively in Spanish, have been effective, although the sample size is somewhat limited.
The combined lessons learned on graph literacy and public surveys need to be brought together into a cohesive argument. One may suggest from this and other studies that there is no “one size fits all” forecast product, given the wide spectrum of interpretations and incorrect responses. It may therefore be instructive to define the groups we wish to target and adopt different methodological approaches for each group. One advantage of public surveys is the ability to reach a very large number of people. Accordingly, it provides the ability to run broad analyses on subgroups (such as a specific demographic) if the population of these subgroups is sufficiently robust for statistical analysis. While we have emphasized the public in this paper, other key stakeholders, such as operational meteorologists on TV and social media, and emergency managers, continue to be targeted for surveys in recent studies.
In our participatory design process, the focus groups and survey were meant to be exploratory in nature before moving toward more advanced surveys and user experiments that explore newer, impact-oriented graphics. Future surveys and experiments will be used to evaluate new prototype visualizations against the benchmark of existing visualizations. Combined with the aforementioned mixed-methods approaches, the public feedback will in turn provide suggestions for future visualizations. This interdisciplinary collaboration is aimed not only at responding to an urgent need to improve the communication of hurricane risk. We also seek to help build a new foundation for parallel collaborations with community groups, agencies, and experts on other weather- and climate-related risks, with emphasis on unequal impacts of weather and climate phenomena on priority populations that continue to be marginalized due to poverty, structural racism, or other factors.
Acknowledgments.
The authors are grateful for funding support from the University of Miami’s Laboratory for Integrative Knowledge (U-LINK) and to Morgan Asmussen for assisting with the interpretation of the survey results. We are also grateful to David Zelinsky and colleagues at NHC for discussions and for providing website statistics.
Data availability statement.
The survey data utilized for this paper are available at https://doi.org/10.17604/XX1X-7X70. The survey comprised 55 mostly multiple-choice and true/false questions, including demographic information and a series of 42 questions aimed at addressing three main research questions.
References
Boone, A. P. , P. Gunalp , and M. Hegarty , 2018: Explicit versus actionable knowledge: The influence of explaining graphical conventions on interpretation of hurricane forecast visualizations. J. Exp. Psychol. Appl., 24, 275–295, https://doi.org/10.1037/xap0000166.
Broad, K. , A. Leiserowitz , J. Weinkle , and M. Steketee , 2007: Misinterpretations of the “cone of uncertainty” in Florida during the 2004 hurricane season. Bull. Amer. Meteor. Soc., 88, 651–667, https://doi.org/10.1175/BAMS-88-5-651.
Cairo, A. , 2010: Why Charts Lie: Getting Smarter About Visual Information. Norton, 256 pp.
Demuth, J. L. , R. E. Morss , B. H. Morrow , and J. K. Lazo , 2012: Creation and communication of hurricane risk information. Bull. Amer. Meteor. Soc., 93, 1133–1145, https://doi.org/10.1175/BAMS-D-11-00150.1.
Eosco, G. , 2008: A study of visual communication: Cyclones, cones, and confusion. M.S. thesis, Dept. of Communication, Cornell University, 153 pp., https://ecommons.cornell.edu/handle/1813/11170.
Lee, S. , B. Benedict , Y. Ge , P. Murray-Tuite , and S. Ukkusuri , 2021: An application of media and network multiplexity theory to the structure and perceptions of information environments in hurricane evacuation. J. Assoc. Inf. Sci. Technol., 72, 885–900, https://doi.org/10.1002/asi.24456.
Meyer, R. , K. Broad , B. Orlove , and N. Petrovic , 2013: Dynamic simulation as an approach to understanding hurricane risk response: Insights from the Stormview Lab. Risk Anal., 33, 1532–1552, https://doi.org/10.1111/j.1539-6924.2012.01935.x.
Millet, B. , A. P. Carter , K. Broad , A. Cairo , S. D. Evans , and S. J. Majumdar , 2020: Hurricane risk communication: Visualization and behavioral science concepts. Wea. Climate Soc., 12, 193–211, https://doi.org/10.1175/WCAS-D-19-0011.1.
Morss, R. E. , J. L. Demuth , J. K. Lazo , K. Dickinson , H. Lazrus , and B. H. Morrow , 2016: Understanding public hurricane evacuation decisions and responses to forecast warning messages. Wea. Forecasting, 31, 395–417, https://doi.org/10.1175/WAF-D-15-0066.1.
Responsive Management, 2018: Public survey on the National Hurricane Center’s arrival of tropical-storm-force-winds product. NOAA Final Rep., 126 pp., https://repository.library.noaa.gov/view/noaa/28905.
Trumbo, C. , L. Peek , M. Meyer , H. Marlatt , E. Gruntfest , B. D. McNoldy , and W. H. Schubert , 2016: A cognitive-affective scale for hurricane risk perception. Risk Anal., 36, 2233–2246, https://doi.org/10.1111/risa.12575.
Zhang, F. , and Coauthors, 2007: An in-person survey investigating public perceptions of and responses to Hurricane Rita forecasts along the Texas coast. Wea. Forecasting, 22, 1177–1190, https://doi.org/10.1175/2007WAF2006118.1.
