How Visual Design of Severe Weather Outlooks Can Affect Communication and Decision-Making

a. MetService Severe Weather Outlook

MetService is the official alerting authority for severe weather in NZ (NEMA 2015). MetService issues three different types of forecasts to notify about severe weather: warnings (next 24 h), watches (1–2 days out), and outlooks (3–6 days out). The MetService Severe Weather Outlook targets widespread (1000 km²) wind, rain, and snow events in NZ that meet any of the following “severe weather” thresholds: rainfall greater than 50 mm within 6 h or 100 mm within 24 h; snowfall below 1000 m on the North Island or in the south Canterbury or Otago region of NZ or below 500 m elsewhere on the South Island of NZ, with a snow depth of 10 cm within 6 h or 25 cm within 24 h; and severe gales, with a minimum mean speed of 90 km h⁻¹ or frequent gusts exceeding 110 km h⁻¹. The events are typically driven by synoptic features, such as low-pressure systems and fronts, that can be identified and predicted several days in advance. Unlike the NOAA SPC Convective Outlooks (which cover 1, 2, 3, and 4–8 days out), they do not assess population exposure or risk, and they do not include phenomena such as thunderstorms, hail, and tornadoes, which occur at a localized scale in NZ and are targeted by the MetService Thunderstorm Outlook (which covers the next 24–36 h). A 2017 survey found that the NZ public is most familiar with the MetService’s Severe Weather Warning and least familiar with the longer-range Severe Weather Outlook (Perceptive 2017). Although web usage statistics indicate that Severe Weather Outlooks are accessed much less frequently than daily weather forecasts, engagement with the outlook graphics on social media can increase greatly in the lead-up to public holiday periods and long weekends. This aligns with previous studies that have found that planning weekend activities and travel is one of the most common uses of weather forecast data among public users (Lazo et al. 2009).

The MetService Severe Weather Outlook shows a national-scale map of NZ, with areas of potential large severe weather events outlined in colored hashed lines, with a connector arrow that lists the date, type of severe weather forecast (heavy rain, strong winds, or heavy snow), and associated confidence (Fig. 1; https://www.metservice.com/warnings/severe-weather-outlook). The outlook includes forecasts for 3–6 days out and is land based only, with limited geographical information (e.g., no roads or city labels) and does not include marine areas. In these outlooks, confidence is defined as one of three different likelihoods that the severe weather event will “actually happen” [low (20%), moderate (40%), or high (60%)], which is assigned through the expert judgment of experienced forecasters working on the day. The color of an outlined forecast area on the map is not explicitly linked to a meteorological process, timing, or level of confidence but is chosen from a fixed set of 15 colors at the discretion of the forecaster. Although there is no formal guidance on how to apply colors, forecasters typically use brighter colors (e.g., red or magenta) for areas of higher confidence or more extreme impacts, and they may apply similar colors to areas of the same meteorological phenomena. There are potential limitations associated with this approach, and little evidence is currently available to guide such design choices. This has in part motivated this study into how alternative visualization choices influence audience perceptions.

View Full Size

MetService Severe Weather Outlook published 9 May 2018. It is the control map used to inform this study.

Citation: Weather, Climate, and Society 15, 4; 10.1175/WCAS-D-23-0010.1

• Download Figure
• Download figure as PowerPoint slide

The outlooks show four days of forecast weather on one composite map that represents all severe weather forecast for 3, 4, 5, and 6 days out from the current day. On the MetService web site, the outlooks are accompanied by a detailed text description of the forecast and a key that defines the confidence levels and likelihoods. However, the outlook is often shared as a standalone image across other social and traditional media platforms, without the key or full accompanying text description. It is therefore important to understand how audiences engage with and understand the outlook image as an individual product. Here, we explore the effect of three outlook design variables: visualization of the time frame, expressions of uncertainty, and representation of the forecast area and weather type.

b. Time frame visualization and temporal uncertainty

The multiday national nature of outlook forecasts requires users to extract complex visual information about different types of events over multiple days and regions in order to understand which type of severe weather is occurring on which days. To assist in making sense of such information, visual variables are processed by an individual’s working memory and mentally organized into groups through a process called chunking (Miller 1956; Egan and Schwartz 1979). Chunking helps to increase the amount of information that can be encoded and stored in our fixed-capacity working memory, which can reach high cognitive loads when large amounts of complex information are being processed (Evans and Stanovich 2013; Thalmann et al. 2019). Scientific visualizations can facilitate chunking of information for readers by using visual variables and design principles, such as color and layout, to drive visual attention to relevant information (Patterson et al. 2014). Chunking can reduce complexity and clutter to help nonexpert users comprehend meteorological data more easily (Harold et al. 2016).

When considering how to present “chunked” information about temporal uncertainty, one design factor to evaluate is whether the map should be displayed in a “univariate” (designing multiple graphics for comparison side by side) or “bivariate” (designing one graphic with extrinsic and intrinsic elements to display different layers of information) manner (Kunz et al. 2011; Kubíček and Šašinka 2011; Gould et al. 2014; Kinkeldey et al. 2014; Doyle et al. 2019). Kubíček and Šašinka (2011) have identified that the latter encourages parallel processing, while the former encourages serial processing, which requires greater information to be held in the working memory for the comparison to occur. The design choices thus each have trade-offs for decision-making, with univariate maps resulting in slower but more accurate interpretations and bivariate maps resulting in quicker but less accurate interpretations.

In this study, we compare methods for perceptually chunking time in outlook forecasts to explore how the different time frame displays affect comprehension of forecast information, perceptions of temporal uncertainty, and ease of reading information from the outlook. We compare the following three conditions, illustrated in Fig. 2: 1) a composite (bivariate) outlook that shows all 4 days of the forecast on one map (the MetService Severe Weather Outlook’s current approach), 2) an animated outlook (univariate) that shows a map for each day displayed one at a time for 3 s each in a loop (similar to the NOAA SPC Convective Outlooks day 4–8 graphic; https://www.spc.noaa.gov/products/exper/day4-8/), and 3) a paneled outlook (univariate) that shows each day presented in an individual panel side by side.

View Full Size

The (a) composite, (b) animated, and (c) panel maps modified from the MetService Severe Weather Outlook control (Fig. 1) and used in the time frame visualization experiment (experiment A).

Citation: Weather, Climate, and Society 15, 4; 10.1175/WCAS-D-23-0010.1

• Download Figure
• Download figure as PowerPoint slide

Understanding the severe weather forecast on each day of an extended forecast and its associated uncertainties is important for planning and decision-making across the outlook time frame. For example, it could inform advance planning about whether to carry out an activity on day 3 or 4 of the forecast window. Previous studies have found that perception of likelihood and uncertainty can vary across multiday forecast windows, but it is unclear how visual representation of the multiday time frame influences this process. People anticipate uncertainty in weather forecasts, but their perception of the uncertainty changes for longer lead times (Morss et al. 2008; Lazo et al. 2009; Joslyn and Savelli 2010). For example, Joslyn and Savelli (2010) found that a forecast of rain or snow for the next day was deemed much more likely to occur than a forecast for rain or snow 3 days out. This discrepancy between perceived likelihood and forecast likelihood may arise due to individuals overlaying the forecast with their mental model of how the weather “works,” such as its likelihood distribution through time (see, e.g., Doyle et al. 2020), similar to the “base rate effect” (Wallsten et al. 1986; Weber and Hilton 1990; Patt and Dessai 2005; Visschers et al. 2009). Such mental models and biases could play a role in the way audiences interpret outlooks over multiple days in the future and how they perceive and respond to the forecasts. We therefore also explore how perceptions of likelihood, level of concern, and decisions change across the time window of the outlook and whether or not this is affected by the way in which days are shown. We test the following null hypotheses about time visualization approaches, comparing the composite outlook with outlooks with each day of weather represented separately (animated and paneled):

• H1: There is no relationship between the way that days are presented on the outlook (composite or separate) and the accuracy of reporting the days on which severe weather will occur.

• H2: There is no relationship between the way that days are presented on the outlook (composite or separate) and the reported likelihood of severe weather occurring on a day that it is not forecast, when severe weather is forecast to occur on adjacent days.

• H3: There is no relationship between the way that days are presented on the outlook (composite or separate) and the level of concern about driving on a day that severe weather is not forecast, when severe weather is forecast to occur on adjacent days.

• H4: There is no relationship between the way that days are presented on the outlook (composite or separate) and the reported likelihood of changing weekend plans.

c. Perception and expression of uncertainty

Uncertainty can also play an important role in how people interpret information from forecast visualizations (Spiegelhalter et al. 2011; Harold et al. 2016). In a systematic review of geospatial visualizations of uncertainty, Kinkeldey et al. (2014) identify five dichotomous categories for uncertainty visualization, including 1) explicit/implicit, 2) intrinsic/extrinsic, 3) visually integral/separable, 4) coincident/adjacent, and 5) static/dynamic. Here, we investigate extrinsic and intrinsic approaches to communicating forecast confidence. Intrinsic approaches alter the existing symbology to represent uncertainty by manipulating visual variables, such as color saturation or hue. Extrinsic approaches add new objects to the display to depict uncertainty, such as symbols or text (see also Bostrom et al. 2008; Kunz et al. 2011; Deitrick and Wentz 2015). The current MetService Severe Weather Outlook graphic uses textual extrinsic expressions of qualitative uncertainty, where “low,” “moderate,” or “high” confidence is defined as a 20% (or one chance in five), 40% (or two chances in five), or 60% (or three chances in five) likelihood that “the event will actually happen,” that is, assigned using a label on the map.

Most previous studies have investigated intrinsic uncertainty depictions, which are considered effective for facilitating an overview of uncertainty (Kinkeldey et al. 2014). Extrinsic expressions, however, have been shown to aid in-depth analysis and support communication of qualitative assessments of uncertainty and have been identified as more suitable for specific locational versus overall uncertainty (Bostrom et al. 2008). Deitrick and Wentz (2015) similarly explored decision-making with “visually integral” and “visually separable” approaches to presenting uncertainty (where the former alters data symbology and the latter overlays uncertainty onto the map with patterns, textures, or shapes). They found that visually integral maps are preferred by decision-makers due to their simplicity, while visually separable images were preferred by researchers and scientists because they could more readily identify and evaluate the uncertainty. This builds on the suggestion by Slocum et al. (2003) that the expressions of uncertainty should consider audience background, as explored by Pang (2008) in a “task-oriented visual mapping” approach that is adapted to different users. The answer to how best to display this visual uncertainty is still unclear due to mixed findings. Thus, Kinkeldey et al. (2014) recommend more empirical studies on how visualization of uncertainty influences decision-making.

Joslyn and Savelli (2010) found that audiences are aware of the inherent uncertainty in meteorological forecasts but are prone to making unjustified assumptions about forecasting biases and uncertainty. For example, participants in their study consistently interpreted an overforecasting bias (perceiving the forecast to be more severe than reality) for extreme weather and an underforecasting bias (perceiving the forecast to be less severe than reality) for rain. Clear communication of uncertainty may be necessary to balance such biases in the interpretation of the forecast (Joslyn and Savelli 2010). Here, we compare comprehension and intended decision-making between extrinsic text expressions of confidence, with the most commonly used intrinsic expression—variations saturation (Kinkeldey et al. 2014; Fig. 3).

View Full Size

The (a) outlines + extrinsic, (b) shapes + extrinsic, and (c) shapes + intrinsic modified from the MetService Severe Weather Outlook control (Fig. 1) and used in the perception and expressions of uncertainty experiment (experiment B).

Citation: Weather, Climate, and Society 15, 4; 10.1175/WCAS-D-23-0010.1

• Download Figure
• Download figure as PowerPoint slide

We also explore underlying understandings and interpretations of the concept of confidence in relation to this outlook through the following open-ended question: “What do you think the different levels of ‘confidence’ in the forecast mean?” The World Meteorological Organization (WMO) recognizes “confidence” as a scale of uncertainty that represents a reliability measure (Gill et al. 2008). The Intergovernmental Panel on Climate Change (IPCC) defines “level of confidence” as the subjective expression of uncertainty based on the degree of understanding in the expert community and degree of agreement across findings and the amount of evidence to support those findings (Mastrandrea et al. 2010). These technical definitions highlight the nuance and complexity of uncertainty terminology and definitions [see review in Doyle et al. (2019)]. While communication of uncertainty is critical for informed decision-making with modeled scientific data, such as severe weather forecasts, the prevalence of many different metrics in the uncertainty lexicon, such as “chance,” “probability,” “confidence,” and “likelihood,” can obfuscate what they refer to, especially for nonspecialist audiences [see review in Hudson-Doyle et al. (2018)]. Here, we aim to better understand how people interpret “confidence” in the Severe Weather Outlook context to better understand the role this plays in making inferences from outlook content.

Visual variables in the forecast graphic design, such as shape, hue, size, and texture of forecast areas, can also affect perception of uncertainty by influencing cognitive and emotional responses to the information (e.g., MacEachren 1995; Hegarty 2011; Severtson and Vatovec 2012; Severtson and Myers 2013; Ash et al. 2014; Patterson et al. 2014; Thompson et al. 2015; Padilla et al. 2018). Visual saliency can affect visual attention to mapped data and influence perceived levels of threat (Grant and Spivey 2003; Clive et al. 2021). For example, forecast areas that capture attention through bold shading, bright color, or large size may be associated with higher perceptions of likelihood or degrees of threat. Previous studies have investigated the use of outlines and shaded areas to express probabilistic warning forecasts and found differences in inferences and decision-making between these approaches (e.g., Ash et al. 2014; Thompson et al. 2015; Cheong et al. 2016). Similarly, a recent study by Ernst et al. (2021) found that the colors and language used in NOAA SPC Convective Outlooks in the United States were misinterpreted by some audiences. Here, we compare approaches for representing forecast areas with different visual saliency (outlines vs solid shapes) to explore effects on perception and interpretation of uncertainty (Fig. 3). We test the following four null hypotheses about perceptions and expressions of uncertainty:

• H5: There is no relationship between the forecast area style (outlines or shapes) and the estimated percent confidence in the severe weather occurring.

• H6: There is no relationship between the expression of uncertainty (extrinsic or intrinsic) and the estimated percent confidence in the severe weather occurring.

• H7: There is no relationship between the expression of uncertainty (extrinsic or intrinsic) and the range of estimates of percent confidence in severe weather occurring for days when confidence is “low” and “moderate.”

• H8: There is no relationship between the expression of uncertainty (extrinsic or intrinsic) and the day chosen to take a walk.

d. Icons

Graphical representations, such as icons and symbols, are increasingly prevalent in modern mobile applications for weather forecasts. Symbols such as a sun, a cloud, or raindrops are commonly used as icons in forecast products around the globe to quickly communicate simple weather information (Zabini 2016). Studies show that combining icons with text information is preferred among users (Carr et al. 2016). However, despite their prevalence, icons are often ineffective at clearly illustrating specific weather phenomena (Zabini et al. 2015; Zabini 2016; Reed et al. 2022). People also tend to assign varying amounts of uncertainty to deterministic weather icons (Sivle et al. 2014; Zabini et al. 2015). While the connotations and nuanced meaning of weather icons is complex and experiential for many forecast users (Sivle et al. 2014; Reed and Senkbeil 2020), icons may play a role beyond phenomenon identification, especially over long-term outlook time frames. Weather icons may serve as visual cues for “chunking” the weather processes by phenomenon to understand overall distributions and frequency of severe weather across the region without having to read text descriptions for each forecast area. To explore the potential value of weather icons in recognizing and distinguishing large-scale weather patterns and trends, we added gale, heavy rain, and snow icons used in MetService’s daily weather forecast to the Severe Weather Outlook for two of the three experiment groups (Fig. 4).

View Full Size

The (a) icons, (b) weather icons, and (c) weather + confidence icons maps modified from the MetService Severe Weather Outlook control (Fig. 1) and used in the icon experiment (experiment C).

Citation: Weather, Climate, and Society 15, 4; 10.1175/WCAS-D-23-0010.1

• Download Figure
• Download figure as PowerPoint slide

Icons have also been shown to assist in the interpretation of uncertainty and can lead to increased decision performance relative to numerical or textual representations alone (Bisantz et al. 2005, 2011). However, much of the existing work on uncertainty icons focuses on intrinsic methods for representing uncertainty, through adjusting visual variables of the icon display, such as saturation, resolution, or shape of the icon (e.g., MacEachren 1992; Bisantz et al. 2005, 2011; Kinkeldey et al. 2014; Kolbeinsson et al. 2015). Here, we explore the potential value of icons to communicate uncertainty extrinsically. For one experiment group, we therefore added both weather icons and an icon to represent the confidence, or uncertainty, where an arrow points to one of three increasingly saturated bars representing low, medium, and high confidence (Fig. 4). We test the following null hypotheses about the use of icons:

• H9: There is no relationship between the icons shown on the outlook and the accuracy in reporting the number of different weather phenomena occurring during the outlook period.

• H10: There is no relationship between the presence of confidence icons and the accuracy in responding to statements about forecast likelihood.

• H11: There is no relationship between the presence of confidence icons and the accuracy in comparing the likelihood of experiencing severe weather between the North and South Islands.

• H12: There is no relationship between the presence of icons and the reported ease of reading information from the outlook.

a. Study design and sampling

We employed a purposive sampling online survey approach in order to reach a large national sample of active weather forecast users. Although purposive sampling does not enable us to test a representative sample of the population of NZ, this nonprobability convenience sampling approach allowed us to target a subset of the primary users of the MetService Severe Weather Outlook in order to increase the relevancy and reliability of findings for this audience. The survey was advertised on the MetService web site and its social media channels (e.g., Twitter, Facebook) and was shared with relevant special interest groups in order to reach audiences who actively use and seek long-term weather forecasts. Data were collected using the Qualtrics online survey platform, and the survey was open for participation for 2 months from mid-October to mid-December 2019.

A between-group mixed-methods study design was adopted, in which participants were randomly assigned to one of three groups in which they answered the same set of multiple-choice and open-ended questions using different maps. Each group saw different modified versions of the MetService Severe Weather Outlook, and responses were compared across groups. The survey was designed around the following three successive experiments in which variables were changed across groups: 1) time frame visualization, 2) perception and expression of uncertainty, and 3) icon inclusion (Table 1). Questions explored comprehension (reporting the correct answer), perceptions of likelihood (Likert scales and percentages), and self-reported ease of reading (Table A1). In the rest of this section, the name of each group within experiments A–C is italicized on first use for clarity.

Table 1.

Experimental groups and outlook variables.

View Table

The outlook design used in experiments A and B was based on a MetService Severe Weather Outlook published on 9 May 2018 that included 4 days (11–14 May) of forecasts for heavy rain and severe gales across the country with low and moderate confidence (Fig. 1). The dates were changed on the outlook to reflect a summer public holiday weekend in New Zealand that is popular for traveling and planning outdoor activities (Waitangi Day long weekend). For the experiments, participants were asked to imagine that they were planning activities several days out from the Waitangi Day long weekend (6–9 February) and then respond to a series of questions about their planning while viewing the modified outlooks. In experiment A, the time frame visualization was modified such that the composite group saw one composite map graphic representing all 4 days of weather (control map; Fig. 2a); the animated group saw one animated map graphic [graphics interchange format (GIF)], which cycled in a loop showing each day (1–4 successively) for 3 s at a time and could not be paused by the user (Fig. 2b); and the panel group saw a four-paneled map graphic that juxtaposed the 4 days beside each other (Fig. 2c). Participants used the maps to respond to questions about the weather and travel plans for a fictional campsite at a starred location that was forecast to experience heavy rain and gales on Thursday and heavy rain on Saturday but no severe weather on Friday and Sunday.

The paneled visualization from experiment A (panel group) was adopted for experiment B. In experiment B, two groups saw outlooks in which the type of weather and confidence level were extrinsically written using text. For the outlines + extrinsic group, forecast areas were represented by hashed outlines of random color with a text label defining the confidence level (current MetService approach; Fig. 3a). For the shapes + extrinsic group, forecast areas were represented by solid shapes of random color with a text label defining the confidence level (Fig. 3b). The shapes + intrinsic group was shown an outlook in which the forecast areas were represented by solid shapes, but a shape’s hue represented the type of weather, and the saturation represented the confidence level intrinsically (Fig. 3c). Participants used the maps to respond to questions about confidence and planned activities for a campsite at a starred location that was forecast to experience heavy rain on Friday (low confidence), Saturday (moderate confidence), and Sunday (moderate confidence).

The composite-style outlook design used in experiment C was based on a MetService Severe Weather Outlook published on 18 May 2018 that included 4 days (20–23 May) of forecasts for heavy rain, severe gales, and heavy snow across the country with low, moderate, and high confidence. The forecast area hue represented the type of weather (blue, heavy rain; light blue, heavy snow; gray, severe gales). For all groups, the type of weather was described in a written text label pointing to the forecast area (Figs. 4a–c). The no icons group had no icons shown on the map. The weather icons group saw an outlook that included weather icons (Fig. 4b). The weather and confidence icons group saw an outlook that included both weather icons and icons that extrinsically represented the level of confidence (Fig. 4c). Participants then responded to questions about the types of weather occurring across the outlook period, forecast confidence, and likelihood.

At the end of the survey, all participants responded to a set of multiple-choice questions about demographics; familiarity, confidence, and use of weather forecasts and the MetService Severe Weather Outlook; and an open-ended question about the perceived meaning of the “confidence” levels.

b. Statistical analysis

We used IBM SPSS Statistics version 24.0.0.1 to carry out nonparametric statistical analyses on our null hypotheses and report findings at the p < 0.05 significance level. We use the χ² test for equality of proportions to determine if participant characteristics are equally distributed between experiment groups (e.g., to identify if one group reported being more familiar with the outlooks than the other groups). We use the Pearson’s χ² test to test for associations in categorical variables across experiment groups and report φ_c as a test of the strength of the association. We use the Mann–Whitney U test to compare differences between independent continuous variables (i.e., percent confidence reported) between groups, and we use the Kruskal–Wallis test to test for differences in the continuous variables (i.e., changes in percent confidence reported for “low” and “moderate”) across all three groups. Where noted, some groups were combined for analysis. For example, the paneled and animated approaches of time representation were grouped together as “separate” days for comparison with the composite days group in some instances to assess differences between bivariate and univariate approaches. We use the Shapiro–Wilks test to test for normality before applying nonparametric tests. A content analysis (Creswell 2013) was carried out on text responses to open-ended questions to identify the presence and frequency of words, themes, and concepts raised by participants. Inductive codes were recognized from the data, through a process of reading and interpreting the data to identify key concepts and to further explore participants’ perspectives and understanding of the design options.

a. Participant overview

The online survey was open for 2 months, and a total of 417 people participated, with 369 completing the survey in full. Of those who responded to a question about language (n = 396), 100% reported being fluent in English. Of those who responded to questions about age and education (n = 369), a majority were between the ages of 20 and 69 [93% (n = 343)] and had completed a university degree [68% (n = 269)]. When participants were asked about where they spent most of their time, responses (n = 417) were distributed across all 15 regions of NZ, with higher representation from regions with large urban centers, such as the capital city of Wellington [42% (n = 177)], Auckland [13% (n = 56)], and Canterbury [12% (n = 48)]. Of the 417 respondents, most were long-term residents, having resided in NZ for 20 years or more [77% (n = 321)] or 10–19 years [11% (n = 44)].

More than one-half of the 417 respondents reported high confidence in their skills for “reading and using weather forecasts” [51% (n = 211)], with 18% (n = 75) “extremely confident” and 33% (n = 136) “very confident” (Fig. 5). A large portion reported moderate confidence, with 42% (n = 175) “reasonably confident,” and 7.4% (n = 31) reported low confidence, with 7.2% (n = 30) “not very confident” and 0.2% (n = 1) “not at all confident.” A majority [88% of the 417 respondents (n = 367)] reported using weather forecasts regularly (daily to weekly), 9.4% (n = 39) occasionally (e.g., planning trips and occasions), and 2.4% (n = 10) rarely (e.g., in storm events); 0.2% (n = 1) reported that they never used weather forecasts.

View Full Size

Self-reported confidence in skills for using and reading weather forecasts.

Citation: Weather, Climate, and Society 15, 4; 10.1175/WCAS-D-23-0010.1

• Download Figure
• Download figure as PowerPoint slide

One-half of the 417 respondents also reported high levels of familiarity with the MetService Severe Weather Outlook [50% (n = 206)], with 28% (n = 116) “extremely familiar” and 22% (n = 90) “very familiar” (Fig. 6). Less than one-third reported moderate levels of familiarity, with 27% (n = 111) “reasonably familiar,” and low levels of familiarity [24% (n = 100)], with 15% (n = 62) “not very familiar,” 8.6% (n = 36) “not at all familiar,” and 0.5% (n = 2) “not sure.” There were no observed differences in the level of confidence with weather forecasts [χ²(4) = 7.66; p = 0.105; φ_c = 0.096] or familiarity with the outlooks [χ²(4) = 5.39; p = 0.249; φ_c = 0.080] between experiment groups at the p = 0.05 level. At the start of the survey, 61% of 417 respondents reported a high likelihood of using the MetService Severe Weather Outlook for planning activities such as family outings, with 32% (n = 134) reporting they were “extremely” and 29% (n = 121) “very” likely to use it. Similarly, most of the 417 respondents [69% (n = 286)] reported that, overall, the Severe Weather Outlook was “extremely” or “very useful” as a tool for planning ahead, with 28% (n = 115) responding it was “reasonably useful” and 4% (n = 16) responding it was “not very” or “not at all useful.”

View Full Size

Self-reported familiarity with Severe Weather Outlook graphics.

Citation: Weather, Climate, and Society 15, 4; 10.1175/WCAS-D-23-0010.1

• Download Figure
• Download figure as PowerPoint slide

b. Experiment A: Time frame visualization

Four null hypotheses were tested in experiment A to explore the effect of time frame visualization on comprehension, perceived likelihoods, and behavioral intent, using an outlook that forecast heavy rain for Thursday and Saturday, but not Friday.

• H1: There is no relationship between the way that days are presented (composite or separate) and the accuracy of reporting the days on which severe weather will occur.

Experiment A failed to reject H1. When participants were asked to report which days were forecast to experience severe weather, there were no observed differences in the proportion of correct responses between the composite group and the groups that saw days presented separately (animated and panel groups) [χ²(2) = 1.72; p = 0.423; φ_c = 0.064]. Of the 417 respondents, a majority of participants in each group were able to correctly identify that the campsite was forecast to experience severe weather only on Thursday and Saturday (composite: 62%, with n = 86 of 139 respondents; animated: 63%, with n = 88 of 140 respondents; and panel: 69%, with n = 43 of 138 respondents).

• H2: There is no relationship between the way that days are presented (composite or separate) and the reported likelihood of severe weather occurring on a day that it is not forecast, when severe weather is forecast to occur on adjacent days.

H2 was rejected. Across all respondents (n = 417), participants in the composite group (n = 139) were more likely to respond with a high or moderate likelihood that heavy rain would occur on Friday 7 February at the campsite than participants in the groups that saw weather for each day presented separately (animated and panel) [χ²(4) = 52.3; p < 0.01; φ_c = 0.250], with 12% (n = 17) reporting that heavy rain was “extremely” or “very” likely to occur and 35% (n = 48) responding that it was “reasonably” likely to occur (Fig. 7). The highest portion of correct responses (low likelihood) occurred among members of the panel group (n = 138), of whom 90% (n = 124) reported that heavy rain was “not very” or “not at all” likely to occur.

• H3: There is no relationship between the way that days are presented (composite or separate) and the level of concern about driving on a day that severe weather is not forecast, when severe weather is forecast to occur on adjacent days.

View Full Size

Estimated qualitative likelihood of heavy rain occurring on Friday.

Citation: Weather, Climate, and Society 15, 4; 10.1175/WCAS-D-23-0010.1

• Download Figure
• Download figure as PowerPoint slide

H3 was rejected. Across all respondents (n = 416), participants in the composite group (n = 139) were more likely to respond with a high or moderate level of concern about driving on the roads near the campsite on Friday 7 February [χ²(4) = 50.06; p < 0.01; φ_c = 0.245], with 24% (n = 33) reporting that they would be “extremely” or “very” concerned, 53% (n = 73) reporting that they would be “reasonably” concerned, and 24% (n = 33) responding “not very” or “not at all” concerned (Fig. 8). Low levels of concern (“not very” or “not at all”) were more frequently reported among most participants in the groups that saw weather for each day presented separately (animated: 50%, with n = 69 of 139 respondents, and panel: 64%, with n = 88 of 138 respondents).

• H4: There is no relationship between the way that days are presented (composite or separate) and the reported likelihood of changing weekend plans.

View Full Size

Reported level of concern about driving on roads on Friday.

Citation: Weather, Climate, and Society 15, 4; 10.1175/WCAS-D-23-0010.1

• Download Figure
• Download figure as PowerPoint slide

H4 was rejected. Across 417 respondents, participants in the composite (n = 139) and animated (n = 140) groups were more likely to respond with a high or moderate likelihood of changing plans based on the forecast relative to the panel group (n = 138) [χ²(4) = 10.269; p = 0.036; φ_c = 0.111]. The panel group was more likely to report a low likelihood of changing plans [22% (n = 30)] relative to the composite group [11% (n = 15)] or animated group [14% (n = 19)] (Fig. 9). The panel-group participants were also less likely to report a high likelihood of changing plans [40% (n = 55)] relative to the composite [56% (n = 78)] and animated [51% (n = 71)] groups. When optionally describing how they would change plans, respondents across all groups generally described that they would either cancel their plans completely or delay the start of their trip to Friday to miss inclement weather on Thursday. Overall, the composite outlook was associated with higher perceived likelihoods of severe weather, levels of concern, and intent to change plans than the outlooks that presented information grouped into separate days (animated and panel style outlooks).

View Full Size

Reported qualitative likelihood of changing plans based on the outlook forecast.

Citation: Weather, Climate, and Society 15, 4; 10.1175/WCAS-D-23-0010.1

• Download Figure
• Download figure as PowerPoint slide

Panel-group participants reported the highest ease of reading information from the outlook [χ²(4) = 10.276; p = 0.036; φ_c = 0.111]. The panel-group respondents (n = 138) were more likely to report a high level of ease [“extremely” or “very” easy; 62% (n = 86)] compared to the composite group [50% (n = 69 of 139 respondents)] or animated group [45% (n = 63 of 140 respondents)]. Low levels of ease (“not very” or “not at all” easy) were most frequently reported among the animated group [16% (n = 23 of 140 respondents)].

c. Experiment B: Perception and expression of uncertainty

Four null hypotheses were tested in experiment B to explore how representation of the forecast area and expressions of uncertainty affect perceptions and estimates of forecast confidence.

• H5: There is no relationship between the forecast area style (outlines or shading) and the estimated percent confidence in the severe weather occurring.

H5 was rejected. The mean estimated percent confidence among those who saw areas displayed as solid shapes (n = 264) was higher than that among those who saw outlines (n = 133; independent-samples Mann–Whitney U test) for both Friday (low confidence; U = 20 486; p = 0.001) and Saturday (moderate confidence; U = 21 108; p = 0.001; Table 2; Fig. 10).

• H6: There is no relationship between the expression of uncertainty (extrinsic or intrinsic) and the estimated percent confidence in the severe weather occurring.

Table 2.

Mean, median, and mode for estimated confidence across groups.

View Table

View Full Size

Reported percent confidence that heavy rain will actually occur at the campsite on Friday (when heavy rain has low forecast confidence) and Saturday (when heavy rain has moderate forecast confidence).

Citation: Weather, Climate, and Society 15, 4; 10.1175/WCAS-D-23-0010.1

• Download Figure
• Download figure as PowerPoint slide

H6 was rejected. Extrinsic uncertainty (e.g., “low”/“moderate”; n = 264) was associated with lower mean estimates of confidence than intrinsic uncertainty (hue saturation; n = 133; independent-samples Mann–Whitney U test) for both Friday (Q12; low confidence; U = 21 322; p < 0.01) and Saturday (Q13; U = 23 354; p < 0.01).

• H7: There is no relationship between the expression of uncertainty (extrinsic or intrinsic) and the range of estimates of percent confidence in severe weather occurring for days when confidence is “low” and “moderate.”

Experiment B failed to reject H7. There were no significant differences in the range between estimates for Friday (“low” confidence) and Saturday (“moderate” confidence) between groups that saw uncertainty displayed extrinsically and intrinsically [independent-samples Kruskal–Wallis test; χ²(2) = 0.340; p = 0.843].

• H8: There is no relationship between the expression of uncertainty (extrinsic or intrinsic) and the day chosen to take a walk.

Experiment B failed to reject H8. Across all responses (n = 397), a majority of participants [79% (n = 315)] responded they would choose to take a walk on Friday, the day with the lowest confidence in the severe weather forecast for the area, and there was no significant difference in the day chosen and the outlook style viewed [χ²(4) = 4.89; p = 0.299; φ_c = 0.79; Fig. 11].

View Full Size

Reported day chosen to take a walk as based on the outlook forecast.

Citation: Weather, Climate, and Society 15, 4; 10.1175/WCAS-D-23-0010.1

• Download Figure
• Download figure as PowerPoint slide

Participants in the shapes + extrinsic group (n = 132; Fig. 3b) reported the highest ease of reading information from the maps [χ²(4) = 9.54; p = 0.049; φ_c = 0.109]. Participants in the shapes + extrinsic group were more likely to report a high level of ease [“extremely” or “very” easy; 57% (n = 75)] relative to participants in the outlines + intrinsic group [n = 133; 47% (n = 63)] or participants in the shapes + intrinsic group [n = 133; 50% (n = 67)] group. Low levels of ease (“not very” or “not at all” easy) were most frequently reported among the shapes + intrinsic group [17% (n = 23); Fig. 3c].

d. Experiment C: Icons

Four null hypotheses were tested in experiment C to explore how the presence of icons affects performance and ease of reading the outlooks.

• H9: There is no relationship between the icons shown on the outlook and the accuracy in reporting the number of different weather phenomena occurring during the outlook period.

H9 was rejected. The addition of weather icons increased the frequency of correct responses among participants who responded to a question asking them to report how many weather phenomena were forecast to occur over the outlook period [n = 378; χ²(1) = 6.37; p = 0.012; φ = 0.156]. While 78% (n = 96) of respondents in the no icon group (n = 124) correctly identified that three different weather phenomena (gales, rain, snow) were forecast to occur, 90% (n = 115) of participants in the weather icons group (n = 128) reported correct answers. However, respondents in the weather + confidence icons group did not show a significant improvement over the no icons group, with 82% (n = 103) of the weather + confidence icons group (n = 126) reporting the correct answer of three [χ²(1) = 0.484; p = 0.487; φ = 0.044].

• H10: There is no relationship between the presence of confidence icons and the accuracy in responding to statements about forecast likelihood.

Experiment C failed to reject H10. When presented with several statements about forecast intensity, accuracy, and likelihood, there were no significant differences in participants’ ability to identify the “true” statement between groups [χ²(2) = 0.005; p = 0.998; φ_c = 0.004]. The correct answer, “the likelihood is forecast to vary between days,” was chosen by less than 40% of participants (n = 151) across all groups (n = 378) as follows: no icons [36% (n = 44 of 124 respondents)], weather icons [37% (n = 47 of 128 respondents)], and weather + confidence icons [37% (n = 46 of 126 respondents)].

• H11: There is no relationship between the presence of confidence icons and the accuracy in comparing the likelihood of experiencing severe weather between the North and South Islands.

Experiment C failed to reject H11. Across participants in all groups (n = 378), there were no significant differences in the ability to correctly identify that the South Island was more likely to experience severe weather than the North Island [χ²(2) = 0.974; p = 0.615; φ_c = 0.051]. A majority of participants reported the correct response across all groups as follows: no icons [79% (n = 98 of 124 respondents)], weather icons [74% (n = 94 of 128 respondents)], and weather + confidence icons [79% (n = 99 of 126 respondents)].

• H12: There is no relationship between the presence of icons and the reported ease of reading information from the outlook.

H12 was rejected. Participants in the no icons group reported the highest ease of reading information from the outlook [χ²(4) = 25.5; p < 0.01; φ_c = 0.184; Fig. 12].

View Full Size

Self-reported qualitative level of ease of reading information from the outlook.

Citation: Weather, Climate, and Society 15, 4; 10.1175/WCAS-D-23-0010.1

• Download Figure
• Download figure as PowerPoint slide

e. Interpretations of “confidence”

When asked, “What do you think the different levels of confidence in the forecast mean?” participants (n = 329) demonstrated a wide range of latent interpretations of the word “confidence,” with the most dominant themes relating to likelihood, statistics, probability, or chance, followed by a consideration of the accuracy of the data and the agreement between models, forecasts, or forecasters and then that the participants were confused, did not know, or believed confidence reflects a subjective best guess or hedging (Table 3). The most common themes related to participants translating “confidence” into either likelihood (n = 156) or occurrence (n = 133) or both (n = 83), translating it directly to “the likelihood of occurrence,” for example, “the likelihood that the severe weather will occur” or “confidence in that weather occurring at that time.” This indicates that nearly a quarter of participants were interpreting the concept the way intended by MetService. Relating the confidence to statistics and probability, or possibility and chance, was raised by less than a fifth of participants (n = 61 and 37, respectively), for example, a “statistical chance of happening” or “possibility of the forecast being correct” or “a longwinded way of saying ‘chance.’” A small number of participants (n = 10) referred to the confidence being subjective and a best guess, based on gut feelings or opinion, for example, “a ‘best guess’ at the likelihood of a particular weather pattern.”

Table 3.

Summary of thematic analysis of participant descriptions of confidence.

View Table

In the next dominant themes, participants considered the forecasts, predictions, data, models, and calculations (n = 69) and the confidence in those (n = 33), for example, “confidence the forecasters have in their prediction being true” and “how precise the info is.” A number of participants inferred that confidence either was related to the uncertainty and disagreement among advice and forecasters (n = 45) or reflected the truth and reliability of the forecast or accuracy of data (n = 41), for example, “how likely the forecast is to match actual weather” and “confidence the forecasters have in their prediction being true” and “a measure of the uncertainty (divergence) in the models.” These interpretations relate closely to how confidence is defined by agencies such as the WMO (Gill et al. 2008) and the IPCC (Mastrandrea et al. 2010).

Thirty-five respondents also indicated they were confused and did not know what “confidence” meant or did not like its use, for example, “a clumsy way of expressing percentage of something being likely” and “I always struggle with the confidence levels; it’s very hard to assess how much to trust it.” This latter quote also highlights an additional theme raised, that is, trust, either the trust the forecasters have in the data and models or participants’ own trust in the forecasts presented to them.

a. Grouping content by day can reduce perceived biases

Outlooks that divide multiday forecast information into a format that presents each day separately may result in more deliberate decision-making that is less influenced by perceived biases. Our results indicate that people who view composite-style maps, which show information for all four days of weather on one map, may incorrectly infer that forecast areas extend into neighboring days of the outlook. In experiment A of our study, this emerged as a perceived underforecasting bias. Participants in the composite outlook group reported higher likelihoods of severe weather occurring on a day when there was no severe weather forecast to occur, despite accurately reporting that there was no severe weather forecast on that day (Fig. 7). They tended to extend the severe weather forecast for Thursday and Saturday into the forecast for Friday (suggesting an assumption that it was underforecast). This perceived bias resulted in higher reported levels of concern about travel on Friday (Fig. 8) and greater likelihood of changing plans for weekend travel (Fig. 9) relative to the groups that saw the information separated into days. This aligns with previous findings that audiences may make unwarranted assumptions about forecasts and uncertainty (Joslyn and Savelli 2010) and provides empirical evidence that forecast visualization choices could play a role in mitigating these types of perceived biases.

While a composite map offers advantages by distilling information about multiple days into a single image, there are trade-offs associated with this approach. When the information was separated out into individual days, people were less likely to perceive that the outlook was underforecasting the weather for days adjacent to those with severe weather. The visual cue of seeing severe weather forecasts illustrated on some days and not on others may have affected perceptions of forecast accuracy and reliability in the animated and panel groups. Displaying forecast information for each day individually in a univariate approach may help reduce the perceived underforecasting bias associated with composite styles. We found that panels, in particular, have a higher likelihood of being interpreted correctly and were also reportedly easier to read and understand than the animated style. A well-designed interactive map, where the user can choose to view individual days, could be an effective alternative to the panel and animated maps tested in this study. Although they are less transferable across media platforms, interactive maps offer many advantages for engaging map users and empowering them to create a representation that meets their needs (Roth et al. 2017), and future work should investigate this approach.

Across all participant groups, 10%–47% of participants reported moderate to high likelihoods of heavy rain occurring on a day that did not have any heavy rain forecast but was between 2 days for which heavy rain was forecast. This indicates complex perceptions of the timing and evolution of severe weather events, which should be further explored in future work.

b. Visual saliency can affect perceived forecast confidence

Forecast areas represented as solid shapes, which are more vivid and eye catching, or more visually salient, than the outlined forecast areas, were associated with higher estimates of forecast confidence. Participants who viewed forecast areas as shapes estimated higher levels of forecast confidence than participants who viewed forecast areas in outlines of the same color. The solid shapes were associated with higher median and mean percent confidence that the heavy rain forecast would “actually” occur for both “low confidence” and “moderate confidence” forecasts (Fig. 10; Table 2). Similarly, confidence visualized intrinsically, using variations in saturation for the visually salient shapes, resulted in higher median and mean percent confidence estimates than the extrinsic (textual) approaches. Estimates of confidence among participants who saw forecast areas as outlines and/or extrinsic expressions of uncertainty more closely matched the forecasters’ intended confidence values. For example, the mean and median values reported for “low” and “moderate” among the outlines and extrinsic groups were closer to the 20% and 40% values to which these words were intended to refer.

Visual saliency is a key variable for capturing attention and driving cognition (Itti and Koch 2000; Cavanagh 2011), and previous studies have shown that it can influence decision-making with maps and graphics (Jarvenpaa 1990; Fabrikant et al. 2010; Clive et al. 2021). Our findings suggest that visual saliency also plays a latent role in how people cognitively assess the uncertainty or confidence in outlook forecasts. The tangible nature of seeing data visualized on a map can result in map readers ascribing higher certainty to the information than warranted (MacEachren 1992; Severtson and Vatovec 2012). Using bold visually salient colors with sharp boundaries to display forecast areas could reinforce this bias by increasing perceived forecast confidence. Shapes also present a number of visualization challenges in the instances of overlapping forecast areas, which may occur in complex forecasts or longer-term outlooks. For example, the area starred in Fig. 2 has overlapping gales and heavy rain on Thursday 6 February. This requires interpretation of whether or not both events are occurring in the overlapping shaded areas and introduces space for divergent interpretations.

In this study, we do not explore the role of color choice in how shape and outlined forecast areas are interpreted. However, we acknowledge that color is widely recognized as having an effect on the way people comprehend and interpret maps (e.g., Robinson 1967; Bryant et al. 2014). We suggest that future iterations of the Severe Weather Outlook develop a consistent and transparent approach to color choice, as people may infer unintended meaning from a random color choice, which could also result in divergent interpretations of the same forecast.

c. Confidence is informative but confusing

Our results indicate that participants anticipate uncertainty with outlook forecasts and can effectively use forecast confidence to inform their decision-making, despite an inconsistent interpretation of the uncertainty expression. Confidence levels presented on the outlook effectively informed participants’ decision-making about which day to plan activities, but participants demonstrated a range of interpretations for what the term “confidence” referred to. Across all groups, participants were most likely to choose to take a walk on a day where the forecast for the area had the lowest confidence (i.e., lower likelihood of the heavy rain actually occurring). However, two-thirds of participants across all groups were not able to correctly report whether variations in confidence referred to variations in intensity, likelihood, or accuracy. When asked to describe “confidence” in their own words, there was a wide range of replies. This suggests that conveying the gist of uncertainty can be useful for decision-making, even if precise interpretation varies from the intended statistical or scientific definition. People can successfully infer meaning from qualitative descriptions of confidence, even if they are not clear on the technical meaning of the term.

For more complex decision-making tasks, clarity and transparency about uncertainty terms can help support more accurate inferences. Participants across all groups tended to report that “moderate” confidence was associated with a 50% value or above, which is greater than the 40% likelihood of the event occurring to which it was intended to refer. The findings suggest that the term “moderate” may be interpreted as a midpoint, or 50% likelihood, and this should be considered when using qualitative explanations of uncertainty. This also highlights the importance of testing the translation terms used to convert from numerical to linguistic or textual probability terms. Research has demonstrated the challenges of selecting appropriate translation thresholds, given that misinterpretation of these terms can occur due to their framing, directionality, and probabilistic format [see review in Doyle et al. (2014)]. Interpretations and translations are also influenced by biases, numeracy, and world view [see review in Doyle et al. (2020); Tversky and Kahneman 1974; Slovic et al. 2004; Joslyn et al. 2009; Kahan et al. 2012]. The inconsistent interpretations of confidence, biased perceptions of “moderate” confidence, and latent beliefs and understanding of what “confidence” means in a severe weather outlook forecast context all indicate the value in clearly defining use of the term, particularly in the context of likelihood, reliability, and consistency.

d. Icons can be helpful, in moderation

Weather icons may assist in mental “chunking” of information about weather processes, facilitating a high-level understanding of the distribution of severe weather in a national long-range outlook. While previous work has found that icons may not effectively describe specific weather phenomena (Zabini et al. 2015; Zabini 2016), we find that they can be helpful on maps for distinguishing patterns of distributed weather phenomena. This may be particularly useful for outlooks that show large regional areas where multiple different weather phenomena may be occurring at once. Weather icons helped participants identify that three different types of weather were occurring across the country, facilitating an accurate high-level understanding of overall severe weather patterns affecting the country. This improved comprehension and awareness could increase engagement with phenomenon-specific preparedness actions, such as seeking out information about preparing for snowfall (Griffin et al. 1999; Paton et al. 2005; Lindell and Perry 2012). However, maps that showed icons for both weather and confidence levels were not associated with increased comprehension, suggesting that too many icons can lead to confusion.

Despite the improved performance in identifying weather patterns from the outlook with the addition of weather icons, participants who viewed the map with weather icons reported a lower ease of reading information from the outlook than those who saw a map with no icons. The results suggest that weather icons can help audiences more effectively infer overall weather patterns on a map but that icons do not necessarily improve ease of reading, and too many icons can be ineffective. Experimental results for the effective use of icons in weather risk communication have been mixed, and further research on this topic could help further elaborate how to cognitively leverage the advantages of icons in such graphics.

e. Limitations and future work

Our targeted sample of engaged weather information users is helpful for the context of this study and application of the findings in NZ, but we acknowledge that the findings may not be transferable to all social and demographic groups. The study sample comprised highly educated long-term residents of urban NZ over the age of 40 years with a relatively high familiarity with the Severe Weather Outlook graphics. Further research with a more diverse sample could reveal deeper insights into social and cognitive processes affecting inferences and decision-making. Similarly, severe weather outlooks will have different design needs in different global and meteorological contexts and with diverse cultural audiences.

The empirical designs tested in this study are not proposed as potential solutions for use in practice. The designs have been created to carefully control for testing of visual representation variables between groups. Development of a product for practice should be designed based on the findings of this and other studies and employ user experience design and accessibility testing that incorporates end users’ perspectives via the application of evidence-based usability guidelines (e.g., Tan et al. 2020). Further empirical studies of other elements of the graphic, such as the presence or absence of geographical markings, color choice, static or interactive formats, and how these large-scale products link with the higher-resolution content in weather warnings and watches, could deepen our understanding of how audiences engage with such products. It is important to note that we explore the high-level influence of outlook design on intended decision-making by comparing differences in comprehension across visual representation variables between groups, but we acknowledge that decision-making is a complex cognitive task and that other psychological biases, heuristics, and sociopolitical factors, such as trust, play into decision-making tasks. Further, we acknowledge that these findings are based on participant responses to a hypothetical scenario, and actual behavior may differ from that reported here under the stress and pressure of a real event. We note that the questions used in experiment A focused on a day in which no severe weather was forecast, and thus, the assumption is that the desired behavior is no mitigation actions need to be taken. It would be valuable to compare this with a situation in which the desired response is an increase in protective action and increased levels of concern are warranted.

Here, we tested the forecast area colors associated with the MetService control map to reduce experimental error. We suggest that future studies could explore forecast areas that are less saturated (e.g., lighter colored) or introduce other visual variables for representing forecast areas and uncertainty, such as transparency or texture. Future studies exploring the role of severe weather outlooks in decision-making could also support and inform more targeted testing about design, such as interactive maps where display variables can be adjusted by the user.