a. Forecast uncertainty

Although the current warning system fails to acknowledge it directly, meteorologists know that the probability of a tornado varies geographically within the warned area and changes over time (Karstens et al. 2015). Nonetheless, at present this information is not made available to members of the public. Whether to communicate probabilistic information such as this is the subject of continued debate. Evidence suggests that people understand that all forecasts involve some level of uncertainty (Joslyn and Savelli 2010), due to their own prior experience with the reliability similar forecasts. For instance, in the tornado season of May 2011 to May 2014, more than one-half (58%) of the 132 tornado warnings were false alarms, that is, no tornado was observed anywhere within the warned area (NWS 2011). Moreover, research links false alarm ratios to subsequent tornado casualties (Simmons and Sutter 2009) suggesting a reduction in compliance as a result of increasing false alarms. Thus, residents may regard some warnings as “wrong” and ignore future forecasts (Ripberger et al. 2015). However, the relationship between false alarms and public perception of the validity such warnings is likely complex. For instance, perceived false alarm rate is not necessarily correlated with the actual false alarm rate and may depend on whether the false alarm is a “close call” (Barnes et al. 2007; Lim et al. 2019). Nonetheless, noncompliance with warnings may be due, at least in part, to lack of trust.¹

If so, it may help to add an uncertainty estimate. Experimental evidence suggests that adding numeric uncertainty information (e.g., 30% chance) attenuates the reduction in trust due to increased error in forecasts (Joslyn and LeClerc 2012). In addition, there is evidence that adding explicit uncertainty information to the forecast preserves trust to a greater degree than does reducing false alarms (LeClerc and Joslyn 2015). Moreover, there is now strong evidence that the advantages for probabilistic forecasts go beyond trust, improving both understanding and decision quality as well (Joslyn et al. 2007, 2009; Joslyn and LeClerc 2012). Indeed, these advantages may be the reason that numeric uncertainty estimates increase trust: better understanding allows users to reach their decision goals.

b. Multiple forecast updates

However, there are some important differences between the experimental settings cited above and real-world weather warnings. In experimental settings participants are generally asked to make a decision on the basis of a single forecast. However, in natural settings forecasts often begin several days in advance of major events and are updated over time. In the case of a tornado, for instance, residents can receive multiple forecast updates in a single day as the situation continues to evolve. With each update the information may change, requiring residents to replace old with current information. Keeping track of rapidly changing information such as this can be challenging due to limitations in what is referred to as “working memory” capacity, our ability to maintain and update information in consciousness (Baddeley 2000). Because uncertainty information constitutes additional and somewhat more complex information than what is currently provided in most warnings, it may not be fully processed when rapid updates are provided. However, to our knowledge, no experimental research has yet tested this question. Therefore, the experiment reported here was designed to test the following hypotheses, in a simulated tornado warning situation with multiple forecast updates, to determine whether probabilistic forecasts remain beneficial when this complexity is added:

1. Participants will have a better understanding of the likelihood of a tornado when forecasts include explicit numeric likelihood expressions as compared with categorical expressions such as the conventional watch and warning format.

2. Participants will have greater trust in forecasts that include explicit numeric likelihood expressions as compared with categorical expressions.

3. Participants will make better decisions (for specific operationalizations² of this construct, see procedure section below) when forecasts include explicit numeric likelihood expressions as compared with categorical expressions.

c. Second-order uncertainty

The processing challenge may be compounded by the fact that assigning a single probability may not be possible for some weather events. In practice, a range of likelihoods such as 10%–20% chance of a tornado may be used. This is referred to as second-order uncertainty, an expression that indicates uncertainty about the uncertainty. Research suggests that users can understand likelihood ranges and sometimes regard them as more credible than single probabilities (Dieckmann et al. 2010). However, likelihood ranges further increase the amount of information that must be processed and may be particularly challenging in a dynamic decision environment. Thus, another goal of the research reported here was to test whether probability ranges provide the same advantages as a single estimate.

d. Color coding

Because of the increased processing demands in dynamic decision environments like a tornado event, numeric expressions of likelihood such as probabilities or probability ranges may be too challenging. Therefore, it may be necessary to simplify likelihood expressions to allow for rapid and easy understanding. Many believe, for instance, that color coding, due to its high salience (Wogalter et al. 2002), may be appropriate. In fact, evidence suggests that a multihue color-coded expression of wildfire likelihood led to better decisions under time pressure, although simple text expressions were more advantageous otherwise (Cheong et al. 2016). Therefore, in the research reported here, in addition to testing numeric expressions of likelihood, we also tested color-coded likelihood expressions.

There are two main types of color coding, multihue as mentioned above, and a single hue, such as red, that varies in value. The research on these formats is mixed, but slightly favors multihue formats. Multihue schemes have been shown to be read faster and sometimes preferred by users although they are not significantly different in terms of reading accuracy than a red scale varying in value (Miran et al. 2017). However, some research suggests that users find a multihue scale less intuitively linked to uncertainty than a single hue with variation in value (MacEachren et al. 2012), although neither reading accuracy or speed of interpretation were tested in that research. Thus, it is not clear which color-coded format is superior overall.

Here we focus on a multihue color-coded scheme, as it has already been integrated into risk communication tools developed by the National Oceanic and Atmospheric Administration in the Forecasting a Continuum of Environmental Threats (FACETs) framework for tornado threats. Our main question was how accurately multihue uncertainty formats communicate likelihood. At present, the research addressing this issue is sparse. There is evidence that people prefer color coding, such as a “traffic light” (red, yellow, and green) configuration to represent uncertainty in such situations (Radford et al. 2013; Tak and Toet 2014). Color-coded likelihood has also been shown to correct the common but incorrect assumption that tornadoes are more likely to occur in the center of the warned area (Ash et al. 2014). Although it is difficult to discern whether this effect is due to the color coding per se, or to the contours that it creates. However, to date, as far as we are aware, there is no research that investigates whether users understand that color coding is intended to indicate likelihood alone when that is the case, or whether their interpretations match the precise level of likelihood intended by forecasters. We address both potential misunderstandings in the paragraphs below.

e. Likelihood–severity confusion

The fundamental question about people’s understanding of color coding is what they construe the colors to represent. This is an issue due in part to the fact that color coding involves an extra step, reading a legend or explanation that translates the colors into an expression of something else, in this case likelihood. Whether or how carefully people engage in this step may be influenced by prior experience with color coding in other contexts that gives rise to expectations about what the colors mean. For instance, in the context of weather, color coding is often used to represent the severity of the event, that is, estimates such as wind speed or the amount of precipitation. In other cases, color coding is used to indicate risk, a combination of the likelihood and severity. Because of these precedents, users may have expectations about the meaning of the color coding that cause them to assume they already understand and misread or ignore the legend. As a result, users may misinterpret color-coded likelihood as an expression as severity or the extent of potential damage, or some combination of likelihood and severity, rather than as likelihood alone.

This inclination may be reinforced by the fact that even when color is not involved, there is evidence for a tendency to misinterpret graphically depicted likelihood information as some deterministic quantity such as wind speed or precipitation amount, referred to as a deterministic construal error (Joslyn and Savelli 2021). For example, users tend to misinterpret visualizations of percent chance of precipitation as duration or geographic extent of precipitation (Joslyn et al. 2009). In another study, bracket visualizations depicting the 80% predictive interval for nighttime low temperature [e.g., 35°–42°F (1.7°–5.6°C)] were misinterpreted as diurnal fluctuation. In other words, participants thought the endpoints of the range depicted two single-value forecasts, one for daytime high and the other for nighttime low (Savelli and Joslyn 2013). Similarly, many people think the cone of uncertainty, intended to show the possible hurricane path, depicts the extent of the wind field (Broad et al. 2007). Therefore, it may be that people have a general tendency to misinterpret likelihood expressions as some expression of severity if the expression permits it. This may have to do in part with cognitive load. A probabilistic forecast indicates that multiple outcomes are possible and thus, requires consideration of more information than a forecast describing the severity of a single outcome. It is possible that users tend to avoid the more difficult probabilistic interpretation and instead choose the easier severity interpretation, referred to as “attribute substitution” (Kahneman and Frederick 2002). One of the major goals of the research reported here is to determine the degree to which the likelihood–severity confusion arises from color-coded likelihood representations. Therefore, we add a prediction (in italics below) to hypothesis 1:

1. Participants will have a better understanding of the likelihood of a tornado when forecasts included explicit numeric likelihood expressions as compared with categorical expressions such as the conventional watch and warning format or color coding. Participants will tend to confuse likelihood and severity when likelihood is color-coded.

f. Understanding the level of likelihood

The other issue about users understanding of color-coded likelihood, even when they understand that it is intended to portray likelihood, is whether color coding conveys the precise level of likelihood intended by forecasters. Here the research is sparse. Much of the work to date has been conducted on color-coded scales intended to indicate risk, a combination of likelihood and severity for which the dependent variable is the order in which participants rank colors to represent risk. The majority of the evidence suggests substantial variability in rank order, suggesting that there may be issues. Indeed, with the exception of red, often found to convey the notion of greatest risk (Borade et al. 2008; Hellier et al. 2010), there is little consensus on the rank order of colors to convey risk (Chapanis 1994; Wogalter et al. 1995; Griffith and Leonard 1997; Rashid and Wogalter 1997). In addition, there are cultural differences. Orange, rather than red, is considered of greatest hazard by Chinese participants (Lesch et al. 2009). Moreover, a study of the now retired Homeland Security Advisory System (HSAS), using color to indicate terrorist threat, showed that more than half of the participants (57.8%) ranked the colors from most threatening to least in an order that conflicted with that intended (Mayhorn et al. 2004). Therefore, color-coded risk may convey different levels of risk to different individual users as well as different levels of risk than what was intended.

It is important to note that the studies reviewed above tested the rank ordering of color-coded risk. It is possible that participants could accurately rank colors and still misunderstand the precise likelihood of the adverse event. In other words, orange may be ranked as the second highest risk and assigned 90% chance by the user when it was intended to represent 50% chance. Indeed, no studies of which we are aware have asked participants to assign likelihoods (e.g., “a 40% chance”) to each color category to determine how they compare with the intended values. Determining whether color coding conveys the likelihood intended is another of the major goals (see hypothesis 1, above) of the research reported here.

In sum, public trust in forecasts may be critical for appropriate and timely precautionary decisions in the face of severe weather. Moreover, trust may be maintained by providing understandable event likelihood information. Likelihood information may also improve decisions based on the forecast. At present however, whether these benefits will be seen in a dynamic decision environment and how best to convey likelihood information in that context remains an open question.

The experimental study reported here investigated these issues using a computerized decision task (based on Schwartz and Howell 1985). Participants decided whether to seek shelter based seven updated tornado forecasts presented sequentially. Although more complex than the experimental scenarios tested previously, it was a vastly simplified version of the real-world task in which many other factors (e.g., geographic knowledge, environmental cues, prior experience), options (e.g., rearranging schedules, increasing monitoring), and information sources (e.g., social media) are considered at different stages (Lindell 2018). However, this simplification allowed for the experimental control necessary to infer direct causal relationships between the way in which the forecast information was expressed and differences in understanding, trust, and hypothetical shelter decisions. The conventional watch or warning format that served as our control condition was compared with numeric probabilistic forecasts (both single probabilities and ranges) and color-coded likelihood expressions.

a. Participants

Of the 489 University of Washington students who participated in the study for extra course credit, 57% were female. All participants were between 18 and 24 years of age, with an average age of 19 years. All were enrolled in a psychology course. Although racial information was not collected for this sample, the composition was likely similar to that of the student population in the year in which the experiment was conducted, 2019 (40% Caucasian, 25% Asian American, 8% Hispanic/Latino, 4% African American, 1% American Indian, and 0.9% Hawaiian/Pacific Islander; University of Washington 2019). Most participants were residents of Washington State, where tornados are rare. Evidence suggests that most University of Washington psychology students (80%) have no experience with tornados (C. Qin et al. 2021, unpublished manuscript).

b. Procedure

Participants performed as computerized decision task (details described below) individually on desktop computers in a laboratory room that accommodated approximately 10 participants per session. The task was presented in an online html platform. The researcher read the instructions aloud (see the online supplemental material) as participants viewed the same instructions printed on the computer screens. The instructions provided background on the tornado hazard including how a tornado is formed as well as the potential damage to homes and occupants. To familiarize participants with the range of tornado hazards, instructions described the wind speeds for both weak (73–112 mi h⁻¹; 1 mi h⁻¹ ≈ 0.45 m s⁻¹) and strong (260+ mi h⁻¹) tornadoes. Then, participants were told that they would receive forecasts for 40 storms with the potential to produce tornadoes. Severity was held constant by informing participants that all storms would produce wind speeds of 90–112 mi h⁻¹, consistent with weak tornadoes. Every storm moved west to east from the same distance, toward “home”—a house in which participants were to imagine they were located. Each storm constituted a single trial. One trial comprised seven successive forecast updates as the storm moved from west to east. Participants performed 10 practice trials followed by 40 data collection trials.

1) Cost–loss structure

To motivate participants to put forth their best effort, they were given a point balance (24 000 points) at the beginning of the simulation to spend on protective actions. The goal was to complete the task with as many points as possible. There were three decision options at each forecast update: wait, take shelter, and not take shelter (see Table 1). Participants could wait for more information at no cost on forecast updates 1–3. However, on forecast updates 4–7, there was a 20-point cost for every wait decision to reflect the increasing danger of the storm approaching home. Participants could choose to go to a nearby tornado shelter at a cost of 303 points at forecast update 1. This cost increased slightly (see Table 1) at each forecast update to reflect the increasing danger of being caught in a vulnerable position when a tornado strikes. All costs were deducted immediately from the onscreen point balance. Third, participants could choose for no cost to not take shelter. However, if a tornado hit home and the participant had chosen to not take shelter or wait as their final decision, a 1500-point penalty was immediately deducted from their balance. Notice that, in this task, as in actual weather hazard situations, the cost to protect oneself is much less than the loss that could result from inaction if a tornado hit the residents location.

Table 1

Cost of decisions (based on Schwartz and Howell 1985).

View Table

2) Storm movement

Participants saw different forecast updates depending on the path of the storm on a virtual grid of longitude and latitude shown in Fig. 1, although no geographic representation of storm movement was shown to participants. On each trial, the storm moved across the grid toward the participant’s home at the far eastern boundary. As the storm reached each new longitude, a forecast update was issued. At the start of every trial, the storm was located at latitude 4, longitude 1. From that point, storm movement was randomly generated in real-time but constrained to proceed from west to east and remain within the boundaries of the grid (see Fig. 1). When the storm was located in a cell between latitudes 2–6, there was a 0.3 probability of moving to the cell northeast or southeast and a 0.4 probability of moving laterally to a cell to the east. When the storm was located along on the top or bottom of the grid (latitudes 1 or 7) it advanced laterally to the east with a 0.7 probability and toward the center (i.e., southeast from latitude 1 or northeast from latitude 7) with a 0.3 probability. Thus, storm movement was more constrained in this experimental than in the real world, and for no storms was there reduced likelihood not related to the storm path (e.g., dissipation) as there might be in the real world. The probability of the storm being in each location on the grid is shown in Fig. 2.

View Full Size

Example of storm movement across the grid. This graphic was not shown to participants.

Citation: Weather, Climate, and Society 14, 4; 10.1175/WCAS-D-21-0186.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Probability of storm being in each location.

Citation: Weather, Climate, and Society 14, 4; 10.1175/WCAS-D-21-0186.1

• Download Figure
• Download figure as PowerPoint slide

3) Trial structure

On each trial participants received a series of updated forecasts each time the storm advanced (seven per trial). At each update they answered two questions that reflected their understanding of the information provided. They rated the likelihood of a tornado hitting their home by clicking on a visual analogy scale (VAS), anchored on left end with “impossible” and the right end with “certain” (see Fig. 3). Participants also rated the damage they would expect if a tornado were to hit home on a similar VAS anchored on the left with “not severe” and the right with “very severe.” On the next screen, the same forecast was displayed, and participants indicated their shelter decision by clicking on one of three radio buttons labeled “wait,” “not-take-shelter,” or “take-shelter,” shown in that order. The cost for each choice was shown beside it (see Table 1). On the same screen, in order to gauge the impact of the information format on trust, participants rated how much they trusted the forecast by clicking on a VAS, anchored on the left with “not at all” and to the right with “completely.”³ Then, an updated forecast was shown, and they answered the same set of questions with respect to the update. A decision to take shelter or not take shelter was a final decision for that storm (trial). However, in order to discourage rushing through trials by deciding early, when participants made a final decision prior to the seventh update, they saw the remaining forecast updates and answered the same questions about likelihood, severity, and trust although they were not allowed to change their decision. Note that because there were seven updates, there were seven opportunities to make a final decision. Therefore, it was possible to evaluate both when the participant made a final decision (to which we refer as “timeliness”) as well as the decision itself. After all seven forecast updates, the outcome screen informed participants whether the tornado hit or missed the home, reminded them of their final choice and the cost, whether sheltering was “necessary” (if the tornado struck home), and whether they incurred or avoided a 1500-point penalty. Another trust rating was taken on the next screen, which also included the final score. Then, the next trial with the same sequence of screens was shown. Participants were told that each trial represented a storm independent from the others to discourage them from deducing trends in the weather conditions. After the last trial, the final screen showed the ending point balance, the number of times a tornado hit home, and a summary of the participant’s choices.

View Full Size

Questions asked at each update on each trial and response mode: VAS.

Citation: Weather, Climate, and Society 14, 4; 10.1175/WCAS-D-21-0186.1

• Download Figure
• Download figure as PowerPoint slide

c. Stimuli

All forecasts were based on the probability of a tornado hitting home from the cell in which the storm was currently located (Fig. 4). The probabilities of a tornado hitting home in this simulation (to which we refer as “actual probability”) were realistic for a geographic region under tornado threat and ranged from 0 to 0.40.⁴

View Full Size

Actual probability of tornado hitting home from each cell on the grid.

Citation: Weather, Climate, and Society 14, 4; 10.1175/WCAS-D-21-0186.1

• Download Figure
• Download figure as PowerPoint slide

Participants saw one of five formats, a watch/warning format that served as a control and four experimental formats that included some expression of tornado likelihood (described in detail below). The watch/warning format was similar to the communication currently in use. Participants were told that a “watch” meant that tornadoes were possible in or near the watch area, and a “warning” meant that a tornado was imminent or occurring and that taking shelter was advised. In this simplified task, a watch was issued if the storm reached a position at which the actual probability of hitting home was greater than or equal to 0.13 and less than or equal to 0.24. In this probability range, based on the cost–loss structure described above in Table 1, the economically optimal decision was to wait. Here, we define the optimal decision as the one for which the participant could expect to lose the least points either in terms of the cost or expected loss (no gains are possible in this scenario). This constitutes the operationalization of the quality or “goodness” or participants decisions. The expected loss was the potential penalty (1500 points) weighted by the probability of receiving it, the probability the tornado hitting home from that position. See Eqs. (A1)–(A3) in the appendix. For positions at which the actual probability of hitting home was less than 0.13, no watch or warning was issued. In this probability range the optimal decision was to not take shelter. A warning was issued for positions at which the actual probability of hitting home was 0.25 or greater, when the optimal decision was to take shelter.

The four experimental formats included some form of explicit uncertainty information conveying the underling percent chance of the tornado hitting home from the current location of the storm (Fig. 5). In the color-coded condition participants received a color-coded forecast and were told in the instructions that warmer colors indicated higher likelihood of a tornado hitting their area. A green bar was shown when the actual probability of a tornado hitting home was less than 0.13, yellow when the actual probability was ≥0.15 ≤ 0.24, and orange when the actual probability was ≥0.25 and ≤0.40. Notice that these values are identical to those defining “no watch or warning,” “watch,” and “warning” and correspond to the optimal decision thresholds.

View Full Size

Forecasts by format (in the columns labeled A–E) and optimal decision threshold ranges. All forecasts were preceded by the phrase “Chance of Tornado Hitting Your Area:.” The percentages shown in column C are the midpoints of the ranges of percentages (identical to that in column D) shown in that condition.

Citation: Weather, Climate, and Society 14, 4; 10.1175/WCAS-D-21-0186.1

• Download Figure
• Download figure as PowerPoint slide

In the percent chance condition participants saw a numeric percent chance of a tornado ranging from 0% to 40% on a continuous scale and rounded to two decimal places. In the percent chance range condition participants saw ranges 0%–12%, 13%–24%, and 25%–40% (shown in Fig. 5, column D). Participants in the color + percent chance range condition saw both the color bar and the corresponding percent chance range (e.g., green and 0%–12%). Each forecast in the experimental conditions was preceded with the text: “Chance of tornado in your area:” (e.g., 6%). Figure 6 shows the storm grid and describes the stimuli that were shown in each condition at each position. It is important to note that in conditions with categorized expressions (watch and warning, color, percent chance range), the category boundaries were aligned to the optimal likelihoods for each of the three decision choices.

View Full Size

Storm location grid, with stimuli shown in each forecast format. Optimal decisions are indicated by shading (white = not take shelter, light gray = wait, and dark gray = take shelter). Cells shaded black are impossible positions or are tornado destinations (column marked as 8).

Citation: Weather, Climate, and Society 14, 4; 10.1175/WCAS-D-21-0186.1

• Download Figure
• Download figure as PowerPoint slide

d. Design

This experiment was a two-factor between/within-mixed design. The two independent variables were 1) forecast format (between groups) and 2) actual probability (within groups). The background information, task goal, cost–loss structure, and underlying probabilities were the same for all participants. The critical difference was the format by which the forecast information was presented (factor 1). Participants were randomly assigned⁵ to one of five format conditions: 1) watch and warning, 2) color, 3) percent chance, 4) percent chance range, and 5) color + percent chance range. Therefore, each participant saw only a single format throughout the experiment (between groups). The second factor was the probability level: 1) 0%–12%, 2) 13%–24%, and 3) 25%–40%—all levels shown to all participants (within groups). There were four dependent variables: forecast understanding (operationalized using likelihood and severity ratings), decision quality (operationalized as expected loss), decision timeliness, and trust (rating) in the forecast.

a. Understanding of the forecast (hypothesis 1)

There were two issues related to understanding. The first was how closely participants’ interpretation of likelihood, as indicated by their response on the VAS, matched the actual likelihood value and whether that varied by forecast format. The second was whether using some formats (e.g., color coding), participants mistook likelihood for severity.

1) Likelihood level

To determine how close participants’ likelihood estimate was to the intended value, the actual probability of a tornado hitting home on a given update was subtracted from the participant’s likelihood rating for that update. Participants likelihood rating was summarized as the percentage of the line between the left anchor (impossible) and the position to which participants moved the handle on the VAS. Then, in order to determine whether the error differed by actual probability level (e.g., higher for 25%–40% range), a mean difference score was calculated for each participant in the three actual probability ranges, (i.e., 0%–12%, 13%–24%, and 25%–40%). Negative numbers indicate that participants’ likelihood ratings were lower on average than the actual probability of a tornado hitting home, while positive numbers indicate that it was higher (see Fig. 7 and Table 2). Then, in order to determine whether estimates differed by condition, a mean was taken across participants in each forecast format condition, in each probability range (e.g., mean likelihood difference across participants in color-coded condition in 0%–12% probability range, color coded green).

View Full Size

Average likelihood difference, by actual probability range and forecast format.

Citation: Weather, Climate, and Society 14, 4; 10.1175/WCAS-D-21-0186.1

• Download Figure
• Download figure as PowerPoint slide

Table 2

Mean differences (with standard deviations) between participants’ likelihood rating and actual probability, by format forecast and actual probability range (0–0.12, 0.13–0.24, and 0.25–0.40). The final column shows the overall mean difference by forecast format.

View Table

Participants overestimated likelihood in all forecast formats but did so to the greatest degree with color-coded forecasts and at the highest actual probability range (Fig. 7; Table 2). A mixed model ANOVA was conducted on mean likelihood difference scores with actual probability range (0–0.12; 0.13–0.24; 0.25–0.40) as the within-groups factor and forecast format (watch and warning, color, color + percent chance range, percent chance range, and percent chance) as the between-groups factor. There was a significant main effect of forecast format F(4, 484) = 47.01, p < 0.001, with ${\text{η}}_p^2=0.28$. Planned contrasts, comparing the control (watch and warning) with experimental conditions revealed that the mean likelihood difference in the watch and warning condition was significantly greater than in all of the other conditions [color + percent chance range: t(484) = 3.30, p < 0.001, and Cohen’s d = 0.30; percent chance range: t(484) = 7.60, p < 0.001, and Cohen’s d = 0.69; and percent chance: t(484) = 10.30, p < 0.001, and Cohen’s d = 0.94] except the color only condition. Furthermore, Tukey post hoc comparisons showed that likelihood differences in both color: t(484) = 8.60, p < 0.001, and Cohen’s d = 0.78, and color + percent chance range: t(484) = 4.30, p < 0.001, and Cohen’s d = 0.39, conditions were significantly greater than in the percent chance range condition, suggesting that color significantly increased overestimation despite the addition of numbers. However, there was no significant difference between the percent chance range and percent chance conditions, p > 0.05, suggesting that ranges confer a similar advantage as individual probabilities.

There was also a significant main effect of actual probability range suggesting that overestimation increased in higher actual probability levels, F(1.53, 739.05) = 932.37, p < 0.001, with ${\text{η}}_p^2=0.66$. The degrees of freedom were corrected for violation of sphericity using the Greenhouse–Geisser correction. There was also a significant interaction between actual probability range and forecast format F(6.11, 739.05) = 8.07, p < 0.001, with ${\text{η}}_p^2=0.06$. As shown in Fig. 7, likelihood difference for watch and warning and color were similar in the bottom two percent chance ranges. However, the overestimation for color was much greater in the upper range [mean M = 42.02; standard deviation (SD) = 19.23] than for watch and warning (M = 34.26; SD = 17.45).

In sum, forecasts that included a numeric estimate of likelihood were less susceptible to overestimation than were those that did not (color and watch and warning). Surprisingly, percent chance range performed in a manner that was comparable to percent chance alone, suggesting that understanding was substantially better not only with single probabilities but also with second-order numeric uncertainty information.

2) Likelihood–severity confusion

We were also concerned that participants would misinterpret the expression of likelihood as an expression of severity, especially when color coding was used. Mistaking likelihood for severity was operationalized in three ways. The first was the difference between severity ratings and the likelihood ratings. The second was the correlation between severity and likelihood ratings. The third was the variability of severity ratings (in fact severity was held constant so less variability represents better understanding). There was evidence for this misunderstanding in all three operationalizations. Severity ratings were summarized as the percentage of the rating line between the left anchor (not severe) and the position to which participants moved the handle.

For the first operationalization, the likelihood rating was subtracted from the severity rating made at the same forecast update (severity − likelihood). Then an average difference score was calculated for each participant. The smaller the difference (positive or negative) was, the more similarly the participants regarded the two constructs and the greater was the confusion (see Fig. 8). An ANOVA conducted on mean difference score showed a main effect for forecast format F(4, 484) = 8.05, p < 0.001, with ${\text{η}}_p^2=0.06$, suggesting that the difference was least (confusion greatest) in the color (M = 2.43; SD = 10.45) and most (confusion least) in the percent chance range (M = 12.92; SD = 22.46) condition. Planned contrasts revealed that the mean difference for color was significantly smaller (more confusion) than percent chance range: t(484) = 4.15, p < 0.001, and Cohen’s d = 0.38, and percent chance conditions (M = 13.14; SD = 20.84): t(484) = 4.15, p < 0.001, and Cohen’s d = 0.38. Although the difference between color + percent chance range (M = 7.76; SD = 17.75) and color approached significance, t(484) = 2.10, p = 0.037, and Cohen’s d = 0.19, it did not fall below the Bonferroni corrected level of 0.0125. Nor did the difference between color and watch and warning (M = 3.06; SD = 15.88) reach significance. This suggests that forecasts that included a numeric estimate of likelihood were less susceptible to the likelihood–severity confusion than were those that did not. However, including color reduced this corrective effect to some degree and gave rise to confusion despite the presence of the numeric expression in the color + percent range condition.

View Full Size

Mean difference between severity and likelihood ratings, by forecast format.

Citation: Weather, Climate, and Society 14, 4; 10.1175/WCAS-D-21-0186.1

• Download Figure
• Download figure as PowerPoint slide

Mistaking likelihood for severity was also operationalized as the correlation between severity and likelihood ratings. Because the sampling distribution for highly correlated variables is skewed, the distributions were submitted to Fisher’s z transformation before conducting the ANOVA. The ANOVA revealed a significant main effect of forecast format F(4, 484) = 6.36, p < 0.001, with ${\text{η}}_p^2=0.05$, showing that correlations were highest in the color condition (M = 0.79; SD = 0.30) and least in the percent chance range condition (M = 0.56; SD = 0.43). Planned contrasts showed that the correlation in the color condition was significantly higher than in the percent chance range: t(484) = 4.50, p < 0.001, and Cohen’s d = 0.41, and percent chance conditions (M = 0.58; SD = 0.46): t(484) = 4.08, p < 0.001, and Cohen’s d = 0.37. However, the correlation in the color condition did not significantly differ from the watch and warning (M = 0.69; SD = 0.36): t(484) = 2.45 and p = 0.015, and color + probability range (M = 0.68; SD = 0.37): t(484) = 2.24 and p = 0.025, conditions. This again suggests that forecasts that included color were most susceptible to the likelihood–severity confusion.

The third operationalization of the likelihood–severity confusion was the variability in severity ratings represented by the mean standard deviation of each participant’s severity ratings. Recall that participants were informed that all storms involved same wind speed implying the same severity. Thus, their severity ratings should be the same throughout the experiment. Smaller mean standard deviation values indicated less variability and therefore less likelihood–severity confusion. A one-way ANOVA on mean severity SD revealed a significant main effect of forecast format F(4, 484) = 9.72, p < 0.001, with ${\text{η}}_p^2\hspace{0.17em}=0.07$ such that the SD was greatest in the color condition (M = 18.46; SD = 7.07) and least in the percent chance (M = 13.46; SD = 6.52) and percent chance range (M = 13.46; SD = 6.08) conditions. Planned contrasts revealed that the severity SD of the color condition was significantly greater than all other conditions: watch and warning (M = 15.44; SD = 6.56): t(484) = 3.20, p = 0.001, and Cohen’s d = 0.29; color + percent chance range (M = 15.96; SD = 6.87): t(484) = 2.66, p = 0.008, and Cohen’s d = 0.24; percent chance range (M = 13.46; SD = 6.08): t(484) = 5.38, p < 0.001, and Cohen’s d = 0.49; and percent chance conditions (M = 13.46; SD = 6.52): t(484) = 5.25, p < 0.001, and Cohen’s d = 0.47. This suggests that participants in the color-coded condition were confusing severity and likelihood to a greater degree than those using other formats.

Thus, analyses of all three operationalizations of this misunderstanding (severity difference, severity–likelihood correlation, severity standard deviation) suggest that color coding tends to promote, to a greater degree than the other formats tested here, a confusion between likelihood and severity. Participants using color-coded likelihood expressions tended to think that they were also receiving information about severity. Including a numeric expression (color + percent chance range) appears to counteract this tendency to some degree as shown in two of the three operationalizations, but not completely.

b. Trust in the forecast (hypothesis 2)

To determine whether explicit numeric likelihood estimates inspired greater trust, we examined trust in the forecast rated after learning the outcome of the storm at the end of each trial. Trust ratings were summarized as the percentage of the rating line between the left anchor, “not at all,” and the position to which participants moved the handle. An ANOVA conducted on mean postoutcome trust rating revealed a main effect of forecast format F(4, 484) = 3.10, p = 0.016, with ${\text{η}}_p^2=0.025$ such that trust was highest in the percent chance condition (M = 50.38; SD = 18.80) and lowest in the color condition (M = 42.82; SD = 15.75). Tukey post hoc comparisons showed that trust ratings for percent chance was significantly greater than for color: t(484) = 2.91, p = 0.031, and Cohen’s d = 0.26. In addition, trust was significantly higher in the color + percent chance range [M = 50.07; SD = 19.00, t(484) = 2.82, p = 0.040, and Cohen’s d = 0.26] than in the color condition. No other significant differences were found.

In sum, the percent chance forecast inspired the greatest trust. Color, which inspired the least trust, also gave rise to the greatest overestimation of likelihood, which may explain the low trust. Because the forecasts were well calibrated, overestimation of likelihood may have led participants to expect more tornado hits than they actually experienced. However, adding the percent chance range to color, helped to preserve trust. We return to this issue in the discussion section below.

c. Decision-making (hypothesis 3)

There were three issues related to decision-making that were of interest, decision quality, cautiousness, and timeliness. We hypothesized that both quality and timeliness would improve with numeric likelihood estimates. We had no specific hypotheses about cautiousness.

1) Decision quality

To determine the effect of forecast format on decision quality, it was operationalized as the expected loss/cost of participants’ final decisions (wait, take shelter, or not take shelter). Each decision was assigned either a point cost (take shelter) or the expected point loss of a penalty (wait or not take shelter). The expected loss can be thought of as the penalty amount weighted by the chance that it would be incurred (probability of the tornado hitting home). See appendix for the expected loss calculations. The cost or expected loss of the participants’ decision was then subtracted from the optimal expected value (or cost) on that trial. The smaller the expected loss difference the better the participants decision.

The ANOVA conducted on expected loss difference revealed a main effect for forecast format F(4, 484) = 5.19, p < 0.001, with ${\text{η}}_p^2=0.041$ such that participants in the percent chance condition had the smallest expected loss difference (M = −4.10; SD = 2.70), and those in the watch and warning condition had the largest (M = −6.65; SD = 4.92) (see Fig. 9). Planned contrasts revealed that decision quality in the percent chance: t(484) = −4.40, p < 0.001, and Cohen’s d = 0.40 and the percent chance range (M = −5.18; SD = 4.11): t(484) = 2.62, p = 0.009, and Cohen’s d = 0.21, conditions were significantly better (smaller difference) than in the watch and warning condition. Tukey post hoc comparisons revealed that percent chance (M = −4.10; SD = 2.70) was also significantly less (better) than color alone (M = −5.75; SD = 4.31): t(484) = 2.89, p = 0.032, and Cohen’s d = 0.26. No other significant differences were found. This suggests that the forecast showing the percent chance of a tornado led to better decisions than the conventional watch and warning as well as color coded likelihood.

View Full Size

Mean expected loss difference, by forecast format. Means closer to zero indicate better decisions, because there is little difference between the participant’s and the optimal expected loss values.

Citation: Weather, Climate, and Society 14, 4; 10.1175/WCAS-D-21-0186.1

• Download Figure
• Download figure as PowerPoint slide

2) Decision cautiousness

Although we had no specific hypotheses about this issue, it was important to evaluate cautiousness. It could be that some forecast formats (e.g., watch/warning) lead to more cautious decisions without leading to economically optimal decisions. That is because greater cautiousness means shelter more often overall, whereas economically optimal means sheltering only when that option constitutes the least point loss. To understand whether participants were more cautious (i.e., shelter more often) with certain forecast formats, a one-factor ANOVA was conducted on the proportion of take-shelter decisions (across the 40 trials regardless of whether take shelter was the optimal choice) with the independent variable forecast format (watch and warning; color; color + percent chance range; percent chance range; percent chance; see Fig. 10). There was a significant main effect of forecast format F(4, 484) = 5.19, p < 0.001, with ${\text{η}}_p^2=0.041$. Participants with the watch and warning [M = 0.37, SD = 0.21, t(484) = 4.04, p < 0.001, and Cohen’s d = 0.37] and color conditions [M = 0.36, SD = 0.18, t(484) = 3.73, p = 0.002, and Cohen’s d = 0.34] had a significantly higher proportion of shelter decisions than those with the percent chance condition (M = 0.26; SD = 0.16). Thus, although participants in the watch and warning and color conditions made fewer good decisions from an economic standpoint, they were more cautious overall.

View Full Size

Proportion of final decision out of 40, by forecast format.

Citation: Weather, Climate, and Society 14, 4; 10.1175/WCAS-D-21-0186.1

• Download Figure
• Download figure as PowerPoint slide

3) Decision timeliness

The final analysis was conducted on decision timeliness, operationalized as the difference in the forecast update number (1–7) at which the participant made a final decision (shelter or not take shelter) and the point at which it was optimal to make a final decision (stopping difference). It was optimal to make a final decision on the first trial on which the cost of sheltering or the expected loss of not taking shelter was the less than waiting. The step at which the participant made a final decision was subtracted from the optimal stopping point on each trial. A negative number means that the participant made a decision prior to the optimal stopping point and a positive number means the final decision was made after the optimal stopping point; zero is indicates that the two are the same (optimal). A mean was taken for each participant across all 40 trials.

Participants on average made timely decisions in all forecast formats although those in the percent chance condition showed a slight delay beyond optimal stopping (see Fig. 11). A one-factor ANOVA conducted on stopping difference revealed a significant main effect for forecast format F(4, 484) = 2.45, p = 0.046, with ${\text{η}}_p^2\hspace{0.17em}=0.02$. The stopping difference in the percent chance condition (M = 0.29; SD = 1.22) was significantly different than watch and warning (M = −0.33; SD = 1.81): t(484) = 2.71, p = 0.007, and Cohen’s d = 0.25. No other significant differences were found. Thus, although percent chance showed a slight delay, those in the watch and warning condition made their decisions slightly too early by about the same amount (about ⅓ of a step).

View Full Size

Mean difference in participants’ final decision and optimal stopping point, by forecast format.

Citation: Weather, Climate, and Society 14, 4; 10.1175/WCAS-D-21-0186.1

• Download Figure
• Download figure as PowerPoint slide

In sum, participants in the percent chance condition waited slightly longer to make a decision; however, they made decisions with the highest expected value. Percent chance range showed comparable decision quality to percent chance alone but did not delay significantly beyond optimal stopping. Participants in the watch and warning and color conditions made more cautious decisions and earlier, although their decisions had the greatest expected loss.

a. Understanding

Participants in the conditions in which numerical expressions alone were provided (percent chance and percent chance range) understood the forecast best both in terms of the intended likelihood and avoiding the potential confusion with severity. However, the benefits of color-coded likelihood are less clear.

b. Trust

The percent chance forecast also inspired the greatest trust. Color, which gave rise to the greatest overestimation of likelihood, inspired the least trust. However, adding the percent chance range to color helped to preserve trust despite inflated likelihood estimates, mirroring the positive impact of uncertainty estimates on trust seen in previous research (LeClerc and Joslyn 2015).

c. Decision-making

Not only did numeric expressions of uncertainty lead to better understanding, but they also led to better decisions in this dynamic environment. Participants with numeric likelihood forecasts made decisions with higher expected value than those using categorical forecasts (i.e., color coded or watch and warning forecast). This was true even though each forecast included multiple updates, each with its own probabilistic estimate and each choice point included three options (wait, take shelter, or not take shelter). In sum, these results suggest that participants were able to evaluate and incorporate numeric uncertainty information despite the increase in processing load represented by these complexities.

The advantage of percent chance over categorical expressions is particularly striking in this experimental context because the boundaries of the categories were aligned with the optimal decision. When the economically optimal decision was to not take shelter, the color was green and no warning was shown. When the economically optimal decision was to wait, the color was yellow and a watch was shown. When the economically optimal decision was to take shelter, the color was orange and a warning was shown. Therefore, those in the categorical conditions had information that was perfectly tailored to the cost–loss structure of the task. If participants had chosen to shelter only when the highest category was forecast (i.e., warning, or orange), the expected value in these conditions would have been superior (i.e., no difference from optimal). This was not the case. Instead, participants in both the color-coded and watch and warning conditions chose to take precautionary action more frequently, often unnecessarily expending resources to avoid a tornado that was unlikely.

While the percent chance expression proved to be advantageous overall, there was a slight delay in making the decision (timeliness), less than a single forecast update on average, which may be the only indication of the extra processing required with numeric expressions of uncertainty. In addition, those using the percent chance expression, although they made better decisions from an economic standpoint, were not as cautious as those using the watch and warning and color-coded forecasts who took precautionary action more often overall. Thus, there might be a trade-off in the choice of forecast format between precision and cautiousness, such that some forecast formats enhance the first while others the second. Some have argued that such trade-offs could be exploited to fit specific situations (Ash et al. 2014). For instance, color coding might be used in situations in which additional cautiousness would be beneficial.

It is important to note, however, that the increased cautiousness observed here is likely directly due to misunderstanding the forecast (as reflected in likelihood ratings). As such, there may be a cost in terms of lowered trust that could affect responses to subsequent warnings. Indeed, postoutcome trust was significantly lower in the color-coded than in the percent chance condition. It may be that trust in color-coded forecasts was diminished because of the frequent and ultimately unnecessary decisions to take shelter, suggesting a “false alarm” effect, in which participants saw themselves as receiving an inaccurate forecast.

d. Second-order uncertainty

Importantly, the advantages for numeric uncertainty expressions extended to the percent chance range condition, which was also categorical in a sense and added a level of uncertainty about the likelihood itself. Nonetheless, those using the percent chance range had a better understanding of the likelihood of a tornado than did those using the watch and warning as well as those using color + percent chance range, which was identical with the exception of color. Those using the percent chance range were also less susceptible to the likelihood–severity confusion than those using color alone (as seen in the severity standard deviation and severity difference operationalizations). Trust in this format was equal to that of the single percent chance expression, despite the fact that it that included uncertainty about the probabilistic forecast itself. Moreover, decision quality in percent chance range was equivalent to percent chance while decisions were made in a timelier fashion. Taken together, these results suggest that in a real-world settings, forecasters may be able to include second-order uncertainty information (e.g., a range such as 0%–12%), without a detriment to understanding, trust or decision quality.

e. Limitations

As an experimental study using a convenience sample of college students there are several limitations that should be mentioned. First, as with all experimental studies there was a trade-off between ecological validity and experimental control. In a natural environment several other factors would also influence peoples’ decisions. Here those factors have been stripped away or held constant in order to determine the impact of forecast format alone. Indeed, research using more realistic storm-tracking tasks suggests that with multiple potential information sources, search strategies must be adjusted (Wu et al. 2015a) and decisions can be delayed (Wu et al. 2015b).

In addition, the consequences in the real-world counterpart of this task would be far more serious. Here the consequences were merely a reduction in points that impacted the monetary reward. Although it is important to note that, unlike many laboratory-based studies, there were actual consequences in the task reported here. Participants who made better decisions received a larger cash bonus.

The college student participants may be more prepared to understand probabilistic forecasts than some members of the public, giving rise to questions about generalizability. However, recent evidence suggests that, although they may not have a theoretical understanding of probability, those with high school education or less can benefit from forecasts that include numeric likelihood estimates to the same degree as those who are college educated (Grounds and Joslyn 2018; Grounds et al. 2017).

f. Next steps

Although these are promising results, future research should replicate them using a more diverse sample to verify that generalization warranted. Research should also explore decision tasks with different kinds of complexities, such as multiple sources of information or multiple decision alternatives, to determine whether the advantages for numeric uncertainty estimates continue to hold. Finally, it would be useful to know if decision quality increases further with a combination of both numeric uncertainty information and explicit warning information in this complex scenario, as has been observed in simplified scenarios in previous research (Joslyn and LeClerc 2012).