What is the best way to communicate the risk of rare but extreme weather to the public? One suggestion is to communicate the relative risk of extreme weather in the form of odds ratios; but, to the authors’ knowledge, this suggestion has never been tested systematically. The experiment reported here provides an empirical test of this hypothesis. Participants performed a realistic computer simulation task in which they assumed the role of the manager of a road maintenance company and used forecast information to decide whether to take precautionary action to prevent icy conditions on a town’s roads. Participants with forecasts expressed as odds ratios were more likely to take appropriate precautionary action on a single target trial with an extreme low temperature forecast than participants using deterministic or probabilistic forecasts. However, participants using probabilistic forecasts performed better on trials involving weather within the normal range than participants with only deterministic forecast information. These results may provide insight into how best to communicate extreme weather risk. This paper offers clear evidence that people given relative risk information are more inclined to take precautionary action when threatened with an extreme weather event with a low probability than people given only single-value or probabilistic forecasts.
Effectively communicating the risk of rare but extreme weather events poses a challenge to forecasters. Forecasts must be given well in advance to allow residents adequate time to take precautionary action. However, at longer lead times, storm-track uncertainty and dramatic fluctuations in intensity often mean that the probability of significant impact for a given area is relatively low. The challenge is to communicate the risk posed by potentially extreme weather events with low probability in a manner that convinces vulnerable residents that the threat is serious and preparation is warranted.
Indeed, in many cases, residents fail to take adequate precautionary action for extreme weather events despite timely warnings (Petrolia and Bhattacharjee 2010). There are numerous potential explanations for low compliance with weather warnings. Of particular concern to emergency managers are false alarms (Breznitz 1985), which are thought to erode trust in the system providing the information (Bliss and Fallon 2006; Bostrom and Lofstedt 2003). Another factor that may contribute to noncompliance is the importance people attribute to the costs and losses associated with taking precautionary action. Some evidence suggests that people overestimate the cost of precautions, such as lost work productivity due to staying home in a winter storm, relative to the potential loss associated with ignoring the warning, such as injury or death caused by an automobile collision on icy roads (Dow and Cutter 2000). Other evidence suggests that people underestimate personal risk (Baker 1995; Drobot 2007). This might be due to cognitive factors, such as use of the availability heuristic (Tversky and Kahneman 1974) to judge one’s own vulnerability to extreme weather. For example, as a winter storm approaches, a person might feel less vulnerable if examples of precautions he or she has taken, such as stocking up on supplies or putting chains on the car, come easily to mind (Fischhoff et al. 1993). Affective factors, such as overoptimism (Nicholls 1999) or wishful thinking, may also contribute to the underestimation of personal risk (Harris et al. 2009). Furthermore, past experience with the rare event itself can influence decision making (Weber 2006). People continuously update their perception of an event’s likelihood based on how recently the event has occurred, and because recent events play a greater role than distant events in decision making and rare events are less likely to have occurred recently, people might underestimate the risk posed by a rare event (Hertwig et al. 2004; Weber 2006). Another factor that may contribute to noncompliance with weather warnings is the belief that forecasts for extreme events are exaggerated, incorporating both probability and event severity (Patt and Schrag 2003). Thus, there are many possible reasons for the low compliance rates with current weather warnings.
Many of these problems might be overcome by adding an explicit uncertainty estimate to the forecast to quantify the risk and potentially increase the plausibility of the forecast. At present, numerical probabilities are usually not included in weather warnings, in part because of the concern that the general public will not understand them. Moreover, as was mentioned above, to give residents adequate time to take precautionary action, forecasts for extreme weather events must be given so early that event probabilities are often low. Emergency managers fear that communicating low numerical probabilities will suggest to people that the forecasted event is unlikely and precautionary action is unwarranted. Instead, in many cases verbal expressions of probability are used. However, evidence suggests that such expressions are rendered largely meaningless by the enormous variability in interpretation (Wallsten et al. 1986; Moxey and Sanford 2000). A forecast that snow is “likely” might suggest a 60% chance of snow to one person and a 90% chance to another. Research also suggests that verbal expressions suffer significantly from effects of context (Windschitl and Weber 1999): a “slight chance” of rain means something very different to a resident of Seattle than to a resident of Los Angeles.
However, some of the concerns about numerical probability estimates may be unfounded. Recent research suggests that weather forecasts with numerical uncertainty estimates lead end users to make better, more economically rational decisions than do single-value deterministic forecasts (Joslyn and LeClerc 2012; Roulston et al. 2006), suggesting that understanding probabilities is not a problem. In one series of experiments (Joslyn and LeClerc 2012), participants played the role of a manager of a road maintenance company. They were presented with forecasts for overnight low temperatures to help them decide whether to treat the town’s roads to prevent icy conditions. The salt treatment cost $1000 per application, but if participants failed to treat the roads and a freezing temperature was observed, they were penalized $6000. The normative decision on any trial was based on a cost–loss ratio (Thompson 1952), the break-even point between costs and potential losses, which in this case was obtained by dividing the cost of salting the roads ($1000) by the potential penalty for not salting if a freezing temperature was observed ($6000), or 0.17. Thus, in the long run, to maximize what is referred to as “expected value” (Von Neumann and Morgenstern 1944), it was more cost effective to salt the roads for trials with a probability of freezing at or above 17%. Results suggested that participants who received probability of freezing forecasts performed significantly better and had greater trust in the forecasts than those who received only deterministic forecasts (Joslyn and LeClerc 2012). Other experiments have produced similar results (e.g., Roulston et al. 2006).
Although decisions in these experiments were significantly better with uncertainty information than without it, participants did not, on average, make perfectly rational decisions (i.e., always following the rule based on the cost–loss ratio). In particular, participants with probability of freezing estimates generally failed to apply treatment at low probabilities of freezing for which it was the “rational” decision (17%–23%), presumably because the probabilities seemed too low to warrant action (Joslyn and LeClerc 2012). Thus, while uncertainty estimates, such as probability of freezing, are helpful for typical seasonal weather, they might not be helpful in situations of extreme weather, such as hurricanes or extreme winter storms, in which precautionary action is warranted at low probabilities.
Instead, for rare but extreme weather events, it may be helpful to augment the forecast with an odds ratio. In the case of a rare October snowstorm, for example, the probability of forecasted snow accumulation and wind speeds might be relatively low, but much higher than on a typical October day. Thus, to emphasize the increased likelihood of a rare weather event, expressing the forecast in terms of an odds ratio, the increase in odds of the event in the present situation over climatological odds, has been suggested (Murphy 1991). Climatological odds are a ratio of the odds of an event occurring to the odds of it not occurring over many years of observations (Zhu and Toth 2001). Therefore, an odds ratio expresses the likelihood of a weather event in the present situation relative to past events, whereas a probability estimate expresses the likelihood of a weather event occurring in the present situation based on data from numerical models (forecast probability; Zhu and Toth 2001). The argument is that an odds ratio will make the threat of a rare, extreme event seem more serious than the probability estimate. A similar approach, relative risk, has been used to express health risks (Lipkus 2007). For example, the likelihood of getting lung cancer from smoking cigarettes is actually quite small, but it is many times greater for smokers than nonsmokers. Indeed, evidence suggests that use of relative risk encourages precautionary action. For example, in one experiment, participants were willing to pay more for products, such as tires, to avoid low-probability events, such as tire blowouts, when pairs of alternatives were compared in terms of relative as opposed to absolute risk (Stone et al. 1994).
It is important to note, however, that end users presented with an odds ratio would not know the actual probability of the weather event on any given occasion, but rather only how much more (or less) likely the event is relative to climatology. Thus, there is some concern that odds ratios will lead to overestimation of risk on the part of the user (Edwards et al. 2001). Therefore, odds ratio forecasts might be best suited for situations in which persuasive techniques are deemed warranted (Lipkus 2007), such as when human life is at stake. To our knowledge, however, the impact of odds ratios on weather-related decisions, compared to other forms of risk communication, has never been tested empirically.
Therefore, in the study reported here, we tested the impact of forecasts expressing the increase in odds over climatology. As in the experimental paradigm described above (Joslyn and LeClerc 2012), participants used temperature forecasts, expressed in degrees Fahrenheit, to decide when to treat the roads to prevent icy conditions. To simulate the rare event situation, in addition to regular salt brine required for subfreezing temperatures, special treatment was required for subzero temperatures, unusual for the region in which the experiment was set (Washington State). Therefore, each decision included three options: withholding treatment, applying regular salt brine appropriate for temperatures between 1° and 32°F, and applying special salt brine appropriate for temperatures 0°F and below. We compared decisions made by participants with conventional deterministic forecasts to those with forecasts that also included probability estimates and to those that also included the increase in odds over climatology (odds ratio). We predicted that participants with probability estimates would do best at the task overall, replicating past experiments, but that odds ratio participants would have an advantage in choosing the optimal treatment on the extreme cold trial.
A total of 294 University of Washington psychology students (48% female) participated for course credit and the chance to earn prize money. Mean age was 19 yr (range 18–32 yr).
The experiment, programmed with Microsoft Excel Visual Basic, was administered on standard desktop computers.
After participants gave informed consent and entered demographic information, they read a set of instructions, which included a description of the task and the cost–loss structure, at the same time that the experimenter read the instructions aloud. Participants were to assume the role of a manager of a road maintenance company contracted to treat the roads in an eastern Washington town to prevent icing at two key thresholds: 32° and 0°F. There were 60 trials representing a hypothetical 2-month period of winter weather. Participants received a virtual monthly budget of $31 000. Two types of treatment were available: a regular salt brine for temperatures at or below 32°F and a special salt brine for extreme temperatures at or below 0°F. Applying regular salt brine cost $1000 per day and applying special salt brine cost $2000 per day. The penalty for failing to apply regular salt brine when a temperature between 32° and 1°F was observed was $6000. The penalty for failing to apply special salt brine when a temperature of 0°F or below was observed was $12 000. Thus, in both cases, the cost–loss ratio was 0.17, meaning that, from an economic standpoint, participants ought to have applied the necessary treatment for probabilities greater than or equal to 17%. Participants were instructed to attempt to maximize profits by minimizing salting expenses and avoiding penalties. At the end of the experiment, participants were paid $2 for every $1000 in their final balance, such that applying special salt on one trial per month and regular salt on all other trials would constitute breaking even.
In each trial, representing one day, a forecast for the next night appeared on the screen. After reading the forecast, participants clicked on one of three boxes marked “Regular Salt,” “Special Salt,” or “No salt.” Finally, participants entered a numeric value in a text box to indicate what they thought the nighttime low temperature would be. Immediately afterward, the observed nighttime low temperature and any budget adjustments appeared on the screen. Participants were able to borrow against the next month’s budget installment if their balance dropped below $0. After 30 trials, representing one month, participants clicked “Next” to continue to the next month’s trials and $31 000 was added to the budget. At the end of the second month, participants with ending budgets above $0 were paid and all participants were awarded course credit points. Experimental sessions included from 1 to 12 participants and lasted approximately 45 min.
A 3 × 4 between-participants design was used. Participants were randomly assigned to one of three forecast formats, all of which included the deterministic forecast. The deterministic condition included only the deterministic forecast. The probability condition also included uncertainty expressed as percent chance, and the odds ratio condition also included uncertainty expressed as the increase in odds for that night compared to typical winter nights; see Table 1a.
To determine the threshold at which participants would choose to apply special salt, participants were also randomly assigned to one of four weather datasets. The four weather datasets were identical except for a single trial, on which the probability of observing temperatures 0°F or less was greater than 5% (target trial). On the target trial, the probability of a nighttime low temperature of 0°F or less was 10%, 17%, 24%, and 31%. The dependent variables were final balance, mean expected value, and binary salt decision.
Participants in all conditions received the same single-value nighttime low temperature forecasts and observed temperatures in the same order. Only the target trial, which was the 22nd forecast for all participants, was different. The ranges of temperature, probabilities of freezing, and forecast error were based on historical forecast data from the cities of Spokane and Yakima in Washington State. On the nontarget trials, the single-value forecasts ranged from 12° to 37°F (M = 32.66°F), the probabilities of freezing (PoF) ranged from 10% to 100% (M = 35.31%), the probabilities of 0°F or colder ranged from 0% to 1%, and observed temperatures ranged from 10° to 41°F (M = 32.90°F). The observed temperatures followed realistic trends that included only natural fluctuations from one night to the next of less than 16°F. Note that the trial immediately preceding the target trial forecasted an abnormally cold night, 8°F (the second coldest night of the forecast set), and had a probability of 0°F of 5%. Even though it did not warrant application of special salt, that trial was excluded from all analyses of nontarget trials because it was still relatively extreme. Therefore, the nontarget trials were all trials except the target trial and the trial that preceded it (the 21st and 22nd trials).
The probabilistic forecasts were reliable. The percentage of trials on which freezing temperatures were observed approximately matched the probabilities of freezing stated in those trials. This was achieved by dividing the probability forecasts into seven range categories (10%–16%, 17%–23%, 24%–33%, 31%–37%, 38%–44%, 45%–51%, and 52%–100%) and ensuring that the percentage of observed temperatures 32°F or less in each category was within that probability range. For example, in the 10%–16% range, temperatures 32°F or less were observed on 2 of 18 (11.1%) days. There were an approximately equal number of forecasts within each of the ranges of probability. Half of all observed temperatures were above their respective deterministic temperature forecasts and half were below.
Climatological odds were calculated from approximately 100 yr of historical weather data from Yakima and Spokane, Washington, provided by the Department of Atmospheric Sciences at the University of Washington. We tallied the number of times between December and March that a low temperature less than or equal to 0°F was observed and divided that by the number of times a low temperature greater than 0°F was observed (637/18 954 = 0.034). Then an increase in odds over climatology was calculated for each forecast in the weather dataset used for the experiment. We took the odds of the 0°F-or-less low temperature that corresponded to the probabilistic forecast on that trial and divided it by the climatological odds. For instance, when the probability of 0°F or less was 17%, it was 0.17/(1 − 0.17)/0.034 = 6; see Table 1b.
First, we examined performance on the regular salt task, focusing on the typical-weather nontarget trials, to determine whether the previously found advantage for decisions based on probabilistic forecasts—that is, that they were economically superior to those based on deterministic forecasts (Joslyn and LeClerc 2012; Roulston et al. 2006)—extended to this new, more complex task (analyses a–c, below). Then we turned to participants’ decisions regarding the extreme weather event, focusing on the target trial (analysis d). Before conducting these analyses, however, we sought to determine whether participants understood the task and were taking it seriously, so we examined their temperature estimates. We compared participants’ temperature estimates to the forecasted temperature on each trial to create a distribution of mean standard errors of temperature estimates. Participants whose standard errors were two standard deviations or more above the mean standard error for temperature estimates within their forecast format were omitted from further analysis. Using this criterion, 8 participants were removed, leaving 286 participants.
a. Final balance (nontarget trials)
We assessed participants’ performance at the road-salting task by computing their final balance over all trials except the extreme weather target trial and the trial that immediately preceded the target. Final balances ranged from −$65 000 to $14 000, with the higher final balance suggesting better performance. A final balance of $4000 reflected following the economically rational course of action on all trials (i.e., applying the appropriate treatment whenever the probability of the event exceeded the cost–lost ratio, and withholding treatment otherwise), although some participants were able to score an even higher final balance by chance. Participants using the probability forecast did best overall. Independent samples’ t tests [see Howell (2008) for information on statistical tests used in the present analyses] revealed that probability participants (M = −$4290.36, SD = $10 691.91) had a significantly higher mean balance than both odds ratio participants (M = −$11 391.75, SD = $11,939.57), t(188) = 4.31, p < 0.01, and participants using deterministic forecasts (M = −$12 291.67, SD = $14 774.03), t(187) = 4.25, p < 0.01; see Table 2. (The statistic p is the probability of a Type I error, in which an effect is said to exist when it really does not.)
b. Mean expected value (nontarget trials)
The final balance is a straightforward measure of performance, but it is influenced at least in part by chance. Participants can be penalized for making normatively correct decisions, for example, withholding regular salt when the probability of freezing is less than 17% but an unlikely freezing temperature is observed. They can be rewarded for making normatively incorrect decisions, for example, withholding regular salt when the probability of freezing is greater than or equal to 17% but a freezing temperature is not observed. A more direct measure of the quality of participants’ decisions is mean expected value because it is not influenced by outcomes. For example, a decision to apply regular salt at 30% probability of freezing should be regarded as a normatively appropriate decision irrespective of the observed temperature on that trial. We calculated the expected values for each decision, except those made on the extreme weather target trial and the trial that immediately preceded it, by assigning the cost of salting ($1000 or $2000) to decisions to apply salt and the potential penalty of $6000 multiplied by the probability of freezing on that trial to decisions to withhold salt. Then we calculated a mean expected value for each participant. Mean expected values ranged from −$1856.90 to −$978.62, with a mean expected value of −$971.03 reflecting following the economically rational course of action on all trials. Finally we compared means calculated over each forecast format condition to one another. Again, participants using probability forecasts did best overall. Independent samples’ t tests revealed that probability participants (M = −$1155.67, SD = $118.63) had significantly higher mean expected value than did both odds ratio participants (M = −$1236.84, SD = $157.26), t(188) = 4.00, p < 0.01, and participants using deterministic forecasts (M = −$1206.61, SD = $161.48), t(187) = 2.47, p = 0.02; see Table 2.
c. Overall salting decision (nontarget trials)
To better understand the decision strategies that led to the advantages for the probability format, we examined individual decisions more closely. To determine whether participants made different decisions in the ranges of PoF above and below the economically rational threshold, we calculated the proportion of salt decisions (regular and special salt) in each of the seven probability ranges and then calculated the average proportion above and below the 17% threshold for each participant and averaged them for each condition; see Fig. 1. Then to determine whether the differences in proportion salting in the categories were statistically significant, we conducted a mixed-model analysis of variance (ANOVA) on mean proportion of “salt” responses per participant. Probability of freezing range (above and below 17%) was the within-groups variable and forecast format (probability, odds ratio, and deterministic) was the between-groups variable. There was a significant main effect for probability of freezing, F(1, 283) = 4328.59, p < 0.01. Participants salted less below (M = 0.13) than above (M = 0.65) the 17% probability of freezing threshold. There was a significant main effect for forecast format, F(2, 283) = 3.29, p = 0.04. Participants using the deterministic forecast salted more often overall (M = 0.42), while those using probability (M = 0.37) and odds ratio (M = 0.38) forecasts salted less often. Most importantly, there was a significant interaction, F(2, 283) = 12.55, p < 0.01, suggesting greater differentiation of weather conditions among participants using probability forecasts (Fig. 1). These results are consistent with previous results using the same experimental paradigm (Joslyn and LeClerc 2012), indicating that participants with probability forecasts made superior decisions overall. Consistent with the original experiment, however, at the lowest probabilities of freezing at which salting was appropriate (17%–23%), participants using the probability forecast in the current study (M = 0.17, SD = 0.24) did not salt as often as did those using deterministic (M = 0.29, SD = 0.25) or odds ratio (M = 0.22, SD = 0.25) forecasts. The first comparison reached statistical significance, t(187) = 3.25, p < 0.01.
d. Special salting decision (target trial)
Finally, we explored participants’ decisions to apply special salt on the target trial. Here, those using the odds ratio forecast were more likely to apply special salt than were participants using other forecast formats; see Fig. 2. We conducted a logistic regression analysis on the application of special salt, coded as a binary variable (apply special salt, apply regular salt, or no salt), with two independent variables: forecast format (odds ratio, probability and deterministic) and probability of 0°F (10%, 17%, 24%, and 31%). We refer to the four different likelihoods of observing 0°F in terms of probability for the sake of simplicity. Bear in mind, however, that the expression of this likelihood depended on the forecast format and only appeared as probability in the probability condition. Participants using odds ratio forecasts were 2.29 times more likely to apply special salt than were participants using the deterministic forecast, Exp(B) = 0.44, p = 0.01, and 14.49 times more likely to apply special salt than participants using probability forecasts, Exp(B) = 0.07, p < 0.01. [Exp(B) is the ratio of the effects of an experimental manipulation to a control.] In addition, participants were more likely to apply special salt above the 17% threshold than below it. Recall that this variable was manipulated between groups. Compared to participants for whom the probability of 0°F was 10% on the target trial, participants in the 17% condition were 1.93 times more likely to apply special salt, Exp(B) = 1.93, p = 0.09 (marginally significant); participants in the 24% condition were 5.95 times more likely to apply special salt, Exp(B) = 5.95, p < 0.01; and participants in the 31% condition were 8.72 times more likely to apply special salt, Exp(B) = 8.72, p < 0.01. In addition, the interaction of forecast format with a probability of 0°F was significant, Wald’s χ2(6) = 34.39, p < 0.01 (see Fig. 2). Notice that the difference in the likelihood of 0°F impacted forecast formats differently. Those in the probability condition salted very little overall, and there was little difference in the four likelihood conditions. Those using the odds ratio and the deterministic forecasts salted more often the higher the likelihood of 0°F was, although the pattern is somewhat different in these two formats. Those using the odds ratio forecast salted more often in all four categories than did those using deterministic forecasts.
These results confirm Murphy’s (1991) hunch that forecasts expressing an increase in odds over climatological norms induce more cautious decisions for rare, extreme weather events than do deterministic or probabilistic forecasts. The majority of those using odds ratio forecasts took appropriate precautionary action and applied special salt in all three likelihood ranges above the 17% probability of 0°F threshold, demonstrating that knowledge of the enhanced odds of observing rare, extreme weather was convincingly motivating for most participants.
However, odds ratios were not effective for regular salt application on typical weather trials. For these situations they provided the decision maker with no additional useful information beyond the deterministic forecast. On almost every trial, the odds ratio forecast stated that compared to a typical winter night, the odds were no greater for observing temperatures below freezing. For these trials, probability forecasts were most effective. Participants using probabilistic forecasts performed better, ending with higher balances and higher mean expected values, than those using deterministic or odds ratio forecasts. Furthermore, they differentiated to a greater degree between situations in which precautionary action was and was not warranted. These results replicate the advantage of probabilistic over deterministic forecasts reported in earlier work (e.g., Joslyn and LeClerc 2012; Roulston et al. 2006) and extend them to a much colder weather situation in which the decision-making task was more complex, involving two different possible treatment alternatives.
However, probability forecasts were not effective in the lowest range of probabilities in which precautionary action was economically warranted. This was most obvious on the extreme weather target trial, for which participants using probability forecasts were less willing to take precautionary action than participants using both other forecast formats. This demonstrates the problem faced by emergency managers who wish to warn the public about an extreme weather event: although a forecast with a realistic uncertainty estimate may be more plausible to many end users, when the probability of the event is low, a warning including a probability estimate may not be taken seriously. In this experiment, probabilities between 17% and 31% did not inspire most participants to take precautionary action of either kind. Perhaps they did not think the risk warranted the cost to apply treatment. It is possible that the low stated probability even discouraged some from taking action, as the proportion of probability participants choosing to apply special salt on the target trial (21%) was less than the proportion among participants using the deterministic forecast (65%).
Thus, for low-probability situations such as this, odds ratios clearly do a better job of encouraging precautionary action. More than 80% of those using odds ratios applied special salt on target trials above the 17% likelihood threshold. However, it is important to note that those using odds ratio forecasts also inappropriately applied special salt more than other participants when the probability of 0°F was only 10%, suggesting that any elevation in the odds over previous trials is regarded as reason to take action. Therefore, forecasts expressing increase in odds over climatology must be used with care. Although such forecasts might well encourage precautionary action initially, it is possible that they will give rise to false-alarm effects over time. Whether this is in fact the case and how it would compare with the false-alarm effects induced by deterministic forecasts is at present unknown. This constitutes a fruitful line of inquiry for future behavioral research.
Future research might also explore the degree to which the present results generalize to both different users and different weather situations. We believe that the majority of college-educated weather consumers would interpret these forecast expressions similarly to what is reported here, although multiple additional factors may influence real-life decisions. In addition, while less educated users may not have a conceptual understanding of the mathematics that give rise to these forecast formats, they may also react to them similarly. Survey research suggests that a wide range of general public end users have an intuitive understanding of the uncertainty inherent in weather forecasts (Joslyn and Savelli 2010). We suspect that regardless of one’s level of education, “6 times greater” will seem more alarming than “17% chance.” In fact, more advanced statistical knowledge may well reduce—rather than enhance—the effect. Furthermore, we see no reason why these results concerning extreme cold weather would not extend to other types of weather events. Clearly, additional research is required to confirm these suspicions.
Thus, the practical implications of this line of research are far reaching. In our opinion, forecasts with odds ratios can be extremely effective in convincing users to take precautionary action. As such, they may be appropriate for a wide range of dangerous weather situations in which prompt public response is required, such as hurricanes and tornados. They may well increase compliance rates and save lives. However, they should be reserved for situations in which persuasion is truly warranted.
This material is based on work supported by the National Science Foundation Grant 0724721.