1. Introduction
Although all weather forecasts involve uncertainty, many important weather-related decisions in diverse domains such as agriculture, water and energy management, and boater safety are based on deterministic forecasts. Nonetheless, there is evidence that many people understand the uncertainty involved in weather predictions and could make use of explicit uncertainty forecasts (Morss et al. 2008; Lazo et al. 2009; Joslyn and Savelli 2010).
For instance, a survey of over 1000 general weather users from Washington and Oregon states (Joslyn and Savelli 2010) demonstrated that everyday users understood many of the factors that increase uncertainty, including lead time and deviations from climatology. Interestingly, their responses suggested that users regarded some forecasts as systematically biased, correctly anticipating the high false alarm rate for wind and winter storm warnings but incorrectly expecting single-value extreme temperature forecasts to verify closer to normal values. This same survey (Joslyn and Savelli 2010) also revealed important psychological properties of uncertainty understanding. For instance, some of the anticipated biases depended on how the question was asked. Some biases were revealed in questions about individual forecasts, but not when respondents were asked to generalize across forecasts. This may have been because respondents were consciously aware of only some biases. Generalizing across forecasts probably involved a deliberate assessment of the proportion of observations anticipated above or below, requiring conscious awareness, whereas evaluating a single weather event likely did not. In addition, respondents anticipated greater uncertainty when the forecast was used to make a decision, suggesting that when one is contemplating a course of action, a broader range of potential outcomes is considered.
The research reported here takes up where the previous research left off. It probes the understanding of forecast uncertainty among those with a vested interest in the weather. Many believe that, because of the challenging nature of uncertainty forecasts, the benefits would be realized mainly among high-stakes users. To that end, the research reported here was based on a survey of recreational boaters in the Pacific Northwest of the United States, a sizable and important user group. The 2009 U.S. Recreational Boat Registration Statistics lists 450 367 boats registered in Washington and Oregon states. Indeed, Seattle, Washington, has more pleasure boats per capita than any other city in the United States (Seattle Parks and Recreation 2011).
Recreational boaters constitute high-stakes end users because weather has a huge impact on almost all aspects of boating. It is crucial for safety as well as cruise planning and navigation. The major body of water in this area, Puget Sound, is known for its highly variable weather and currents, making trip planning critical to boater safety. In 2010, the U.S. Coast Guard (2010)1 reported 4730 boating accidents, which resulted in 736 fatalities. Hazardous waters (91 deaths) and weather (74 deaths) ranked second and third, respectively, as the primary contributing cause of death after alcohol use (120 deaths) in the recreational boating statistics of 2009.
Arguably, the weather parameter with the highest impact on boating safety is wind. Thus, although other weather parameters were tested as well, our focus was on wind speeds, wind warnings, and wind advisories. Little or no research of which we are aware investigates the attitudes of high-stakes users, such as these, toward weather warnings. Do they regard such forecasts as involving uncertainty? If so, how does that impact their response to them?
There is, however, considerable research investigating the response of general public end users to warnings for high impact events, especially hurricanes. For most warnings the compliance rate is less than desirable, often around 50%. For instance, interviews of a sample of respondents under mandatory evacuation for Andrew and Hugo, both category-4 hurricanes, revealed that only 42% of them evacuated their homes (Riad et al. 1999). For Hurricane Floyd, the evacuation rate among those sampled was 64% (Dow and Cutter 2000). Higher rates are sometimes noted. For Hurricane Ike, 75% of those interviewed reported evacuating in response to a dramatic warning stating that those who stayed faced “certain death” (Morss and Hayden 2010). In addition, there was an unusually high compliance rate for Hurricane Rita, experienced only one month after the widely publicized devastation of Hurricane Katrina. Of those polled, 90% reported that they evacuated (Zhang et al. 2007). However, these last two examples are exceptional. Thus, many believe that including uncertainty estimates in weather warnings would further reduce the compliance rate, as the probabilities are often fairly low when evacuation is required.
It is important to note, however, that although warnings have an effect on evacuation decisions (Morss and Hayden 2010; Baker 1995; Zhang et al. 2007), so do many other factors such as vulnerability of family members (Morss and Hayden 2010), individual household circumstances (Dow and Cutter 2000; Zhang et al. 2007), and trust in the forecast (Lazo et al. 2010). Many believe that current evacuation decisions are especially influenced by past experience with warnings that resulted in false alarms, decreasing willingness to comply. However, the evidence for this hypothesis is mixed. Among studies that investigated false alarms directly, there is some weak evidence to support the hypothesis (Dow and Cutter 1998), although sometimes no effect was observed (Burnside 2006). Other studies exploring the effect of prior experience with warnings in general sometimes found a positive relationship: those who evacuated in the past were more likely to evacuate for the current warning (e.g., Riad et al. 1999). However, sometimes no relationship was observed between previous and current evacuation decisions (Lazo et al. 2010).
Among the primary goals of the research presented here was to explore the impact of prior experience with critical forecasts on users’ understanding of the inherent uncertainty as well as on their precautionary decisions. However, our approach was different from that of the research reviewed above. First, we sought a user group with increased expertise to determine whether clearer patterns would emerge. Because boating leads to considerable practice in evaluating weather forecasts for the purpose of making serious safety decisions, recreational boaters were targeted. Secondly, we investigated decision making as well as the parameter values that respondents expected to observe. This allowed us to directly compare expectations and decisions to determine the level of consistency in boaters’ decisions. In particular, boaters may correctly anticipate the high false alarm rate of warning forecasts and choose to ignore them in some cases. Experienced boaters may believe that such forecasts are not tailored to their specific situations and attempt to “second guess” the warning, thinking, for instance, that it applies to others who are in a more vulnerable situation. Such reasoning may serve to justify less cautious decisions, especially if the forecast is regarded as a potential false alarm.
The questionnaire used to explore these issues targeted weather warnings as well as everyday weather forecasts and included two different kinds of questions. One kind of question asked about individual forecasts (single-forecast questions) and the other required generalizations across multiple forecasts (multi-forecast questions). The questions targeted weather parameters known to be relevant to decisions regarding trip planning and boat protection, including nighttime low temperature, wind speed, and precipitation. When asking about single forecasts, we specified 1-day or 3-day forecast lead times and, in most cases, a related decision.
Although some of the questions were identical to those tested in our own previous study (Joslyn and Savelli 2010) to allow for a comparison between them, the boater questionnaire emphasized wind speed forecasts that are particularly relevant for boaters, including the standard marine advisories and warnings for small craft advisories (sustained winds of 21 to 33 kt, potentially in combination with wave heights exceeding 10 feet or wave steepness values exceeding local thresholds), gale warnings (winds ranging from 34 to 47 kt are forecast for the area), and storm warnings (winds ranging from 48 kt or higher are forecast for the area) (NOAA 2007). Although these usually include some information about wave height, we focused here on the wind element for the sake of simplicity. Notice that although marine advisories and warnings are defined as ranges of wind speeds, they do not include specific uncertainty estimates. Many of the questions about warnings and advisories also described specific locations to determine whether boaters regarded the forecasts for some locations as more reliable. We targeted three main geographic areas (Fig. 1) that are referred to in standard marine forecasts:
Admiralty Inlet,
Northern inland waters including San Juan Island, and
Puget Sound and Hood Canal.
Map of the three main areas targeted in the survey.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-11-00025.1
Map of the three main areas targeted in the survey.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-11-00025.1
Map of the three main areas targeted in the survey.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-11-00025.1
Thus, the research reported here sought to answer two main questions about these high-stakes forecast consumers: 1) How well do they understand the uncertainty inherent in weather forecasts and 2) how does perceived forecast uncertainty impact their decision making? In the following sections we will explain the methodology, describe boaters’ responses, and evaluate their implications. Finally, we will discuss recommendations for providing uncertainty forecasts to end users with a vested interest in weather forecasts.
2. Method
a. Respondents
Sixty-six respondents were recruited through the University of Washington Boater Information System (http://bis.apl.washington.edu) Web site from June 2009 to November 2009 and an additional 120 respondents were recruited through a Seattle-area weather blog (www.cliffmass.blogspot.com) in July 2010 for a total of 186 respondents. Of those, a total of 166 boaters completed the questionnaire with valid responses and indicated that they were indeed boat owners. Eighty-five percent of respondents were male. Twenty-three percent were between the ages of 18 and 40, 48% were between the ages of 41 and 60, and 29% were 61 yr of age or older. Compared with the respondents of previous studies (e.g., Joslyn and Savelli 2010) and the general population of Washington State, there were more male respondents and more of them were over the age of 61. However, this is similar to the general boater population in Washington State, which is predominantly male (66%) and older (M = 50.6 yr) (Responsive Management 2007).
b. Boat categories
The boats owned by respondents were classified in terms of their vulnerability to high winds. They were assigned to one of five categories shown in Table 1. Those respondents who provided a description for more than one boat were categorized according to the boat that fell into the category least susceptible to high winds. The majority (67%) of respondents described their boat as being a category 3, “slightly vulnerable to high winds,” or higher (i.e., less vulnerable).
Number of respondents by boat category and boat category descriptions.
c. Questionnaire
There were 35 total questions (see the appendix). They asked about nighttime low temperature; wind speed, which was expressed as categories indicating ranges as described above; and precipitation, which was a binary forecast. The parameter values were selected to be representative of the local climate and to be relevant to decisions made by boat owners in the Pacific Northwest.2 Precipitation is the one parameter for which uncertainty forecasts are commonly available, although the expressions vary widely in public forecasts. Because of the variety of ways in which it is expressed, and because the focus here was on boaters’ understanding of deterministic forecasts, precipitation forecasts were simplified to “rain” or “no rain.” While this approach might be inadequate to evaluate people’s understanding of forecast wording (e.g., showers likely) it provided information about boaters’ general expectations in a manner comparable to the other parameters tested here.
There were two kinds of questions designed to probe uncertainty expectations, 18 asking about specific forecasts (single-forecast questions; questions 1–6 in the appendix) and 15 asking respondents to generalize over a group of forecasts (multi-forecast questions; questions 8–11 in the appendix). The single-forecast questions specified a day and a month of the year as well as the parameter value and asked respondents to indicate what they expected to observe and what they would “not be surprised” to observe. Indicating a range of values in answer to these questions implied that the respondent expected uncertainty in the forecast. Each single-forecast question was asked twice, once with a 1-day and once with a 3-day lead time.
To determine how uncertainty expectations impacted decision making, most of the specific questions also included a binary decision (appendix). We created eight realistic scenarios, with the help of several subject matter experts, all of which involved a decision. While the consequences varied, each case required some form of precautionary action according to our subject matter experts. Each scenario involved either a typical boat protection task or trip planning decision (appendix questions 2–6). The boat protection questions included pumping out water lines to prevent freeze damage, covering the boat to prevent water damage, and securing the boat to prevent wind damage. The trip planning scenarios involved small craft advisories. Each trip planning scenario described a specific route involving one of three of the geographic areas for which standard marine forecasts are provided (appendix question 4). Each scenario also included a specific motivation (e.g., meet former classmates not seen in many years). Then, boaters were asked if they would alter their plans based on the advisory.
To probe uncertainty expectations another way, respondents were asked to indicate the number of days, out of either 10 or 100, that they expected the observation to fall within specified ranges above and below the deterministic forecast (multiforecast questions; appendix questions 8–11). The same three parameters—nighttime low temperature, wind speed, and precipitation—were tested. A reference class of 100 was used for questions about common forecasts, a number which participants could reasonably be expected to have experienced on previous occasions. A reference class of 10 was used for warning and precipitation forecasts, which are experienced less often. In addition to asking respondents to consider multiple forecasts, both contextual information and specific forecast values were absent from multiforecast questions. Finally, in order to determine the probability threshold at which respondents would alter their plans, there were two questions in which they were asked to indicate the percent chance at which they would begin to take precautionary action given extreme wind speed and precipitation forecasts (appendix question 7) and what precautionary action they would take.
Each boater responded to a subset of the total questions, including at least one question for each parameter, several multiforecast questions, and an equal number of single-forecast questions with 1-day and 3-day lead time. However, no respondent saw the same basic question twice (e.g., 1- and 3-day lead time for the same forecast). There were four unique versions of the questionnaire, each with a different subset of questions. Each version included between 14 and 16 questions. The order in which the forecast parameters were mentioned was the same for all versions. In addition, the multiforecast questions and the questions asking respondents to indicate a probability threshold were asked last to avoid any influence on answers to previous questions. On average, it took respondents approximately 23 min (SD ≈ 17 min) to complete the questionnaire.
d. Procedure
The link to the questionnaire explained that it was designed to investigate “how boat owners interpret weather forecasts.” Respondents were informed that they must be boat owners, 18 years of age or older, and that the information collected would remain anonymous. Consenting respondents were asked to provide a description of the size and kind of boat they owned, where they obtained weather information, their age, and their gender. The questions and the forecast to which they referred were presented individually on a computer screen. Boaters responded to each question and then clicked a “continue” button in the bottom right-hand corner of the screen to advance to the next question. Respondents were not able to go back and change their answers to previous questions. For most questions, a single response was required. There were also some questions that were used to gather more detailed explanations that allowed free format text responses.
3. Results
a. Single-forecast question
First, we examined the values that respondents indicated they would “not be surprised” to observe for nighttime low temperature. Only 3 of the 166 respondents (2%) entered the same temperature for both the “as high as” and “as low as” questions, suggesting that they had no uncertainty expectations. The vast majority of respondents anticipated a range of values, suggesting that they regarded the forecast as uncertain. The range was calculated by subtracting the temperature provided in answer to the “as low as” question from the temperature provided in answer to the “as high as” question to arrive at an expectation range for each respondent. The mean expectation range for the 32°F forecast, averaged across lead times, was 10.08°F (SD = 5.66). The mean expectation range for the 29°F forecast, averaged across lead times, was 8.70°F (SD = 5.91).
For the categorical wind speed forecasts we compared respondents’ “as high as” and “as low as” estimates to the values indicated in the forecast definition. Only 1.2% of respondents who answered the small craft advisory questions expected the exact range indicated by the definition (i.e., 21–33 kt). Only 6.8% of respondents who answered the gale warning questions expected the exact range indicated by the definition (i.e., 34–47 kt).
Then, respondents’ expectation ranges were calculated for each of the categorical wind speed forecasts by subtracting respondents answers to the “as high as” question from their answers to the “as low as” question and averaging across respondents, lead times, and geographic locations (Figs. 2a,b). On average respondents expected significantly wider ranges than were indicated by the definition they were given. For the small craft advisory, defined as a 12-kt range, respondents expected a mean range of 18.16 kt (SD = 7.06), t(73) = 6.55, p < 0.001. For the gale warning, a 13-kt range, respondents expected a mean range of 20.49 kt (SD = 9.84), t(165) = 11.25, p < 0.001. The storm warning forecast was defined as sustained winds greater than 47 kt so there was no upper bound given. For this forecast, respondents expected the widest range of values (M = 42.89 kt, SD = 8.14 kt). Thus, on average, respondents expected both different boundary values and wider ranges than were specified in the definition for the categorical wind speed forecasts.
(a) Mean expectation range for gale and storm warnings (no locations specified). All four biases shown here were significant at the Bonferroni corrected value of p < 0.0125. Error bars show one standard error above and below the mean. (b) Mean expectation range for small craft advisories for the San Juan Islands and Admiralty Inlet. None of the biases shown here are significant at the Bonferroni corrected value of p < 0.0125. Error bars show one standard error above and below the mean.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-11-00025.1
(a) Mean expectation range for gale and storm warnings (no locations specified). All four biases shown here were significant at the Bonferroni corrected value of p < 0.0125. Error bars show one standard error above and below the mean. (b) Mean expectation range for small craft advisories for the San Juan Islands and Admiralty Inlet. None of the biases shown here are significant at the Bonferroni corrected value of p < 0.0125. Error bars show one standard error above and below the mean.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-11-00025.1
(a) Mean expectation range for gale and storm warnings (no locations specified). All four biases shown here were significant at the Bonferroni corrected value of p < 0.0125. Error bars show one standard error above and below the mean. (b) Mean expectation range for small craft advisories for the San Juan Islands and Admiralty Inlet. None of the biases shown here are significant at the Bonferroni corrected value of p < 0.0125. Error bars show one standard error above and below the mean.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-11-00025.1
Although respondents anticipated a wider range of values for the 3-day as compared to the 1-day lead time for every forecast except for the November 29°F nighttime low temperature forecast, none of the lead-time comparisons reached statistical significance. This was probably because of the relatively small sample size answering each lead-time question. These sample sizes were theoretically capable of detecting only large effect sizes (Cohen’s d < 0.82), whereas the lead-time effect size observed in the previous study (Joslyn and Savelli 2010) was small (Cohen’s d = 0.14).
1) Biases
Perhaps the most interesting result was the fact that respondents expected systematic biases in several forecast parameters. For the most part, boaters, like general public end users (Joslyn and Savelli 2010), thought the nighttime low temperature forecasts were too low. This was revealed in the fact that for three of the four nighttime low temperature forecasts (Table 2), the difference between the deterministic forecast and the mean temperature that would not surprise respondents on the high end (“as high as” question) was significantly greater than the difference between the deterministic forecast and the mean temperature that would not surprise respondents on the low end (“as low as”). This suggests that respondents thought the nighttime low temperature forecast was more likely to verify higher (i.e., it had a low bias). Indeed, the nighttime low temperature forecasts for the Pacific Northwest of the United States have a slight low bias of about half a degree Fahrenheit (Baars and Mass 2005).
Asymmetry in range of values above (as high as) compared to below (as low as).
In addition, respondents regarded all of the wind warnings as too high. In fact, the more extreme the warning, the greater was the anticipated bias. This was revealed by the fact that the mean unsurprising wind speed on the low end was much lower than the minimum wind speed indicated in the definition for the warning. On the other hand, the mean unsurprising wind speed on the high end was either lower or only slightly higher than the maximum wind speed in the definition (Figs. 2a,b). For storm and gale warnings (Fig. 2a), this asymmetry in respondents’ expectations was statistically significant for every question. This suggests that respondents regarded such warnings as being prone to false alarms. Although there are no records on the verification of small craft advisories in the Pacific Northwest, there are verification records for gale and storm warnings. According to the Weather Forecasting Office in Seattle, between 1998 and 2010 the false alarm rate (26%) was greater than the rate of misses (16%), suggesting that boaters correctly identified an overforecasting bias.
By contrast, precipitation, presented here as a binary forecast, was regarded as fairly accurate and unbiased. Summarized over the 1- and 3-day lead times, 88% of the respondents expected a rain forecast to verify and 78% of respondents expected a no rain forecast to verify, a difference that was not statistically significant, Exp(B) = 2.03, p = 0.10. For precipitation forecasts, respondents expected significantly more uncertainty at the longer lead times (Fig. 3). A logistic regression analysis revealed that they were more than 10 times more likely to think that a next day forecast for rain would verify as compared to a forecast for rain 3 days in the future, Exp(B) = 10.25, p = 0.03. Likewise, for a no rain forecast, respondents were 3 times more likely to think the next day forecast would verify as compared to a 3-day lead-time forecast; however, this difference was only marginally significant, Exp(B) = 3.21, p = 0.07. Indeed, next-day precipitation forecasts tend to be quite accurate and the error at longer lead times tends to be unbiased (Baars and Mass 2005).
Percent of boaters who would alter their plans given a high wind speed forecast by boat category.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-11-00025.1
Percent of boaters who would alter their plans given a high wind speed forecast by boat category.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-11-00025.1
Percent of boaters who would alter their plans given a high wind speed forecast by boat category.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-11-00025.1
2) Decisions
The above analyses suggest that respondents anticipated uncertainty in all of the forecasts tested. Respondent understanding of the uncertainty inherent in deterministic forecasts was also reflected in the decisions they made based upon the forecasts. Because the majority of the decisions were only relevant to owners with boats in category 2 or higher, the 32 participants who indicated that they owned the smaller category-1 boats were not included in these analyses. Recall that all decisions were embedded in scenarios for which subject matter experts had determined that precautionary action was warranted for most boaters. Nonetheless, the percent of respondents choosing to take precautionary action ranged from 23% to 93% across questions, suggesting that respondents regarded the forecasts as containing some uncertainty.
We conducted a consistency analysis, categorizing respondents as consistent either when the value they expected to observe exceeded the nominal threshold for precautionary action and they decided to take precautionary action (e.g., thought wind speeds would reach at least the lower bound of the advisory definition and cancelled trip) or when the value they expected to observe was below the threshold and they decided against precautionary action (e.g., thought wind speeds would be less than the lower bound of the advisory definition and pursued trip). Summarized over all scenarios, respondents tended to be consistent (60%). In addition, the majority decided to take precautionary action or cancel trip plans (75%). Most of those (69%) did so because they thought that the threshold for action would be exceeded. This level of consistency was true of all decisions except preparation for the storm warning, for which most boaters expected much slower wind speeds. The vast majority of respondents who indicated that they would take precautionary action for the storm warning (92%) were doing so even though the wind speed they expected to observe was less than the lower boundary of the warning definition, suggesting that they decided to err on the side of caution. Summarized over all of the scenarios, a smaller proportion of respondents indicated that they would not take precautionary action (25%), and only a third of those were consistent (32%), suggesting that when they decided against cautiousness it was not because of the weather parameter per se but for other reasons.
There was evidence, for instance, that the severity of the consequences may have been a factor in decision making. In general more boaters indicated they would take precautionary action when consequences were more serious. For boat protection decisions averaged over lead times, the smallest proportion said they would drain the water to prevent freeze damage (M = 49%), followed by covering the boat to prevent water damage (M = 63%). The greatest percentage, 86%, said they would protect against the more serious potential damage from wind when gale or storm warnings were forecasted. As noted above, this is interesting because the storm and gale warnings were the forecasts for which boaters indicated that they expected the greatest amount of uncertainty, as well as the greatest amount of bias. In general, respondents would not be surprised if the winds were much less strong than what was forecasted. Nonetheless, the vast majority chose to take precautionary action.
Slightly smaller proportions, approximately 79%, averaged over lead times and locations, would alter their trip plans and stay in port in the face of a small craft advisory. However, this appeared to be related to the vulnerability of the boat they were operating. The vast majority of respondents with “slightly vulnerable” boats indicated that they would cancel their plans (86%). They were more than 3 times, Exp(B) = 3.32, p = 0.020, as likely to indicate that they would alter their plans than were respondents with boats in the “tolerant” category (65%) and almost 5 times as likely, Exp(B) = 4.93, p = 0.047, to indicate they would alter their plans as were respondents with “stable” boats. Just over half of those with stable boats (56%) would cancel their plans and stay in port given a small craft advisory; however, this proportion was not significantly different from the proportion of respondents with tolerant boats (65%), Exp(B) = 1.49, p = 0.584.
In general, boaters made precautionary decisions more often in response to a forecast with a 1-day as compared to a 3-day lead time, suggesting that respondents were sensitive to the increase in uncertainty that accompanies longer lead times; however, these differences did not reach significance (Table 3). Contrary to our expectations, the decision about pumping the water out of a boat had the opposite pattern. Boaters were more than 7 times more likely to pump the water out of the boat in response to a forecast with a 3-day lead time compared to a forecast with a 1-day lead time, Exp(B) = 7.59, p < 0.001. The explanations that respondents gave suggested that some of them had inferred abnormally low temperatures during the intervening days so that low temperatures would be observed for a longer time period, increasing the chance for damage.
Percent of respondents who would “take action” given a particular forecast and the chance (in odds) of taking action for 1- vs 3-day lead times.
b. Multiforecast question
Respondents’ understanding of forecast uncertainty was also revealed in the multiforecast questions. Recall that respondents were asked to indicate the percentage of forecasts expected to verify within specific ranges above and below the deterministic forecast (appendix questions 8–11). For the most part, the uncertainty expectations revealed by these questions, confirmed those revealed in the single-forecast questions.
For nighttime low temperature, significantly more observations were anticipated above the forecast value (M = 34%, SD = 15%) than below it [(M = 29%, SD = 13%), t(165)= 4.00, p < 0.001], again suggesting that respondents expected a low bias for nighttime low temperatures. Similarly, significantly more slower (M = 36%, SD = 17%) as compared to faster [(M = 26%, SD = 14%), t(165) = −6.00, p < 0.001] observed wind speeds were anticipated, suggesting that respondents anticipated a high bias in wind speed forecasts (Fig. 4).
Mean number of days out of 100 that respondents indicated the wind speed forecast would verify. Error bars show one standard error above and below the mean.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-11-00025.1
Mean number of days out of 100 that respondents indicated the wind speed forecast would verify. Error bars show one standard error above and below the mean.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-11-00025.1
Mean number of days out of 100 that respondents indicated the wind speed forecast would verify. Error bars show one standard error above and below the mean.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-11-00025.1
Categorical wind warnings (small craft advisory, gale warning, storm warning) elicited the same anticipated high bias. On average respondents expected observed wind speeds to fall below the defined range on 3 out of 10 days, whereas they expected observed wind speeds to be above the defined range on fewer than 2 out of 10 days (Fig. 5). This anticipated high bias was significant for the small craft advisory, t(71) = 6.74, p < 0.001, the gale warning, t(113) = 5.94, p < 0.001), and the storm warning, t(101) = 7.14, p < 0.001.
Mean number of days out of 10 that respondents indicated the wind speed warnings would verify. Error bars show one standard error above and below the means.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-11-00025.1
Mean number of days out of 10 that respondents indicated the wind speed warnings would verify. Error bars show one standard error above and below the means.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-11-00025.1
Mean number of days out of 10 that respondents indicated the wind speed warnings would verify. Error bars show one standard error above and below the means.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-11-00025.1
Boaters expected precipitation forecasts to verify about 70% of the time for both rain and no rain (Fig. 6). Notice that this is lower than the percentage of respondents who expected precipitation forecasts to verify in the specific questions above (98% for rain forecasts and 89% for no rain forecasts), perhaps because no month was identified in the general questions.
Mean number of days out of 10 that respondents indicated the rain forecast (expressed would verify. Error bars show one standard error above and below the means.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-11-00025.1
Mean number of days out of 10 that respondents indicated the rain forecast (expressed would verify. Error bars show one standard error above and below the means.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-11-00025.1
Mean number of days out of 10 that respondents indicated the rain forecast (expressed would verify. Error bars show one standard error above and below the means.
Citation: Weather, Climate, and Society 4, 1; 10.1175/WCAS-D-11-00025.1
These results demonstrate that respondents anticipated uncertainty in all of the deterministic forecasts tested here regardless of how the question is asked, suggesting that boaters are psychologically prepared to understand uncertainty forecasts. To explore the potential need for uncertainty information among boaters, a final question was asked about the probability thresholds at which they would take precautionary action. Two weather events were tested: extreme rain and high winds. Notice in Table 4 that boaters had a wide range of thresholds at which they would take action. For neither forecast do more than 22% of boaters have the same threshold. It is clear from these data that boaters would benefit from having explicit uncertainty information to accommodate these individualized thresholds.
Percent of respondents taking precautionary action at specific probability thresholds.
The threshold questions had also been asked of general public end users (Joslyn and Savelli 2010). Comparing the two groups, it appears that boaters were more cautious than general users for similar forecasts, as more of them had lower probability thresholds for taking action. Between 0% and 50% probability of extreme wind speeds, only 38% of the general users would take action compared to 71% of boaters. Similarly, between 0% and 50% probability of extreme amounts of rain, only 29% of general users would take action compared to 59% of boaters. There are a number of potential explanations for this difference including experience and age as well as the kinds of choices involved, which may be more serious for boaters than everyday users. Most of the precautionary actions listed by boaters for wind (70%) and many of those for rain (28%) were in some way related to boating. However, general users listed a wide variety of actions.
4. Conclusions
These results suggest that boaters, like general public end users, recognized that many kinds of deterministic forecasts involve uncertainty. Moreover, these high-stakes forecast consumers demonstrated remarkably sophisticated understanding of both forecast uncertainty and forecast bias. Boaters expected wind speed warnings to verify lower, nighttime low temperatures to verify higher, and no bias for precipitation forecasts, all of which correspond with actual verification data for the region. Boaters’ experience with weather and their reliance on weather forecasts to make safety decisions may well have provided them with a highly developed understanding of these issues, thereby improving the accuracy of their predictions.
Boaters’ awareness of the high false alarm rate for gale and storm warnings is particularly interesting as it sheds light on a critical issue concerning severe weather forecasts in general. Some have argued against including uncertainty estimates in forecasts for severe events because acknowledging uncertainty may reduce willingness to take precautionary action. These data suggest the opposite. The boaters queried here were well aware of the tendency for wind warnings to result in false alarms, as reflected in their answers to both the specific and the general questions. Nonetheless, they reported being quite willing to take precautionary action based on such forecasts, at a rate in some cases of almost 90%. In other words, boaters already know that the actual likelihood of observing high winds is much less than what is implied by the forecast, but they choose to take action anyway. Although responses to the hypothetical scenarios tested here may differ from decisions taken in real-world situations (Teper et al. 2011), one would expect the latter to be more rather than less cautious in most cases.
Thus, it appears that boaters were using an economically rational approach to making decisions, taking into account anticipated forecast uncertainty as well the severity of the outcome. When the potential loss is great, the expected loss, the loss weighted by the probability (Bernoulli 1954), outweighs the cost of precautionary action even when the probability is low. This approach was evidenced in the boat protection scenarios, for which boaters decided to take action most often in response to the severe potential consequences of gale and storm warnings, despite the fact that these were the forecasts for which they expected the most uncertainty. A rational approach was also evidenced in the trip planning decisions in which operators of vulnerable boats were significantly more likely to cancel the trip than were those with less vulnerable boats, despite expecting a high false alarm rate. This suggests that boaters’ motivation to make good decisions combined with their considerable direct experience of weather (Shanteau 1992), has allowed them to develop a sophisticated approach to both uncertainty assessment and weather-related decision making.
This level of sophistication among boaters was also evident in the comparison between question types. Unlike general public end users (Joslyn and Savelli 2010), boaters’ answers to multiforecast questions were similar in most respects to their answers to single-forecast questions. In both question types, respondents correctly identified the low bias for nighttime low temperatures, the high bias for wind speeds, and the lack of bias in precipitation forecasts. Thus, boaters did not show the dissociation between questions types observed among general users in which biases were mainly observed in the single-forecast questions (Joslyn and Savelli 2010). This suggests that boaters, unlike general users, were consciously aware of their bias expectations and reported them even when engaging in the deliberate calculation that was required for the multiforecast questions.
However, there were also some differences in responses that appeared to be due to question type. In the single-forecast questions the anticipated bias for wind warnings appeared to be exaggerated. For both storm and gale warnings, even the highest wind speed anticipated by respondents was below the upper limit of the defined range. For small craft advisories expectations were only a few knots above the range, but many knots below it. On the other hand, the expected wind speeds reflected in the multiforecast questions for wind warnings were much higher overall, overlapping completely with the defined range, although they extended much farther below it than above. For precipitation, more uncertainty was anticipated in response to the multiforecast as compared to the single-forecast questions. Responses to multiforecast questions indicated that boaters expected only 70% of rain forecasts to verify whereas almost 90% of respondents indicated that the single-forecast for rain would verify. Thus, for both of these parameters the extreme expectations, extreme bias in one case and high accuracy in the other, elicited in the single-forecast questions seem to have been attenuated in the multiforecast questions. This could be due either to a rudimentary understanding of the principles of regression to the mean, or to the lack of specific details (e.g., season) in the multiforecast questions.
Finally these results suggest that well-calibrated uncertainty forecasts would be useful to boaters. Boaters expressed a wide range of threshold probabilities at which they would take precautionary action. Specific uncertainty estimates would allow them to easily tailor the forecast to their own needs. For boaters, like general public end users, part of the variation in probability thresholds is likely due to differences in personal tolerance for risk. However, part of it is due to the boats that they operate, which differ a great deal in their vulnerability to weather, making uncertainty forecasts particularly useful for this special interest group. Interestingly, boaters indicated willingness to take precautionary action at fairly low probability levels, perhaps because of the safety-related consequences to the decision that they made.
In addition, forecasts with specific uncertainty estimates might help narrow boaters’ expectations. Although they anticipated uncertainty with deterministic forecasts, boaters’ expectation ranges were quite wide. This was especially true of wind warnings. Despite the fact that the forecasts indicated a range of values, boaters’ expectations consistently exceeded the range of values given, and sometimes barely overlapped with them. This suggests that providing range forecasts alone, without specifying the uncertainty, does not satisfy users’ needs for uncertainty information. Perhaps uncertainty estimates, in the form of 80% predictive intervals could narrow and focus expectations, providing more precise and useful information for decision making. This might allow users to adapt the forecast to their specific boating situation with greater confidence.
Acknowledgments
This material is based on work supported by National Science Foundation Grants 0724721 and 1023354.
APPENDIX
Survey Questions
On a day [a Wednesday] in December, you notice that the predicted nighttime low temperature for the next night is 32°F.
1a. What do you think the nighttime low temperature will be on the next [Saturday] night? ______ °F.
1b. I would not be surprised if the nighttime low temperature on the next [Saturday] night was as high as ______ °F.
1c. I would not be surprised if the nighttime low temperature on the next [Saturday] night was as low as ______ °F.
Assume that you usually pump out all of the water on your boat (i.e., the head, water lines, etc.) when temperatures are predicted to fall below freezing. Imagine that it is [a Wednesday in] November and you notice that the predicted nighttime low temperature for the next [Saturday] night is 29°F.
Would you pump the water out of your boat before the next [Saturday] night? (Check one) Yes___ No___
Why?
Questions 1a, 1b, and 1c are repeated for this question.
Imagine that it is [a Wednesday in] October and you notice that a gale warning (34–47 kt) [storm warning (sustained winds greater than 47 kt)] is posted for your area for the next day [Saturday].
3a. Would you check to make sure your boat was secure? (Check one) Yes___ No___
3b. What do you think the maximum sustained wind speed will be the next day [Saturday]? ____ knots.
3c. I would not be surprised if the maximum sustained wind speed was as high as ___ knots.
3d. I would not be surprised if the maximum sustained wind speed was as low as ___ knots.
Imagine that it is [a Wednesday in] October and you plan to leave on your boat the next day [on Saturday] to go from Shilshole Bay Marina to San Juan Island [though Puget Sound to Port Townsend] for a long weekend. You will meet former classmates that you and your crew have not seen in many years. This trip has been planned for a year. You check the weather forecast and notice that a small craft advisory (sustained winds of 21–33 kt) is posted for the northern inland waters, including the San Juan Islands, for the next day [Saturday].
Would you alter your plans? (Check one) Yes___ No___
Questions 3b, 3c, and 3d are repeated for this question.
Imagine that it is [a Wednesday in] June and you have left your boat at a marina 20 min away without rain protection. You notice that heavy rain is predicted for the next day [Saturday] in that area.
Would you return to your boat to put on protective covering? (Check one) Yes____ No____
On a day [Wednesday] in October, you notice that rain [NO rain] is predicted for the next day [Saturday that week].
Do you think it will rain the next day [Saturday that week]? (Check one) Yes____ No____
Why or why not?
If you knew that 48-kt sustained winds [extreme amounts of rain] were predicted (possible power outages, downed trees, bridge closures, etc.), what would you do to prepare if you had this information in advance?
For which of the following forecasts would you begin preparation (select one): 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, or 100% chance of 48-kt sustained winds?
Assume that every day you look at the nighttime low temperature forecast for the next night [for the Puget Sound area]/[for the San Juan Island area]/[Admiralty Inlet]. Out of 100 consecutive nighttime low temperature forecasts (make sure your answers to the following sum to 100)…
How many times do you think the actual nighttime low temperature will be what was predicted___, 1–2 degrees higher___, 1–2 degrees lower___, 3–4 degrees higher___, 3–4 degrees lower___, 5 or more degrees higher___, 5 or more degrees lower___?
Out of 10 consecutive small craft advisories (sustained winds of 21–33 kt)/[gale warnings (sustained winds of 34–47 kt]/[storm warnings (wind speeds greater than 47 kt)] made for Puget Sound and Hood Canal [Admiralty Inlet]/[the northern inland waters including the San Juan Islands] (your answers should sum to 10)…
How many of them do you think will be correct (observed sustained wind speed of 21–33 kt) ___, slower than predicted ___, faster than predicted?
Assume that every day in the spring, you look at the precipitation forecast for the next day. Out of 10 forecasts for NO rain (make sure your answers to the following sum to 10) …
How many of them do you think will be correct (NO rain when NO rain is predicted) ___
How many of them do you think will be wrong (rain when NO rain has been predicted)? ___
Assume that every day in the spring, you look at the precipitation forecast for the next day. Out of 10 forecasts for RAIN (make sure your answers to the following sum to 10) …
How many of them do you think will be correct (rain when rain is predicted) ___
How many of them do you think will be wrong (NO rain when rain has been predicted)? ___
REFERENCES
Baars, J. A., and Mass C. F. , 2005: Performance of National Weather Service forecasts compared to operational, consensus, and weighted model output statistics. Wea. Forecasting, 20, 1034–1047.
Baker, E. J., 1995: Public response to hurricane probability forecasts. Prof. Geogr., 47, 137–147.
Bernoulli, D., 1954: Exposition of a new theory on the measurement of risk. Econometrica, 22, 23–36.
Burnside, R., 2006: Leaving the Big Easy: An examination of the hurricane evacuation behavior of New Orleans residents before Hurricane Katrina. J. Pub. Manage. Soc. Policy, 49–61.
Dow, K., and Cutter S. L. , 1998: Crying wolf: Repeat responses to hurricane evacuation orders. Coastal Manage., 26, 237–252.
Dow, K., and Cutter S. L. , 2000: Public orders and personal opinions: Household strategies for hurricane risk assessment. Global Environ. Change, 2B, 143–155.
Joslyn, S., and Savelli S. , 2010: Communicating forecast uncertainty: Public perception of weather forecast uncertainty. Meteor. Appl., 17, 180–195.
Lazo, J. K., Morss R. E. , and Demuth J. , 2009: 300 billion served: Sources, perceptions, uses, and values of weather forecasts. Bull. Amer. Meteor. Soc., 90, 785–798.
Lazo, J. K., Waldman D. M. , Morrow B. H. , and Thatcher J. A. , 2010: Household evacuation decision making and the benefits of improved hurricane forecasting: Developing a framework for assessment. Wea. Forecasting, 25, 207–219.
Morss, R. E., and Hayden M. H. , 2010: Storm surge and certain death: Interviews with Texas coastal residents following Hurricane Ike. Wea. Climate Soc., 2, 174–189.
Morss, R. E., Demuth J. , and Lazo J. K. , 2008: Communicating uncertainty in weather forecasts: A survey of the U.S. public. Wea. Forecasting, 23, 974–991.
NOAA, cited 2007: National Weather Service marine forecasts. [Available online at http://www.weather.gov/om/marine/cwd.htm.]
Responsive Management, cited 2007: Washington boater needs assessment: Data compendium. [Available online at http://www.rco.wa.gov/documents/rcfb/boating/Data_Compendium.pdf.]
Riad, J. K., Norris F. H. , and Rubak R. B. , 1999: Predicting evacuation in two major disasters: Risk perception, social influence, and access to resources. J. Appl. Soc. Psych., 25, 918–934.
Seattle Parks and Recreation, cited 2011: Motorized boat launches. [Available online at http://www.seattle.gov/parks/boats/motorized.htm.]
Shanteau, J. M., 1992: Competence in experts: The role of task characteristics. Organ. Behav. Hum. Decis. Processes, 55, 252–266.
Teper, R., Inzlicht M. , and Page-Gould E. , 2011: Are we more moral than we think? Exploring the role of affect in moral behavior and moral forecasting. Psychol. Sci., 22, 553–558.
U.S. Coast Guard, cited 2010: Recreational Boating Statistics 2009. [Available online at http://www.uscgboating.org/assets/1/workflow_staging/Publications/394.PDF].
Zhang, F., and Coauthors, 2007: An in-person survey investigating public perceptions of the responses to Hurricane Rita forecasts along the Texas coast. Wea. Forecasting, 22, 1177–1190.
These figures include all states, the District of Columbia, Puerto Rico, Guam, the Virgin Islands, American Samoa, and the Commonwealth of the Northern Mariana Islands.
Some questions were based upon those used in Joslyn and Savelli (2010) and others were developed using input from subject matter experts.
