When extreme river levels are possible in a community, effective communication of weather and hydrologic forecasts is critical to protecting life and property. Residents, emergency personnel, and water resource managers need to make timely decisions about how and when to prepare. Uncertainty in forecasting is a critical component of this decision-making, but often poses a confounding factor for public and professional understanding of forecast products. A new suite of products from the National Weather Service’s Hydrologic Ensemble Forecast System (HEFS) provides short- and long-range forecasts, ranging from 6 h to 1 yr, and shows uncertainty in hydrologic forecasts. To understand how various audiences use and interpret ensemble forecasts showing a range of hydrologic forecast possibilities, a research project was conducted using scenario-based focus groups and surveys with community residents, emergency managers, and water resource managers in West Virginia and Maryland. The research assessed the utility of the HEFS products, identified barriers to proper understanding of the products, and suggested modifications to product design that could improve the understandability and accessibility for a range of users. There was a difference between the residential users’ reactions to the HEFS compared to the emergency managers and water resource managers, with the public reacting less favorably to all versions. The emergency managers preferred the revised HEFS products but had suggestions for additional changes, which were incorporated. Features such as interactive text boxes and forecaster’s notes further enhanced the utility and understandability of the products.
As forecast technology advances, the challenges and opportunities associated with providing information about forecast uncertainty become both more germane and more complicated. While meteorologists have substantial information about forecast uncertainty—both in general and in specific situations—much of it is not easily accessible by the public (Morss et al. 2008). In theory, uncertainty information is very useful to both weather forecasters and to the general public as individuals decide how to prepare for weather situations. However, with the exception of the probability of precipitation, forecast uncertainty is not usually communicated formally to the public (Joslyn et al. 2007). In fact, most forecasters remain deterministic in their forecasting, in part because it is unclear whether people can successfully make use of uncertainty information (Joslyn et al. 2007).
Even though individuals in the meteorological community have sometimes discouraged probability forecasts for significant weather events because the public often misinterprets them, there can be many potential benefits to using such forecasts (Murphy et al. 1980). Providing uncertainty information to the public in an accessible format may help people decide how much confidence to place in a given forecast (Morss et al. 2008), and communicating information about data uncertainty has the potential to increase trust in results and to support decision-making that uses that data (Kinkeldey et al. 2014). Communicating forecast uncertainty is important because it avoids falsely portraying certainty in forecasts and may help forecast users make more informed decisions (Morss et al. 2008, 2010). On the forecast side of the equation, uncertainty forecasts can be used to improve deterministic forecasts (Joslyn et al. 2007).
Indeed, practitioners who make decisions based on weather forecasts (such as emergency managers and others) understand that uncertainty is an unavoidable factor that is part of decisions they must make every day, in a complex and ever-evolving environment (Morss et al. 2005). The consequence of conveying only single-value, deterministic information to practitioners is that poorer decisions may be made because they do not have the benefit of knowing and accounting for the forecast uncertainties and risks on which their decisions are based (Hirschberg et al. 2011). To that end, some practitioners seek to link probabilistic information within more traditional deterministic decision-making (Pappenberger et al. 2013). Currently, however, decisions by users at all levels, and most critically those individuals associated with the protection of life and property, are most often being made without the benefit of knowing uncertainties of the forecasts upon which they rely (National Research Council 2006).
Important questions remain regarding how uncertainty should be represented to decision-makers (Finger and Bisantz 2002). Research has found a disconnect between the ability to create probabilistic, ensemble forecasts and the ability to communicate and use them (Demeritt et al. 2016). In some studies, practitioners have been found to have biases and misunderstandings of probability (Demeritt et al. 2007, 2010; Morss 2010). Research has suggested, therefore, that the decision-making process is best served when uncertainty is communicated as precisely as possible, but no more precisely than warranted (Wallsten et al. 1993).
Several studies have addressed the role of, and preference for, uncertainty information in forecasts. In one, respondents were given forecasts of the percentage chance of exceeding a specific threshold. More respondents took protective action as that percentage increased, which suggests that the respondents understood the tested probabilistic forecasts well enough to make decisions (Morss et al. 2010). In another study, the clear majority of respondents was willing to receive forecast uncertainty information, and 45% of respondents preferred a weather forecast that expressed uncertainty (Morss et al. 2008). While the respondents were not able to provide a correct definition of what the uncertainty meant, the information helped them make better decisions, as was similarly found by Gigerenzer et al. (2005) in their study about public understanding of the probability of precipitation. Further, Joslyn and Nichols (2009) found that uncertainty presented as probability resulted in better decisions than did the use of relative frequency. These and other studies provide some evidence that probabilities may be applicable for conveying uncertainty in forecasts to the public (Morss et al. 2010).
In contrast, forecasters have expressed concerns with conveying uncertainty information, as indicated by an interviewee in a study in Europe who said that “this is too much information that the public can’t use to make decisions” (Demeritt et al. 2010). Additionally, a shift from deterministic to probabilistic forecasts also entails shifting onto forecast recipients more of the responsibility for dealing with uncertainty (Demeritt et al. 2010; Stephens et al. 2012).
Like meteorological models, hydrological models are uncertain, and hydrologists are very conscious about the uncertainty associated with these models. Hydrologists want to see and digest as much uncertainty information as possible to help inform the production of their own forecasts (Demeritt et al. 2010), but the presentation of uncertainty to practitioners should be “as precise as warranted by the available information” (Finger and Bisantz 2002). Sophisticated estimates of scientific uncertainty may only complicate practitioners’ already difficult jobs, without benefiting the people they serve (Morss et al. 2005), with some concerned about the potential for false alarms as well as institutional constraints that limit the usefulness of such estimates (Demeritt and Nobert 2011; Rayner et al. 2005).
In addition to questions of whether and what uncertainty information to provide, meteorologists often find the “how” of communicating uncertainty in a way that users will understand to be difficult, particularly when serving users with varying interests and levels of expertise. As Kox et al. (2015) found, one means of presenting and communicating ensemble predictions does not meet all needs, even those within individual user groups. One significant problem in the cognition of uncertainty involves the need to address how uncertainty information can be communicated so it can be understood and used most effectively (Ruginski et al. 2016). Communicating forecasts effectively requires understanding how intended audiences interpret and use forecast information presented in different ways (Morss et al. 2010). Sophisticated forecast products are of little value if they are misunderstood, used inappropriately, or simply ignored by their recipients (Demeritt et al. 2010), and other research has shown that considerations about the design and presentation of uncertainty information can strongly influence how various audiences understand and respond to uncertainty information (Hogan Carr et al. 2016a,b).
Ensemble flood forecasts in particular are new and sophisticated. There are not yet any universally agreed upon methods for communicating these uncertainty forecasts (Demeritt et al. 2010). Additional research is needed to understand how users will likely interpret and act upon probabilistic warning forecasts; such research will contribute to the development of warning and decision-making products that promote appropriate response behaviors from the public (Morss et al. 2008). The research reported here is one effort to that end. In particular, this study is centered on answering several research questions related to NWS flood forecast improvements and specifically the Hydrologic Ensemble Forecast System (HEFS) products, a new suite of products that provides probabilistic short- and long-range hydrologic forecasts, ranging from 6 h to 1 yr. These include the following:
What improvements to NWS flood forecast products would better motivate people to take flood preparedness and response actions?
How do residents, emergency managers, and water resource managers identify the utility of HEFS products? How will they use these products?
What barriers do each of these audiences identify in understanding and accessing the HEFS products?
What modifications to the product design will help improve the utility, understandability, and accessibility of the products?
This research seeks to understand the diverse audiences that may have use for the HEFS products, specifically looking at two sets of professionals and the public, and to understand these users’ responses to different elements of the products and ways of communicating the HEFS information in a storm preparedness scenario.
The HEFS products were tested through three methods and with three different audiences. In-person focus groups, online surveys, and webinars were conducted with residents, emergency managers, and water resource managers in West Virginia and Maryland. These three groups were selected because of the potential salience of the products to them: residents and emergency managers with respect to flooding, and water managers who must plan for both low- and high-flow situations. Two rounds of in-person focus groups were conducted in Jefferson County, West Virginia, and Frederick County, Maryland. These communities were chosen because they are located within the partner Weather Forecast Office area (Sterling, Virginia), have a history of flooding and active emergency management offices, and though geographically somewhat close, represent a mix of rural and urban/suburban settings with differing hydrologic conditions. Resident participants were recruited with assistance from local emergency managers and municipal leaders, as well as through social media and news outlets. Flood experiences differed among the participants, as is shown later, but all were sufficiently concerned about flooding in their communities to devote several evening hours to address the issue. Public participants were provided with nominal monetary compensation for their time. Professional participants were recruited directly through official channels as well as social media.
The first round of focus groups was held in October 2016, with a resident session and an emergency manager session in each county for a total of four sessions. The second round of focus groups was held in April 2017 with a resident session in each county and one emergency manager session in Frederick County for a total of three sessions. A third round of focus groups was held via webinar with two different groups of water resource managers in June 2017. Following these focus groups, an online survey was administered to all previous participants.
During the first round of focus groups, the participants (14 residents and 13 emergency managers total for both counties) were led through a four-day tropical storm scenario that showed flood forecast products, including HEFS, each day leading up to the storm. Each scenario was tailored to the specific community but all contained the following products presented at each day in the storm scenario:
NHC hurricane forecast cone,
hazardous weather outlook,
Advanced Hydrologic Prediction Service (AHPS) hydrograph,
emergency briefing document,
Weather Forecast Office rainfall forecast,
Weather Forecast Office flash flood watch,
Weather Forecast Office river flood watch, and
Weather Forecast Office river flood warning.
Only the 15-day probabilistic guidance HEFS product was shown to the residential group, while the 15-day HEFS, 90-day exceedance plot for discharge and 90-day exceedance plot for cumulative volume were shown to emergency managers and water resource managers (Fig. 1). The scenario described the progression of a hypothetical hurricane moving northward from Cuba with sustained maximum winds that started at 85 mi h−1 dropping to 40 mi h−1 by the time the storm hit the study area with 6–10 in. of rain and major flooding likely. Questions posed to the groups included how they understood the products, what actions they were motivated to take, and how they accessed information on forecasts and preparation. Thus, participants were encouraged to discuss both the scenario and their previous experiences with flooding, as the latter informs their discussion of the former. Focus group participants completed pre- and postsession surveys and this information, along with focus group notes and transcripts, were used for data and NVivo analysis, a qualitative data analysis software package designed for working with rich-text-based data (QSR International).
The HEFS products tested in this study (Fig. 1, top) included 15-day probabilistic guidance of possible stage levels (ft) and discharge (cfs) with observed, median, most likely (25%–75%), likely (10%–90%), and less likely (5%–95%) lines as well as minor, moderate, and major flood levels demarcated. The 90-day exceedance plot for discharge (Fig. 1, middle) showed the exceedance probability of a range of discharge levels (cfs) for a historical simulation (average conditions) and a conditional simulation (what might occur). These two lines, when interpreted together, can suggest whether the subsequent months will be wetter or drier than usual. The 90-day cumulative volume exceedance plot (Fig. 1, bottom) is similar to the discharge exceedance plot but with cumulative volume as the y variable instead of discharge.
In response to what was learned in the first round of focus groups, graphical and design revisions were made to the 15-day HEFS products (Fig. 2, top). Revisions included changes to the color scheme, design, legends, and title, as well as removing the percentage labels and simplifying the categorizations of likelihood. The 90-day products were not modified because there was no interest in using this product among the groups tested, and as such, there was no meaningful input into the product design. Instead, the 90-day products were tested later with water resource managers who have more utility for those products in this time frame.
The second round of focus groups was conducted with new participants (22 residents and four emergency managers1 total for both counties) led through the same 4-day storm scenario as the first round with the exception that the 15-day HEFS graphics were the revised versions. Participants were presented with the same questions as in the first round and the same pre- and postsession survey data were collected and analyzed. Transcripts were analyzed with NVivo.
Following the analysis of the second round of focus groups, the 15-day HEFS graphics were revised a second time (Fig. 2, bottom), and an online survey was developed with a link sent to all previous focus group participants. Revisions included reintroducing percentages into the graphic, this time as part of a second legend at the bottom. Additional components including text boxes and a forecaster’s note were also tested with the online survey.
To assess the utility of the 90-day HEFS products, two online focus groups were held with water resource managers including Sterling, Virginia, Weather Forecast Office partners and the Middle Atlantic River Forecast Center’s (MARFC) customer advisory board. The participants were shown some examples of HEFS, including modified versions of the 15-day HEFS and unmodified versions of the 90-day discharge and cumulative graphs. Participants were asked how the products would be useful in their decision-making, what they like and do not like about the graphics, how they use uncertainty information, and the utility of the time frame.
a. Presession survey results: Demographics and experiences of focus group participants
Surveys completed by participants before the focus group session elicited demographic and flood experience information. The demographics of all groups for both rounds of focus groups are shown in Table 1. Males dominated the first round of focus groups, but this changed in the second round. There was a larger percentage of college graduates among residents in round 1 than in round 2, though for the emergency managers, the percentages are similar. Round 2 participants were generally older than those in round 1.
In an attempt to understand participants’ situations with respect to flooding, the presession survey asked if they, a family member, or a close friend had experienced one or more significant flood events. More than half (57%) of round 1 residents responded affirmatively compared to 50% in round 2, while most (85%) emergency managers who participated in round 1 had experienced a flood, 38% of which had been in the last two years. This contrasts with the emergency managers in the second round, 75% of whom reported they had not experienced a flood.
Despite this experience, in response to a presession survey question asking them to rate their chance of being flooded, all round 1 residents and two-thirds of round 2 residents chose the options of very little risk or no risk. In round 1, just over half of emergency managers chose very little risk or no risk, as well, while almost all round 2 emergency managers chose very little risk. Because the survey did not distinguish the direct flood experience of the respondents, it is difficult to determine exactly why so many reported being at very little or no risk. It could be because it was not they but rather a family member or friend who experienced flooding, it could be due to cognitive dissonance, it could be because of a shared sense of community experience, or some mix of all of these.
In round 1, a third of residents (36%) said they need 3 days advance notice to prepare for a flood and 43% reported needing notice 1–2 days prior, while no round 2 residents chose 3 days’ notice, with most (59%) preferring 1 day or less. Most emergency managers stated they are constantly monitoring the situation and wanted information as soon as possible. Not surprisingly, the emergency managers are more likely to have responded to warnings than the residents, irrespective of rounds.
b. Focus group analysis: Round 1
As detailed above, focus groups were held with emergency managers (broadly defined but largely representing county and municipal emergency management offices and fire departments) and with members of the public, using the HEFS products provided by MARFC in a scenario leading participants through an impending event. Although several NWS products were included in the scenario to provide context, the emphasis of this analysis is on the HEFS, which is the focus of the discussion below.
1) Emergency managers
The participants in these focus groups were presented in the scenario with both the 15-day and, following the scenario, the 90-day HEFS. With respect to the 15-day HEFS, the overall reaction was mostly positive, with comments such as “this would be very helpful” and “that’s what we want to see.” Interest was expressed in looking at it for situational monitoring with one participant categorizing the information as “more useful and actionable,” and another stating that, given the situation shown, it is “not enough to excite me at this point.” Participants recognized both the complexity and the uncertainty associated with the products, with one emergency manager (EM) noting “I find this extremely helpful as an emergency manager because I can see the worst-case scenario.” On the other hand, because of those same characteristics, several mentioned the need for training if people are going to be using it so that they understand it. Indeed, one EM admitted, “I may or may not interpret these kinds of graphs correctly, so I’m going to lean on the professionals [National Weather Service forecasters] for that.” Some questioned how much it could or would be used, and all agreed they would not send it out to the general public in their briefings or notices. Despite these concerns, participants saw it as one of several products they would use, including one EM who said he could use it to “tell stories to the public.” They had few suggestions on revisions, though most said that the discharge data are irrelevant to their needs; it is the stage of the river that is critical to their responsibilities.
Participants found the 90-day HEFS to be of little value to them, though some recognized its potential utility for water managers. Many of the statements echoed the sentiment shared by one EM, who said “For emergency management purposes, I don’t see the value.”
2) The public
In contrast to the emergency managers, reaction by the public to the 15-day HEFS, as presented in the scenario, was largely negative. Comments centered on how complicated and difficult it is to understand. One representative comment is, “Yeah the more I’m looking at this, the more I think I’m confusing myself.” In addition, many remarked that they did not understand it or did not care about the information provided, with some wondering about the intended audience.
Features of the product, besides the overall topic, that created confusion included the use of color, which made it difficult to figure out what to “take seriously” and the use of discharge along with flood stage. Some had difficulty understanding the title and what the lines meant, particularly when trying to sort through the percentages, suggesting that both interpretation of the information on the graphic as well as the way the probabilistic information is visualized were problematic.
This input, along with that from the emergency management focus groups, led to the revisions that were presented in a second round in which the same scenario was tested with different participants. Suggested changes from this first round included providing better visual clarity on confidence, such as differentiating colors. River location and title simplification were suggested as well as a vertical legend instead of a horizontal one. For residents, reducing the extra information they did not need, including the percentages and discharge information, was crucial.
c. Focus group analysis: Round 2
1) Emergency managers
Once again, the response among the participants on the now revised 15-days HEFS was mostly positive, with emphasis on internal and partner use of the product rather than sending it out to the public. Emergency managers showed a distinct preference for probabilistic ranges because they are more useful to them than deterministic forecasts. At the same time, they preferred “having more quantitative information and not just less likely and least likely.” While there was an understanding that NWS is trying to make the product suitable for everyone, some questioned why this is the case, given the difficulty they see in getting people to understand uncertainty. For instance, there was discussion as to whether the use of both river levels and discharge was essential in a product emergency managers would use.
2) The public
Reaction to the revised 15-days HEFS was similar to the original, with most saying that it remains too complicated and therefore useful for only “some” people. While one participant stated that, “It gives you a good visual feel of what could happen,” others said, “That is actually kind of confusing” and “Yeah it just makes no sense.” Further, some questioned the utility of a 15-day product, with comments such as “15 days out is just too much.” Although participants in the first round seemed to have problems with percentages, in this round, participants asked for definitions of less likely, least likely, and likely, suggesting “something just a little more quantitative.” Overall discussion about the product indicated continued misunderstanding of what was conveyed and difficulty interpreting the products. In addition, in round 2, residents appeared to become increasingly distrustful of the 15-day HEFS as the session progressed. This was especially evident in the second Frederick focus group where initial reaction to the HEFS was more positive than in either of the round 1 focus groups or in the other round 2 session. However, as the scenario progressed, participants began to identify and remark upon the large differences between the deterministic hydrograph forecast and the “most likely line” in the HEFS, which ultimately led them to decide that both were untrustworthy.
d. Postsession survey results: Rounds 1 and 2
Following the focus group discussions, participants were asked to rate the specific forecast products they saw during the focus group scenario with respect to the usefulness to them in assessing their flood situation. The average ranking from residents in round 1 and round 2 is shown in Table 2. The 15-day HEFS consistently ranked last and the national hurricane cone product consistently ranked first (perhaps because of familiarity, which was clear from focus group discussions). Most products maintained their same relative order ranking from round 1 to round 2, except for the AHPS hydrograph and the hazardous weather outlook, which both decreased in rank.
Emergency managers rated the products based on the likelihood they would use the product. Almost all products increased in their ratings of usefulness from round 1 to round 2 (Fig. 3). One exception was the 90-day HEFS, which decreased in usefulness. Notably, after revisions to the 15-day HEFS product, usefulness ratings increased significantly for the emergency managers, while residents continued to rank the product as the least useful.
It is apparent that the 15-day HEFS has little perceived utility for most of the residential audiences tested. Emergency managers see value in the product, especially after being presented with the revised version. Emergency managers had little to no use for the 90-day HEFS, noting the time period was not applicable to the time scales needed for emergency response. The 90-day graph was seen to be more suited for water resource managers.
e. Water managers
As mentioned above, a webinar was held with water managers, using both the round 1 and round 2 versions of the 15-day HEFS and the original 90-day HEFS. Most participants had either not seen or used the 15-day HEFS and all thought it would be useful, whether for drought monitoring or for those dealing with floods. The time frame is helpful for their decisions, and most use discharge rather than river stage. The participants were positive about the revised product, though they need the probabilities in the legend because of their importance to the decisions they must make. One item that was mentioned as missing for their purposes was the minimum value of the ensemble, particularly for those dealing with droughts.
In their view, the 90-day exceedance plot would be a good piece of information, especially if there was a probability of low-flow conditions, which would support decisions to conserve the water in reservoirs, among others. It provides a quick idea of conditions that are likely to be wetter or drier than normal, which can be used to corroborate other data, guide other model output interpretations, serve as a quality control check, and complement ground-truthing models. One participant publishes a water supply outlook that includes a model about past water flow, and thought the 90-day exceedance plot would be useful to include as well. Suggested improvements included adding more text if intended for the public and the ability to hover over a point to get flow information.
None of the participants had previously seen the cumulative volume for 90-day exceedance plot, and they thought it would be useful for monitoring both drought and flooding. None use acre feet, preferring millions of gallons, but they acknowledged the conversion was simple enough. Again, interactivity with the ability to hover over points to see values was seen as particularly useful to them and their stakeholders.
f. Summary of focus group discussion and session survey results
In round 2, emergency managers and residents suggested adding back in the detailed probability levels, adding floods of record as a line on the graph, and using deeper color variations on the charts. It is important to note that in the first round of focus groups, the percentages were confusing to residents so the decision was made to simplify the legend with “likely” classifications and remove the percentages. However, in the second round of focus groups this percentage information was missed, especially by emergency managers and water resource managers, so the decision was made to keep the simplified legend for public audiences and add a second legend at the bottom with the percentage information for more technical users (Fig. 4). These were then used in the online survey.
g. Online survey
The online survey administered to all focus group participants as a third round of testing showed two graphics, including a high-flow and a low-flow graphic, and asked about new, additional features including a forecaster’s note and text box, added as a result of analysis of focus group and survey feedback. The revised high- and low-flow graphics depicted an instance where major flooding was predicted and an instance where minor to no flooding was forecast. These HEFS graphics were taken from earlier versions of those used in the focus group sessions and revised based on feedback from those sessions and surveys. The forecaster’s note is a space above the graph area (see Fig. 4) that allows the issuing forecaster to provide text information such as potential critical impacts, key takeaways, or action steps the public should take. The text box is shown when a user hovers over a point on the forecast line and provides details of the anticipated levels, a suggestion derived from the water managers in particular. A total of 21 participants completed the survey but not everyone answered all questions. The small sample size limits the robustness of the findings, but the results from the online survey provide important insights meriting further investigation. The group consisted of both members of the public (11), emergency and water resource managers (6), and other professionals (4). Jefferson and Frederick Counties were about equally represented, with nine and seven participants, respectively, with an additional five from other locations. About two-thirds (14) of the respondents were definitely interested in products that provide guidance on river levels, while only one was not interested.
When given a list of six options and asked what the graphics are showing, a majority correctly interpreted the information being conveyed in both graphics and, in a subsequent question, correctly gauged the level of risk. Thirteen (65%) viewed the shown flood risk as high or somewhat high after interpreting the high-flow graphic, and 12 (71%) viewed the shown flood risk as very low based on the low-flow graphic. These findings are encouraging and supportive of the fact that the graphics are understandable. However, there were two respondents who saw the graphics as “confusing and not clear at all” or who could not read them. Interestingly, even though the majority assessed a high risk of flooding upon viewing the high-flow graphic, only about half (11) said they would take any actions as a result. Those actions included about a third (nine) keeping an eye on the river, eight (27%) seeking out more information, and only two (6%) having an emergency preparedness kit. Given that most respondents reported themselves to be at a relatively low level of flood risk as shown in the presession surveys, lack of action may not be due to the information conveyed in the product, but rather a lack of concern about flooding and its impact on daily operations.
About half (9) of the respondents felt the high-flow product was very useful (Fig. 5), while the low-flow product was considered very to somewhat useful by 12 (75%) respondents. Half of respondents (nine) were extremely likely to use both the high- and low-flow products in the future.
Breaking the responses into two user groups, professionals (including emergency managers and water resource managers) and the public, the results show that the professional users see the products as very useful and they are extremely likely to use them compared to members of the public who were more mixed. Specifically, two-thirds of the public respondents (six) reported finding the products very to somewhat useful with an even distribution of responses with respect to likelihood to use: three were extremely likely to use, three were somewhat likely to use, and three were neither likely nor unlikely (Fig. 5).
When asked about specific product components, most respondents identified the discharge on the right axis as not useful, followed by the percentages and time period. The “most likely” line and the flood levels (minor, major) were useful, as were the river level and range of probable levels (Fig. 6). These findings echo results from the focus groups that discharge was not useful to most potential users. Revisions to the HEFS graphic to include a most likely and range of probable levels appear to be effective and useful to viewers.
Some respondents provided additional explanation of why some of the graphic components were less useful. One respondent felt there were too many elements in the product. A few took issue with the time period, noting that they pay attention to a 3–5-day span and that 15 days is too far out in the future. Another suggested focusing on flooding, getting rid of the top-right legend and calling the product “probability of river flooding.” A few respondents were unsure how to interpret and match the percentages to the top and bottom of the color range and one suggested having simple lines for the various chances of exceedance, with text in the legend.
There was very little difference between the two groups of respondents with respect to the graphics’ components. The most likely line, colors, legends, range of probable levels, and river level were favored by all groups while percentages and the time period were elements not seen as useful among most groups. Only a very small number of professionals felt discharge was useful (Fig. 6). It is important to note that respondents may find specific components useful but be neutral about the overall HEFS graphics.2
h. The utility of additional components: Forecaster’s note and text box
The online survey also asked about additional components to the graphic, specifically a forecaster’s note and text box. The forecaster’s note was rated as very useful with 14 (88%) of the respondents reporting it as very or extremely useful, compared to 9 (56%) for the text box. This usefulness rating was reflected in each element’s influence on decision-making (Fig. 7). Forecaster’s notes were expected to influence decisions to a large extent while the potential influence of text boxes was seen to be of moderate impact. Suggestions for information that would be useful in the notes were a link to historical flooding and the type of flooding that would occur at different levels (100 or 500 year).
Assessing differences between professional and public user groups shows that professionals found the forecaster’s note useful (with six respondents, about 85%, indicating it was extremely or very useful and none indicating is was not at all useful) and an influence on decision-making (with six, about 85%, indicating to a very large or large extent and none indicating not at all). Text boxes were viewed slightly differently, especially by professionals. Although three (43%) reported the text box to be very to extremely useful and four (58%) reported its influence on decision-making as being of a large to a very large extent, one in each case indicated not at all. Interestingly, none of the public respondents chose not at all. The one respondent who did not find the overall HEFS graphics useful at all reported that the forecaster’s note and text box were very useful, and the two who were neutral about the HEFS graphics found the forecaster’s note and text box very useful and moderately useful.
i. Influence of the focus groups
Overall, the findings from the focus group discussions and survey responses suggest that HEFS has limited utility for public audiences but is valuable information to professional users. Graphic revisions and additional components were seen to enhance the understandability and likelihood of use by both groups. Such revisions may foster decision-making relating to preparedness for extreme weather events and may encourage the use of uncertainty forecasts. This is reflected in the responses from focus group participants when asked what actions they were likely to undertake after attending the focus group session and seeing the HEFS information. A higher percentage of emergency managers in round 2 compared to round 1 were very likely to use uncertainty forecasts in decision-making, seek NWS information about extreme weather risks, and share what they learned with others. Similarly, a higher percentage of round 2 residents were very likely to better understand the uncertainty in flood forecasts and seek out NWS information about extreme weather risks compared to those in round 1, suggesting that the changes to the HEFS graphics improved the ability of residents to understand uncertainty. Further, these results suggest that, as users become more familiar with the products, their utility may increase, though not without guidance for interpretation.
Several sets of conclusions emanate from this project, centering on the utility of the information provided by the HEFS, whether 15 or 90 day, and the preferred format and content of the products. The variation in findings among the user groups should inform the next stage in product development, as well as inform future research on the communication of probabilistic forecasts.
The results of this project suggest that both the 15- and 90-day versions of HEFS have the potential for rather widespread use by specific groups, notably emergency managers and water resource managers for the 15-day products and water resource managers for the 90-day products. This was the case with both the original and revised versions of HEFS. It became clear during the focus groups that there is some potential for misinterpretation, yet the more familiar that users, particularly emergency managers and water resource managers, become with the products, the more use they may derive from them. However, even these users suggested that training and guidance will be needed. The findings are somewhat more complicated for the public.
Improvements in understanding the 15-day HEFS were evident in the second round of resident focus groups, but this did not translate into increased preference for the product. It appears, then, that it is the information and not the design of the product that is the issue. Fifteen days out may well be too long a time frame for many users. In addition, the observation in one round 2 focus group that the deterministic AHPS and probabilistic HEFS most likely lines were markedly different, suggests that residents using the HEFS tool as part of a suite of information will compare it to deterministic forecasts for developing situational understanding.
An important question that the issue above raises is how far apart a deterministic and a probabilistic forecast can be at one point and still be issued safely without causing confusion. This is much less a problem for professionals who deal with both deterministic and probabilistic forecasts on a daily basis, but it harkens back to the public’s ability to understand probabilistic forecasts and the most appropriate means of communicating uncertainties and probabilities (Morss 2010; Kox et al. 2015; Joslyn and Nichols, 2009). Results from previous projects suggest that the hydrograph is the public’s “go to” product in some regions (Hogan Carr et al. 2016a), yet even then it was misinterpreted by some. This is not to suggest that the 15-day HEFS should not be issued except to specific audiences (which is very difficult if it is on a NOAA website), but rather that difficulties that it may create should be recognized and perhaps detailed explanations provided.
The revisions in colors, legends, and text were seen by all to improve the understanding of, if not the preference for, the 15-day HEFS. Attempts to meet the needs of one group, the residents, were met with some concern by the professional groups, particularly with regard to the use of probabilities in the legend. Subsequent revisions tested in the online survey addressed this concern and received positive responses. This example illustrates the difficulties in developing one product to meet the needs of users with varied responsibilities, interests, and levels of knowledge. The additions of a text box and, especially, a forecaster’s note were seen to be particularly useful and potentially helpful to decision-making. The text boxes provide the ability to hover over a point for specific information, as suggested by the professionals in the focus groups. (Note that the text boxes were not dynamic in the survey and thus may have received lower rankings as a result.) What is not known, however, is why these elements, especially the forecaster’s note, generated such positive responses. Is it because it presumes a connection to the forecaster and thus a human connection that, in turn, causes an increase in trust? Is it seen to provide the guidance that was suggested in the focus groups by using plain text to provide clarification? Or is it something else? Wider dissemination of the online survey and final revised products is warranted and will provide further insights into the utility of the graphics, specifically the forecaster’s note and text boxes, and to the barriers to understanding. This is particularly important given the small number of participants (which was intentional to elicit detailed discussion) and the specific locational context.
The utility of the HEFS products to motivate preparedness actions varies with user groups, with the least potentially positive effect on residents, in large part because of the time frames on which they make their decisions. All users in this project see the products as useful for situational awareness, but not necessarily as ones on which they would take specific actions. The water resource managers seemed to see the greatest utility in the products, given the types of decisions they make. Most participants in the project had some difficulty understanding what the products were showing, particularly with respect to the way in which the probabilities are depicted. Changes in design elements, including colors and legends, improved the understandability, but not the preference for the products, especially among the public participants. It appears that one product may not be suitable for all audiences and, instead, should be targeted to specific stakeholders and partners, while at the same time, providing sufficient guidance to avoid confusion with other products such as the hydrograph.
The results of this project speak to broader issues associated with how to increase the use of probabilistic information in both short-term forecasts and longer-term climate products by addressing the challenges posed by the level of knowledge required, the communication format and source, as well as experiential, institutional, and political influences that come into play with different sets of users (Kirchhoff et al. 2013; Porter et al. 2015; Tang and Dessai 2012). This project complements others that have explored, for example, what information specific groups of users believe ensemble forecast products should contain and how it should be visualized (Pappenberger et al. 2013), if and how ensemble forecasts have been used in various situations (Demeritt et al. 2016), and reasons for resistance to seasonal forecasts (Dilling and Lemos 2011). All have started to build a foundation that emphasizes both the importance of ensemble probabilistic forecasts for decision-making and the difficulties in communicating that information to diverse users. This requires more than pushing information from the scientists to the users (Dilling and Lemos 2011), but rather working to understand both users’ diverse needs and the contexts in which they make decisions.
This paper was prepared by East Carolina University and the Nurture Nature Center under Award NA16NWS4680004, a social science research project funded as part of the Collaborative Science, Technology, and Applied Research (CSTAR) Program, a NOAA/NWS effort under the Office of Science and Technology. The authors thank the National Weather Service offices and staff who helped implement this project. Specifically, for their help in drafting the scenario for the project, reviewing proposed revisions for technical accuracy, and disseminating findings to professional audiences, a sincere thank you is offered to the NWS Middle Atlantic River Forecast Center and Sterling, Virginia, Weather Forecast Office. The authors also thank Jefferson County Homeland Security and Emergency Management (WV) and Frederick County Division of Emergency Management (MD) for their assistance in coordinating focus groups. The statements, findings, conclusions, and recommendations are those of the authors and do not necessarily reflect the view of NOAA or the U.S. Department of Commerce. The authors declare no conflict of interest related to this submission.
Note that the small number of emergency managers was due to most emergency managers in the counties already participating in the first round of focus groups. For evaluation purposes, the participants in the second round of focus groups could not be the same as the first round.
One respondent did not like the overall products and did not find any components useful, but two respondents were neutral about the HEFS products and reported they found the colors, the most likely line, and the flood levels useful. This has implications for components the NWS might consider for other products.