A role-playing approach to better understand the challenges of using monthly probabilistic forecasts in sequential decision-making in water management.
Seasonal climate forecasts are used in a large number of water resources applications ranging from droughts (Anderson et al. 2000), urban water (Chiew et al. 2000), hydropower operations (Block 2011), water supply (Bracken et al. 2010), water allocation (Mushtaq et al. 2012), agriculture (Ghile and Schulze 2008), and groundwater levels (Guo et al. 2009). Without exception, seasonal forecasting systems are probabilistic. They incorporate uncertainties about the future state of the climate (Brown and Ward 2013, chapter 3), which is especially useful for risk assessment.
Coelho and Costa (2010) defined several challenges for integrating seasonal climate forecasts in operational management, ranging from the production of the forecasts to the effective implementation into user applications (e.g., Hartmann et al. 2002; Lemos et al. 2002). In water resource applications, this often requires translating predictions of precipitation and temperature into predictions of streamflow or inflows to reservoirs (Regonda et al. 2011). Practitioners can then make management decisions that are informed by the hydrologic forecasts.
Coelho and Costa (2010) particularly emphasize end-user decision-making as a key challenge when implementing seasonal forecasts for water management, even given the common existence of quantitative water decision support systems (e.g., Dutta et al. 2013; Regonda et al. 2011). The decision-making process and the policies for water management are extremely complex, for they have to satisfy a large range of possibly conflicting objectives, including technical, socioeconomic, and environmental issues, and also cover a considerable range of hydrologic conditions (Simonović and Marino 1982; Welsh et al. 2013). A number of studies connect forecast performance to decision-making (e.g., Golembesky et al. 2009), but few recognize that a skillful forecast does not necessarily lead to the forecast actually used by decision-makers (Chiew et al. 2003; Kiem and Verdon-Kidd 2011; Ritchie et al. 2004).
In addition, almost all studies are unable to incorporate all operating rules or key decisions because of the complexity of the task. Technical complexities together with intricate governance settings contribute to barriers that lower the uptake of seasonal climate forecasts in water resource management [for a detailed review see Kirchhoff et al. (2013) and Lemos (2008)]. Among others, these barriers include the challenge of incorporating information into the decision-making process, and the often insufficient human and institutional capacities. Kirchhoff et al. (2013) highlight individual water manager’s behavior and risk perception as important areas that need to be addressed to realize the value of probabilistic seasonal forecasts (Block and Goddard 2012).
These issues can be addressed through an improved decision-making process, using structured decision-making models, and training. It is however extremely complex to replicate a full reality, with all possible consequences. The use of role-playing games can aid in this process, while allowing the investigation of key research questions, such as how decision rules can be formulated or whether probabilistic forecasts lead to better decisions (Ramos et al. 2013). The decision process includes a range of potential actions, a number of possible events, various consequences for each combination of action and event, and a set of probabilities for each combination (Faber and Stedinger 2001; Sankarasubramanian et al. 2009). These can be controlled in a game setting, while providing an experience that can be close to reality (Cannon-Bowers and Bell 1997) and helpful in enhancing understanding.
GAME SETTING
Each participant plays the role of a water manager for “Lake Dual,” which is a reservoir with a capacity of 500 Mm3 that serves two primary functions: water supply for “Swof Town” and flood control for “Safe Town.” The residents of Swof Town would like to see the reservoir full (500 Mm3) on 1 August to ensure an adequate drinking-water supply until the end of summer, while the residents of Safe Town are interested in keeping monthly releases below 60 Mm3 to prevent flood damage to their homes. Probabilistic forecasts of monthly inflows are available and updated on the first day of each month during the management period running from April to August. At the beginning of each month, participants have to decide on the monthly reservoir releases for the remaining months in this period.
This paper presents a game experiment that focuses on a realistic decision sequence for managing a reservoir—one based on actual hydrology, seasonal forecasts, and typical reservoir management objectives from a setting in the western United States. The game was played with different groups of students, researchers, operational hydrologists, forecasters, decision-makers, and water managers during conferences and meetings. Players were asked to manage a reservoir used for flood control and water supply, for a 4-month period (referred to as the management period). They were presented with a monthly series of probabilistic inflow forecasts for runoff in the coming 1–4 months: such forecasts are referred to as seasonal forecasts in practice because of their monthly to seasonal lead times. Participants were required to plan outflows at each decision step while complying with reservoir capacity and release constraints. The objectives of this paper are to present the results obtained and to analyze the decision-making process of the participants.
MANAGEMENT OBJECTIVES
The goal of each participant is to have the reservoir volume as close as possible to 500 Mm3 on 1 August without ever exceeding this maximum capacity. Participants also have to maintain a minimum release of 15 Mm3 for environmental flow and their maximum release cannot exceed 60 Mm3. As a penalty, participants are fired from their management role in the game if they fail to meet the constraint of maximum capacity. At the end of the game, the “winner” is the manager with the highest reservoir volume on 1 August without having exceeded its maximum capacity during the management period.
In the following sections, we present the game setup, the way it unfolded, and how participants managed their reservoirs. The last section is dedicated to discussion and conclusions.
MATERIAL.
Game setup.
The setting of the game is a reservoir in a watershed with a pronounced annual runoff cycle defined by a winter/spring snow accumulation and spring/summer melt period, followed by a low-runoff regime in the summer and fall. The reservoir management objectives are typical of many managed systems: water supply, minimum environmental releases, and flood control. The game was adapted from training material for a short course given at the 91st Annual Meeting of the American Meteorological Society in Seattle, Washington. It was modified to be played in an auditorium, in 20–25 min, and to be accessible for audiences ranging from students to researchers, operational hydrologists, forecasters, decision-makers, and water managers.
Playing the game.
To collect the results of each participant, a worksheet was distributed at the beginning of the game. It allowed participants to record their releases and update their reservoir volume. It also provided information on the long-term flow climatology for each month (the median inflow in Mm3 over the past 30 yr). Before starting the game, an example was given for the month of March. Participants were guided on how to fill in the worksheet. Finally, a volunteer was sought to play the game in front of the group, bringing a livelier atmosphere to the experience. Since computations for the volunteer could not be executed in real time, the volunteer was presented with three predefined options of releases. They were designed after several test-plays with small groups. For the volunteer, all possible combinations of sequential decisions were precalculated, but only the chosen sequence was displayed. Furthermore, the volunteer’s choice and play did not interfere with the play of the other participants, because participants had to define their releases before the choices available for the volunteer were presented.
To illustrate the forecast-decision procedure, Fig. 1 shows the slides presented for the first month. First, the inflow forecasts are displayed with the help of boxplots showing the 5% (minimum), 25%, 50%, 75%, and the 95% (maximum) percentiles (Fig. 1a), and participants are given a few minutes to decide on their reservoir releases. On the next slide (Fig. 1b), it is the volunteer’s turn to make a decision, choosing among options A, B, and C. The next slide (Fig. 1c) displays the actual inflow of the month and participants update their reservoir levels accordingly, balancing the inflow with the release they had decided on for that month. In the last slide of the round (Fig. 1d), we assess the volunteer’s decision, by showing the calculation of reservoir volume at the end of the month for the release option chosen by the volunteer. It also indicates whether the volunteer still has a job. This marks the end of a forecast-decision-update sequence and the game moves on to the next decision (i.e., next month).
Example of a sequence of a forecast decision in the game: forecast issued on 1 Apr and results for a volunteer that chose option C as the release schedule.
Citation: Bulletin of the American Meteorological Society 97, 4; 10.1175/BAMS-D-14-00270.1
The game is a repetition of this sequence of steps for forecasts issued once a month from 1 April to 1 July until we reach 1 August. If participants are fired before the last round, they are encouraged to keep playing and try to recover their jobs by lowering the reservoir volume below its maximum capacity.
A major flow event was included to occur in June to test the participants’ capacity to hedge for the possibility of high inflows to the reservoir. In addition, the probabilistic forecasts were designed such that their median values were below the actual inflow, to discriminate whether participants became sensitive to the risk represented by the upper tail of the forecast distribution. To help participants spot this pitfall, forecasts displayed both observed and forecast inflows for the past month.
RESULTS.
Worksheets collected.
We collected a total of 203 worksheets through eight distinct presentations of the game (Table 1). Seventy-five percent presented release schedules filled in for each month (and not only for the first month). Thirty-two percent showed a miscomprehension of the release constraints and had releases greater than 60 Mm3 or less than 15 Mm3. Miscalculations were observed on 63 worksheets: most were small computational errors that did not impact the reasoning process and few were miscalculations of the reservoir inflow–outflow balance. We discarded the worksheets that did not respect the constraints in the releases for the first month of the schedule and those that presented miscalculations that could have led to erroneous decisions. In the end, the analysis considered 162 worksheets, 126 of which had complete release schedules. The volunteer’s play was not considered.
Characteristics of the applications of the game.
Decision-makers’ behavior during the game: Who won and who lost?
In terms of the main decision-makers’ constraint (i.e., keep the reservoir level below 500 Mm3), we observe (Fig. 2) that 20% of the participants (32) never exceeded the reservoir capacity during the management period and kept their jobs until the end of the game. Eighty percent of the participants (130) exceeded the reservoir capacity at some point during the game. Among them, 103 exceeded the reservoir capacity once, in June, and lost their jobs, but then managed to recover (i.e., released enough in July to reset the reservoir level back below 500 Mm3 on 1 August), and 27 exceeded the reservoir capacity in June, but were unable to bring the volume below 500 Mm3 in the end.
Results of participants in terms of the main reservoir constraint: not to exceed the maximum reservoir capacity of 500 Mm3.
Citation: Bulletin of the American Meteorological Society 97, 4; 10.1175/BAMS-D-14-00270.1
We show the monthly evolution of reservoir volumes (Fig. 3a) and the distribution of releases at 1-month lead (Fig. 3b), for participants who exceeded the maximum reservoir capacity and for those who never did. All participants who won decreased their reservoir volumes by 10–50 Mm3 in the first two months. These two months were crucial in separating the winners and losers. April and May inflows summed to 73 Mm3, which, given the minimum compulsory release (15 Mm3 for each month) and the initial reservoir level (450 Mm3), forced participants to reach at most a reservoir volume of 493 Mm3 at the end of May. This level left little flexibility to keep the volume lower than the maximum capacity in the subsequent months.
(a) Evolution of reservoir volumes at the end of each month and (b) distribution of releases for the coming month (1-month lead) for participants who exceeded the reservoir capacity (participants who lost, 130 worksheets, in yellow) and participants who did not (participants who won, 32 worksheets, in blue). In (a), the gray and red horizontal lines mark the reservoir volume at the beginning of the game (450 Mm3) and the constraint of reservoir capacity (500 Mm3), respectively. In (b), the gray horizontal lines mark the minimum (15 Mm3) and the maximum (60 Mm3) allowed for releases. Box plots display minimum value; 25th, 50th, and 75th percentiles; and maximum value over 162 participants.
Citation: Bulletin of the American Meteorological Society 97, 4; 10.1175/BAMS-D-14-00270.1
In the group of participants who won, releases are higher than 30 Mm3 in April, combined with, in most cases, additional relatively high releases in May (above 50 Mm3). In contrast, more than half of the participants who lost opted to release the minimum allowed in April and, consequently, saw their reservoir volumes increase by the end of the month since the April inflow (18 Mm3) was greater than their releases. In May, 75% of these participants had their reservoir volumes greater than or equal to the initial volume of 450 Mm3, with the highest value being the maximum possible of 493 Mm3.
In June, the month for which the high-runoff event was forecast, 80% of all participants released the maximum 60 Mm3. This represents 62% of the participants who won and 84% of the participants who lost. At the end of the month, the reservoir volumes of participants who lost were between 503 and 553 Mm3, while the volumes of participants who won were between 473 and 498 Mm3.
All participants who won decreased their reservoir levels in the first two months, anticipating the peak runoff month, which was reflected in both the forecasts and the climatology. In the beginning of June, their reservoirs were then low enough to collect the water from the high inflow without overtopping. Participants who lost had a general tendency toward increasing their reservoir volumes after the first decisions, reflecting an emphasis on the goal of having the reservoir level as close to 500 Mm3 as possible on 1 August, rather than on the risk of high inflow in June. Therefore, in June, they suffered the consequence of overtopping the reservoir. The maximum reservoir volume obtained by a participant reached 553 Mm3 at the end of June, which corresponds to releasing the minimum 15 Mm3 in April and in May, and the maximum 60 Mm3 in June. This is the profile of a decision-maker who has ignored or failed to comprehend the implications of the June inflow forecasts and climatology, and became trapped by release constraints in June.
To recover their role as reservoir managers, participants who had exceeded the reservoir capacity by the end of June had to release enough in July to decrease their reservoir volumes by the end of the month. Almost 75% of this group released more than 40 Mm3 in July, while 75% of the participants who never exceeded the maximum capacity released, in the same month, less than 40 Mm3. The return to a reservoir volume below 500 Mm3 was achieved by 79% of the participants who had lost their jobs with the high inflow in June. Despite their efforts, 27 unfortunate players saw their reservoirs remain above the “spill” volume.
The urgency to decrease the reservoir volume after June to avoid further overtopping, or maybe just to secure their jobs, led participants who had lost to release large quantities of water in July. Out of the 32 participants who had not exceeded 500 Mm3 after the event in June and who had then more flexibility to adjust their reservoir volume so as to bring it as close as possible to the goal on 1 August, 11 were able to increase their reservoir volumes in July by managing their releases.
And the winner is…: Optimal 1-month lead release schedule.
Figure 4 presents the reservoir volumes of each participant at the end of July as a function of their 1-month lead releases. We consider, progressively, their releases in April (Fig. 4a), the summed releases for the months of April and May (Fig. 4b), and the summed releases for all the months prior to the high-inflow event (i.e., April, May, and June; Fig. 4c). This figure allows a retrospective analysis of which decisions led to overtopping the reservoir.
Reservoir volumes of 162 participants at the end of Jul as a function of the 1-month lead releases for (a) Apr, (b) Apr–May, and (c) the sum of Apr–Jun. Participants who won (never exceeded the maximum reservoir capacity) are indicated in blue, and participants who lost in yellow. The darker the dot, the more participants adopted the volume/release scenario. Shaded areas highlight the strict separation between winners and losers by indicating the domain occupied exclusively by participants who won, and by participants who lost in the blue and yellow colors, respectively. A percentage indicates the proportions of dots in these areas.
Citation: Bulletin of the American Meteorological Society 97, 4; 10.1175/BAMS-D-14-00270.1
Figure 4a shows that participants who released 25 Mm3 or less in April (61% of the 162 participants) lost their jobs on 1 July. The same is observed for participants who released an accumulated volume of 80 Mm3 or less by May (Fig. 4b). By this time, the total already represented 72% of all participants. By the end of June, participants who had released an accumulated volume of 140 Mm3 or less (80% of all participants; Fig. 4c) had lost the game. On the opposite end of the release spectrum, all participants who released 120 Mm3 by May (i.e., that released the maximum allowed in April and May) did not overtop their reservoirs during the game (Fig. 4b). At the end of the first three sequential decisions, all participants who had released more than 145 Mm3 had their reservoirs prepared for the high-runoff event in June (Fig. 4c).
In order not to overtop their reservoirs, participants had to lower the initial reservoir volume of 450 Mm3 by at least 10 Mm3 when making their first two decisions. In terms of releases, this means that they had to release at least 83 Mm3 over the first two decisions (given the 18- and 55-Mm3 inflows of April and May, respectively). Given the constraint of the maximum release (60 Mm3), these two months were essential to adjust the reservoir volume prior to the high-inflow event. From the worksheets, it emerges that the high-score winner applied the following sequence of releases—35, 50, 60, and 24 Mm3—to achieve the following volumes: 433, 438, 498, and 496 Mm3. This winner was among the audience of the Hydrological Ensemble Prediction Experiment (HEPEX) 10th Anniversary Workshop, which was, most probably, the venue with the most specialized users of probabilistic forecasts in the audience.
As noted earlier, the game was designed such that the seasonal forecasts were underpredicting the coming high-inflow event if judged on the 50% percentile. In fact, the observed peak inflow of June was close to the 95% percentiles of the first three (April–June) probabilistic forecasts for June. These forecasts foreshadowed that an upcoming major event was possible, albeit with a low forecast probability of occurring. The flow climatology information in the worksheets indicated that June was historically a month of high inflows and also provided a warning for participants to be cautious and to prepare for the coming event with an appropriate release strategy.
Evolution of release schedules.
The way participants were planning their releases months ahead, and how they changed their planning or not as the June high inflow approached, was investigated with the help of the 126 worksheets (out of 162) that had release schedules fully completed. In both groups, that is, participants who lost (102 worksheets) and participants who won (24 worksheets), there were approximately the same proportions of players that fully filled in their release schedules (78% and 75%, respectively).
We observe that participants who won had basically planned their releases for the month of May and June already in the first decision on 1 April. When 1 May and later 1 June arrived, the majority confirmed their previous decisions or just increased their releases by approximately 5–10 Mm3 more to accommodate the high inflows in the reservoir without overtopping.
On the other hand, participants who lost were, in general, planning very low releases on the first months and already, since the first decision in April, planning to release the maximum in June. What they had not anticipated was that this would not be enough to accommodate the high-inflow event and that a better strategy would have been to gradually empty the reservoir beginning in April and May. On 1 June, players from this group may have been frustrated by the fact that they had their reservoir volumes too high but could not release more than the maximum allowed.
In general, the planned releases for July were progressively increased when moving from April to June in both groups. On 1 July, however, when the only release they had to plan was for the coming month and the highest inflow had already passed, the majority in the group of participants who won were able to decrease the values they had planned to release previously on 1 June and, therefore, better target the final goal of having the reservoir volume as close as possible to 500 Mm3 on 1 August. On the other hand, in the group of participants who lost, half of the players had to decide on releasing more than what they had scheduled in the previous decisions to have a chance of getting their reservoirs below 500 Mm3 and, consequently, their jobs back.
How might participants have used the probabilistic forecasts and the flow climatology when making decisions?
It was left to participants to choose how they would take into account forecast and climatology information in their decisions. We could not, unfortunately, follow this process within each participant’s mind. Nevertheless, using the worksheets only, we tried to identify which forecast quantile participants based their releases on. To do so, we calculated the 1 August reservoir volumes they would have obtained if the observed inflows had consistently matched one of the forecast quantiles or, alternatively, the flow climatology. Volumes obtained with the observed inflows were also estimated. This was done for each decision step (month), so that we could also evaluate the release planning strategy. Figure 5 shows the results.
Volumes that participants would have in their reservoirs on 1 Aug by applying their release programs during each decision month (Apr–Jul) to the cases where inflows are equal to the forecast quantiles Q5%–Q95% (shades of green), the flow climatology (orange), and the observed inflows (red). Boxplots represent statistics over (top) 24 participants who won and (bottom) 102 participants who lost.
Citation: Bulletin of the American Meteorological Society 97, 4; 10.1175/BAMS-D-14-00270.1
The players who took proactive actions as early as April and May to balance the game objective with the overtopping risk likely focused on upper quantiles of the forecasts (Q75 and Q95), with maybe also some support from the flow climatology. Indeed, most of these participants would not have overtopped their reservoirs had the Q75 and even the Q95 forecast quantiles been verified as observed inflows throughout the game, as early as in the first decision step. By the third and fourth decisions, they seemed to have set aside flow climatology and rather used an “adjusted” upper forecast quantile (Q95) to evaluate the possible observed inflow and optimize their releases to the goals of the game. This might have been the result of having previously noted that the forecasts were, in general, underpredicting the observed inflows. The fact that part of the distribution for August volumes given actual observed flows is above the reservoir maximum during the first two decisions (April and May) means that participants tolerated or were not able to eliminate the overtopping risk early in the management period, but took steps to eliminate it when the potentially high-inflow month was imminent.
In the group of participants who lost, most participants might have been guided by the medium to lower quantiles (Q50 and Q25) in their first decisions. Figure 5 shows that if the upper quantile forecasts or even the flow climatology values had verified as observed inflows, these participants would have reached reservoir volumes much higher than the allowed 500 Mm3 in August. In contrast, if lower quantiles had verified as observed inflows, most would have been safe from overtopping their reservoirs. Later, when making their third and fourth decisions, these participants might have acted based on the upper forecast quantiles (Q75 and Q90) and, eventually, on the flow climatology to plan their releases.
DISCUSSION AND CONCLUSIONS.
This paper presented the results of a game experiment designed to mimic risk-based decision-making in water management using probabilistic forecasts of inflows to a reservoir. From the analysis of the worksheets collected during the application of the game within eight different contexts, we were able to illustrate key issues related to the use of probabilistic forecasts in sequential decision-making.
In the game setup, seasonal forecasts had to be viewed as a whole and not as independent monthly forecasts. Even though the forecast time evolution warned the participants about the potential for a high-inflow event, monthly 50% percentile forecast values were underestimating flows. Given the reservoir constraints, as expressed in the rules of the game, and the goals of the management, it was necessary to look at forecasts months ahead before deciding on the reservoir releases. A winning strategy would be the one that would gradually empty the reservoir two months ahead of the expected high inflow to accommodate its potential volume without overtopping. Although the worksheets were designed to invite participants to adopt this long-term approach, notably by asking them to schedule their releases for the whole management period, still about 25% of the participants only filled their releases for the coming month.
Approximately 20% of the participants in the game were able to finish the management period without exceeding the maximum reservoir capacity at any time during the sequenced decisions, while approximately 17% of the participants not only caused the reservoir to overtop, but also were unable to bring its volume back below the threshold. Winners were those who had programmed the releases in a way that they succeeded in adequately decreasing their reservoir volumes in the first two months, anticipating the potential high inflows. Losers of the game were those who did not recognize early enough the significance of the high-inflow risk or did not comprehend its potential impact on the reservoir volume.
In this paper, answers that showed signs of miscomprehension of the balance equation were not used in the interpretation. Therefore, it has been implicitly assumed in the analysis of the remaining results that the participants were able to take full advantage of the proposed information, once given the necessary tools to read the probabilistic graphs and understand the climatological values at the beginning of the game. The results based on the remaining answers may however still be influenced by the comprehension of the proposed tools and the way they were presented. It is possible that given the timed nature of the game (participants had to decide within a limited amount of time), some participants could understand the way forecasts were displayed but were not fast enough in applying their understanding. Communicating forecasts and, more generally, probabilities is still a major challenge in hydrometeorology, and can be a barrier to the widespread use of ensemble forecasts within operational contexts.
The information collected from the worksheets shows which decisions were taken, but not why they were taken. Therefore, care has to be taken in the interpretation of the results of this game experiment. We can only speculate on what could have been the reasons for many participants to make a certain sequence of release decisions, and on what strategy winners had with respect to the forecast inflows. Therefore, even if the results indicate that the participants who won the game might have considered the upper quantiles of the forecasts, at least in their first decisions, information on why they followed this procedure is not available through the game setup. A possible solution to this limitation could be to expand the game by asking these relevant questions either orally or directly in the worksheets. Recording whether participants had prior water resource management or forecasting experience could also help indicate the value of training toward improving the application of probabilistic information. Collecting specific information on the main occupation and background of each participant could also be helpful in evaluating how strategies may vary between different groups (e.g., students, managers, or forecasters).
Finally, this game is a simplified representation of reality and does not intend to reproduce the full context of operational environments in reservoir management. Indeed, a participant who had real-life experience managing reservoirs noted during one game session that the winning strategy (described in the section discussing the decision-makers’ behavior during the game) could have raised alarms in practice for having allowed too much flooding risk en route to achieving a near-perfect target level. Despite the simplifications, however, we received positive feedback after the different applications, even though howls of indignation were often heard in the rooms when the observed June inflow was revealed. The role-playing approach, and the penalty experienced by participants of being fired from their jobs, added a light touch to the experience and created a pleasant atmosphere to discuss the challenges of using probabilistic seasonal forecasts in sequential decision-making, where choices have delayed consequences. Notably, the game has been successfully used as support material during teaching and training activities.
ACKNOWLEDGMENTS
This game is part of the Hydrologic Ensemble Prediction Experiment and is available online to be freely used for teaching or training (www.hepex.org/). Special thanks to Kevin Werner, a key designer of the original game on which this one is based; Fredrik Wetterhall for playing the game at ECMWF; and Micha Werner and Robert Hartman for kindly playing the volunteer. The lead author was partly funded by the EU Interreg IVB NWE project DROP.
REFERENCES
Anderson, M. L., M. D. Mierzwa, and M. L. Kavvas, 2000: Probabilistic seasonal forecasts of droughts with a simplified coupled hydrologic-atmospheric model for water resources planning. Stochastic Environ. Res. Risk Assess., 14, 263–274, doi:10.1007/s004770000049.
Block, P., 2011: Tailoring seasonal climate forecasts for hydropower operations. Hydrol. Earth Syst. Sci., 15, 1355–1368, doi:10.5194/hess-15-1355-2011.
Block, P., and L. Goddard, 2012: Statistical and dynamical climate predictions to guide water resources in Ethiopia. J. Water Resour. Plann. Manage., 138, 287–298, doi:10.1061/(ASCE)WR.1943-5452.0000181.
Bracken, C., B. Rajagopalan, and J. Prairie, 2010: A multisite seasonal ensemble streamflow forecasting technique. Water Resour. Res., 46, W03532, doi:10.1029/2009WR007965.
Brown, C., and M. N. Ward, 2013: Managing climate risk in water supply systems. IRI Tech. Rep. 10–15, International Research Institute for Climate and Society, Palisades, NY, 133 pp. [Available online at http://iri.columbia.edu/resources/publications/pub_id/1048/.]
Cannon-Bowers, J. A., and H. R. Bell, 1997: Training decision makers for complex environments: Implications of the naturalistic decision making perspective. Naturalistic Decision Making, C. Zsambok and G. Klein, Eds., Lawrence Erlbaum Associates, 99–110.
Chiew, F. H. S., T. A. McMahon, S.-L. Zhou, and T. Piechota, 2000: Streamflow variability, seasonal forecasting and water resources systems. Applications of Seasonal Climate Forecasting in Agricultural and Natural Ecosystems, G. L. Hammer, N. Nicholls, and C. Mitchell, Eds., Atmospheric and Oceanographic Sciences Library, Vol. 21, Springer, 409–428.
Chiew, F. H. S., S.-L. Zhou, and T. A. McMahon, 2003: Use of seasonal streamflow forecasts in water resources management. J. Hydrol., 270, 135–144, doi:10.1016/S0022-1694(02)00292-5.
Coelho, C. A. S., and S. M. S. Costa, 2010: Challenges for integrating seasonal climate forecasts in user applications. Curr. Opin. Environ. Sustain., 2, 317–325, doi:10.1016/j.cosust.2010.09.002.
Dutta, D., K. Wilson, W. D. Welsh, D. Nicholls, S. Kim, and L. Cetin, 2013: A new river system modelling tool for sustainable operational management of water resources. J. Environ. Manage., 121, 13–28, doi:10.1016/j.jenvman.2013.02.028.
Faber, B. A., and J. R. Stedinger, 2001: Reservoir optimization using sampling SDP with ensemble streamflow prediction (ESP) forecasts. J. Hydrol., 249, 113–133, doi:10.1016/S0022-1694(01)00419-X.
Ghile, Y. B., and R. E. Schulze, 2008: Development of a framework for an integrated time-varying agrohydrological forecast system for southern Africa: Initial results for seasonal forecasts. Water S. Afr., 34, 315–322.
Golembesky, K., A. Sankarasubramanian, and N. Devineni, 2009: Improved drought management of Falls Lake Reservoir: Role of multimodel streamflow forecasts in setting up restrictions. J. Water Res. Plann. Manage., 135, 188–197, doi:10.1061/(ASCE)0733-9496(2009)135:3(188).
Guo, W., J. Zhao, and F. Wang, 2009: The seasonal forecast method of Sanjing Plain underground water level. J. Northeast Agric. Univ., 5, 104–107.
Hartmann, H. C., T. C. Pagano, S. Sorooshian, and R. Bales, 2002: Confidence builders: Evaluating seasonal climate forecasts from user perspectives. Bull. Amer. Meteor. Soc., 83, 683–698, doi:10.1175/1520-0477(2002)083<0683:CBESCF>2.3.CO;2.
Kiem, A. S., and D. C. Verdon-Kidd, 2011: Steps toward “useful” hydroclimatic scenarios for water resource management in the Murray-Darling Basin. Water Resour. Res., 47, W00G06, doi:10.1029/2010WR009803.
Kirchhoff, C. J., M. C. Lemos, and N. L. Engle, 2013: What influences climate information use in water management? The role of boundary organizations and governance regimes in Brazil and the U.S. Environ. Sci. Policy, 26, 6–18, doi:10.1016/j.envsci.2012.07.001.
Lemos, M. C., 2008: What influences innovation adoption by water managers? Climate information use in Brazil and the United States. J. Amer. Water Resour. Assoc., 44, 1388–1396, doi:10.1111/j.1752-1688.2008.00231.x.
Lemos, M. C., T. Finan, R. Fox, D. Nelson, and J. Tucker, 2002: The use of seasonal climate forecasting in policymaking: Lessons from Northeast Brazil. Climatic Change, 55, 479–507, doi:10.1023/A:1020785826029.
Mushtaq, S., C. Chen, M. Hafeez, J. Maroulis, and H. Gabriel, 2012: The economic value of improved agrometeorological information to irrigators amid climate variability. Int. J. Climatol., 32, 567–581, doi:10.1002/joc.2015.
Ramos, M. H., S. J. van Andel, and F. Pappenberger, 2013: Do probabilistic forecasts lead to better decisions? Hydrol. Earth Syst. Sci., 17, 2219–2232, doi:10.5194/hess-17-2219-2013.
Regonda, S., E. Zagona, and B. Rajagopalan, 2011: Prototype decision support system for operations on the Gunnison basin with improved forecasts. J. Water Resour. Plann. Manage., 137, 428–438, doi:10.1061/(ASCE)WR.1943-5452.0000133.
Ritchie, J. W., C. Zammit, and D. Beal, 2004: Can seasonal climate forecasting assist in catchment water management decision-making?: A case study of the Border Rivers catchment in Australia. Agric. Ecosyst. Environ., 104, 553–565, doi:10.1016/j.agee.2004.01.029.
Sankarasubramanian, A., U. Lall, F. A. Souza Filho, and A. Sharma, 2009: Improved water allocation utilizing probabilistic climate forecasts: Short-term water contracts in a risk management framework. Water Resour. Res., 45, W11409, doi:10.1029/2009WR007821.
Simonović, S. P., and M. A. Marino, 1982: Reliability programing in reservoir management: 3. System of multipurpose reservoirs. Water Resour. Res., 18, 735–743, doi:10.1029/WR018i004p00735.
Welsh, W. D., and Coauthors, 2013: An integrated modelling framework for regulated river systems. Environ. Modell. Software, 39, 81–102, doi:10.1016/j.envsoft.2012.02.022.
