1. Introduction
This paper presents the history of a computer model and decision-support system (DSS) that facilitates coordination across multiuse reservoirs while incorporating climate information (CI). The history shows that both informal and formal modes of collaborative science may be critical to develop and advance technologies that assist with climate adaptation in water and energy governance. There is an urgent need to adapt to climate change, demanding that we reexamine natural resources governance and the tools needed to influence governance. Institutional arrangements for water and hydropower governance can make those systems more vulnerable to climate impacts (e.g., Hanemann 2006; Willis et al. 2011; Viers 2011; Bedsworth et al. 2018). The network of laws governing water and energy resources can also make altering hydropower or large reservoir operations difficult in the United States (e.g., Ziaja 2017). Models and DSS offer alternative venues for adapting water and hydropower systems, through the incorporation of CI (e.g., seasonal forecasts) and linking reservoir operations (Medellín-Azuara et al. 2008; Viers 2011; Willis et al. 2011; A. Georgakakos et al. 2012; Georgakakos et al. 2013). However, despite the proliferation of models and DSS, they are rarely adopted for actual use by water and energy decisionmakers [Ziaja 2017; see also Garfin et al. (2008), discussing use of DSS in the public sector]. This is consistent with the CI “use gap” (Lemos et al. 2012), especially in water management (Rayner et al. 2005; Kirchhoff 2013; Kirchhoff et al. 2015), and the modeling “relevance gap” (McCown et al. 2009; Prost et al. 2012). There are nonexclusive theories regarding what causes these gaps: governance structure (Dilling et al. 2015; Flagg and Kirchhoff 2018); cultural context (Roncoli 2006; Roncoli et al. 2009; Peterson et al. 2010; cf. Bolson and Broad 2013), conflicting time scales (Rayner 2019), and the models themselves (Prost et al. 2012; Ziaja 2017; Lindblom et al. 2017; cf. Etkin et al. 2015). Theoretical and empirical work suggest that deliberate coproduction of science may be a solution (e.g., Termeer et al. 2011; Dewulf et al. 2013; McNie 2013; Lemos et al. 2014a; Moser 2016; Guido et al. 2016; Furman et al. 2018; Lemos et al. 2018, 2019).
There is a need for successful case studies of deliberate coproduction as benchmarks for future adaptation efforts (Meadow et al. 2015; Lach and Rayner 2017). Studies are needed that can add to understanding how knowledge networks influence the usability of knowledge (Kalafatis et al. 2015). Similarly, there is a need for analysis of the institutional arrangements and science policy decision processes that support coproduction (McNie 2007; Ziaja 2017). This paper responds to these needs with a case study of coproduction and the role of a knowledge network in fostering the development and implementation of the Integrated Forecast and Reservoir Management (INFORM) system—a model and DSS that incorporates CI into hydropower and reservoir operations in Northern California.
The history of INFORM demonstrates that despite complex institutional arrangements over a contentious topic—balancing competing demands for water for consumption, energy, agriculture, recreation, and environment—formal and informal collaboration changed the process of coproduction, improved the model and DSS, and assisted in implementation at the California Department of Water Resources (DWR). This paper proceeds by first providing background on climate change impacts and multiuse reservoirs and hydropower in California, and how INFORM aids in climate adaptation. This paper then introduces literature on coproduced science for model development and water-energy governance. After presenting the methods, this paper relates the history of the development of INFORM.
This history is focused on the changes in institutions that supported collaboration, organized by phases of interaction (and noninteraction) between researchers and a semi-defined community of hydropower and reservoir stakeholders. Collaborations between managers and researchers were crucial to the success of the project, by building knowledge networks and incorporating policy preferences of end users into the model. The knowledge network led to substantive changes in the INFORM model, DSS, and collaborative process (Fig. 1). The history also shows how even successful collaborative projects do not always follow the usual expectations for coproduction; even when those guidelines are followed, external circumstances—for example, changes in political climate and funding availability—can threaten the adoption of research products. Ultimately, this paper introduces the model of bimodal coproduction and argues for building strong knowledge networks alongside more formal processes for collaborative engagement.
Examples of connection between knowledge network, OIC, and improvements in the INFORM products (model and DSS) and process. Most of the footnotes reference the appendix volume of Georgakakos et al. (2007).
Citation: Weather, Climate, and Society 11, 4; 10.1175/WCAS-D-19-0007.1
2. Background on climate and multiuse reservoirs and hydropower in California
California is rapidly transitioning its energy system to respond to climate change (CEC 2017; CNRA 2017) and has encouraged subnational jurisdictions to do likewise (Climate Group 2015). Hydropower in California is part of its climate mitigation strategy, facilitating intermittent renewable energy adoption (e.g., Gleick 2015). Studies suggest that climate change impacts and human responses will make hydropower generation less reliable (Voisin et al. 2016; Tarroja et al. 2016; Voisin et al. 2018) (Table 1). Moreover, there is an array of cultural and economic values and uses associated with large multiuse reservoirs, well discussed in water governance literature [e.g., Wandschneider 1986; Postel and Richter 2003; Bauer 2004; Ingram 2006; Hanemann 2006; Doremus and Tarlock 2008; Moore et al. 2010; for U.S. water resources and politics generally, see Schlager and Blomquist (2008)]. However, water-related complexities of hydropower governance are not generally incorporated into energy research or planning (e.g., Karambelkar 2017).
Overview of climate impacts and threats to multiuse reservoirs and hydropower (see text and/or Table 2, below, for expansions of agency acronyms).
a. California hydropower, reservoir governance, and climate change
Physical characteristics of hydropower make it sensitive to climate change and political contexts. Hydropower interrupts streamflow, altering characteristics of the water,1 which is relied on by nonenergy needs—including the needs of threatened and endangered species (Postel and Richter 2003). The governance of hydropower facilities is sensitive to competing uses, inexorably tied to its landscape, and affected by the values of other water users (Prieto and Bauer 2012; Ziaja 2017) and their social-ecological systems (see Ostrom 2009; Dietz et al. 2003).
Institutional and political environments are major factors in use and adoption of CI for water resources (Flagg and Kirchhoff 2018). Institutional context of hydropower in California is built on a history of conflicts among electricity development interests and water interests—including environmental protection, navigation, and flood protection (Swiger et al. 2015). Each hydropower facility is governed by its own mix of overlapping jurisdictions (Table 2) and operating rules (Swiger et al. 2015; Ziaja 2017).
Hydropower jurisdiction across federal and state agencies. An asterisk indicates required coordination in FERC licensing. Note that the chart is only relevant to hydropower in California. The array of relevant agencies changes somewhat based on location. For example, the Bonneville Power Administration would be a relevant federal agency on the Columbia River. There are exceptions to subject matter jurisdiction of federal agencies over hydropower and reservoirs. For example, federal authority generally only extends to navigable rivers. These exceptions are not enumerated in the table. For a thorough review of hydropower law and jurisdiction of federal agencies see Swiger et al. (2015).
It is well known, though rarely discussed outside specialist groups, that rules for some hydropower dams are ill-suited to safe, economical, and sustainable operation in a changing climate (CEC 2017). The historic drought in the western United States and subsequent record rain years, however, are bringing renewed attention to hydropower governance (CEC 2017). For hydropower reservoirs with a flood control function, the U.S. Army Corps of Engineers (USACE) sets operational parameters for how much water a reservoir can store and when (Willis et al. 2011). These parameters were developed over 60 years ago, based on observed climate, and do not allow for changes in operation based on currently observed weather or short- or long-term weather forecasts (Willis et al. 2011). Models demonstrate that hydropower reservoirs in California perform better for energy generation, water management, and environmental protection when operational rules are able to incorporate weather and probabilistic climate forecasts (Willis et al. 2011; K. Georgakakos et al. 2012). Changes to reservoir rules and practices are necessary to adapt the system to climate change (Viers 2011; CEC 2017).
b. What Is INFORM?
INFORM is a model and DSS for reservoir and hydropower operations, designed to incorporate CI and facilitate coordinated operations and planning for reservoir and hydropower systems in Northern California. Its software program is designed to assist reservoir and hydropower decision-makers by offering information on what combination of reservoir operations in a river basin—or across basins—is optimal for multiple purposes, across multiple time scales, given current and projected weather and climate. DWR, in coordination with the Bureau of Reclamation (hereinafter simply “Reclamation”), uses INFORM in operations and planning in the Sacramento River basin,2 covering a territory of 15 counties and influencing decisions for the key reservoirs of the State Water Project (SWP) and the Central Valley Project. If INFORM were to be extended to the San Joaquin River basin—another critical river system in California’s statewide water infrastructure—the project would need to build on existing INFORM components and develop “the necessary real-time databases, quality control and ingest mechanisms to allows INFORM operations [to expand]” (Georgakakos et al. 2018, p. 17).
INFORM’s implementation, as of June 2018, overcomes some of the shortcomings of traditional administration of hydropower and multiuse reservoirs. The modeled operational rules were optimized for environmental concerns, flood control, water conservation, water supply, and hydropower generation. A multiyear demonstration project, in which INFORM was run, virtually, in real time and compared against actual operations of reservoirs, concluded that the virtual operations fared far better to meet competing demands for water (Georgakakos et al. 2013).
The INFORM system—a water-supply forecast component and a water management DSS component—allows DWR staff to access the two components through an interactive graphical interface. The DSS is not intended to replace human decision-making. Rather, it provides more complete information about the temporal fluctuations in water availability and the implications of that variability and management decisions on a systemwide and reservoir-specific scale. Validation, training, and adjustments are ongoing, with partnerships between DWR and the INFORM principal investigators (PIs).
A person using INFORM first develops a network map of the river system through a GIS interface (Georgakakos et al. 2018, their appendix D). “Arcs” are used to define streams, channels, diversions, and returns of water. These are matched against “nodes”—which include reservoirs, watersheds, demand nodes, river nodes, and a delta node (for the delta of the Sacramento–San Joaquin Rivers). Through selections of arcs and nodes, the user creates a virtual river. The virtual rivers, like real ones, have multiple management objectives—like power production and habitat protection—along with infrastructure to influence and measure those objectives. The user next inputs specific data relevant to the selected nodes and arcs, such as the number of turbines or generator capacity. Once the data are input, the user can access the “Long Range Decision Support module”—picking the defined time horizon (from months to years) for which they want to consider planning and management decisions. The module uses inflow forecasts provided in an ensemble form. The output of the module is a suite of options “that meet multiple user-defined system objectives and requirements as best as possible” (Georgakakos et al. 2018, p. 97). The module includes both optimization and simulation models—allowing the simulated system conditions under various management options to reflect “actual” conditions—i.e., those that “would occur in the system under the same management options” (Georgakakos et al. 2018, p. 97) (Fig. 2).
INFORM DSS workflow (adopted from Georgakakos et al. 2018, p. 27).
Citation: Weather, Climate, and Society 11, 4; 10.1175/WCAS-D-19-0007.1
Several articles describe findings from INFORM (Carpenter and Georgakakos 2001; Yao and Georgakakos 2001; Georgakakos et al. 2005; Georgakakos and Graham 2008; Graham and Georgakakos 2010; A. Georgakakos et al. 2012; K. Georgakakos et al. 2012). Very little has been written on the process of coproduction behind INFORM. A report by the U.S. Climate Change Science Program (CCSP 2008) comes nearest, highlighting INFORM as an experiment in DSS, integrating CI into water management. The report stresses the importance of two-way “science–society collaboration” (Ingram et al. 2008a, p. 5) and creating credible and reliable information through “familiarity and repeated interaction between information collaborating and the working and reworking of relationships” (Ingram et al. 2008b, p. 16). The report’s summary of INFORM, however, covers the technical details of the product rather than the process of collaboration, only once mentioning the agencies involved and the existence of an Oversight Implementation Committee (OIC), which was responsible for providing feedback for the INFORM model and DSS (Feldman et al. 2008).
3. Coproduction: Boundary organizations and knowledge networks as means to create actionable science for water-energy governance
Collaborative approaches to develop “usable” CI have gained traction across disciplines since the late 1990s (e.g., Pulwarty and Redmond 1997; Callahan et al. 1999; Pulwarty and Melis 2001; Redmond 2004; Fraisse et al. 2006; Garfin 2006; Kiker and Linkov 2006; McNie 2008; Carbone et al. 2008; Breuer et al. 2009; Prokopy and Power 2015; Stevenson et al. 2016). The core attributes of coproduced usable science are salience to the community, credibility among scientists, and legitimacy to both (Cash et al. 2003, 2006; Moser 2016). Social science investigation of coproduction and stakeholder participation to improve the “usability” of science products notes that coproduction depends on active dialogue and engagement between science and society (Cash et al. 2003, 2006; Lemos and Morehouse 2005; Feldman and Ingram 2009; NRC 2009; Dilling and Lemos 2011; Lemos et al. 2012; McNie 2013; Bartels et al. 2013; Meadow et al. 2015; Prokopy and Power 2015; Beier et al. 2016; Buizer and Cash 2016; Gordon et al. 2016; Guido et al. 2016; Furman et al. 2018; Lemos et al. 2018, 2019). Coproduction has been hailed as a means to incorporate interdependencies and tradeoffs in the governance of the water–energy nexus (Polk 2014; Zhang and Vesselinov 2016; Howarth and Monasterolo 2017) and the social complexities of water governance (Kiparsky et al. 2012; Flagg and Kirchhoff 2018), especially for water allocation choices [Rice et al. 2009 (southwestern United States); Peterson et al. 2010 (Uganda and Brazil); Kirchhoff et al. 2013 (Brazil and United States); Kirchhoff and Dilling 2016 (United States); Bolson and Broad 2013 (southern Florida)], water quality choices [(Kalafatis et al. 2015 (Great Lakes, United States)], and approaches to integrated water resources management (IWRM) [(Roncoli et al. 2016 (Burkina Faso); Falconi and Palmer 2017 (Zimbabwe; the southeastern United States; Las Vegas, Nevada; the Solomon Islands; Senegal); Lemos 2015].
Models of coproduction emphasize participation from stakeholders to overcome challenges to the acceptability of scientific research results (Cash et al. 2006, p. 484). Meadow et al. (2015) draw from agricultural research (e.g., Biggs 1989) to provide some structure to the field, matching approaches to collaboration with “modes of engagement” that define the type of relationship between researchers and practitioners or stakeholders. The approaches and modes show various ways to deliberately implement coproduction (Table 3) (Meadow et al. 2015).
Approaches to deliberate coproduction and modes of collaboration (adapted from Meadow et al. 2015, p. 184).
Boundary organizations and knowledge networks are institutional arrangements that can facilitate engagement and help to mediate between the demands of salience and credibility (Flagg and Kirchhoff 2018; Guido et al. 2016). Knowledge networks are informal networks of people and organizations from different disciplinary backgrounds and missions, but who are at least temporarily “linked together in an effort to provide close, ongoing, and nearly continuous communication and information dissemination among multiple sectors of the society involved in technological and policy innovations for manag[ement]” [Feldman and Ingram 2009, p. 10; also see Sarewitz and Pielke 2007; Guido et al. 2016; cf. Kalafatis et al. 2015 (including objects as network nodes)]. Knowledge networks are characterized by person-to-person sharing that produces “blended knowledge”—which depends on transdisciplinary communication (Feldman and Ingram 2009, p. 13; Aslin and Blackstock 2010). They rely on flexible processes for exchange, iterative learning (Feldman and Ingram 2009, p. 14; Bartels et al. 2013), and “information brokers” to repackage, translate, and disseminate information (Buizer and Cash 2016; Lemos et al. 2012, 2014b; Guido et al. 2016). The characteristics of information brokers and end users—e.g., education (Rice et al. 2009), position within their organizations (Bolson and Broad 2013; Guido et al. 2016), or age (Rayner et al. 2005)—influence adoption of CI use (Flagg and Kirchhoff 2018).
By contrast, boundary organizations are formal groups that translate between stakeholders and users, mediating the needs of salience and credibility (Guston 2001; Cash 2003; Dilling and Lemos 2011; McNie 2013; Ziaja and Fullerton 2015; Meadow et al. 2015; Guido et al. 2016; Feldman and Ingram 2009). Boundary organizations can perform multiple functions, including convening, translation, collaboration, and mediation (Meadow et al. 2015; Feldman and Ingram 2009). They communicate with both stakeholders and scientists, but do not necessarily contain either natively within the organization (Cash et al. 2006).
These two categories of coproduction are related, but distinct. Knowledge networks may include multiple conversations across an array of participants in the network, whereas boundary organizations are characterized by the facilitation of two-way dialogue (Feldman and Ingram 2009). A good example of a boundary organization is the Cooperative Extension Service [Breuer et al. 2010; see also Meadow (2017) on the NOAA Regional Integrated Sciences and Assessments programs (RISAs)]. In a conceptual model of the relationships among a network, boundary organization, and information brokers, developed by Guido et al. (2016), the boundary organization provides a venue for a network to form, and “supports and develops” that network, while information brokers emerge from the network, “who help connect information from upstream sources . . . to downstream users” (Guido et al. 2016, p. 295). The presentation of INFORM below adds to this work and suggests an alternative conceptual model of the relationship between networks and boundary organizations (Fig. 3).
Bimodal collaboration, showing the process of evolution of INFORM’s knowledge network through amplifying effects of its boundary organization (OIC).
Citation: Weather, Climate, and Society 11, 4; 10.1175/WCAS-D-19-0007.1
Significant scholarship on water/energy governance and coproduction discusses incorporation of CI (water: Lemos 2015; Kalafatis et al. 2015; Flagg and Kirchhoff 2018; water/energy: Howarth and Monasterolo 2017). Bolson and Broad (2013), in their investigation of the adoption of seasonal climate forecasts by the South Florida Water Resources Management District (SFWRMD), offer another approach, building from “technology transfer” literature. Their conclusions are consistent with the research described above: communication, trust, the forecast’s “fit” with the decision-making process, attributes of the technology, and the traits of the technology adopters all influence the adoption of CI. They suggest that the social dimensions of collaborative research and implementation outweigh the influence of the CI itself, finding that “[i]n fact, trust in forecast provider, not the skill of forecasts, appeared to be the driving factor in selection of [Climate Prediction Center] forecasts.” (Bolson and Broad 2013, p. 278). This fits in a long line of social science research on climate forecasts for water management (e.g., Rayner et al. 2005; Jacobs et al. 2005; Ingram et al. 2008a, b; Lemos 2008; Dilling and Lemos 2011; Peterson et al. 2010; Kiparsky et al. 2012; Lemos 2015).
Less attention has been paid to the role of specific design of computer models in adapting governance and coproduction (cf. Falconi and Palmer 2017; Ziaja 2017). Falconi and Palmer (2017) argue that models can be “boundary objects” facilitating collaboration and leading to improved salience, credibility, and legitimacy (see also Kiker and Linkov 2006). However, parameterization of models can misrepresent institutional and legal aspects of water and hydropower governance, rendering them questionable to decision-makers (Ziaja 2017). Research in agronomic modeling is informative here; agronomic modeling has a “gap in relevance” (McCown et al. 2009; Prost et al. 2012) and “problem of implementation” (Lindblom et al. 2017). Prost et al. (2012) found a lack of consideration of end users during model development. They note that “the use of the [agricultural] models for action is not well established, although it is often claimed by the authors.” (Prost et al. 2012, p. 582). Critiques of agronomic models posit that participation of end users in design could improve the models (Cerf et al. 2012) and use outcomes (McCown 2002; Carberry et al. 2002; Breuer et al. 2008; Lindblom et al. 2017). Cerf et al. (2012) offer a framework for collaborative (“dialogical”) design and development of agronomic decision-support tools: 1) diagnosis of uses (i.e, how might the tool help solve a problem) and 2) use of a prototype of the tool under development, allowing for debriefing after experimental use.
This is different from technological adoption and collaboration shown in Bolson and Broad’s study of changes in the regulation of Lake Okeechobee to incorporate CI (Bolson and Broad 2013). Tools and CI were available; adoption, not development, was the key question. Cerf et al. (2012) speak more to the challenges facing California reservoirs when INFORM was developing, despite coming from agronomic studies. Modeling could theoretically address challenges facing California reservoirs, but there was no “off the shelf” tool that could be adopted. A new tool needed to be developed to incorporate CI and coordinate multiuse reservoir and hydropower operations for competing uses.
4. Method
This paper results from qualitative research undertaken from 2014 to 2019 in California. The INFORM PIs and the chief climatologist of DWR constituted the initial selection of interviewees. Guido Franco, head of the California Climate Change Research Program at the California Energy Commission (CEC), and the author’s supervisor from 2014 to 2017, assisted with introductions. From the initial set, snowball sampling was used as a purposive method to reach and reveal the knowledge network for the INFORM project (Bernard 1988). Interviews with staff from federal and state agencies were conducted in person in Sacramento and Davis, California, or via telephone. Follow up questions and verification were done by e-mail or additional interviews by telephone. Initial interviews lasted on average 45 min.
These interviews are bolstered by a significant literature review and content analysis. Happily, the INFORM researchers were meticulous note takers, keeping records of presentations to management and funding agencies and the OIC, discussions among OIC members, and final decisions of meetings, covering 2003–06. Summaries of OIC meetings are publicly available through the CEC.
This paper adopts the Furman et al. (2018) recommendation to go beyond investigation of formal workshops and collaborative meetings. Materials for this study cover the period from the inception of INFORM as a research project in the late 1980s, through its collaborative phase, and into its early implementation in 2018.
5. History of the development of INFORM
This study examines the macro-, meso-, and microscale characteristics (Flagg and Kirchhoff 2018) of INFORM’s development, from inception to implementation. The history of INFORM presented below builds on proceeding scholarship and is most interested in the relationships among the institutional arrangements that influenced INFORM. Specifically, how did institutional arrangements—formal rules and informal norms—across scales shape INFORM and its knowledge network?
A review of the available documents and interviews with researchers, funders, and OIC members highlights an important dynamic among the knowledge network and boundary organization models for coproduction. Formal processes (procedural rules that set expectations and rules for engagement, from boundary organizations) and the knowledge network (positive relationships among participants) can build on one another to foster the success of the project (Fig. 3). Based on interviews and content analysis, formal rules for collaboration initially came from funding agencies and were developed later through dialogue between researchers and stakeholders. The procedures provided a framework for trust relationships to develop. Relationships within the knowledge network carried the project through gaps in processes and in funding. The boundary organization and network influenced the collaborative process and the development of the INFORM products (model and DSS) (Fig. 1). One of the INFORM PIs noted that, while the products are useful, the process was indispensable, stating “What this exercise is really about is what the stakeholders value and what they can trade off.”3 In this sense, the INFORM products were boundary objects (Falconi and Palmer 2017) allowing stakeholders to better understanding the competing uses and constraints on the river.
This section is organized by funding source (Fig. 4). Funding is predicated on contractual agreements. Through contract terms, funding agreements formalize minimum processes for research coordination and dissemination of results. Changes in the procedural rules for engagement come with substantive changes to research, and varying levels of dialogue and collaboration with end users and stakeholders. The role of funding requirements in participation has been noted in other contexts. The World Bank required participatory decision-making for water with a broad range of in-basin stakeholders as a condition for a loan to develop reservoirs and canals in Brazil (Peterson et al. 2010). Note that the base requirement for participation alone did not produce positive outcomes in that case (Peterson et al. 2010).
Timeline of the INFORM process from initial stage to development to implementation.
Citation: Weather, Climate, and Society 11, 4; 10.1175/WCAS-D-19-0007.1
a. Forming the intent to engage; limited knowledge network: 1987–2001
INFORM did not begin with coproduction. A decade of preparatory work preceded outreach. It is unclear whether this period would have been helped or hindered by engagement between researchers and stakeholders. Taking a longitudinal (Furman et al. 2018) and pragmatic (Meadow et al. 2015) perspective, however, it is notable that not all coproduction efforts start out with engagement. This is consistent with the dialogical method of coproduced modeling (Cerf et al. 2012). In the late 1980s, brothers K. and A. Georgakakos received a 5-yr grant from the National Science Foundation (NSF) to study the use of scientific information for risk-based management, resulting in resolution of multiscale forecast problems across multiple reservoirs. At this earliest phase, research for INFORM did not seek to engage with potential users. Rather, research took place cloistered from potential users. The type of funding for the research also facilitated this incubation period. Unlike later funders, NSF did not require outreach to end users at the time.
As funds decreased, the researchers grew concerned about a gap between the state of hydrologic modeling and the state of hydrologic management—akin to that noted by Prost et al. (2012). They also noted a related gap in funding for research not perceived to be “useful.” Their response was to create a boundary organization to bridge those gaps—the Hydrologic Research Center (HRC)—in 1993. The boundary organization’s translation function (Cash et al. 2003; Bolson and Broad 2013; Meadow et al. 2015) was most important. The Georgakakos brothers and their colleagues needed more information about actual reservoir operations and needed a way to better present their information to make it useful. The HRC had a “technology transfer and science mission”4 at its core. As one PI put it “[We] thought that field application was lagging very significantly in the use of science research results. Our experience was that the main reason is that scientific research results created for ‘test cases’ do not always represent the real world but need additional research and adjustment before they are appropriate for the field. This requires cooperation.”5
b. Building the knowledge network to increase legitimacy: 2001–03
The HRC facilitated discussions with stakeholders—including potential end users, like DWR and Reclamation, and interested government staff, knowledgeable about climate and hydrologic modeling. Those discussions expanded the knowledge network, helped to translate early findings, and provided new information to the PIs. The boundary organization served as the main convener, drawing in outside “consumers” or potential “end users” who began to form a network—much like the conceptual model of networks and boundary organizations offered by Guido et al. (2016).
The researchers anticipated that in-person presentations would help to secure funding and develop partnerships with reservoir management agencies. The team was concerned that the mathematics developed for the DSS function for reservoir operation would be misunderstood by relevant agencies.6 They also thought that they could get needed hydrologic information by engaging directly with stakeholders who were knowledgeable about existing river operations. When asked to reflect on their approach to collaboration, one PI stated that “[he] can’t say enough about how important it is to know what the local issues are…what their aspirations are.”7 So, the PIs sought out and created opportunities to meet key staff within potential end-user and funding agencies—for example, the California–federal “CALFED” Bay-Delta Program (hereinafter CALFED) and DWR—and communicate the project’s goals and methods. Here the model of network and boundary organization interaction begins to depart from the Guido et al. (2016) model. For example, INFORM researchers spoke at a CALFED Science Forum panel, where HRC staff “presented materials related to new integrative forecast-management capacity related to climate that could be of interest to CALFED, targeting the improved management of water resources at major reservoir sites in California.” (https://www.hrcwater.org/wp-content/uploads/2015/11/AR-01-02.pdf, p. 8). Depending on whether one views the individual researchers as acting as a part of the HRC, this kind of engagement could be viewed as interlinked boundary organizations (Lemos et al. 2014a; Kirchhoff et al. 2015; Kalafatis et al. 2015; Meyer et al. 2015), with CALFED as a boundary organization relying on the HRC for new information.
INFORM researchers also sought feedback from potential end users and opportunities to start collaborations. Researchers also met directly with “several agency representatives in the US…to discuss HRC’s vision for the initiatives and to receive feedback from technical and management staff of the agencies on project goals, objectives, design, data and technical issues” (https://www.hrcwater.org/wp-content/uploads/2015/11/AR-01-02.pdf, 8–9). They also held focused trainings for agency staff. For example, they gave a training “on state estimators suitable for use with operational database and hydrologic models” (https://www.hrcwater.org/wp-content/uploads/2015/11/AR-01-02.pdf, p. 8).
The PIs’ suspicion that their mathematics would be dismissed or misunderstood was not unfounded. The need for translation is often cited in literature on coproduction and adoption of CI for water systems (Gibbons 1999; Rayner et al. 2005; Feldman and Ingram 2009; Bolson and Broad 2013). As with water, it is also true for energy systems that the more complex the problem is, the more communication is needed (e.g., Howarth and Monasterolo 2017). These axioms of coproduction proved true in this phase of INFORM’s development. Translation was problematic for DWR. At the time, CEC was creating a research roadmap for climate change, including water resources.8 The CEC’s Franco saw the abovementioned presentation at the CALFED Forum in 2002.9 At that time, Franco was responsible for overseeing the development of a climate research roadmap for California. The presentation and related publications were sufficient to convince Franco to try to include INFORM in the roadmap but were not enough for DWR. At DWR, M. Roos was responsible for the water section of the climate research roadmap (PIEREA 2003). He was reluctant to support INFORM because he suspected that water managers would not accept the results and because the mathematics were too complex to evaluate.
Translation was not the solution to the communication problem or the mathematics. Instead, the solution was to expand the knowledge network. Franco invited a third party from a well-respected water think tank, the Pacific Institute, to give an opinion on the project to DWR and the INFORM team. Just as trusted individuals were critical to the success of incorporating CI into reservoir systems in the Bolson and Broad (2013) study, here the trust DWR placed in this individual was critical to the development of INFORM. The third party endorsed the project, which gave DWR sufficient assurances to follow suit, and it did (PIEREA 2003). The third party acted as an “information broker” (Buizer and Cash 2016; Dilling and Lemos 2011; Lemos et al. 2012; Guido et al. 2016), able to “share and transform information” (Guido et al. 2016, p. 295). Franco’s efforts to enlist another information broker were not on behalf of a boundary organization. Instead of the top-down model (Guido et al. 2016) or the interlinked model of boundary organizations (e.g., Lemos et al. 2014a), INFORM at this stage was aided by a budding knowledge network, working outside of a boundary organization.
External science reviews scored the application more favorably, finding the project to be “excellent.” External science reviews also noted that there was some uncertainty regarding the outcome of the research but, unlike regional reviewers, did not treat that uncertainty as dispositive. For example, one reviewer wrote thatThe Bay Region did not feel the proposal was very applicable to their region, and had little connection with Bay restoration efforts. The Delta Region saw little linkage to restoration activities, but acknowledged that increased water availability could translate to improved environmental conditions. The Sacramento Region agreed with the conceptual approach of the projects, but thought the project should be done by the agencies involved with managing these reservoirs. Reviewers were concerned with the uncertainty of the outcome of the modeling efforts. (CALFED Review 2002, p. 5).
The reluctance of water management staff to see models and especially CI as useful in the United States has been well documented (Rayner et al. 2005, cf. Lach and Rayner 2017; Rayner 2019). A 2005 study noted that individual characteristics like age, level of tenure within the organization, and education factored in whether staff perceived climate information to be useful to their water management (Rayner et al. 2005). In the case of INFORM, it was not possible to determine these characteristics for all the network members, or even just for the six CALFED reviewers. However, the split between supportive scientists and skeptical management staff was notable.…the challenges will be significant…as with any research project, unknowns abound. …I have some doubts that integrated management will succeed even if the project demonstrates benefits because of institutional and territorial interests. However, if downscaling of [global climate models] and the inclusion of climate indicators is shown to work even on the single reservoir operations, that may cause that technique to be adopted, and that alone could yield major benefits in these very large, multi-year storage reservoirs. (CALFED Review 2002, p. 19).
c. Evolution of procedural rules from required minimums to negotiated agreements: 2003–11
The funding organizations had distinct approaches regarding requirements for engagement. The Public Interest Energy Research (PIER) program, the CEC’s funding mechanism for climate research, was expansive in scope. NOAA funding was targeted, with specific requirements for predetermined needs based on the agency’s mission. Contracts with each of the agencies also set certain procedural requirements for engagement. CEC funding required periodic reviews of the project and a technical advisory committee. NOAA funding likewise required regular review meetings. Both required participation from the relevant project managers in the agencies. Contractual terms provided the de jure rules for minimum acceptable participation, which formed the basis for clearly defined processes, shown to be important for coproduction (Aslin and Blackstock 2010). Rules for coordination within funding contracts do not appear to be derived from legislation; rather, they appear to have been developed by the funding agencies—although none of the interviewees from funding agencies recalled how they were developed.
Formal rules from funding contracts were not the only structures that determined how collaboration proceeded. The researchers also brought their own ideas about participation, as evinced by their presentations to stakeholders, outlining potential roles and responsibilities for participation. There were limits to these rules on engagement. For example, there were no rules for how decisions would be made by the group.
Through dialogue between the researchers and the funding agencies, additional stakeholders joined the collaborative effort, including for example, technical staff from DWR, the California Bay-Delta Authority, and the California–Nevada River Forecast Center (CNRFC) of the National Weather Service (NWS) (Table 4). Altogether, the network developed practices that became de facto rules for engagement—powers of committee members, processes for decision-making, and the ability to nominate and appoint new members. There were no specific rules regarding the powers of such stakeholders, provided by funding contracts. Rather, the de facto roles developed organically during meetings of the OIC.10
OIC oversight and implementation committee meeting participation (available in meeting notes; http://ww.drecp.org/2006publications/CEC-500-2006-109/CEC-500-2006-109-APA.PDF, pp. APA-2, APA-9, APA-14, APA-22, and APA-26). Participants are listed by initials only.
The OIC was a boundary group and was the main formal means of collaboration, comprising individuals from multiple disciplines, including funders, researchers, and end-user agency staff (Guston 2001). It aimed to translate science and agency needs (Feldman and Ingram 2009) and to mediate between salience and credibility (Cash et al. 2003) to coproduce usable and acceptable information (Dilling and Lemos 2011). The OIC also fulfilled the CEC’s requirement for a technical advisory committee. It also allowed researchers to gain information from potential users while providing a forum to educate users.
The PIs tended to be the first movers. They came with specific requests and inquiries for the OIC members: requests for data, verification of assumptions, and open questions about what parameters to include in the model. They also proposed a “protocol for collaboration,” identifying contacts for technical matters for each basin and reservoir from relevant agencies (OIC 1 meeting notes; http://ww.drecp.org/2006publications/CEC-500-2006-109/CEC-500-2006-109-APA.PDF, p. APA-8), which was agreed upon by the other members of the OIC. They also set the initial expectations for the OIC. In their first presentation to the OIC in October of 2003, the PIs outlined roles and duties for the OIC (Table 5).
Roles and duties of the OIC as presented by the INFORM PIs; the presentation is reflected in the meeting minutes (http://ww.drecp.org/2006publications/CEC-500-2006-109/CEC-500-2006-109-APA.PDF, p. APA-6).
Moreover, to facilitate information exchange across the multiagency OIC, the PIs created a “secure web site for the exchange of data and information among OIC members and Co-PIs” (OIC 1 meetings notes; http://ww.drecp.org/2006publications/CEC-500-2006-109/CEC-500-2006-109-APA.PDF, p. APA-4). OIC input changed the DSS, the model, and the research process (Fig. 1).
OIC members raised their own questions and concerns, provided feedback, and helped to expand the membership of the OIC. Here, the formal boundary group operated to expand the network (Guido et al. 2016) and link with other boundary organizations (Lemos et al. 2014a). Even in the early stages in 2003, nonresearch members expanded the membership of the OIC, changed the scope of INFORM to include additional reservoirs, set parameters of the model and DSS, and worked together with the PIs to approach technical experts and practitioners (Fig. 1). Discussions at the first OIC led to agreement that the PIs would seek OIC membership approval to publish any findings (OIC meetings notes; http://ww.drecp.org/2006publications/CEC-500-2006-109/CEC-500-2006-109-APA.PDF, p. APA-4). As the project progressed, OIC nonresearch members took an active role in shaping the research. By 2005 the project was at a critical point, short on time, long on requests, and about to run out of funding; the OIC had expanded the scope previously, and it now helped to focus INFORM given the limitations.
The scope of work changed based on the needs expressed at OIC meetings. The PIs gave targeted trainings to individuals and general trainings to the entire OIC on “climate science, hydrologic science, and decision science.” (OIC meetings notes; http://ww.drecp.org/2006publications/CEC-500-2006-109/CEC-500-2006-109-APA.PDF, p. APA-5). PIs and nonresearch OIC members jointly strategized on how to best model specific reservoirs and regions, who should participate, how to obtain information and inputs for the model, and how to harmonize the INFORM model with those used by reservoir management agencies (OIC 2 meetings notes; http://ww.drecp.org/2006publications/CEC-500-2006-109/CEC-500-2006-109-APA.PDF, APA-1–APA-29).
PIs took action based on what was said at the OIC meetings. Dialogue within the OIC led to changes in the INFORM model, again consistent with the Cerf et al. (2012) dialogical model. For example, the team changed course and diverted the original scope of work based on new information provided by OIC members. In 2004, a branch of NOAA was beginning to undertake modeling of upstream reservoirs to incorporate them into their operations. The OIC suggested that the modeling effort be incorporated, even though upper reservoirs were not part of the initial scope of work (OIC 2 meetings notes; http://ww.drecp.org/2006publications/CEC-500-2006-109/CEC-500-2006-109-APA.PDF, p. APA-10).
PIs also met individually with OIC members and third-party stakeholders. As noted by K. Georgakakos, the discussions with operators allowed the team to incorporate realistic nuances into INFORM. There are “rulebooks” for hydropower operation and irrigation deliveries—with specific requirements for what operational choices can be made under specific circumstances. Besides written rules, there are informal rules and common practices by which operators abide. The researchers, through discussions with operators, codified those informal rules and incorporated them into INFORM.
During this period of collaboration, the PIs wanted to keep outreach limited. They were concerned that if the public, nonexperts, or press became involved, they would misinterpret the project and its specific application of terms, especially “risk.” The Fine (2007) study of meteorologists provides a useful comparison. Modelers of reservoir and hydropower systems, like meteorologists, are engaged in “future work” (Fine 2007, 99–134)—creating simulations and scenarios for better planning. Eliminating the public perception of risk was central to the job of meteorologists; “[m]isfortune”, Fine wrote, “is easier to cope with than ambiguity” (Fine 2007, p. 106). Meteorologists depended on legitimation to satisfy the public’s desire to feel that the meteorologists gave them risk-reducing information (Fine 2007, p. 248). The same imperative to preserve legitimation by limiting behind-the-scenes interaction may apply to the INFORM PIs.
Although the broader public was not part of the INFORM process, a second round of funding from the CEC supported a multiyear demonstration phase of INFORM (Georgakakos et al. 2013). In the demonstration phase the DSS was run alongside actual operations (Georgakakos et al. 2013). That second round did not formally require the continuance of the OIC.
d. The persistence of knowledge networks after the dissolution of boundary groups: 2012–15
Eventually, funding ran out. Apparent interest from agencies waned, even as the staff involved in the OIC remained committed. The formal partnerships—the OIC boundary group—between researchers and stakeholders on the OIC came to an end. Without the formal partnerships, only the knowledge network remained. That network proved critical to securing future funding and implementation of INFORM.
Macro- and mesolevel context (Flagg and Kirchhoff 2018) changed. CALFED had ceased to exist (e.g., Lubell et al. 2013). A report reviewing PIER from the Legislative Analysis Office (LAO) concluded that the research funded by PIER was unlikely to lead to actionable results. Armed with the report, prominent legislators ended the PIER program.11
Meanwhile, INFORM’s knowledge network, bolstered through interaction facilitated by the OIC, persisted. DWR remained interested in finding a way to integrate INFORM into reservoir operations. Despite good demonstration results, DWR too did not have the means to fund changes to INFORM or implementation itself. Informal communication among former members of the OIC and PIs continued. During this time, several of the participants from the OIC looked for opportunities to fund the necessary steps to incorporate INFORM DSS into reservoir operations.
e. Knowledge network facilitates implementation funding: 2015–18
By 2015, years of severe drought created a new political justification to spend funds to update reservoir operations—changing the macrolevel context and creating a new “policy window” (Kingdon 1984). This is consistent with literature suggesting that crises are often critical, though insufficient by themselves, to accelerate changes in resource governance (e.g., Solecki and Michaels 1994; Bolson and Broad 2013; Pulwarty and Sivakumar 2014; Mockrin et al. 2018). During the drought, DWR finally secured funding to integrate INFORM into operations. The funds came from a water bond, passed a decade earlier (Proposition 84; http://bondaccountability.resources.ca.gov/Program.aspx?ProgramPK=10&Program=Flood%20Corridor%20Program&PropositionPK=4; http://bondaccountability.resources.ca.gov/PDF/Prop1E/PROPOSITION_84_fact.pdf). The amount was minimal; DWR triaged, and they limited implementation to the Sacramento River—with the possibility of later expanding and further developing INFORM for the San Joaquin River and the rest of the state.
Although not an energy agency, DWR’s integration of INFORM retained the hydroelectric generation details and output, to share information with its partners (e.g., Reclamation). The inclusive OIC encouraged agencies to work across disparate missions, which otherwise tended to drive their interests in INFORM.
In this last phase the INFORM remained unused for several years, even though the collaborative research shaped the model to be more usable, the demonstration phase of INFORM showed promising results for climate adaptation (A. Georgakakos et al. 2012; K. Georgakakos et al. 2012; Georgakakos et al. 2013), and there was interest in implementing it. Adoption of climate adaptation measures—even if coproduced—is not necessarily speedy, let alone immediate (see Hanemann 2000).
6. Discussion and conclusions
The history shows how the combination of an informal knowledge network alongside formal boundary organizations aided coproduction that created a useful DSS, led to changes in reservoir and hydropower management, and incorporated CI in reservoir governance. The network and the boundary organization influenced INFORM itself (e.g., how the model and DSS worked, what was included, and how it could be used) and the process (e.g., scheduling meetings and trainings, funding, expansion of the network). The network was able to seize on the opportunity to implement the tool. This is a new example of what can be called a bimodal collaboration, in which boundary organizations and knowledge networks strengthen one another, although both do not need to exist simultaneously (Fig. 3). INFORM also offers insights into attributes of coproduction processes. These are discussed below, with special attention to the influence the bimodal collaboration had on INFORM’s successes and failures.
a. Networks can influence rules and roles within formal boundary organizations
Well-defined processes for participation are important for collaboration (e.g., Aslin and Blackstock 2010; Jolibert and Wesselink 2012; Meadow et al. 2015). For INFORM, boundary organizations were used selectively; there was no consistent forum for engagement. There were times when the researchers and network actors collaborated outside of a boundary organization—especially at the beginning of the project and during the phase shortly before implementation. Whether there were clear rules for participation is questionable. There were clear minimum expectations set by the funding agencies (e.g., meetings between the project managers and PIs) but nothing more. Still, those requirements fostered the growth of knowledge networks and person-to-person communication that carried the project through. In addition, the practiced rules for participation changed throughout the life of the project. There were no rules when the DSS was first being developed in the 1990s. Similarly, by the end of the OIC, there were no longer any formal rules for participation. The lack of formal rules does not appear to be positive (facilitating adoption) or negative (constraining adoption) by itself.
From a longitudinal perspective (Furman et al. 2018), INFORM’s process looks more like the “dialogical” design and development of DSS (Cerf et al. 2012) than other participatory modes. Cerf et al. (2012) argue for using a prototype tool to allow for experimental use and debrief. For INFORM, the PIs developed a prototype, which they then modified with the help of the OIC, ran experimental simulations to demonstrate its effectiveness, and then further refined the DSS to fit the needs of end users.
b. Formal organizations can set the minimum bounds of iterative collaboration, but knowledge networks can move past those minimums
Iterativity—ongoing communication with learning and adjustment—is a significant factor in the success of other boundary organizations and knowledge networks. Iteration combined with communication is the main means to produce science that is relevant, reliable, and trusted (Feldman and Ingram 2009). Dilling and Lemos (2011) find that iterativity is a result of someone (person or persons/organization) “owning” or “shepherding” the task of producing usable science; institutional design influences the prevalence and quality of iterativity. Critical of assumptions that connections between users and producers happen automatically, Dilling and Lemos (2011) argue for deliberate and ongoing efforts to connect the two.
INFORM relied on iterative collaboration. Iterativity was an explicit requirement of the funding contracts with the CEC and CALFED, but that only accounts for three meetings. The PIs and the members of the OIC urged further meetings, workshops, and training throughout the project. In the INFORM case study, funding agencies encouraged collaboration, as in Peterson et al. (2010), but there were active participants within the knowledge network who were committed to repeated engagement and iterative learning, as in Bolson and Broad (2013).
c. Knowledge networks can combine with boundary organizations to overcome some transaction costs
Coproduced research can require considerable time and investment [Lemos et al. 2012; McNie 2013; Lemos et al. 2014a,b; Lemos 2015; Jacobs et al. 2016; Furman et al. 2018; also, see Poteete et al. (2010) regarding multidisciplinary work]. Interlinked boundary organizations (“boundary chains”) can decrease these transaction costs (Lemos et al. 2014a; Kirchhoff et al. 2015; Kalafatis et al. 2015; Meyer et al. 2015). At the same time, the long-term viability of specific boundary chains is questionable; organizations may not be stable, financially viable, or politically acceptable (Meyer et al. 2015). The history of INFORM offers another possible avenue to bolster existing boundary organizations. Knowledge networks can persist beyond the convening of formal boundary organizations, and still act to take advantage of policy windows to implement new climate adaptation tools.
d. Context matters for both knowledge networks and boundary organizations
Coproduction itself is not a guarantee to success (Lövbrand 2011; Meehan et al. 2018; Lemos et al. 2018, 2019). Institutional, organizational, and personal contexts matter significantly (Crane et al. 2010; Peterson et al. 2010; Flagg and Kirchhoff 2018). Salience, credibility, and legitimacy are determined by technical and normative aspects of practitioners and scientists (Crane et al. 2010, p. 56). In water management, “we are likely most effective at bridging the knowledge-use gap when we couple usable knowledge production processes to those contexts where micro [individual], meso [organizational], and macro [political and institutional] factors support use” (Flagg and Kirchhoff 2018, p. 7). The level and scope of participation of potential end users and stakeholders in knowledge production processes depend on sociocultural context (Peterson et al. 2010; Bartels et al. 2013) and the social dynamics of power (e.g., Roncoli 2006; Roncoli et al. 2011). For INFORM, disruption occurred at the mesolevel. CALFED ceased to exist during development. CEC’s ability to fund the project ceased because of legislative changes. The network, however, kept the project going. The process of the OIC, building on the contractual requirements from funding agreements, built a more robust network, able to weather barriers of lack of funding and lost support from the CEC and CALFED.
At the microlevel, tacit knowledge may have played a role in group cohesion (Fine 2007). The individuals who participated in the OIC had high levels of expertise and training. Even those outside the OIC, who were part of the knowledge network, all had at least one advanced degree. OIC members themselves were from large state and federal resource management agencies. Notably, the major concern expressed by M. Roos at DWR prior to funding the project was that the policy-focused reservoir management staff would not accept that a model and DSS for reservoir management were possible. And reviews of CALFED scoring documents from the initial bid show that his assessment was not far off the mark. The scoring documents show that water resources management staff viewed the project as infeasible and/or unlikely to help in reservoir operations decision-making; technical staff, however, viewed the project more favorably. Attributes of the INFORM network are consistent with findings about the attributes of water and energy management agencies that adopt CI (Bolson and Broad 2013; Bolson et al. 2013; Bruno Soares and Dessai 2016).
Although INFORM included significant engagement with potential end users, the collaboration between scientists and agency staff was largely technocratic—strikingly similar to critiques of public participation in IWRM efforts (e.g., Roncoli et al. 2016). There were no representatives from the community that lives in and relies on the watershed. From the beginning, there was doubt that the INFORM project would garner engagement. As the 2002 CALFED initial review panel noted, “This is not the kind of proposal that is likely to draw a lot of public attention or involvement” (CALFED Review 2002, p. 3).
Could public participation have helped production and implementation of INFORM, by potentially creating an earlier policy window? There is a considerable literature on the effectiveness of including public participation in resource management policy issues, which suggests that possibility (e.g., Beierle and Cayford 2010; Roncoli et al. 2016; Falconi and Palmer 2017). Yet, as Meadow et al. (2015) point out, the level and mode of participation need to fit the resources and purpose of engagement (see also van Kerkoff and Lebel 2006; Engle and Lemos 2010; Jacobs et al. 2016) and sociocultural context of stakeholders (Peterson et al. 2010; Bartels et al. 2013). Future work on coproduction and DSS for water and energy resources should consider whether intuitional requirements for broad versus targeted participation in coproduction facilitates or hinders development of useful climate adaptation governance tools.
7. Conclusions
The history of INFORM demonstrates that 1) informal knowledge networks can outlast formal boundary organizations, 2) boundary organizations and funding agencies can help to build robust networks, and 3) networks and boundary organizations are important to the development of a usable and useful climate adaptation tool, although 4) all of the above do not necessarily need to exist simultaneously. The history also reaffirms a more basic lesson: coequal status of researchers and community is important for coproduced science. Coequal status is frequently overlooked in examples of processes for coproduction; rather, there is a tendency to stress the importance of the product (e.g., Dilling and Lemos 2011). There is good reason for this: it is imperative to adapt to and mitigate climate change and to do so quickly. There is a need for science to inform policy in this matter. However, in an era of “alternative facts” and changes in the role of federal scientists, there is also now an additional concern about the place of experts, scientists, and science in society (Toumey 2017; Iyengar and Massey 2019). A less-emphasized benefit to coproduced research is that it may help to keep scientists and researchers employed and funded.12 Building knowledge networks that outlast formal processes can open doors for further research and support. What the history of INFORM offers is a guide to recognizing that the researchers need the community. It was clear that everyone had a stake in the process. What is needed for effective coproduction is not just clearly defined rules or an iterative framework for communication, but first to come to the table as equal partners. The researcher needs the community as much as the community needs the researcher.
Acknowledgments
The author thanks the three anonymous reviewers as well as Helen Ingram, Carl Bauer, Edella Schlager, Diana Liverman, and Surabhi Karambelkar for their comments and suggestions on earlier drafts.
REFERENCES
Aslin, H. J., and K. L. Blackstock, 2010: ‘Now I’m not an expert in anything’: Challenges in undertaking transdisciplinary inquiries across the social and biophysical sciences. Tackling Wicked Problems through the Transdisciplinary Imagination, V. A. Brown, J. A. Harris, and J. Y. Russell, Eds., Earthscan, 117–129.
Bartels, W.-L., and Coauthors, 2013: Warming up to climate change: A participatory approach to engaging with agricultural stakeholders in the southeast US. Reg. Environ. Change, 13, 45–55, https://doi.org/10.1007/s10113-012-0371-9.
Bauer, C., 2004: Siren Song: Chilean Water Law as a Model for International Reform. RFF Press, 173 pp.
Bedsworth, L., D. Cayan, G. Franco, L. Fisher, S. Ziaja, 2018: Statewide summary report. California’s Fourth Climate Change Assessment Rep. SUMCCCA4-2018-013, 133 pp., https://www.energy.ca.gov/sites/default/files/2019-07/Statewide%20Reports-%20SUM-CCCA4-2018-013%20Statewide%20Summary%20Report.pdf.
Beebe, J., 2001: Rapid Assessment Process: An Introduction. Altamira Press, 199 pp.
Beier, P., L. Hansen, L. Helbrecht, and D. Behar, 2016: A how-to guide for coproduction of actionable science. Conserv. Lett., 10, 288–296, https://doi.org/10.1111/conl.12300.
Berk, M. M., J. G. van Minnen, B. Metz, W. Moomaw, M. G. J. den Elzen, D. P. van Vuuren, and J. Gupta, 2002: Climate Options for the Long Term (COOL) global dialogue—Synthesis report. National Institute for Public Health and the Environment (RIVM) Rep. 490200003 2001/2002, 57 pp., https://www.pbl.nl/sites/default/files/cms/publicaties/490200003.pdf.
Bernard, R. H., 1988: Research Methods in Cultural Anthropology. Sage, 824 pp.
Beierle, T., and J. Cayford, 2010: Democracy in Practice: Public Participation in Environmental Decisions. Routledge, 160 pp.
Biggs, S. D., 1989: Resource-poor farmer participation in research: A synthesis of experiences from nine agricultural research systems. International Service for National Agricultural Research Special Series on the Organization and Management of On-Farm Client-Oriented Research Comparative Study Paper 3, 37 pp., http://ebrary.ifpri.org/cdm/ref/collection/p15738coll11/id/92.
Bolson, J., and K. Broad, 2013: Early adoption of climate information: Lessons learned from south Florida water resources management. Wea. Climate Soc., 5, 266–281, https://doi.org/10.1175/WCAS-D-12-00002.1.
Bolson, J., C. Martinez, N. Breuer, P. Srivastava, and P. Knox, 2013: Climate information use among southeast US water managers: Beyond barriers and toward opportunities. Reg. Environ. Change, 13, 141–151, https://doi.org/10.1007/s10113-013-0463-1.
Breuer, N., V. E. Cabrera, K. T. Ingram, K. Broad, and P. E. Hildebrand, 2008: AgClimate: A case study in participatory decision support system development. Climatic Change, 87, 385–403, https://doi.org/10.1007/s10584-007-9323-7.
Breuer, N., C. Fraisse, and P. Hildebrand, 2009: Modeling the pipeline into a loop: The participatory process of developing AgroClimate, a decision support system for climate risk reduction in agriculture. J. Service Climatol., 13, 1–12.
Breuer, N., C. Fraisse, and V. Cabrera, 2010: The Cooperative Extension Service as boundary organization for diffusion of climate forecasts: A 5-year study. J. Ext., 48 (4), https://www.joe.org/joe/2010august/rb7.php.
Bruno Soares, M., and S. Dessai, 2016: Barriers and enablers to the use of seasonal climate forecasts amongst organizations in Europe. Climatic Change, 137, 89–103, https://doi.org/10.1007/s10584-016-1671-8.
Buizer, J. K. J., and D. Cash, 2016: Making short-term climate forecasts useful: Linking science and action. Proc. Natl. Acad. Sci. USA, 113, 4597–4602, https://doi.org/10.1073/pnas.0900518107.
CALFED Review, 2002: CALFED Bay-Delta, Proposal Reviews, #85: INFORM—Integrated Forecast and Reservoir Management demonstration for northern California water resources. CALFED Doc., 44 pp., https://nrm.dfg.ca.gov/FileHandler.ashx?DocumentID=5600.
Callahan, B., E. Miles, and D. Fluharty, 1999: Policy implications of climate forecasts for water resources management in the Pacific Northwest. Policy Sci., 32, 269–293, https://doi.org/10.1023/A:1004604805647.
Carberry, P. S., and Coauthors, 2002: The FARMSCAPE approach to decision support: Farmers’, advisers’, researchers’ monitoring, simulation, communication and performance evaluation. Agric. Syst., 74, 141–177, https://doi.org/10.1016/S0308-521X(02)00025-2.
Carbone, G. J., J. Rhee, H. P. Mizell, and R. Boyles, 2008: A regional-scale drought monitoring tool for the Carolinas. Bull. Amer. Meteor. Soc., 89, 20–28.
Carpenter, T. M., and K. P. Georgakakos, 2001: Assessment of Folsom Lake response to historical and potential future climate scenarios: 1. Forecasting. J. Hydrol., 249, 148–175, https://doi.org/10.1016/S0022-1694(01)00417-6.
Cash, D. W., W. C. Clark, F. Alcock, N. M. Dickson, N. Eckley, D. H. Guston, J. Jäger, and R. B. Mitchell, 2003: Knowledge systems for sustainable development. Proc. Natl. Acad. Sci. USA, 100, 8086–8091, https://doi.org/10.1073/pnas.1231332100.
Cash, D. W., J. C. Borck, and A. G. Patt, 2006: Countering the loading-dock approach to linking science and decision making—Comparative analysis of El Niño/Southern Oscillation (ENSO) forecasting systems. Sci. Technol. Human Values, 31, 465–494, https://doi.org/10.1177/0162243906287547.
CCSP, 2008: Decision-Support Experiments and Evaluations using Seasonal-to-Interannual Forecasts and Observational Data: A Focus on Water Resources. Synthesis and Assessment Product 5.3, N. Beller-Simms et al., Eds., U.S. Climate Change Science Program and the Subcommittee on Global Change Research, 190 pp., https://downloads.globalchange.gov/sap/sap5-3/sap5-3-final-all.pdf.
CEC, 2017: 2016 Integrated Energy Policy Report Update. California Energy Commission. Publication Number: CEC-100-2016-003-CMF, 197 pp., http://docketpublic.energy.ca.gov/PublicDocuments/16-IEPR-01/TN216281_20170228T131538_Final_2016_Integrated_Energy_Policy_Report_Update_Complete_Repo.pdf.
Cerf, M., M. H. Jeuffroy, L. Prost, and J. M. Meynard, 2012: Participatory design of agricultural decision support tools: Taking account of the use situations. Agron. Sustain. Dev., 32, 899–910, https://doi.org/10.1007/s13593-012-0091-z.
Climate Group, 2015: Baden-Wurttenberg and California welcome 43 new signatories to the Under 2 MOU initiative. Accessed 15 May 2019, https://www.theclimategroup.org/news/baden-wurttemberg-and-california-welcome-43-new-signatories-under-2-mou-initiative.
CNRA, 2017: Draft Report: Safeguarding California Plan: 2017 Update— California’s Climate Adaptation Strategy. California Natural Resources Agency, 175 pp., http://resources.ca.gov/wp-content/uploads/2017/05/DRAFT-Safeguarding-California-Plan-2017-Update.pdf.
Crane, T., C. Roncoli, J. Paz, N. Breuer, K. Broad, K. T. Ingram, and G. Hoogenboom, 2010: Forecast skill and farmers’ skills: Seasonal climate forecasts and agricultural risk management in the southeastern United States. Wea. Climate Soc., 2, 44–59, https://doi.org/10.1175/2009WCAS1006.1.
Dewulf, A., M. Brugnach, C. Termeer, and H. Ingram, 2013: Bridging knowledge frames and networks in climate and water governance. Water Governance as Connective Capacity, J. Edelenbos, N. Bressers, and P. Scholten, Eds., Interscience, 229–247.
Dietz, T., E. Ostrom, and P. Stern, 2003: The struggle to govern the commons. Science, 302, 1907–1912, https://doi.org/10.1126/science.1091015.
Dilling, L., and M. C. Lemos, 2011: Creating usable science: Opportunities and constraints for climate knowledge use and their implications for science policy. Global Environ. Change, 21, 680–689, https://doi.org/10.1016/j.gloenvcha.2010.11.006.
Dilling, L., and Coauthors, 2015: What stakeholder needs tell us about enabling adaptive capacity: The intersection of context and information provision across regions in the United States. Wea. Climate Soc., 7, 5–17, https://doi.org/10.1175/WCAS-D-14-00001.1.
Doremus, H., 2009: CALFED and the quest for optimal institutional fragmentation. Environ. Sci. Policy, 12, 729–732, https://doi.org/10.1016/j.envsci.2009.06.004.
Doremus, H., and A. Tarlock, 2008: Water War in the Klamath Basin: Macho Law, Combat Biology, and Dirty Politics. Island Press, 280 pp.
Dutterer, A. D., and R. D. Margerum, 2015: The limitations of policy-level collaboration: A meta-analysis of CALFED. Soc. Nat. Resour., 28, 21–37, https://doi.org/10.1080/08941920.2014.945054.
Engle, N. L., and M. C. Lemos, 2010: Unpacking governance: Building adaptive capacity to climate change of river basins in Brazil. Global Environ. Change, 20, 4–13, https://doi.org/10.1016/j.gloenvcha.2009.07.001.
Etkin, D., and Coauthors, 2015: Stochastic linear programming for improved reservoir operations for multiple objectives in Burkina Faso. J. Water Resour. Plann. Manage., 141 (3), 04014056, https://doi.org/10.1061/(ASCE)WR.1943-5452.0000396.
Falconi, S., and R. Palmer, 2017: An interdisciplinary framework for participatory modeling design and evaluation—What makes models effective participatory tools? Water Resour. Res., 53, 1625–1646, https://doi.org/10.1002/2016WR019373.
Feldman, D. L., and H. Ingram, 2009: Making science useful to decision makers: Climate forecasts, water management, and knowledge networks. Wea. Climate Soc., 1, 9–21, https://doi.org/10.1175/2009WCAS1007.1.
Feldman, D. L., and Coauthors, 2008: Decision-support experiments within the water resource management sector. Decision-Support Experiments and Evaluations using Seasonal-to-Interannual Forecasts and Observational Data: A Focus on Water Resources. Synthesis and Assessment Product 5.3, N. Beller-Simms et al., Eds., U.S. Climate Change Science Program and the Subcommittee on Global Change Research, 65–100, https://downloads.globalchange.gov/sap/sap5-3/sap5-3-final-all.pdf.
Fine, G. A., 2007: Authors of the Storm: Meteorologists and the Culture of Prediction. University of Chicago Press, 280 pp.
Flagg, J., and C. Kirchhoff, 2018: Context matters: Context-related drivers and barriers to climate information use. Climate Risk Manage., 20, 1–10, https://doi.org/10.1016/j.crm.2018.01.003.
Fraisse, C. W., and Coauthors, 2006: AgClimate: A climate forecast information system for agricultural risk management in the southeastern USA. Comput. Electron. Agric., 53, 13–27, https://doi.org/10.1016/j.compag.2006.03.002.
Furman, C., W. Bartels, and J. Bolson, 2018: Participation, process, and partnerships: Climate change and long-term stakeholder engagement. Anthropol. Action, 25, 1–12, https://doi.org/10.3167/aia.2018.250301.
Garfin, G. M., 2006: Arizona drought monitoring. Extended Abstracts, 2006 North American Drought Monitor Workshop, Mexico City, Mexico, NOAA, https://www.climas.arizona.edu/sites/default/files/pdfgarfin-nadm-ext-abstract-2006.pdf.
Garfin, G. M., and Coauthors, 2008: Regional knowledge transfers: Coping with drought through collaboration. Extended Abstracts, 17th Conf. on Applied Climatology, Whistler, BC, Canada, Amer. Meteor. Soc, 7.3, https://ams.confex.com/ams/13MontMet17AP/webprogram/Paper141284.html.
Georgakakos, A. P., H. Yao, M. Kistenmacher, K. P. Georgakakos, N. E. Graham, F. Y. Cheng, C. Spencer, and E. Shamir, 2012: Value of adaptive water resources management in Northern California under climatic variability and change: Reservoir management. J. Hydrol., 412-413, 34–36, https://doi.org/10.1016/j.jhydrol.2011.04.038.
Georgakakos, K. P., and N. E. Graham, 2008: Potential benefits of seasonal inflow prediction uncertainty for reservoir release decisions. J. Appl. Meteor. Climatol., 47, 1297–1321, https://doi.org/10.1175/2007JAMC1671.1.
Georgakakos, K. P., N. E. Graham, T. M. Carpenter, A. P. Georgakakos, and H. Yao, 2005: Integrating climate-hydrology forecasts and multi-objective reservoir management for Northern California. Eos, Trans. Amer. Geophys. Union, 86, 122–127, https://doi.org/10.1029/2005EO120002.
Georgakakos, K. P., N. E. Graham, and A. P. Georgakakos, 2007: Integrated Forecast and Reservoir Management (INFORM) for northern California: System development and initial demonstration. Hydrologic Research Center–Georgia Water Resources Institute California Energy Commission Publ. CEC-500-2006-109, 244 pp., https://ww2.energy.ca.gov/2006publications/CEC-500-2006-109/CEC-500-2006-109.PDF.
Georgakakos, K. P., N. E. Graham, F. Y. Cheng, C. Spencer, E. Shamir, A. P. Georgakakos, H. Yao, and M. Kristenmacher, 2012: Value of adaptive water resources management in northern California under climatic variability and change: Dynamic hydroclimatology. J. Hydrol., 412–413, 47–65, https://doi.org/10.1016/j.jhydrol.2011.04.032.
Georgakakos, K. P., and Coauthors, 2013: Integrated Forecast and Reservoir Management (INFORM): Enhancements and demonstration results for northern California (2008–2012). Hydrologic Research Center–Georgia Water Resources Institute California Energy Commission Publ. CEC-500-2014-019, 529 pp., https://ww2.energy.ca.gov/2014publications/CEC-500-2014-019/CEC-500-2014-019.pdf.
Georgakakos, K. P., T. M. Modrick, J. A. Sperfslage, C. R. Spencer, R. J. Banks, A. P. Georgakokos, M. Kistenmacher, and X. Liu, 2018: Integrated Forecast and Reservoir Management (INFORM): Implementation of a Stand-Alone Operational INFORM System for the California Department of Water Resources (DWR). Hydrologic Research Center–Georgia Water Resources Institute Final Rep., 123 pp., https://www.hrcwater.org/wp-content/uploads/2018/08/HRC-GWRI_DWRINFORM_FINALREPORT_20180627.pdf.
Gibbons, M., 1999: Science’s new social contract with society. Nature, 402, C81–C84, https://doi.org/10.1038/35011576.
Gleick, P., 2015: Impacts of California’s ongoing drought: Hydroelectric generation. Pacific Institute Rep., 13 pp., http://pacinst.org/publication/impacts-of-californias-ongoing-drought-hydroelectricity-generation/.
Gordon E., and Coauthors, 2016: Navigating scales of knowledge and decision-making in the Intermountain West: Implications for science policy. Climate in Context: Science and Society Partnering for Adaptation, 1st ed. A. Parris et al. Eds., John Wiley and Sons, 235–253.
Graham, N. E., and K. P. Georgakakos, 2010: Toward understanding the value of climate information for multiobjective reservoir management under present and future climate and demand scenarios. J. Appl. Meteor. Climatol., 49, 557–573, https://doi.org/10.1175/2009JAMC2135.1.
Greenwood, D. J., and M. Levin, 2007: Introduction to Action Research: Social Research for Social Change. 2nd ed. Sage, 301 pp.
Guido, Z., V. Rountree, C. Greene, A. Gerlak, and A. Trotman, 2016: Connecting climate information producers and users: Boundary organization, knowledge networks, and information brokers at Caribbean climate outlook forums. Wea. Climate Soc., 8, 285–298, https://doi.org/10.1175/WCAS-D-15-0076.1.
Guston, D. H., 2001: Boundary organizations in environmental policy and science: An introduction. Sci. Technol. Human Values, 26, 399–408, https://doi.org/10.1177/016224390102600401.
Hanemann, W. M., 2000: Adaptation and its measurement—An editorial comment. Climatic Change, 13, 571–581.
Hanemann, W. M., 2006: The economic conception of water. Water Crisis: Myth or Reality? P. Rogers, R. Llamas, and L. Martinez-Cortina, Eds., Taylor and Francis, 61–91.
Howarth, C., and I. Monasterolo, 2017: Opportunities for knowledge co-production across the energy–food–water nexus: Making interdisciplinary approaches work for better climate decision making. Environ. Sci. Policy, 75, 103–110, https://doi.org/10.1016/j.envsci.2017.05.019.
Ingram, H., 2006: Water as a multi-dimensional value: Implications for participation and transparency. Int. Environ. Agreement Polit. Law Econ., 6, 429–433, https://doi.org/10.1007/s10784-006-9021-3.
Ingram, H., D. Feldman, N. Mantua, K. L. Jacobs, A. M. Waple, and N. Beller-Simms, 2008a: Executive summary. Decision-Support Experiments and Evaluations using Seasonal-to-Interannual Forecasts and Observational Data: A Focus on Water Resources. Synthesis and Assessment Product 5.3, N. Beller-Simms et al., Eds., U.S. Climate Change Science Program and the Subcommittee on Global Change Research, 1–6, https://downloads.globalchange.gov/sap/sap5-3/sap5-3-final-all.pdf.
Ingram, H., D. Feldman, N. Mantua, K. L. Jacobs, D. Fort, N. Beller-Simms, and A. M. Waple, 2008b: The changing context. Decision-Support Experiments and Evaluations using Seasonal-to-Interannual Forecasts and Observational Data: A Focus on Water Resources. Synthesis and Assessment Product 5.3, N. Beller-Simms et al., Eds., U.S. Climate Change Science Program and the Subcommittee on Global Change Research, 7–28, https://downloads.globalchange.gov/sap/sap5-3/sap5-3-final-all.pdf.
Iyengar, S., and D. Massey, 2019: Scientific communication in a post-truth society. Proc. Natl. Acad. Sci. USA, 116, 7656–7661, https://doi.org/10.1073/pnas.1805868115.
Jacobs, K., S. N. Luoma, and K. A. Taylor, 2003: CALFED: An experiment in science and decisionmaking. Environment, 45, 30–41, https://doi.org/10.1080/00139150309604521.
Jacobs, K., G. Garfin, and M. Lenart, 2005: More than just talk: Connecting science and decisionmaking. Environment, 47, 6–21, https://doi.org/10.3200/ENVT.47.9.6-21.
Jacobs, K., and Coauthors, 2016: Linking knowledge with action in the pursuit of sustainable water resources management. Proc. Natl. Acad. Sci. USA, 113, 4591–4596, https://doi.org/10.1073/pnas.0813125107.
Jahn, T., M. Bergmann, and F. Keil, 2012: Transdisciplinarity: Between mainstreaming and marginalization. Ecol. Econ.,