1. Introduction
Scientists have declared with widespread consensus that our global climate is changing, getting warmer, and causing widespread perturbations in ecosystems around the world (Alley et al. 2007). The impacts of climate variability and change vary widely by geographic, temporal, economic, social, and cultural scales (Alley et al. 2007), and responding to such variability requires policy options that expand alternatives and clarify choices for decision makers at the relevant scale and context (Pielke 2007; McNie et al. 2007). Our current efforts in producing useful information to address climate adaptation and other coupled human–environmental problems are inadequate, and we need to improve our abilities to produce useful information for decision support (NRC 2012, 2009a,b, 2007a,b, 2005a,b, 1999; CCSP 2008; Committee on Science 2002; Mayden 2002; Ehlers 1998).
At present, producing more useful information often involves increasing research funding and producing more of the same kind of information, thus increasing the overall supply of information. Yet rarely does this unilateral supply-side push result in more useful information, and it often lacks any relevance to what users actually need (Lahsen and Nobre 2007; Sarewitz and Pielke 2007; Dilling and Lemos 2011), or they may be unaware of potentially useful information that is already available in the public domain (NRC 1999; Stone et al. 2001; NRC 2007b, 2009b). Even when researchers strive to produce information that users need—and have the ability to produce it—significant barriers remain, effectively blocking the ability to link the supply of scientific information with users’ demands. Barriers include cultural differences between science and society (e.g., communication styles, epistemologies, delays between when science is produced versus when it is needed by users), policies that favor “pure basic research” over “use-inspired research” (Stokes 1997), political and power asymmetries between scientists and users (Escobar 1994), differences in what constitutes expertise, and other barriers (McNie 2007).
Understanding how to provide effective decision support for climate adaptation remains a key goal in global climate-change research (NRC 2012, 2010; National Climate Assessment 2011; NRC 2009b; Alley et al. 2007). Many gaps in our knowledge and understanding of use-inspired climate research and how it informs policy decisions remain including understanding the formal and informal linkages between scientists and decision makers necessary to enhance information utility (NRC 2005b; McNie et al. 2007) and how scientists and researchers decide what use-inspired research to undertake (Pyke et al. 2007).
This paper presents one element of a multifaceted research project called Science Policy Assessment and Research on Climate that was funded as a Decision Making Under Uncertainty program sponsored by the National Science Foundation. Broadly construed, this research examined how scientists and researchers prioritized research agendas, engaged with stakeholders, packaged or promoted climate-science information, supported stakeholder decision making, ensured decision makers understood and could integrate the information into existing decision frameworks, and evaluated the researchers’ own efforts to produce useful information for decision support. Based on these overarching themes, this paper identifies specific processes and approaches utilized by climate-service organizations to produce useful information for decision support.
a. Methodology
This project examined programs managed through the Climate Program Office (CPO) of the National Oceanic and Atmospheric Administration (NOAA) called the Regional Integrated Sciences and Assessments (RISA) programs. The aim of the program is to provide integrated scientific assessments for place-based decision support related to climate variability for a variety of users and decision makers (henceforth called “stakeholders”) who work in areas of water management, public health, fisheries, hydropower production, forestry, natural hazards, agriculture, natural resources management, and other areas. RISA research involves a variety of disciplines including climate, terrestrial ecosystem, hydrological, agricultural, civil engineering, economic, and human dimensions research undertaken by scientists and researchers (henceforth called “members”). Every RISA program shares the aforementioned goals, yet each was designed individually with its own set of priorities, research agendas, and stakeholders. Studying RISA programs offer ample opportunities to conduct robust case and cross-case comparisons between programs given similar institutional constraints but different context-sensitive approaches to fulfilling the RISA mission.
This research utilized a small-n, controlled-case comparison approach in which dependent variables between cases are homogeneous (George and Bennett 2005)—that is, whether RISA members produced climate science information that stakeholders found useful. Utilizing process-tracing techniques in a small-n study has obvious advantages in that the findings mapped out “the pathways linking the independent variables to the dependent variable” (Sambanis 2004, p. 263) and identified new variables and hypotheses (George and Bennett 2005) to characterize necessary causes and outcomes (Bennett and Elman 2006). These results contribute much-needed empirical data to research on decision support for climate adaptation (Pyke et al. 2007) and refine our understanding of the processes necessary and/or sufficient to produce useful climate science information.
Research began with extensive literature reviews, archival research, a workshop with representatives from every RISA program (in existence at that time), semistructured interviews with RISA members (N = 22) and stakeholders (N = 23), and in-depth case studies of each of the following three RISA programs: the Climate Impacts Group (CIG) at the University of Washington; Climate Assessment of the Southwest (CLIMAS) at the University of Arizona; and the Pacific RISA at the East–West Center in Hawaii (see Table 1 for details on each RISA’s program activities). These three programs had significant experience, consisting of the first and oldest programs (two out of three were 10 yr old at the time) and so had substantial experience and time in which to “fine tune” their approaches and methods.
Stakeholders were queried about what RISA-sourced information they found useful in satisfying various value demands and what was not useful, as were members whose responses were highly correlated with the stakeholders’ responses. Both groups were also queried about the engagement processes and approaches that contributed to the coproduction of useful information. Stakeholders were asked about their climate information needs and unmet needs. Members were asked about how they designed their research agendas, promoted and packaged information, and ultimately shared climate knowledge with stakeholders.
b. Defining useful information
Despite a chorus calling for more useful climate information, neither scientists nor policy makers have arrived at a consensus regarding what constitutes useful information or policy-relevant science (Dilling and Lemos 2011). For this research project “useful information” was defined as fulfilling stakeholders’ value demands in which “values” consist of a desired situation, object, or condition (Clark 2002; Lasswell 1971; Lasswell and McDougal 1992), including power, enlightenment, wealth, well-being, skill, affection, respect, and rectitude (see Table 2 for more detailed descriptions). For example, climate information educates and informs decision makers who learn about how climate change may impact their sector (enlightenment), results in opportunities to reduce costs or avoid human losses (wealth, well-being), informs the design of models that clarify options for optimizing water system efficiency (skill), and supports previous decisions or justifies current policy preferences (power). Coproduction of climate information enhances trust and increases social capital (affection), enhances the linkages between members and stakeholders, and improves the efficiency of the knowledge system (McNie 2008). Knowledge may also lead to moral justification and sense of responsibility for specific actions or problems (rectitude). Climate information can satisfy multiple value demands simultaneously and consists of both content and process (McNie 2007; Haas 2004).
Identifying specific value demands that were satisfied helped to describe how information was useful, but three additional criteria helped clarify the conditions about why the information was useful. Generally speaking, useful information met three criteria that shaped its utility: salience, describing how information was context sensitive and relevant to the appropriate temporal and spatial scale; credibility, describing the quality, accuracy, and validity of the information; and legitimacy, describing stakeholders’ belief that the information was produced by trusted sources that had not been captured by special interests (Cash and Buizer 2005; Cash and Clark 2001; Cash et al. 2002, 2003; Guston 2001). Useful information also had a procedural dimension, providing “a mechanism for transmitting knowledge from the scientific community to the policy world” (Haas 2004, p. 573) effectively linking members and stakeholders together with sustained, iterative, and frequent interactions that improved the likelihood of producing useful information (CCSP 2008; Lemos and Morehouse 2005; McNie 2007).
c. Evaluating useful information
Evaluating the usefulness of climate information for decision support is challenging because the same information could be used differently by different users. Measuring outcomes of decisions is difficult with the long temporal delays inherent in changes to social and environmental systems; and identifying causal links from a single unit of information to a decision is unrealistic given that other social, economic, political, and scientific inputs also inform decisions. While notable accomplishments in evaluating forecasting skill with seasonal and interannual forecasts exist (CCSP 2008), less is known about how such information is useful in the context of actual policy decisions. Some suggest using “internal criteria” to evaluate environment decisions rather than relying solely on outcomes (NRC 2005a; Beierle and Cayford 2002; Shaw et al. 1997), or identifying and qualifying users’ satisfaction with the information with “ex-post satisficing” or “ex-post evaluation” (Deelstra et al. 2003; Herrick and Sarewitz 2000).
d. Research design limitations
The research design used in this project resulted in some limitations. Identifying stakeholders was somewhat problematic given the researcher’s reliance on each RISA program to provide stakeholder contact information, although snowball-sampling techniques were used to mitigate some of the potential bias (Babbie 2004). Focusing on the dependent variable (the production of useful climate science) and not the failure to produce useful climate science led to some missed opportunities to provide more context and detail in the findings. Relying on qualitative methods alone enhanced the richness in understanding some of the data but also reduced opportunities to analyze and compare outputs and products in other ways. Finally, assessing the production of useful information as broadly construed, in lieu of tracing individual and specific knowledge outputs in detail from conception to production to dissemination to specific policy outcomes, limited our understanding of how information was transformed at each step in the “end-to-end” research process.
2. Evaluating the production of useful information
This section identifies and describes climate information produced by members that satisfied various value demands according to the stakeholders. The useful information is analyzed using the criteria of salience, credibility, and legitimacy described above.
a. Useful information produced
1) Salience of information
Useful products included technical reports explaining likely impacts of short-term and long-term climate variability and change, paleo-climate histories, drought predictions, climate-forecast evaluation tools, streamflow and reservoir forecasts, vulnerability assessments, monthly or quarterly newsletters reporting on the latest climate and resource information, climate fact sheets, and guides for incorporating climate information into government policymaking. Many of the useful products placed climate-change variability into the geographic, political, and economic contexts of the regions represented by each RISA. This contextualization served an important role in educating stakeholders about the importance of considering climate in place-based policy decisions. Some products were useful because they provided enough lead time for decisions related to extreme events to be made, such as in the Pacific where large geographic distances between islands and mainland suppliers, coupled with high transportation costs, necessitated early and convincing communication of hazards such as impending drought coupled with the El Niño–Southern Oscillation (ENSO). Useful information also spanned spatial and temporal scales, from extreme weather forecasts (several days) through monthly, seasonal, and annual streamflow and climate forecasts (1 to 18 months) to longer-range climate predictions (from 10 to 100 yr).
Many of the useful “products,” however, were actually events, in the form of workshops, meetings, conferences, and one-on-one (or one-on-few) meetings between members and stakeholders. While most of these informal meetings were never archived or listed as products of RISA activities, they nonetheless were significant activities both for the conveyance of information and for building social capital and capacity for future integration of climate information. Indeed, much of the Pacific RISA’s work can be categorized as these events. Pacific RISA members provided important leadership and support with the Pacific ENSO Application Center and in the Pacific Risk Managers ′Ohana (PRiMO), and created workshops to train and educate stakeholders about climate change impacts and vulnerabilities. With CIG, one of the watershed moments in its history was a workshop called The Future Ain’t What It Used To Be, which included over 700 participants (400 more than expected), marking a clear acknowledgment from the stakeholder community that they needed and valued the climate information provided.
2) Credibility
RISAs addressed the question of credibility in their information through a variety of means. Information that the RISAs shared was based on peer-reviewed research conducted by the RISA or by other scientists, or the information was produced through similarly rigorous standards, particularly in the CLIMAS and CIG programs that were university-based. The Pacific RISA also utilized peer-reviewed research of their own but, given its limited resources, also relied heavily on the research of others, particularly the Pacific ENSO Application Center, National Climate Data Center (NCDC), and other research produced at the University of Hawaii, Manoa. In this regard the Pacific RISA served more as a broker of information, communicating actively between members and stakeholders and mediating the process of knowledge production.
Many stakeholders valued information that was translated from scientific language into user-friendly vernacular, yet also valued having direct access to peer-reviewed literature, or at least knowing that the products they received from the RISAs were based on peer-reviewed research conducted through rigorous scientific methods. In other cases, stakeholders eventually came to trust that the members had adequately vetted the information and judged it to be credible. It appeared that once members earned the trust of their stakeholders through consistent provision of credible and useful information, the stakeholders were less likely to demand evidence of peer-reviewed products and relied on members to “take care of business” in providing credible information. One factor that facilitated the creation and maintenance of credibility was the transparency surrounding source material and peer-reviewed information.
3) Legitimacy
Stakeholders trusted that the RISAs would provide information of the highest quality, and that using RISA information “inoculates us from criticism” from others in their organization or elsewhere. One stakeholder explained, “We’re on the cusp of an era where we have to make really tough decisions about how we respond to climate change. So, we need trusted sources of information to make decisions based on long-term projections. That’s the tension.” Over time, members and the RISA organizations came to be viewed as “honest brokers of information” (Pielke 2007). Some stakeholders did not necessarily know of the RISA organization by name, but spoke of members they trusted for providing useful information. Yet, other stakeholders spoke about the RISAs as trusted organizations, identifying their affiliation with well-respected universities that were believed to be “above the fray” as an important source of legitimacy.
b. Missed opportunities
Every RISA also had clear “misses” when it came to producing information that was not needed, or used by stakeholders, particularly during the early years. Often the problem behind the lack of usefulness was one of scale. For example, research on the Pacific decadal oscillation (PDO) focused on basinwide impacts to salmon fisheries, yet fisheries management focused on rivers, a much smaller scale. Another example involved climate forecasts. Members learned that for forecasts to be useful to their stakeholders they had to “translate” the forecasts into “resource vulnerability and impacts” products relevant to Stakeholder concerns. Another example involved research that linked climate change with streamflow, salmon, and thus hydropower production. While the information was valuable to stakeholders in part, they were unable to fully incorporate the information into their own hydrological models because of specific legal constraints. Through iterative processes of engagement, Members eventually modified the information to fit within the constraints of the stakeholders’ models.
These misses occurred because members unilaterally believed their information would be useful. In some cases, such products represented significant contributions to climate research, yet were not useful in the original form. Overall, identifying misses was difficult given that dissatisfied stakeholders stopped engaging with the RISA and thus were difficult to identify, or because stakeholders who benefitted from their relationship with the RISA tended to forget the misses. Furthermore, what began as a missed opportunity often evolved into useful information through ongoing engagement with stakeholders and by reframing or editing the information, adding more contextual content, or adapting the information into appropriate tools. As one member explained, “Lots of successful things [products] went through this phase.”
Other missed opportunities occurred because RISAs were unable to meet specific stakeholder demands because of lack of fiscal resources, lack of personnel who could do the work, and, more importantly, by current limitations of our knowledge about climate change. For example, at present we are not capable of producing climate forecasts that can predict seasonal rainfall intensity, produce fairly accurate longer-range climate forecasts, or downscale climate models to a resolution fine enough to inform decision makers about more localized problems. Some stakeholders mentioned needing longer lead time for ENSO-related events, improved identification of the different “flavors” of El Niño, and sea level-variation models. One member explained that stakeholders wanted climate information scaled down “to x forest or y watershed” but that the science just “wasn’t there” yet. In one case, civil engineers who were in the process of designing future sewage and storm-drain systems wanted predictions about likely changes in 2- to 3-day severe-storm events and how these events would vary from the current average. Members were unable to provide that information, but instead spent time educating the engineers about constraints in climate and precipitation predictions.
Other stakeholders wanted information with uncertainty levels significantly lower than what science could produce. Members, unable to fill those needs, instead focused on other ways to increase the stakeholders’ comfort with the available information and how it could be useful, or provided additional information as a “suite” of data. Problems such as reducing uncertainty suggest that cultural differences between science and policy were partly to blame in these missed opportunities. Still in other cases members complained about the lack of sophisticated satellite monitoring systems as a major barrier to providing stakeholders with relevant information.
3. Processes and approaches
This section describes the necessary and most important processes and approaches that contributed to the coproduction of useful climate information. These include assessing users’ needs; translating, communicating, and sharing knowledge; supporting the production of social capital; capacity building; and leadership and organizational design. This section concludes with a discussion of important similarities and differences in approaches utilized by the three RISA programs.
a. Assessing users’ information needs
Research indicates that organizations and individuals are more likely to use information that they have requested (Oh 1996; Oh and Rich 1996; Landry et al. 2003). RISAs determined what information to produce from both formal and informal needs assessments. At the formal end, robust social science methods were deployed to determine users’ needs through focus groups, survey instruments, and individual stakeholder queries. Less formally, some RISAs took advantage of opportunities when stakeholders were convened to pass out questionnaires and informal needs assessments. For one RISA, such questionnaires were a “part of every workshop” they held. Many members learned about users’ needs, and thus priorities for their own research agendas, through informal, frequent, and direct personal contact with stakeholders. These informal and iterative conversations enabled Members to fine tune their research agendas to better address stakeholders’ needs. One RISA identified “big issues” or problems shared by many stakeholders or across sectors in order to guide research. At other times, members asked simply, “can they [the stakeholders] use this information to defend their decision to their bosses?”
Other external factors shaped research agendas such as availability of fiscal resources, including both the lack of resources and external funding opportunities directed toward specific research. A lack of resources was particularly germane with the Pacific RISA, which operated with a very small budget, significantly constraining choices and alternatives for programmatic activities. The limited budgets compelled many members to seek out research activities with the highest potential “bang for the buck” and opportunities to collaborate with other agencies. Sometimes RISAs were able to leverage external funding (e.g., state monies directed toward climate adaptation research). In this respect the RISAs operated as quasi-consulting organizations. Another important factor shaping research agendas was serendipity and the random occurrence of various clarifying events and policy windows of opportunity (Kingdon 1997). One such event was the El Niño of 1997–98, which elevated problems of climate variability into the public discourse. Another was the Southwest drought of 2002–05, which sparked interest among stakeholders to seek out climate information relevant for their sector and region. As with most research, members’ curiosity about particular problems also influenced decisions, as well as the likelihood for “high impact” results within their scientific-peer community.
One factor that appeared consistent across the RISAs was the importance of a clear vision and mission guiding members’ decisions about research priorities. Understanding the “big picture” and the overarching purpose of each RISA guided research agendas. The CIG used a five-year plan to guide research and held quarterly meetings with all members. Regular principal investigator (PI) meetings served as additional opportunities to fine tune overarching goals yet also allowed for enough autonomy for members to pursue their own research interests. CLIMAS utilized an executive committee consisting of senior PIs that was developed to resolve any disputes about research priorities, should they arise, and to ensure that research agendas continued to match the core questions of the RISA program. At the time of the research, however, it was unclear what affect this committee had on resolving research agendas.
b. Translating, communicating, and sharing knowledge
RISA members determined how best to package and translate information through both formal and informal mechanisms. Formally, some RISAs utilized social science research in order to answer fundamental questions about how stakeholders perceived information and how best to package and present the information. These efforts consisted of empirical studies, experiments, workshops, and survey instruments. CLIMAS implemented one project by which they queried stakeholders on a monthly basis for an entire year in order to assess how they wanted the climate information packaged, presented, and laid out, including vernacular preferences. Other data indicated that multiple presentations were often useful with a single product to address different learning styles and levels of understanding in order to reach the widest audience.
RISAs learned that translating scientific material into lay vernacular was important in order to speak to a wide audience, but in many instances so too was maintaining the formal scientific language reflecting its peer-reviewed source. Some stakeholders suggested that this connection added to the credibility of the information, particularly in cases where decision stakes were high or the information contested. This work came at some price, however—for example, with an early Pacific ENSO Application Center product called the Rainfall Atlas. The original product was confusing to many stakeholders because it was presented in probabilistic terms using terciles. Communication and feedback from stakeholders led to the development of a “Historical Analog for the Atlas” that presented the information in more deterministic terms and in language and styles more easily understood by stakeholders. Some personnel involved in the product development received criticism from their scientific colleagues who believed the information had been “dumbed down.” Members, however, were convinced that they had produced a more important product that, according to stakeholders, actually got used.
RISA Members needed to be flexible about the language they used in communicating their research, for example with the phrase “climate change.” Because of early stakeholder resistance to the phrase, many members used the phrase “climate variability” to avoid unnecessary political conflict or resistance from stakeholders. Some members were criticized by peers as “selling out” to climate deniers, or even perpetuating misunderstandings about the gravity of climate change. Given the long-range goals of the RISAs, however, members realized that in order to deliver useful information, they needed to tailor the information to suit the users’ context. While some stakeholders were not ready to deal with climate change as a problem, they were open to working with RISAs over issues of climate variability relevant to their decision needs. Members recognized that using the stakeholders’ language was a necessary means in developing relationships with them while simultaneously moving forward on producing information the stakeholders could use.
One of the most important activities for RISA success was early, frequent, and iterative communication with stakeholders (Lemos and Morehouse 2005; Cash and Buizer 2005). The importance of robust communication cannot be overstated, particularly in producing and sustaining social capital and trust. Such attention to communication, however, took time and resources to deploy and was not always acknowledged as worthwhile investments by some researchers or managers in the members’ peer community. RISA organizations and their leaders, however, recognized the need for robust lines of communication with stakeholders and provided members with additional time and resources necessary for these tasks.
c. Producing and situating social capital
Members universally discussed the importance of establishing relationships with their stakeholders based on trust and mutual respect. These relationships created the social capital that facilitated the successful creation, sharing, and integration of information that was useful, both in terms of the credibility of the information and its perceived legitimacy. As one member explained, “I can’t imagine not being concerned about trust.” Other members indicated that, “Trust is huge … It’s everything. Trust encompasses scientific accuracy, rigor, relationships.” Remarking about trust, another member said, “It’s mine to lose.” Members and stakeholders were also explicit about the fact that “good science” and trust were “bound together” in the aim of producing useful information. Members discussed the importance that their work continued to be of the highest quality now that stakeholders turned to them for answers and information.
Some relationships required significantly more attention for the production and maintenance of social capital, particularly in those cases with significant power and resource asymmetry or significant geographic and cultural distance, for example with the Pacific RISA and the work they did with stakeholders in the U.S.-Affiliated Pacific Islands (USAPI). In most Polynesian societies, clan affiliation and family ties matter more than the pan-Polynesian similarities, political connections, and national affiliation (Lefale 2002), making the uptake and adoption of knowledge from “outsiders” particularly challenging. Stakeholders and members placed extremely high value on face-to-face meetings and personal connections. One official with the National Weather Service who was also an affiliate with the Pacific RISA called this work “eyeball-to-eyeball” and emphasized how important traveling to the islands was in order to share information directly with his stakeholders. Another affiliate working with the Pacific RISA said, “Don’t even bother bringing your briefcase for the first two years. It takes that long before they [in this case, Marshallese] get to know you and will work with you.” Pacific RISA members found that once they had formed robust relationships with their stakeholders, identifying and delivering useful climate information was much easier to accomplish. In the case of the drought predicted at the onset of the significant 1997–98 El Niño, all it took was the word of a Pacific RISA affiliate member for policy makers in Pohnpei to implement emergency-planning contingencies.
Social capital resided within the RISA organization or with individual members, but more often with both. In the Pacific Affiliated Islands, where trust was inversely correlated with stakeholders’ power and their access to resources, social capital was often grounded in an individual relationship with a member or affiliate of the Pacific RISA. Indeed, few of the Pacific RISA’s stakeholders were aware of the organization called the “Pacific RISA”—and some had never heard of it—yet they found great value in the information they received from its members. In most cases, social capital was grounded in the RISA organization, which leveraged the capital by “branding” certain products and information as coming from that RISA. In such cases, stakeholders identified the RISA as a trusted source of information based on its reputation without necessarily having any significant knowledge or relationship with individual members affiliated with the RISA.
These findings beg the question: What is the best place in which to situate the social capital? Research suggests that both locations have benefits, but it is important to understand the limitations and opportunities inherent in locating social capital in each place. For example, the Pacific RISA will have to wrestle with the question of how to transfer social capital from member to member (due to changes in personnel) since the organization itself possesses only limited “brand recognition.” CIG and CLIMAS, alternatively, tended toward both types of social capital, forging individual relationships and building their “brand” as trusted sources of climate information. Further research is necessary to understand the tradeoffs involved with deciding where, when, and how to situate the social capital, but acknowledging its value as a resource is the first step.
d. Capacity building
Creating and maintaining social capital was also necessary to support capacity-building efforts with stakeholders, particularly when their organizations found limited value in climate information, had not yet incorporated climate information into their operations, or had preconceived or inaccurate notions about climate science. RISA Members spent time “pushing” climate information to stakeholders not only in the belief that such information could be useful, but also because they wanted to build stakeholders’ capacity to absorb, understand, and, at a later time, utilize the information. As one member explained, “If we had waited for the demand [to arise], we would have been behind the curve.” “Undemanded” capacity-building often laid the ground for future information demands by stakeholders because they often “did not know that they needed the information.”
Encouraging stakeholders to engage the material was challenging not only because of significant differences in knowledge about climate science, but also because of different mental models and paradigms about how the world operated. The Pacific RISA spent a lot of time engaging in capacity-building activities, traveling to numerous islands, and educating stakeholders not only about how climate change affected various sectors of the economy and society but also about the basics of climate-change science. The Pacific RISA’s activities were further complicated by the fact that their stakeholders spoke multiple languages, requiring multiple translations of educational material. Work to build capacity and “change mental models” constituted a significant portion of early RISA activities. Stakeholders also indicated they valued this outreach and information and that it provided important “grounding in climate variability” at a time when they were just beginning to incorporate such information into their decision making, or had not yet considered using it at all. Some stakeholders made explicit requests for RISA outreach activities such as workshops or information sessions.
e. Leadership and organizational design
Strong leadership by senior scholars in climate science, climate impacts and adaptation, social sciences, or extensive programmatic development experience in climate services was a characteristic feature of the RISA programs. Leadership proved critical during the start-up phases of the organizations for several reasons. First, such leaders were “champions” both outside the organization (with academic departments and leaders in the host universities and with NOAA) and within it. Leaders were instrumental in conceptualizing the RISA program and were tireless in pushing for its creation. Leaders were also instrumental in planning each RISA and conceptualizing each RISA’s mission and vision. Leadership and support by senior PIs also helped junior members and researchers who lacked institutional support from their home academic departments in universities because they were engaged in problem-driven research. Members articulated strong support for such leadership that helped foster a feeling of community in the organization due to the leaders’ “dedication,” “inspiration,” and “commitment to the cause.”
RISAs resembled rather young, early phase entrepreneurial organizations in the context of an organization’s life cycle (Greiner 1997). Finding stakeholders, developing products, conducting research, securing resources, establishing social capital and credibility, and otherwise finding their research and product niches were extremely important activities during the RISAs’ earlier years, not unlike the challenges experienced by new businesses. RISAs were fairly flat and decentralized organizations in which most members worked autonomously, making decisions independently, albeit collaboratively at times and within constraints of the organizational vision and mission. Decentralization not only enabled members in all the RISAs to make their own decisions regarding what research to conduct but also, more importantly, members could respond quickly to changing contexts, issues, and windows of opportunities that arose. Greater attention to record-keeping and institutionalizing some processes would be beneficial to RISAs’ long-term development, but not at the expense of maintaining organizational adaptability and nimbleness in responding to opportunities. Decentralization enabled members to engage directly with stakeholders and respond in real time to demands, questions, concerns, or feedback.
RISAs also shared many characteristics with what Senge (1994) calls a “learning organization” that processes its experiences, adapts to external stimuli, and integrates new understanding of its environment into its decision-making processes. Senge describes a learning organization as one “where collective aspiration is set free, and where people are continually learning to see the whole together” (p. 3). Providing climate services is a developing paradigm, a “voyage of discovery” as one RISA described its work (Miles et al. 2006, p. 19 621). RISAs incorporated “double-loop learning” (Argyris and Schön 1978), or colloquially what can be called “learning from experience.” One Member described this processes as “mutual learning” because in addition to their stakeholders learning about climate, the members learned about stakeholder needs, opportunities, constraints, integration of information, etc. RISAs also had frequent meetings in which members reflected on their experiences—not only with regard to science but also the process of their work—to integrate learning into practice and continuous improvement. All of the RISAs published extensively sharing their experiences and observations about their work, process, and service activities.
f. Cross-case comparisons
Each of the three RISA programs studied for this project was created by different PIs who focused on different resource problems based on different climatic conditions, and with significantly different resources available for program support. While all were bound within the overall goal of producing place-based climate-science information for decision support, each program was organized and operated differently. Together, these RISA programs shared many more characteristics than not, suggesting that the processes and approaches identified may be characteristics that are essential for operating successful climate-service organizations.
The biggest difference between the programs concerns how they produced or otherwise acquired the climate information. CIG and CLIMAS, both housed in major universities and supported with generous funding from NOAA and other sources, were the primary producers of the climate information, doing the work in-house or collaboratively with nonmember researchers. In these cases, the members served as both producers of the climate information and as the mediators and negotiators, working directly with stakeholders, building relationships, and tending to the social process. Of these two organizations, CLIMAS had more social-science members who helped to clarify and understand stakeholders’ needs and understand the challenges of creating and integrating climate information into stakeholders’ knowledge systems.
The Pacific RISA was the smallest of the three and operated with a very small budget. Fiscal constraints significantly limited its ability to have in-house researchers and members who specialized in physical and natural science research. The Pacific RISA did have a few full-time researchers trained primarily in the social sciences, which, given the political, cultural, and epistemological challenges of producing and integrating useful climate science in the U.S. Affiliated Islands, proved quite beneficial. But for climate information informed by natural and physical sciences, the Pacific RISA had to access other resources in the surrounding area. Because of the lack of resources, Pacific RISA members often functioned as “brokers” of climate information, engaging both sides of the climate information spectrum. Members spent significant time and energy to understand their stakeholders’ needs, knowledge of climate science, capacity to integrate climate information, and sociocultural barriers to using climate information. In addition to doing their own social science research, members also actively engaged other researchers at the Pacific ENSO Application Center or University of Hawaii, for example, to help produce the climate information needed by their stakeholders. Acting as an information broker was an effective strategy for the Pacific RISA, although it also took significant time and resources, resulting in some obvious trade-offs.
4. Analysis and conclusions
The approaches and processes identified in this research raise important science-policy questions concerning research design and implementation, design and operation of climate-service organizations, and how to evaluate the outputs and outcomes of climate-science research for decision support and climate services more broadly.
a. Science policy
Producing useful climate information for decision support compelled the RISA organizations to employ additional processes and approaches that fell outside of the standard, basic-research paradigm typically found in academia. These problem-driven approaches are often described as use-inspired basic research, user-driven research, mode 2 research, and problem-oriented research; despite subtle conceptual and operational differences, they share the overarching objectives of shaping research agendas that respond to the information needs and priorities of society, what some have called a new social contract for science (Stokes 1997; Nowotny et al. 2003; Clark 2002; Guston 2000). Problem-driven research objectives include the discovery of new knowledge but, unlike basic research, must also satisfy stakeholders’ various value demands (Stokes 1997; Clark 2002). In addition to disseminating findings in peer-reviewed publications, researchers must also communicate findings in a variety of ways and ensure that stakeholders have the capacity to integrate the information into existing knowledge systems. These problem-driven approaches represent an end-to-end paradigm of research in that knowledge production is undertaken with its end use in mind, actively linking science and decision makers (Agrawala et al. 2001).
Problem-driven research is a more resource-intensive process than doing basic research alone. Findings from this research point to the importance of tending to the social process, building and maintaining social capital, developing robust lines of communication, forging relationships based on mutual trust and respect, educating stakeholders, and building capacity in their organizations to utilize climate information—in addition to conducting research that results in high-quality and credible science. These activities require substantial commitments to time, allocation of additional resources, and even personnel who function as outreach and education specialists. Members indicated that these additional activities are often not recognized or rewarded in the tenure, retention, and promotion process in the academy; that funding cycles are often too short to build relationships with stakeholders or fully assess their needs; and that evaluating problem-driven research using the same methods used for basic research falls short.
The linear model of science policy still informs the vast majority of climate research in the United States (Dilling and Lemos 2011), privileging basic research over problem-driven approaches. A frequent critique of problem-driven research is that it “drives out” basic research and the discovery of new knowledge, but research suggests that producing new knowledge and creating useful information for society are not mutually exclusive activities (Stokes 1997; Sarewitz and Pielke 2007). Members continued to create new knowledge as a matter of course, including, for example, significant contributions to our understanding of the Pacific decadal oscillation (Mantua et al. 1997). While basic research will always have a large and important role to play in climate research, science-policy decision makers need to give serious consideration to expanding support for problem-driven research design, implementation, and integration into existing knowledge and policy systems, particularly if the objective is to produce useful climate science for place-based decision support.
b. Future evaluation of the RISAs
Outputs from basic research are typically evaluated using a variety of bibliometric methods such as evaluating the quantity and quality of peer-reviewed publications. Some outputs from problem-driven research can also be evaluated using these methods; however, they fall short in evaluating reports, models, white papers, forecasts, workshops, training sessions, etc. In addition to evaluating outputs, programmatic outcomes must also be evaluated, such as improved understanding of climate science, policies enacted, resources saved or conserved, decisions made, models designed, stakeholder networks created, and social capital developed. Temporal delays between producing outputs and observing outcomes, difficulties in quantifying changes in complex coupled human–environmental systems, attributing policy-related decisions to specific RISA outputs, time and resources necessary to implement formal programmatic evaluation, measuring changes in attitudes and beliefs concerning climate risks, and concern over “stakeholder burnout” due to frequent and ongoing monitoring add to the challenges of evaluating each RISA program and climate services in general.
At the time of this research, RISAs provided relatively limited reporting on various outputs as required by the NOAA CPO, but utilized few if any additional methods to assess the full range of RISA activities and production of useful climate information for decision support. One limitation to enhancing evaluation metrics, methods, and monitoring concerns the lack of bureaucratic mechanisms within each RISA needed to record detailed information about research agendas and activities, stakeholder engagement, communication, outputs, and programmatic outcomes—tasks that will prove challenging given limited existing resources. Another limitation stems from the NOAA CPO office, which is also exploring how best to evaluate RISA programs, a question that is constantly evolving. Until such time, however, evaluating useful information based on how it satisfies stakeholders’ value demands and use of ex-post satisficing will have to do. What is clear is that both quantitative and qualitative methods for evaluating climate services need to be developed and deployed.
c. Conclusions
RISAs exist to produce quality climate information for decision support, but they also function as quasi-experimental programs for testing various approaches for producing climate-science information and delivering climate-information products to a variety of decision makers in numerous domains. Findings from this research indicate that producing useful climate information that satisfies users’ needs is a complex, highly contextual social process. Producing high-quality and credible natural, physical, and social-science information alone is insufficient to ensure the production of useful climate information for decision support.
Producing and delivering useful information, which included both products and events, involved identifying stakeholders’ specific research needs using both formal and informal research methods; communicating early and iteratively with stakeholders using multiple strategies; building capacity in the stakeholder community so they understood how climate information could be used in their decisions; building and maintaining social capital as a means to facilitate these processes and activities; and utilizing strong leadership coupled with flat, decentralized organizational design. This research clarifies the importance of the design and delivery of climate services, whereby adequate time and attention needs to be allocated to build relationships and tend to social systems.
Although focused on just three RISAs, this research confirms many previous findings, adds additional empirical research and understanding about the function of climate-service organizations, and develops several hypotheses than can be tested in future research. This research also raises additional questions concerning how climate services should be evaluated and, more importantly, how climate information actually gets used in the policy process. What is clear from the research is that significant questions still remain and additional research needs to be done to improve our use of qualitative and quantitative methods to evaluate the emerging field of climate services.
REFERENCES
Agrawala, S., Broad K. , and Guston D. H. , 2001: Integrating climate forecasts and societal decision making: Challenges to an emergent boundary organization. Sci. Technol. Human Values, 26, 454–477.
Alley, R. B., and Coauthors, 2007: Summary for policymakers. Climate Change 2007: The Physical Science Basis, S. Solomon et al., Eds., Cambridge University Press, 1–18.
Argyris, C., and Schön D. , 1978: Organizational Learning: A Theory of Action Perspective. Addison Wesley, 356 pp.
Babbie, E., 2004: The Practice of Social Research. 10th ed. Thomson Wadsworth Publishing, 608 pp.
Beierle, T. C., and Cayford J. , 2002: Democracy in Practice: Public Participation in Environmental Decisions. Routledge, 158 pp.
Bennett, A., and Elman C. , 2006: Qualitative research: Recent developments in case study methods. Annu. Rev. Polit. Sci., 9, 455–476.
Cash, D., and Clark W. , 2001: From science to policy: Assessing the assessment process. John F. Kennedy School of Government Faculty Research Working Papers Series RWP01-045, 21 pp.
Cash, D., and Buizer J. , 2005: Knowledge-action systems for seasonal to interannual climate forecasting: Summary of a workshop. The National Academies Press, 44 pp.
Cash, D., Clark W. , Alcock F. , Dickson N. , Eckley N. , and Jager J. , 2002: Salience, credibility, legitimacy and boundaries: Linking research, assessment and decision making. John F. Kennedy School of Government Rep. RWP02–046, 24 pp.
Cash, D., Clark W. , Alcock F. , Dickson N. , Eckley N. , Guston D. H. , Jager J. , and Mitchell R. B. , 2003: Knowledge systems for sustainable development. Proc. Natl. Acad. Sci. USA, 100, 8086–8091.
CCSP, 2008: Decision-support experiments and evaluations using seasonal-to-interannual forecasts and observational data: A focus on water resources. A report by the U.S. Climate Change Science Program and the Subcommittee on Global Change Research. N. Beller-Simms et al, Eds., National Climatic Data Center, 192 pp. [Available online at http://www.climatescience.gov/Library/sap/sap5-3/final-report/.]
Clark, T. W., 2002: The Policy Process: A Practical Guide for Natural Resource Professionals. Yale University Press, 215 pp.
Committee on Science, 2002: Hearing on New Directions for Climate Research and Technology Initiatives, U.S. House of Representatives, 17 April.
Deelstra, Y., Nooteboom S. G. , Kohlmann H. R. , van den Berg J. , and Innanen S. , 2003: Using knowledge for decision-making purposes in the context of large projects in the Netherlands. Environ. Impact Assess. Rev., 23, 517–541.
Dilling, L., and Lemos M. C. , 2011: Creating usable science: Opportunities and constraints for climate knowledge use and their implications for science policy. Global Environ. Change, 21, 680–689.
Ehlers, V., 1998: Unlocking our future: Toward a new national science policy. The United States House of Representatives Committee on Science, 100 pp.
Escobar, A., 1994: Encountering Development: The Making and the Unmaking of the Third World. Princeton University Press, 290 pp.
George, A., and Bennett A. , 2005: Case Studies and Theory Development in the Social Sciences. MIT Press, 352 pp.
Greiner, L. E., 1997: Evolution and revolution as organizations grow: A company’s past has clues for management that are critical to future success. Fam. Bus. Rev., 10, 397–409.
Guston, D. H., 2000: Retiring the social contract for science. Issues in Science and Technology, Summer 2000. [Available online at http://www.issues.org/16.4/p_guston.htm.]
Guston, D. H., 2001: Boundary organizations in environmental policy and science: An introduction. Sci. Technol. Human Values, 26, 399–408.
Haas, P. M., 2004: When does power listen to truth? A constructivist approach to the policy process. J. Eur. Public Policy, 11, 569–592.
Herrick, C. N., and Sarewitz D. , 2000: Ex-post evaluation: A more effective role for scientific assessments in environmental policy. Sci. Technol. Human Values, 25, 309–331.
Kingdon, J., 1997: Agendas, Alternatives, and Public Policies. 2nd ed. Pearson Education, 254 pp.
Lefale, P. F., 2002: Traditional knowledge of weather and climate prediction: The Samoa experience. Proc. APN Workshop on Ethnographic Perspectives on Resilience to Climate Variability in the Pacific Island Countries, Christchurch, New Zealand, Macmillan Brown Centre for Pacific Studies, University of Canterbury, 30 pp.
Lahsen, M., and Nobre C. A. , 2007: Challenges of connecting international science and local level sustainability efforts: The case of the Large-Scale Biosphere–Atmosphere Experiment in Amazonia. Environ. Sci. Policy, 10, 62–74.
Landry, R., Lamari M. , and Amara N. , 2003: The extent and determinants of the utilization of research for health-care policy and practice. Online J. Knowl. Synth. Nurs., 9 (7).
Lasswell, H. D., 1971: A Preview of the Policy Process. Elsevier, 187 pp.
Lasswell, H. D., and McDougal M. , 1992: Jurisprudence for a Free Society. Vol. 1, Yale University Press, 722 pp.
Lemos, M. C., and Morehouse B. J. , 2005: The co-production of science and policy in integrated climate assessments. Global Environ. Change, 15, 57–68.
Mantua, N. J., Hare S. R. , Zhang Y. , Wallace J. M. , and Francis R. C. , 1997: A Pacific interdecadal climate oscillation with impacts on salmon production. Bull. Amer. Meteor. Soc., 78, 1069–1079.
Mayden, S., 2002: RISA: Regional Integrated Sciences and Assessment. Institute for the Study of Planet Earth, 12 pp.
McNie, E. C., 2007: Reconciling the supply of scientific information with user demands: An analysis of the problem and review of the literature. Environ. Sci. Policy, 10, 17–38.
McNie, E. C., 2008: Co-producing useful climate science for policy: Lessons from the RISA Program. Ph.D. dissertation, University of Colorado at Boulder.
McNie, E. C., Pielke R. Jr., and Sarewitz D. , 2007: Climate science policy: Lessons from the RISAs. Proc. SPARC Reconciling Supply and Demand Workshop, Honolulu, HI, SPARC. [Available online at http://cstpr.colorado.edu/sparc/research/projects/risa/workshop_report.html.]
Miles, E. L., Snover A. K. , Whitely Binder L. C. , Sarachik E. S. , Mote P. W. , and Mantua N. , 2006: An approach to designing a national climate service. Proc. Natl. Acad. Sci. USA, 103, 19 616–19 623.
National Climate Assessment, 2011: Climate change impacts and responses: Societal indicators for the National Climate Assessment. NCA Rep. Series, Vol. 5C, 122 pp.
Nowotny, H., Scott P. , and Gibbons M. , 2003: Mode 2 revisited: The new production of knowledge. Minerva, 41, 179–194.
NRC, 1999: Global Environmental Change: Research Pathways for the Next Decade. National Academies Press, 192 pp.
NRC, 2005a: Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program. National Academies Press, 162 pp.
NRC, 2005b: Decision Making for the Environment: Social and Behavioral Science Research Priorities. National Academies Press, 296 pp.
NRC, 2007a: Evaluating Progress of the U.S. Climate Change Science Program. National Academies Press, 170 pp.
NRC, 2007b: Research and Networks for Decision Support in the NOAA SARP. National Academies Press, 86 pp.
NRC, 2009a: Restructuring Federal Climate Research to Meet the Challenges of Climate Change. National Academies Press, 254 pp.
NRC, 2009b: Informing Decisions in a Changing Climate. National Academies Press, 200 pp.
NRC, 2010: Informing an Effective Response to Climate Change. National Academies Press, 348 pp.
NRC, 2012: A Review of the Global Change Research Program’s Strategic Plan. National Academies Press, 72 pp.
Oh, C. H., 1996: Linking Social Science Information to Policy-Making. Jai Press, 201 pp.
Oh, C. H., and Rich R. F. , 1996: Explaining use of information in public policymaking. Knowl. Technol. Policy, 9, 3–35.
Pielke, R., Jr., 2007: The Honest Broker: Making Sense of Science in Politics and Policy. Cambridge University Press, 198 pp.
Pyke, C. R., Bierwagen B. G. , Furlow J. , Gamble J. , Johnson T. , Julius S. , and West J. , 2007: A decision inventory approach for improving decision support for climate change impact assessment and adaptation. Environ. Sci. Policy, 10, 610–621.
Sambanis, N., 2004: Using case studies to expand economic models of civil war. Perspect. Polit., 2, 259–279.
Sarewitz, D., and Pielke R. Jr., 2007: The neglected heart of science policy: Reconciling supply of and demand for climate science. Environ. Sci. Policy, 10, 5–16.
Senge, P., 1994: The Fifth Discipline: The Art & Practice of the Learning Organization. Currency Doubleday, 423 pp.
Shaw, C. G., III, and Coauthors, 1997: Conservation and resource assessments for the Tongass Land Management Plan Revision: Evaluation of the use of scientific information in developing the 1997 Forest Plan for the Tongass National Forest. U.S. Forest Service, 9 pp.
Stokes, D. E., 1997: Pasteur’s Quadrant: Basic Science and Technological Innovation. Brookings Institution, 196 pp.
Stone, D., Maxwell S. , and M. Keating, 2001: Bridging research and policy. 50 pp. [Available online at http://www2.warwick.ac.uk/fac/soc/csgr/research/keytopic/other/bridging.pdf.]
