1. Introduction
Damage and disruption from fluctuations in seasonal climate that bring on droughts, hurricanes, and flooding is well documented (IPCC 2012; Howitt et al. 2014). In some regions of the world, it is possible to forecast aspects of seasonal climate, including the chances for extreme conditions, providing an opportunity for decision-makers to use this information in their risk management preparations. Regional Climate Outlook Forums (RCOFs) represent one effort to bring seasonal climate information to decision-making at regional and national levels (Ogallo et al. 2008).
RCOFs were first organized in 19971 to provide seasonal climate information to help decision-makers reduce climate-related risks, develop technical forecasting capacity, and strengthen connections between science providers and decision-makers. They have since become a major international climate service effort (Scaramella et al. 2012; Kadi 2012; Garcia-Solera and Ramirez 2012; Della Cruce 2012; WMO 2008). Currently, RCOFs are routinely convened in 19 regions around the world, serve mainly developing and least-developed countries, and are coordinated in part by the WMO (WMO 2016).2 The experience of RCOFs presents a learning opportunity that can inform the nascent and widespread efforts in climate services.
Two previous evaluations of the RCOFs conducted by the WMO concluded that RCOFs lacked user orientation, failed to understand and document the socioeconomic benefits of the forecasts, and required increased coordination and management (Basher et al. 2000; WMO 2008). There has also been extensive reflection on the skill of forecasts issued at the RCOFs (Berri et al. 2005; Hyvärinen et al. 2015; Mason and Chidzambwa 2009). However, these efforts have provided only a partial view of the success and challenges of RCOFS and do not constitute a systematic approach to track progress, improve operations, and document lessons learned. In fact, Vaughan and Dessai’s (2014) review of climate services more broadly identifies a lack of metrics and methodologies to evaluate these aspects of climate services.
To address this gap, we propose a simple evaluation framework for the RCOFs based on a process-oriented approach. Process evaluations elucidate how programs, systems, and relationships operate, and they identify the mechanisms of change (Rossi et al. 2004). We identify six conditions routinely cited as important for facilitating information use, and these serve as the elements of our framework. We elaborate on each in a case study of the Caribbean Regional Climate Outlook Forum (CariCOF), where the authors have been engaged in a collaborative effort to integrate and evaluate climate information and decision processes for regional climate resilience. We draw from document analysis, small group interactive discussions, interviews, and an online survey with CariCOF participants. Based on the analysis, we provide recommendations that can serve to guide implementation and evaluation for RCOFs and other organizations charged with the provision of climate services. We recognize that many leaders of the RCOFs are not evaluation experts. The process-oriented framework is advantageous in this regard because RCOF leaders can still implement process monitoring in order to improve outcomes.
2. Process-oriented evaluation framework
The processes that underpin a program’s implementation are recognized as vital to understanding the impact and efficacy of programs and activities (Oakley et al. 2006). Process evaluations typically explore the implementation and setting of an activity (Chen and Rossi 1989). They focus on the types, qualities, and quantities of services delivered; the beneficiaries of those services; and the resources applied to deliver the services. Therefore, process-oriented evaluations identify the mechanisms that lead to outcomes.
In comparison to an impact evaluation, which attempts to characterize the extent to which climate services lead to desired outcomes in terms of improved lives or livelihoods, a process evaluation is more interested in outputs—checking, for instance, that the actions that were planned are carried out. Qualitative surveys may also be used to determine whether beneficiaries used project inputs and whether there is evidence that the program beneficiaries were satisfied by the program. While a great deal of focus has been placed lately on impact evaluation of climate services, process evaluation is clearly essential and should be part of any program evaluation (Duflo 2004).
Our process approach resonates with evaluation approaches that speak to the contextual elements of evaluation, stakeholder engagement, and action-orientation that provokes an intentional course of action (Guba and Lincoln 1989). Given the development nature of the Climate Outlook Forums and other climate services, especially around technical capacity development and capacity associated with communication (e.g., Dilling and Lemos 2011; Lemos et al. 2012), our approach also benefits from research on empowerment in evaluation (Fetterman and Wandersman 2005, 2007).
We explicitly build our process-oriented framework around six conditions routinely cited in the rich literature on decision support. These are identified in the literature for their importance in enhancing the production and use of scientific information. Table 1 summarizes the six conditions, the processes that they support, and their corresponding objectives. While the same process can support multiple conditions and objectives, we discuss each separately for clarity. In our case study, we focus on understanding how these conditions are supported.
Conditions and processes that support the use of climate information.
a. Quality and credibility of climate information
The degree to which climate information provides an accurate representation of what is happening in the climate system depends critically on the quality of data and the methods used to analyze those data. The climate system itself has inherent uncertainties that combine with uncertainties in data and analytical assumptions. If conclusions are not properly framed by researchers, poor data quality, high uncertainty, and/or inappropriate analysis can reduce the credibility of the information. Perceptions of credibility are also entwined with salience (the relevance of the information) and its legitimacy (whether the process that created the information is perceived unbiased and fair) (Cash et al. 2002). Salience, credibility, and legitimacy can be optimized by understanding end-user decision contexts, conducting rigorous scientific analyses, ensuring proper communication, and engaging end users in knowledge creation processes.
b. Tailoring of climate information
The use of climate information depends largely on how well it fits within decision-making contexts and is understood by end users (Lemos and Morehouse 2005; Dilling and Lemos 2011). Tailoring climate information products requires that information producers understand user needs and then package, contextualize, and communicate the climate information in appropriate language, formats, and spatial resolutions (Lemos et al. 2012; Moser 2009). Enabling ongoing interaction between users and producers and customizing existing knowledge to meet users’ needs are seen as key strategies to improving usability.
c. Provide specific climate products
People and sectors experience different climate risks and respond differently to these risks. A diverse suite of information, provided in different formats, can sometimes be useful in satisfying the needs of a broad range of diverse actors. For example, some users find historical and climate change information more useful than seasonal forecasts, and vice versa (Goddard et al. 2012; Lowe et al. 2016; Meinke et al. 2006; Vincent et al. 2015); others prefer climate information that is oriented toward reducing vulnerability and sectoral impacts. However, the production of more information does not necessarily translate to more or better use (Lemos and Dilling 2007). Moreover, developing a diverse portfolio that provides tailored information to user and sector communities is a resource intensive process (Kirchhoff et al. 2013; McNie 2013). Therefore, processes that lead to specific knowledge of what is needed, by whom, and under what conditions will help prioritize product development and the allocation of limited resources.
d. Communication of climate information
Effective communication between producers and users of climate information can improve access to and comprehension of such information. Communication is important because seasonal climate information is often presented in a probabilistic form that end users find difficult to understand (NRC 1999). The usability of climate information depends on users’ perception of information fit, how it interplays with other kinds of information they consult, and the level and quality of interaction between producers and users (Lemos et al. 2012, p.789). Trained information brokers can help address these challenges by synthesizing and disseminating information (Dilling and Lemos 2011; Guido et al. 2013; Guido et al. 2016; Buizer et al. 2016). This must be approached thoughtfully because misinterpretation and miscommunication have discredited forecasts and forecast producers in the past (Lemos and Dilling 2007).
e. Stewarding knowledge networks
Climate information can be made useful through a “knowledge network” through which researchers interact and build relationships with the stakeholders and decision-makers (Bidwell et al. 2013; Feldman and Ingram 2009; Goddard et al. 2014; Henry and Vollan 2014). Networks can help disseminate knowledge efficiently across broad communities of users and provide important value-added functions (Bidwell et al. 2013; Lemos et al. 2014; Guido et al. 2016). Networks also help inform and strengthen regional efforts and bring to scale the production of usable information (Kalafatis et al. 2015).
3. Background: The Caribbean Climate Outlook Forum
The first CariCOF was convened by NOAA in 1998 in Jamaica as a response to the strong El Niño event of 1997/98 that coincided with drought and crop losses across the region. The development of modeling capabilities that connect atmospheric patterns to tropical Pacific Ocean sea surface temperatures was starting to deliver the promise of improved seasonal climate forecasts. The Caribbean Institute for Meteorology and Hydrology (CIMH)—a regional organization that supports research and service activities for 16 National Meteorological and Hydrological Services (NMHS)—was a lead convener in this forum. The event formulated a consensus precipitation forecast for the Caribbean for the period of June–August 1998.
The second and third CariCOFs were held in Barbados in April 1999 and the Dominican Republic in May 2000, respectively. The aims of the second forum were to discuss the regional climate events since the first forum, to consider the recommendations of that forum, to develop a consensus climate outlook for May–July 1999, and to evaluate the 3-month precipitation outlooks that CIMH had produced.
After the third forum, however, the CariCOF failed to become routine because of a lack of interest and inadequate institutional and financial support, according to some involved in the process. Nevertheless, CIMH continued to produce and disseminate the seasonal climate forecasts (SCFs). From 1999 to 2011, the CIMH produced 0-month lead and 3-month tercile precipitation outlooks every two months, starting in January—a total of six information products per year.
Interest returned for the CariCOF in 2010 as a result of the confluence of regional and international factors. At the international level, funding for activities like the CariCOF became available associated with the new WMO Global Framework for Climate Services. Additionally, in 2009 and 2010, the Caribbean region experienced its worst drought in 50 years (Farrell et al. 2010), galvanizing demand for climate information. With the gained regional governance and international programmatic momentum, CariCOF was reestablished with the organization of a forum in 2010.
The reestablishment led to a follow-up CariCOF event consisting of two distinct parts. The first part was a training exercise for meteorologists and climatologists in the art of forecasting using the International Research Institute for Climate and Society’s (IRI) Climate Predictability Tool (CPT) in February 2012. One outcome of the training exercise was a consensus precipitation outlook produced by all participating NMHSs and CIMH. The training preceded a stakeholder forum that brought together key providers and users of climate information; they discussed the precipitation outlook and its implications, as well as other climate information needs and gaps.
By 2012, the CariCOF was institutionalized as a routine event. It has since been convened at least once annually around 1 May, prior to the onset of the wet season. In 2014, the CariCOF began being held twice a year; the second forum occurs around 1 December, prior to the onset of the dry season. Each CariCOF has taken place in a different Caribbean nation.
The CariCOF brings together national, regional, and international scientists and sectoral decision-makers (e.g., agriculture, disaster, health, tourism, water) across the Caribbean to achieve four goals: 1) formulate and communicate seasonal climate outlooks; 2) identify information and capacity gaps; 3) facilitate research cooperation and data exchange; and 4) improve coordination within the Caribbean climate forecasting community (CIMH 2016). The CariCOF is preceded by a training event for meteorologists representing the NMHSs that work closely with CIMH. At the training events, meteorologists learn analysis techniques and develop the seasonal climate forecasts for the Caribbean that are presented at the CariCOF. Following, decision-makers from diverse sectors participate with meteorologists in a series of presentations and discussions during the 1–2-day CariCOF.
4. Case study methods
We use a case study approach to document the activities of the CariCOF and assess them within our framework (Yin 1994). We draw from document analysis, small group interactive discussions, interviews, and an online survey to generate insights on the six conditions that support information use (Table 2). This methodological diversity helps bolster the validity of the results (Lieberman 2005).
Framework and methods adopted.
The documents we reviewed included CariCOF strategic plans, agendas, and participant lists from forums convened between 2012 and 2015. We documented the country and sector of participants, the agenda activities, and the time allocated to categories of activities. We also examined the climate service products issued at the CariCOFs as well as those that are electronically accessible.
We conducted 12 key informant interviews with participants at the 2014 wet season CariCOF in Jamaica. Interviewees included decision-makers from a variety of Caribbean sectors and personnel with CIMH, IRI, and Caribbean NMHSs. Interviews addressed issues of user needs and perceptions of the quality, diversity, and tailoring of climate information.
In addition, we conducted a participatory network mapping exercise with small groups at the 2014 CariCOFs in Jamaica and in Antigua to understand the dissemination of the seasonal climate outlook in the Caribbean [see Guido et al. (2016) for details].
Finally, we conducted an online survey of 40 participants from the 2014 Jamaica CariCOF (response rate of 61%). The survey covered topics related to the history of the CariCOF, the barriers to using climate information, the CariCOF goals, and the dissemination of seasonal climate outlooks.
5. Findings
a. How is the CariCOF improving the quality and credibility of the climate information it offers?
The production of the seasonal climate outlooks has evolved over time in the Caribbean. Currently, the outlooks are produced through a collaborative effort between the NMHSs and CIMH. This collaboration takes place during the training event preceding the CariCOF as well as electronically every month. Meteorologists use the Climate Predictability Tool (CPT)—a software package that facilities the generation of the outlooks—to produce a range of objective, probabilistic, national and regional seasonal outlooks using canonical correlation analysis (Mason and Tippett 2016). For each of these outlooks, forecasters use a set of predictor fields to build an objective ensemble of experiments. Each national objective outlook represents the arithmetic averages of the forecast probabilities of all ensemble members, whereas each regional objective outlook represents the arithmetic averages of a compilation of national and regional ensemble experiments. These objective outlooks are then shared among all forecasters and compared to each other. They are also compared to forecasts from global forecasting centers including the European Centre for Medium-Range Weather Forecasts, WMO’s Lead Center for Long-Range Forecast Multimodel Ensemble, IRI, and NOAA’s Climate Prediction Center. The meteorologists discuss these results to reach a consensus on the Caribbean outlook.
Since 2012, there have been two improvements in the quality of operations. First, the scientific rigor of the consensus outlook improved. Initially, precipitation outlooks were made using results from the WMO Global Producing Centres (GPCs)3 that were downscaled by Caribbean meteorologists; however, the knowledge of appropriate downscaling methods varied greatly by these meteorologists, and, consequently, the meteorologists often relied heavily on subjective and visual assessments of the GPC products. The incorporation of the CPT in 2012 automated the downscaling process with advanced statistical techniques. Furthermore, CIMH introduced the CariCOF Outlook Generator (CAROGEN) in 2015, which increased the number of experiment simulations and standardized the experimental setup; this allowed results to be compared objectively across the region. These automations, however, were not designed to eliminate the subjective input to the forecast. The latter input consists of a process of which the perceived increase in forecast accuracy outweighs the relatively short additional time requirements.
The second improvement has occurred from regular assessments of forecast skill. In an unpublished report, CIMH compared the skill of the IRI precipitation outlooks to those of CIMH (including those produced at the CariCOF) to distinguish which systems, seasons, and subregions are more accurately forecasted (Bedward and Van Meerbeeck 2013). The results have guided research to improve skill, particularly targeted at poor-performing regions and seasons. To support these efforts, capacity building on forecast verification was added to the pre-CariCOF training events beginning in 2014.
b. How is the CariCOF tailoring climate information for specific contexts?
The tailoring of climate information has largely been approached by CIMH through activities to understand the decision contexts in which the information could be used and through seeking feedback from CariCOF participants. At the 2014 CariCOF in Jamaica, CIMH introduced a new outlook that conveyed alert levels for drought at select locations in the Caribbean. Over the course of several CariCOFs and other workshops, and with input from stakeholder groups, CIMH modified the language of the alert levels and the visual display of the product. CIMH has evolved the presentation of the tercile-based forecasts. These forecasts show the probability of above-normal, below-normal, and normal precipitation levels or temperatures and are among the most common forecast products in many regions. The tercile formats, however, have been often critiqued because they present communication and comprehension challenges for users (e.g., Pagano et al. 2002; Lemos et al. 2002; Ziervogel and Calder 2003; Vogel and O’Brien 2006). In an attempt to assuage some of the challenges, these tercile forecasts have been placed—and contextualized—within descriptive bulletins and not simply issued as stand-alone products. Furthermore, several outlook products converted the tercile-based probabilistic output into occurrence and frequency shifts of extreme rainfall events.
An illustration is given in Fig. 1 that shows an outlook for the frequency shift in the occurrence of extreme 3-day wet spells during the period September–November 2016, as compared to the climatological norm for 1985–2014. An extreme 3-day wet spell is defined as 3 consecutive days with a rainfall sum exceeding the 99th percentile of all rolling 3-day rainfall sums in the climatological period. The colored scale seen in the mapped dots on Fig. 1 represents tercile-based forecast probabilities, with blue, green, and red/brown hues identifying increased chances of above-normal, normal, and below-normal levels, respectively. The occurrence of such spells is a Boolean variable, so an increase in probabilities can straightforwardly be expressed in terms of an increase in frequency, which forms the semiquantitative scale found in the legend. In the CariCOF Wet Days/Wet Spells Outlooks, maps are accompanied by textual information on the climatological number of occurrences; a compiled precipitation, wet days, and wet spells forecast; and key implications of the climatology and forecast for the season of interest.
CariCOF’s September–November 2016 frequency of extreme (top 1%) 3-day wet spell forecast map, issued on 1 Sep 2016.
Citation: Weather, Climate, and Society 10, 2; 10.1175/WCAS-D-17-0029.1
In 2015, organizations from different sectors in the Caribbean formed the Consortium of Regional Sectoral Early Warning Information Systems across Climate Timescales (EWISACTS) with the goal to facilitate “mechanisms to champion the design, development and delivery of tailored climate products and services in the agriculture and food security, disaster risk management, energy, health, tourism and water sectors” (CIMH 2015, p. 12). Monitoring the functioning of this group and its impact with regard to the tailoring of climate products will be central to future process evaluations.
c. How is the CariCOF diversifying climate products for particular applications?
The diversity of outlook products has increased over time, which, in principle, helps respond to a larger range of users and their needs. In 2012, the CariCOF provided only a seasonal rainfall outlook issued immediately prior to the forecasted season (i.e., 0-month lead). In response to feedback from CariCOF participants, CIMH developed additional products to better meet users’ needs. For example, agriculture decision-makers identified a need for longer lead-time forecasts at the 2012 CariCOF, and water managers suggested information on extreme rainfall events and the frequency of wet days would be useful at the May 2015 CariCOF. These and other requests instigated new temperature outlooks for maximum, minimum, and mean temperature at both 0- and 3-month leads (added in 2013); a drought outlook (added in 2014); an outlook for the frequency of wet periods (added in 2015); a coral reef watch (added in 2015); and a climate impacts database (added in 2015).
However, the process for obtaining feedback has been somewhat ad hoc. Moreover, relatively few of the region’s decision-makers attend the CariCOF, and it is unclear how representative the opinions expressed by those attending the CariCOF are compared to the broader Caribbean community. It is unclear how many people benefit, and adding work to NMHS staff presents resource tradeoffs. Not surprisingly, representatives from the NMHSs voiced concerns that their capacity was inadequate to develop and steward new products. CIMH has tried to address this challenge by supporting the research and development of prototype regional products. However, in the case where demand for a product has broad appeal, it is unclear if the regional products can be subsequently tailored by NMHSs to the national or local context, where decisions are typically made.
d. How is the CariCOF fostering effective communication?
The CariCOF aims to foster dialogue and mutual learning between meteorologists and decision-makers. In theory, meteorologists learn how decision-makers use seasonal climate information, as well as the challenges to using the forecasts and the impacts of the forecast on different sectors, while decision-makers learn technical aspects of the forecasts and the climate conditions that underpin them. This information can help meteorologists and decision-makers communicate the information to local and national actors. For example, about 61% of CariCOF participants reported adding information to the outlooks to enhance value when they shared them with others (Guido et al. 2016). Moreover, participants identified “translate and contextualize information” and “discuss sectoral impacts” as important goals of the CariCOF.
To facilitate greater social learning, and with encouragement from its academic participants, the CariCOF has undergone a shift to more active forms of engagement. Half of the agenda time has been dedicated to discussions and engagement activities since 2014, whereas before the majority of time was dedicated to presentations (Fig. 2).
Time allocated to presentations and discussions at CariCOFs, 2012–15.
Citation: Weather, Climate, and Society 10, 2; 10.1175/WCAS-D-17-0029.1
In addition, the CariCOF has experimented with creative ways to communicate climate information. Beginning in 2014, for example, CIMH staff has lead a performance-based “theater” in which NMHS participants act out potential impacts of the forecast and issues related to forecasting and stakeholder responses and awareness. The play is used as a vehicle to introduce and solicit feedback on potential new climate service products.
e. How is the CariCOF stewarding a knowledge network that contributes to information generation and dissemination?
The Caribbean comprises numerous small islands, many of which experience similar climate and weather risks. Nevertheless, each has its own climate vulnerabilities as a result of unique social and geophysical conditions (Farrell et al. 2010). Building a robust network at a regional scale therefore requires engaging people from different countries and across and within many sectors. It also requires promoting information exchange beyond the actual CariCOF. Building such a network presents a considerable challenge, given funding limitations and geographic dispersion.
To better understand the network of actors involved, we investigated CariCOF participation. Table 3 reports on stakeholder participation between 2012 and 2015. Approximately 8 or 9 different sectors were represented at each of the CariCOFs from 2012 through 2014. In 2015, the diversity of stakeholders increased. The NMHSs represent a plurality of participants by design. Agriculture, water, and disaster risk management are consistently the sectors most represented; health, manufacturing, media, energy, transport, and tourism, on the other hand, are not often as represented. The underparticipation of decision-makers working in tourism was surprising to many, given tourism’s importance to the region’s economy.
Participants at CariCOFs, 2012–15.
Though the use of information does not automatically follow from participation, the number of people attending CariCOF events is growing (Fig. 3). CIMH strives for 1) a balance between repeat and new participants in order to build awareness, 2) a broad set of stakeholders to inject new opinions and experiences, 3) increased familiarity with the information, and 4) building relationships among participants.
Participation in the CariCOF.
Citation: Weather, Climate, and Society 10, 2; 10.1175/WCAS-D-17-0029.1
Participants at the 2014 CariCOF in Jamaica indicated that they anticipated sharing information presented at the CariCOF with 4.3 individuals, on average. In most cases, participants reported including additional information that adds value to the outlook. For example, sectoral actors most often translate climate information into potential impacts and guidance, whereas NMHS often translates the general seasonal climate forecast categories of above-, below-, or near-normal into country and subcountry specific information, such as the chances that precipitation will be above 120% of average. This enables stakeholders and NMHSs to interact but also helps to build the CariCOF network at the national level because the NMHSs act as a bridge between different sectors within the country (see Fig. 4a). There are also organizations that tie the region together by communicating across national boundaries (Fig. 4b). CIMH is the most obvious bridge, being the central hub for the production and dissemination of regional climate information, as noted above. Other regional organizations that are sectorally focused have this function as well, including the Caribbean Public Health Agency (CARPHA) and the Caribbean Disaster Emergency Management Agency (CDEMA).
Communication network at the Kingston, Jamaica, CariCOF.
Citation: Weather, Climate, and Society 10, 2; 10.1175/WCAS-D-17-0029.1
6. Discussion: Notable achievements and persistent challenges in providing the Caribbean Climate Outlook Forum
Our process evaluation finds that CariCOF activities are evolving toward more advanced climate science analysis, wider ranges of tailored products, greater efforts to collect and respond to feedback, and increased participant diversity. However, there are also limitations; these include the need for continued improvements in participant diversity, an incomplete understanding of best practices to tailor information, undeveloped market research in the use of climate products, insufficient experimentation and vetting of communication mechanisms, and the absence of a way to steward a diverse network of regional actors.
The scientific credibility and perceptions of credibility by information users are evolving at the CariCOF in several ways. CIMH is building the capacity of national and regional organizations to quality control data, develop new monitoring activities, and train in analysis techniques. However, verification of the forecasts and communication of the forecast skill to stakeholders is generally acknowledged to have room for improvement. More discussion on limits of the information and its uncertainties could help build perceptions of salience and legitimacy among end users, as could processes that enable users to be more part of the knowledge creation process (Doblas-Reyes et al. 2013; Kirchhoff et al. 2012). Capacity-building efforts at the CariCOF are more focused on the NMHS community than on the attending stakeholders. To address this gap, we propose convening a less technical climate-training session at the CariCOF with decision-makers.
Presenting and testing tailored products at the CariCOF can be an important element of market research and customizing products for distribution to a wider network. However, historically, we find that tailoring has consisted largely of asking participants what they think of the climate products. We know from earlier studies in the Caribbean that even when high-quality meteorological data are shared, usable information around impacts and interventions to a particular sector like agriculture may vary from one country to another (USAID 2014). Because user needs vary by place and sector, grouping all feedback from participants together is not optimal.
The selection of products has expanded from tercile precipitation forecasts offered in 2012 to products that also convey information about temperature, drought, extreme precipitation, and coral reef health. However, the extent to which these products reflect good investments of time and resources is unclear, and this signifies an important contribution of process monitoring for future research and development. Simply increasing the number of products can undermine the goals of developing regional capacity to manage climate risks; producing more products requires more research and development, involves greater stewardship, and may actually confuse stakeholders. Clearly, there are resource tradeoffs in satisfying diverse needs; market research should be a more important element of product development.
We observed efforts by CIMH to find new ways to foster communication between producer and user communities to enhance familiarity of both groups with the challenges and opportunities faced by their counterparts and come to some mutual understanding of how to best support climate-informed decision-making. Given challenges people have with using probabilistic information (Stern and Easterling 1999), training sessions to help stakeholders interpret the probabilistic information might allow them to more effectively use the forecasts, make more informed decisions, and more accurately communicate the messages to others.
Beginning in 2014, we can see a shift toward more participatory activities and greater focus on stakeholder input at the CariCOF. But new, experimental mechanisms to engage CariCOF participants—like the CariCOF Theatre—represent untested methods of communicating climate information. In addition, we observed that communication at the CariCOF is solely in English, despite the fact that the CariCOF and CIMH also serve French-, Spanish-, and Dutch-speaking communities.
There is also a need for further study of how stakeholders understand and subsequently use the information from the CariCOF, and to what extent their networks find it useful. Many RCOFs, for example, have not explored users’ comprehension of the information (Mason and Chidzambwa 2009; Ogallo et al. 2008). Understanding and documenting how people use information in specific contexts would be valuable for assessing impacts of the information. Considering the extent to which users understand the outlook products and find them useful is also important. Initial efforts to implement some of these recommendations are underway at the CariCOF, but they will require careful study and evaluation.
Although a knowledge network is being built in the Caribbean, some sectors have participated less/more infrequently than others, notably water management, health, and tourism. As the lead meteorological and hydrological regional organization in the Caribbean, CIMH plays an important role by serving as a boundary organization and capacity builder and by facilitating interaction, often between people or groups operating within different institutional and professional cultures (Feldman and Ingram 2009; Guido et al. 2016; Robinson and Gilfillan 2017). Over the past several CariCOFs, and especially since 2015, CIMH has been engaging the historically less-represented sectors, and the EWISACTS Consortium is expected to further this effort.
7. A path forward: Building a common process-oriented evaluation framework
Although the specific context and climate conditions differ across RCOFs and other emerging climate services, these services are similar in that they produce and disseminate technical forecast information, involve cross-disciplinary experts including decision-makers and meteorologists, develop interest in and capacity to understand climate information with attending user groups, and steward a regional network. Developing ways to take stock based on the six elements of our process-oriented framework can help shed light on advances and persistent challenges across the RCOFs. Ideally, this proposed framework can be used to structure inquiry around the implementation of RCOFs, help establish a move toward a more comprehensive evaluation framework, and contribute to the broader evaluation of climate services. We discuss how the six process elements may serve as a starting point for other RCOFs and organizations charged with the provision of climate services more broadly. Based on our study of the CariCOF, we propose an additional element to our conceptual framework: the development of flexible mechanisms to understand and respond to changing needs and conditions.
a. Quality of climate information
A process-oriented evaluation of the RCOFs and climate services largely should identify activities undertaken to assess scientific quality and perceptions of the credibility of information. Having a longitudinal perspective is important for assessing improvements over time; the quality of information continually evolves as new methods arise, data availability changes, and user groups change in RCOFs. Taking stock of the rigor and objectivity of this process is an important step in improving the quality of consensus-based seasonal climate outlooks.
b. Range and diversity of climate products
A process-oriented evaluation should identify the diversity of climate information the RCOF currently supports, the activities employed to assess whether they are used and useful, and the processes by which new information is deemed necessary. Moreover, RCOFs can identify the ability of the RCOF participants—including the meteorological services—to meet this demand.
c. Tailoring of climate information
Evaluation of the RCOF should consider whether products are tailored to meet specific user needs; it should explore the extent to which they are developed in conjunction with specific users or make use of contextual sectoral information, as well as how that tailoring is enabled. With regard to the latter, documenting feedback opportunities and enabling diverse participation can help avoid responding to the loudest voice and being led down a path that only benefits a few individuals. Evaluators can also gauge the extent to which users report being able to understand and apply the information in question.
d. Communication of climate information
Communication at the RCOFs can be advanced by inviting media participants, training stakeholders (including media) on the technical nuances of the information (including its limitations and uncertainties), continually assessing user understanding of information, testing different communication strategies, and promoting activities that strengthen contextual understanding of the forecasts. Considering the extent to which users understand the information products and find them useful is also important.
e. Stewarding knowledge networks
If RCOFs are to support communication of climate information at scale, they must consciously build and nurture networks that can facilitate two-way feedback, promote diversity of participants, and inform both the development of relevant climate information products and the accessibility and use of products by relevant user groups. Although NMHS representatives are critical to the success of such events, without representative user groups participating in RCOF events and accessing its products, the utility of that information is reduced.
f. Developing flexible mechanisms to understand and respond to changing needs and conditions
RCOFs are a work in progress, and the relevant organizing bodies should explicitly strive to learn and improve products and processes over time by understanding the ways that information is used and digested by various user groups. User needs change over time, so the RCOF process should be flexible to adapt to changing information and needs. Mechanisms to engage participants about how best to evolve their activities and gauge the effectiveness of their strategies are critical. Further, coordinating organizations like CIMH can adopt flexible mechanisms and pathways to support continued learning in their own organizational practices (Pelling et al. 2008). One strategy may be to integrate reflexive learning into regular project meetings and updates. In this way, a more process-oriented approach is elevated, and ongoing evaluation serves to build participation and ownership through the process (Denton 2009, p. 118), thereby serving as a mechanism for learning.
In conclusion, our process-oriented framework can serve as a guidepost to direct short-term improvements in the forums and to inform longer-term efforts in assessing which activities help build resilience to climate variability and change. Process-oriented evaluations can help RCOFs identify for themselves their target sectors and locations, scale up their utility and effectiveness, and, ultimately, support climate resilience.
Acknowledgments
This research was conducted within the International Research and Applications Project, a joint effort by the University of Arizona and Columbia University's International Research Institute for Climate and Society. It was supported by the National Oceanic and Atmospheric Administration Grant NA13OAR4310184 with support from the U.S. Agency for International Development.
REFERENCES
Basher, R., C. Clark, M. Dilley, and M. Harrison, 2000: Coping with the climate: A way forward—Summary and proposals for action. International Research Institute for Climate Prediction Pub. IRI-CW/01/2, 31 pp., http://www.wmo.int/pages/prog/wcp/wcasp/documents/PretoriaSumRpt2.pdf.
Bedward, S., and C. J. Van Meerbeeck, 2013: Assessing the skill of seasonal rainfall outlooks for the Caribbean. Geophysical Research Abstracts, Vol. 15, Abstract EGU2013-5968, http://meetingorganizer.copernicus.org/EGU2013/EGU2013-5968.pdf.
Berri, G., P. Antico, and L. Goddard, 2005: Evaluation of the Climate Outlook Forums’ seasonal precipitation forecasts of southeast South America during 1998–2002. Int. J. Climatol., 25, 365–377, https://doi.org/10.1002/joc.1129.
Bidwell, D., T. Dietz, and D. Scavia, 2013: Fostering knowledge networks for climate adaptation. Nat. Climate Change, 3, 610–611, https://doi.org/10.1038/nclimate1931.
Buizer, J., K. Jacobs, and D. Cash, 2016: Making short-term climate forecasts useful: Linking science and action. Proc. Natl. Acad. Sci. USA, 113, 4597–4602, https://doi.org/10.1073/pnas.0900518107.
Cash, D. W., W. Clark, F. Alcock, N. Dickson, N. Eckley, and J. Jäger, 2002: Salience, credibility, legitimacy and boundaries: Linking research, assessment, and decision making. KSG Working Papers Series RWP02-046, 25 pp., https://dx.doi.org/10.2139/ssrn.372280.
Cash, D. W., W. Clark, F. Alcock, N. Dickson, N. Eckley, D. Guston, J. Jäger, and R. Mitchell, 2003: Knowledge systems for sustainable development. Proc. Natl. Acad. Sci. USA. 100, 8086–8091, https://doi.org/10.1073/pnas.1231332100.
Chen, H., and P. Rossi, 1989: Issues in the theory-driven perspective. Eval. Program Plann., 12, 299–306, https://doi.org/10.1016/0149-7189(89)90046-3.
CIMH, 2015: Terms of reference for the Consortium of Regional Sectoral Early Warning Information Systems across Climate Timescales (EWISACTs) coordination partners. BRCCC Programme Rep., 18 pp., https://rcc.cimh.edu.bb/files/2016/10/TOR-Sectoral-EWISACTs-Consortium-and-5-Sectoral-Partners.pdf.
CIMH, 2016: Caribbean Climate Outlook Forum (CariCOF) fact sheet. Regional Climate Outlook Forums (RCOFs), World Meteorological Organization, https://library.wmo.int/opac/doc_num.php?explnum_id=3191.
Della Cruce, G., 2012: Data sharing and collaboration: Regional and national climate outlook forums in South America. International Research Centre on El Niño Rep., 6 pp., http://www.climate-services.org/wp-content/uploads/2015/09/CIIFEN_WCSACOF_Case_Study.pdf.
Denton, F., 2009: Challenges for evaluating adaptation to climate change within the context of Africa. Evaluating Climate Change and Development, R. D. van den Berg and O. N. Feinstein, Eds., World Bank Series on Development, Vol. 8, Transaction Publishers, 115–136.
Dilling, L., and M. Lemos, 2011: Creating usable science: Opportunities and constraints for climate knowledge use and their implications for science policy. Global Environ. Change, 21, 680–689, https://doi.org/10.1016/j.gloenvcha.2010.11.006.
Doblas-Reyes, F., J. García-Serrano, F. Lienert, A. Biescas, and L. Rodrigues, 2013: Seasonal climate predictability and forecasting: Status and prospects. Wiley Interdiscip. Rev.: Climate Change, 4, 245–268, https://doi.org/10.1002/wcc.217.
Duflo, E., 2004: Scaling up and evaluation. Annual World Bank Conf. on Development Economics, Washington, DC, International Bank for Reconstruction and Development, 341–369.
Farrell, D., A. Trotman, and C. Cox, 2010: Drought early warning and risk reduction: A case study of the drought of 2009–2010. Global assessment report on disaster risk reduction. United Nations International Strategy for Disaster Reduction (ISDR), 22 pp., http://www.preventionweb.net/english/hyogo/gar/2011/en/bgdocs/Farrell_et_al_2010.pdf.
Feldman, D., and H. Ingram, 2009: Making science useful to decision makers: Climate forecasts, water management, and knowledge networks. Wea. Climate Soc., 1, 9–21, https://doi.org/10.1175/2009WCAS1007.1.
Fetterman, D. M., and A. Wandersman, 2005: Empowerment Evaluation Principles in Practice. Guilford Press, 231 pp.
Fetterman, D. M., and A. Wandersman, 2007: Empowerment evaluation: Yesterday, today, and tomorrow. Amer. J. Eval., 28, 179–198, https://doi.org/10.1177/1098214007301350.
Garcia-Solera, I., and P. Ramirez, 2012: Central America’s seasonal climate outlook forum. Regional Water Resources Committee Rep., 8 pp., http://www.climate-services.org/wp-content/uploads/2015/09/CRRH_Case_Study.pdf.
Goddard, L., J. Hurrell, B. Kirtman, J. Murphy, T. Stockdale, and C. Vera, 2012: Two time scales for the price of one (almost). Bull. Amer. Meteor. Soc., 93, 621–629, https://doi.org/10.1175/BAMS-D-11-00220.1.
Goddard, L., W. Baethgen, H. Bhojwani, and A. Robertson, 2014: The International Research Institute for Climate & Society: Why, what and how. Earth Perspect., 1, 10, https://doi.org/10.1186/2194-6434-1-10.
Guba, E. G., and Y. S. Lincoln, 1989: Fourth Generation Evaluation. SAGE Publications, 296 pp.
Guido, Z., D. Hill, M. Crimmins, and D. Ferguson, 2013: Informing decisions with a climate synthesis product: Implications for regional climate services. Wea. Climate Soc., 5, 83–92, https://doi.org/10.1175/WCAS-D-12-00012.1.
Guido, Z., V. Rountree, C. Greene, A. Gerlak, and A. Trotman, 2016: Connecting climate information producers and users: Boundary organization, knowledge networks, and information brokers at Caribbean Climate Outlook Forums. Wea. Climate Soc., 8, 285–298, https://doi.org/10.1175/WCAS-D-15-0076.1.
Henry, A., and B. Vollan, 2014: Networks and the challenge of sustainable development. Annu. Rev. Environ. Resour., 39, 583–610, https://doi.org/10.1146/annurev-environ-101813-013246.
Howitt, R., J. Medellín-Azuara, D. MacEwan, J. R. Lund, and D. A. Summer, 2014: Economic analysis of the 2014 drought for California agriculture. Center for Watershed Sciences Tech. Rep., 20 pp., https://watershed.ucdavis.edu/files/content/news/Economic_Impact_of_the_2014_California_Water_Drought.pdf.
Hyvärinen, O., L. Mtilatila, K. Pilli-Sihvola, A. Venäläinen, and H. Gregow, 2015: The verification of seasonal precipitation forecasts for early warning in Zambia and Malawi. Adv. Sci. Res, 12, 31–36, https://doi.org/10.5194/asr-12-31-2015.
IPCC, 2012: Managing the Risks of Extreme Events and Disasters to Advance Climate Change Adaptation. Cambridge University Press, 582 pp.
Jacobs, K., and Coauthors, 2016: Linking knowledge with action in the pursuit of sustainable water resources management. Proc. Natl. Acad. Sci. USA, 113, 4591–4596, https://doi.org/10.1073/pnas.0813125107.
Kadi, M., 2012: Climate information and development: Regional Climate Outlook Forums in Africa. African Center of Meteorological Application for Development Rep., 5 pp., http://www.climate-services.org/wp-content/uploads/2015/09/RCOF_Africa_Case_Study.pdf.
Kalafatis, S. E., M. Lemos, Y. Lo, and K. Frank, 2015: Increasing information usability for climate adaptation: The role of knowledge networks and communities of practice. Global Environ. Change, 32, 30–39, https://doi.org/10.1016/j.gloenvcha.2015.02.007.
Kirchhoff, C., M. Lemos, and N. Engle, 2012: What influences climate information use in water management? The role of boundary organizations and governance regimes in Brazil and the U.S. Environ. Sci. Policy, 26, 6–18, https://doi.org/10.1016/j.envsci.2012.07.001.
Kirchhoff, C., M. Lemos, and S. Dessai, 2013: Actionable knowledge for environmental decision making: Broadening the usability of climate science. Annu. Rev. Environ. Resour., 38, 393–414, https://doi.org/10.1146/annurev-environ-022112-112828.
Lemos, M., and B. Morehouse, 2005: The co-production of science and policy in integrated climate assessments. Global Environ. Change, 15, 57–68, https://doi.org/10.1016/j.gloenvcha.2004.09.004.
Lemos, M., and L. Dilling, 2007: Equity in forecasting climate: Can science save the world’s poor? Sci. Public Policy, 34, 109–116, https://doi.org/10.3152/030234207X190964.
Lemos, M., T. Finan, R. Fox, D. Nelson, and J. Tucker, 2002: The use of seasonal climate forecasting in policymaking: Lessons from Northeast Brazil. Climatic Change, 55, 479–507, https://doi.org/10.1023/A:1020785826029.
Lemos, M., C. Kirchhoff, and V. Ramprasad, 2012: Narrowing the climate information usability gap. Nat. Climate Change, 2, 789–794, https://doi.org/10.1038/nclimate1614.
Lemos, M., C. Kirchhoff, S. Kalafatis, D. Scavia, and R. Rood, 2014: Moving climate information off the shelf: Boundary chains and the role of RISAs as adaptive organizations. Wea. Climate Soc., 6, 273–285, https://doi.org/10.1175/WCAS-D-13-00044.1.
Lieberman, E., 2005: Nested analysis as a mixed-method strategy for comparative research. Amer. Polit. Sci. Rev., 99, 435–452, https://doi.org/10.1017/S0003055405051762.
Lowe, R., M. García-Díez, J. Ballester, J. Creswick, J. Robine, F. Herrmann, and X. Rodó, 2016: Evaluation of an early-warning system for heat wave-related mortality in Europe: Implications for sub-seasonal to seasonal forecasting and climate services. Int. J. Environ. Res. Public Health, 13, 206, https://doi.org/10.3390/ijerph13020206.
Mason, S., and S. Chidzambwa, 2009: Position paper: Verification of RCOF forecasts. IRI Tech. Rep. 09–02, 26 pp., https://doi.org/10.7916/D85T3SB0.
Mason, S., and M. Tippett, 2016: Climate Predictability Tool Version 15.3.9. Columbia University: International Research Institute for Climate and Society, https://doi.org/10.7916/D8668DCW.
McNie, E., 2007: Reconciling the supply of scientific information with user demands: An analysis of the problem and review of the literature. Environ. Sci. Policy, 10, 17–38, https://doi.org/10.1016/j.envsci.2006.10.004.
McNie, E., 2013: Delivering climate services: Organizational strategies and approaches for producing useful climate-science information. Wea. Climate Soc., 5, 14–26, https://doi.org/10.1175/WCAS-D-11-00034.1.
Meinke, H., R. Nelson, P. Kokic, R. Stone, R. Selvaraju, and W. Baethgen, 2006: Actionable climate knowledge : From analysis to synthesis. Climate Res., 33, 101–110, https://doi.org/10.3354/cr033101.
Moser, S., 2009: Making a difference on the ground: The challenge of demonstrating the effectiveness of decision support. Climatic Change, 95, 11–21, https://doi.org/10.1007/s10584-008-9539-1.
Moser, S., and A. Luers, 2008: Managing climate risks in California: The need to engage resource managers for successful adaptation to change. Climatic Change, 87, 309–322, https://doi.org/10.1007/s10584-007-9384-7.
NRC, 1999: Making Climate Forecasts Matter. National Academies Press, 192 pp.
NRC, 2009: Informing Decisions in a Changing Climate. National Academies Press, 200 pp.
Oakley, A., V. Strange, C. Bonell, E. Allen, and J. Stephenson, 2006: Process evaluation in randomised controlled trials of complex interventions. BMJ, 332, 413–416, https://doi.org/10.1136%2Fbmj.332.7538.413.
Ogallo, L., P. Bessemoulin, J. Ceron, S. Mason, and S. Connor, 2008: Adapting to climate variability and change: The Climate Outlook Forum process. WMO Bull., 57, 93–102, https://public.wmo.int/en/bulletin/adapting-climate-variability-and-change-climate-outlook-forum-process.
Pagano, T., H. Hartmann, and S. Sorooshian, 2002: Factors affecting seasonal forecast use in Arizona water management: A case study of the 1997–98 El Niño. Climate Res., 21, 259–269, https://doi.org/10.3354/cr021259.
Pelling, M., C. High, J. Dearing, and D. Smith, 2008: Shadow spaces for social learning: A relational understanding of adaptive capacity to climate change within organisations. Environ. Plann., 40A, 867–884, https://doi.org/10.1068/a39148.
Robinson, S., and D. Gilfillan, 2017: Regional organisations and climate change adaptation in small island developing states. Reg. Environ. Change, 17, 989–1004, https://doi.org/10.1007/s10113-016-0991-6.
Rossi, P., M. Lipsey, and H. Freeman, 2004: Evaluation: A Systematic Approach. 7th ed. SAGE Publications, 480 pp.
Sarewitz, D., and R. Pielke Jr., 2007: The neglected heart of science policy: Reconciling supply of and demand for science. Environ. Sci. Policy, 10, 5–16, https://doi.org/10.1016/j.envsci.2006.10.001.
Scaramella, C., R. Choularton, and K. Krishnamurthy, 2012: Early warning systems for food security in Eastern Africa: Linking the Food Security Outlook with the Climate Outlook Forum. United Nations World Food Programme Rep., 4 pp., http://www.climate-services.org/wp-content/uploads/2015/09/WFP_Case_Study.pdf.
Stern, P., and W. Easterling, 1999: Making Climate Forecasts Matter. The National Academies Press, 192 pp.
Tschakert, P., and K. A. Dietrich, 2010: Anticipatory learning for climate change adaptation and resilience. Ecol. Soc., 15, 11, http://www.ecologyandsociety.org/vol15/iss2/art11/.
USAID, 2014: The Caribbean Agrometereological Initiative (CAMI): An evaluation of climate services. USAID Tech Rep., 43 pp., https://www.climatelinks.org/sites/default/files/asset/document/Caribbean.Agrometeorological.Initiative.An_.Evaluation.of_.Climate.Services.pdf.
Vaughan, C., and S. Dessai, 2014: Climate services for society: Origins, institutional arrangements, and design elements for an evaluation framework. Wiley Interdiscip. Rev.: Climate Change, 5, 587–603, https://doi.org/10.1002/wcc.290.
Vincent, K., A. Dougill, J. Dixon, L. Stringer, and T. Cull, 2015: Identifying climate services needs for national planning: Insights from Malawi. Climate Policy, 17, 189–202, https://doi.org/10.1080/14693062.2015.1075374.
Vogel, C., and K. O’Brien, 2006: Who can eat information? Examining the effectiveness of seasonal climate forecasts and regional climate-risk management strategies. Climate Res., 33, 111–122, https://doi.org/10.3354/cr033111.
Wall, T. U., A. M. Meadow, and A. Horganic, 2017: Developing evaluation indicators to improve the process of coproducing usable climate science. Wea. Climate Soc., 9, 95–107, https://doi.org/10.1175/WCAS-D-16-0008.1.
WMO, 2008: RCOF Review 2008: An international expert review meeting on Regional Climate Outlook Forums. World Meteorological Organization, Geneva, Switzerland.
WMO, 2016: Regional Climate Outlook Forums. World Meteorological Organization Rep., 52 pp., http://library.wmo.int/opac/doc_num.php?explnum_id=3191.
Yin, R., 1994: Case Study Research. Design and Methods. 2nd ed. SAGE Publications, 170 pp.
Ziervogel, G., and R. Calder, 2003: Climate variability and rural vlivelihoods: Assessing the impact of seasonal climate forecasts. Area, 35, 403–417, https://doi.org/10.1111/j.0004-0894.2003.00190.x.
RCOFs were first conceptualized in 1997 by a team at NOAA's Office of Global Programs, in preparation for the impending El Niño. The concept was introduced and widely accepted by participants at a “Workshop on Reducing Climate-Related Vulnerability in Southern Africa” held in Victoria Falls, Zimbabwe, on 1–4 October 1997 [J. Buizer, 2017, personal communication (18 Jan 2017)].
For a portfolio of enclosed factsheets detailing the 19 RCOFs, see http://library.wmo.int/opac/doc_num.php?explnum_id=3191.
To learn more about Global Producing Centres for Long-Range Forecasts, visit http://www.wmo.int/pages/prog/wcp/wcasp/gpc/gpc.php.
