A Workshop on Improving Our Methodologies of Selecting Earth System Models for Climate Change Impact Applications

Andrew J. Newman National Center for Atmospheric Research, Boulder, Colorado;
National Center for Atmospheric Research, Boulder, Colorado;

Search for other papers by Andrew J. Newman in
Current site
Google Scholar
PubMed
Close
,
Jeffrey R. Arnold Responses to Climate Change Program, U.S. Army Corps of Engineers, Seattle, Washington

Search for other papers by Jeffrey R. Arnold in
Current site
Google Scholar
PubMed
Close
,
Andrew W. Wood National Center for Atmospheric Research, Boulder, Colorado;

Search for other papers by Andrew W. Wood in
Current site
Google Scholar
PubMed
Close
, and
Ethan D. Gutmann National Center for Atmospheric Research, Boulder, Colorado;

Search for other papers by Ethan D. Gutmann in
Current site
Google Scholar
PubMed
Close
Full access

© 2022 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Andrew J. Newman, anewman@ucar.edu

© 2022 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Andrew J. Newman, anewman@ucar.edu

Workshop on Earth System Model Selection for Water Security Applications

What:

Participants from operational water agencies and Earth system model (ESM) development and applications centers met to discuss new methods to evaluate the performance and suitability of ESMs for selecting and weighting ESMs in hydroclimate applications.

When:

12 and 16 February 2021

Where:

Virtual Workshop

The recent completion of most Coupled Model Intercomparison Project phase 6 (CMIP6) experiments necessitates new methods to evaluate the performance and suitability of Earth system models (ESMs) for selecting and weighting ESMs for hydroclimate applications. The continued proliferation of ESMs and the extremely large data volumes now present in the CMIP experiments (e.g., Eyring et al. 2016) could greatly complicate nascent stakeholder efforts to use new ESM outputs in their updated climate vulnerability and impact assessments. This is due to the combinatorial effect of including multiple models and analysis methods at each link in the impact modeling chain (ESMs–downscaling models–impact models), which is becoming a significant point of concern. Without subselecting ESMs in a clear and rational way, the newest science contained in the ESMs may not be available for impacts modelers to improve their own models and results. That, in turn, would severely limit the ability of water and energy sector managers to see and understand climate threats to their planning and operations, and to enhance the resilience of their plans and operations against those threats.

Historically, most water security focused ESM evaluations have used monthly or longer regionally focused hydroclimate metrics to assess ESM performance (e.g., Rupp et al. 2013; Ahmadalipour et al. 2017; Walsh et al. 2018). Crucially, this practice rarely assesses ESM performance for the very criteria most relevant to planning decisions—the fidelity of an ESM’s representation of future change across a range of metrics, nor for extreme precipitation events, event frequency, or other critical aspects of regional hydroclimate.

To begin to address these two core issues, the National Center for Atmospheric Research and the U.S. Army Corps of Engineers (USACE) sponsored a virtual workshop on 12 and 16 February 2021 to gather stakeholders from operational water agencies (USACE and the U.S. Bureau of Reclamation) and ESM model and application developers from NCAR, the Department of Energy, NOAA/Geophysical Fluid Dynamics Laboratory, the Pacific Climate Impacts Consortium, and several universities.

Goals and framing

The goals of the workshop were 1) to advance our understanding of the structures and behaviors of these complex models for projecting hydroclimate change and other water security endpoints, 2) to increase our confidence in using ESMs for specific water security applications sensitive to climate change through the development of an advanced, scientifically defensible model evaluation and selection scheme, and 3) to provide a forum for stakeholders, impact modelers, and ESM developers to interact. Short talks were used to frame each day, followed by full group discussion of the presented material, then small group breakout discussions using specific framing questions as guides and finally full group breakout reports and daily synthesis discussion. A preworkshop reading list (available upon request) and survey questions (Table 1) were also developed to provide additional background context and guide the participants toward focused thinking about ESM evaluation and selection for our workshop specific scope.

Table 1.

Preworkshop survey questions used to condition workshop participant thinking and for workshop discussions with typical anonymized responses from workshop participants.

Table 1.

ESM evaluation framework

During the first day, we focused on principles of a coherent evaluation framework for a water security use case. The day began with talks from ESM developers highlighting their decision-making processes to define their CMIP-class model configurations through the lenses of ESM-centric and application-based metrics. Lively full group discourse followed considering what metrics from both categories could be useful for ESM evaluation for water security applications. Small group discussion was guided by questions around ESM evaluation including the following:

What are the most useful principles for evaluating models used in regional and subregional assessments of water security problems under climate changed futures?

  1. 1)How much weight should be attached to model performance measured against observations, particularly model performance for representing historical change, and how should we measure and compare performance with other model behaviors?
  2. 2)How much does independence of model structure or components matter, and how do we best measure that?
  3. 3)How much fidelity to what we know of processes in the real world should we expect of different types of model processes, and how much should we reward “high” process fidelity?

What other aspects of model structure and behavior can we evaluate that will give us more confidence in selecting and using models for water and energy security questions? Can we learn enough now to move on from considering all models roughly equal for our applications?

ESM Selection

During the second day, we focused on defining objective, scientifically defensible model selection and ensemble design. Identifying and culling obviously low-performing models emerged as a key first step, along with the idea of using global warming levels to organize the future changes (rather than as a series of annual temperatures or anomalies). In addition, participants supported development of storylines as paths forward to reduce dimensionality of the problem for water managers during the morning short talks and plenary session. Small group discussion on day 2 was guided by the following questions:

  1. 1)List three key processes for the ocean, land, atmosphere, sea ice, and cryosphere that need to be represented to give you confidence for future change studies.
  2. 2)What spatiotemporal scales (e.g., global, zonal, continental, subcontinental; annual, seasonal) would we be able to distinguish change signals above noise for temperature, precipitation, extreme precipitation, sea ice, teleconnections (e.g., ENSO, MJO), and synoptic-scale variability (e.g., ETCs, blocking)? (Premise: We accept the use of global mean temperature trend behavior as a metric for usability of an ESM in CMIP experiments—can we go beyond this in scale/variables/processes?)
  3. 3)Design a mock evaluation and weighting/selection scheme including metrics (along performance, process fidelity, model independence, or other axes) and hypothetical importance of each metric (e.g., 1–5 scale, with 5 being the most important) for a use case focused on PNW annual total and seasonality of streamflow. You are limited to a small set [O(10)] of GCMs from CMIP6.

Model culling and weighting generated the most debate in the small group and plenary report-out and summary session, reflecting the current state of the science. Two key objections to any model weighting schemes were advanced by many participants: all weighting methods appear ad hoc, and weighting schemes often cannot change the final projection results significantly except when the final weighted ensemble becomes unreliable because of including only a few ESMs and giving them high weights (e.g., Sanderson et al. 2017). Thus, model culling is most palatable using the principle of first identifying “bad” models—often easier than identifying “good” ones. The workshop participants agreed that, practically speaking, policy-makers are working now with mod­els weighted by some means, most often not declared or tested, and that policy-makers and other users of model outputs and model developers, too, would benefit from a transparent, well-documented process identifying the decision points and explaining how the weighting decisions are made.

Key messages

We identified the following key themes for consideration when creating evaluation methodologies designed to identify ESMs fit for stakeholder specific applications. Inclusion of these themes in ESM evaluation methodologies advance the goals of the workshop where all four themes map onto the workshop goals to varying degrees. The key themes are as follows:

  1. 1)Deeper information sharing between model developers and users to understand ESM process deficiencies, which requires more interactive and iterative approaches to ESM design.
  2. 2)Focusing on diversity of model response rather than just diversity of model genealogy.
  3. 3)Using more metrics from science fields and at scales not often applied for water resource applications: global-scale metrics (e.g., sea ice, oceanic heat content) before shifting to regionally focused metrics. Regional evaluations should include the traditional spatiotemporally averaged metrics and include event (e.g., frequency)-based, multivariable correlation, and spatial metrics among other more advanced evaluation options.
  4. 4)Specific emphasis on improving the evaluation of the ESM representation of the forced climate change response both globally and regionally where possible.

Finally, Figs. 1 and 2 highlight the current and needed impact assessment paradigms first highlighting the more traditional funnel approach with information flowing from ESM development through CMIP-class simulations to impact modelers to decision-makers (where decision-making/-makers here is used as a high-level grouping of all levels of decision-making), which limits interactions between decision-makers and ESM developers (Fig. 1), and a more iterative process, which includes a core iteration loop of the key constituent groups that are responsible for product development (Fig. 2). In Fig. 2, circles represent groups or institutions that develop products represented by rectangles. Two-way arrows with gradient shading highlight iterative discussions and knowledge sharing and transfer between groups, which ideally influence characteristics such as usefulness and usability (e.g., Findlater et al. 2021).

Fig. 1.
Fig. 1.

Current ESM development to climate resilience and adaptation decision-making transition “funnel” paradigm.

Citation: Bulletin of the American Meteorological Society 103, 4; 10.1175/BAMS-D-21-0316.1

Fig. 2.
Fig. 2.

Prototype workflow of an interactive and iterative ESM development to climate resilience and adaptation decision-making paradigm.

Citation: Bulletin of the American Meteorological Society 103, 4; 10.1175/BAMS-D-21-0316.1

Acknowledgments.

We thank all the workshop participants for their willingness to contribute their time, energy, and thoughts to this workshop. They made the event a success. We would also like to thank Chris Frans for his helpful comments and checking the accuracy of this meeting summary.

References

  • Ahmadalipour, A., A. Rana, H. Moradkhani, and A. Sharma , 2017: Multi-criteria evaluation of CMIP5 GCMs for climate change impact analysis. Theor. Appl. Climatol. , 128, 7187, https://doi.org/10.1007/s00704-015-1695-4.

    • Search Google Scholar
    • Export Citation
  • Eyring, V., S. Bony, G. A. Meehl, C. A. Senior, B. Stevens, R. J. Stouffer, and K. E. Taylor , 2016: Overview of the Coupled Model Intercomparison Project phase 6 (CMIP6) experimental design and organization. Geosci. Model Dev. , 9, 19371958, https://doi.org/10.5194/gmd-9-1937-2016.

    • Search Google Scholar
    • Export Citation
  • Findlater, K., S. Webber, M. Kandlikar, and S. Donner, 2021: Climate services promise better decisions but mainly focus on better data. Nat. Climate Change, 11, 731737, https://doi.org/10.1038/s41558-021-01125-3.

    • Search Google Scholar
    • Export Citation
  • Rupp, D. E., J. T. Abatzoglou, K. C. Hegewisch, and P. W. Mote , 2013: Evaluation of CMIP5 20th century climate simulations for the Pacific Northwest USA. J. Geophys. Res. Atmos. , 118, 10 88410 906, https://doi.org/10.1002/jgrd.50843.

    • Search Google Scholar
    • Export Citation
  • Sanderson, B. M., M. Wehner, and R. Knutti , 2017: Skill and independence weighting for multi-model assessments. Geosci. Model Dev. , 10, 23792395, https://doi.org/10.5194/gmd-10-2379-2017.

    • Search Google Scholar
    • Export Citation
  • Walsh, J. E., and Coauthors , 2018: Downscaling of climate model output for Alaskan stakeholders. Environ. Modell. Software , 110, 3851, https://doi.org/10.1016/j.envsoft.2018.03.021.

    • Search Google Scholar
    • Export Citation
Save
  • Ahmadalipour, A., A. Rana, H. Moradkhani, and A. Sharma , 2017: Multi-criteria evaluation of CMIP5 GCMs for climate change impact analysis. Theor. Appl. Climatol. , 128, 7187, https://doi.org/10.1007/s00704-015-1695-4.

    • Search Google Scholar
    • Export Citation
  • Eyring, V., S. Bony, G. A. Meehl, C. A. Senior, B. Stevens, R. J. Stouffer, and K. E. Taylor , 2016: Overview of the Coupled Model Intercomparison Project phase 6 (CMIP6) experimental design and organization. Geosci. Model Dev. , 9, 19371958, https://doi.org/10.5194/gmd-9-1937-2016.

    • Search Google Scholar
    • Export Citation
  • Findlater, K., S. Webber, M. Kandlikar, and S. Donner, 2021: Climate services promise better decisions but mainly focus on better data. Nat. Climate Change, 11, 731737, https://doi.org/10.1038/s41558-021-01125-3.

    • Search Google Scholar
    • Export Citation
  • Rupp, D. E., J. T. Abatzoglou, K. C. Hegewisch, and P. W. Mote , 2013: Evaluation of CMIP5 20th century climate simulations for the Pacific Northwest USA. J. Geophys. Res. Atmos. , 118, 10 88410 906, https://doi.org/10.1002/jgrd.50843.

    • Search Google Scholar
    • Export Citation
  • Sanderson, B. M., M. Wehner, and R. Knutti , 2017: Skill and independence weighting for multi-model assessments. Geosci. Model Dev. , 10, 23792395, https://doi.org/10.5194/gmd-10-2379-2017.

    • Search Google Scholar
    • Export Citation
  • Walsh, J. E., and Coauthors , 2018: Downscaling of climate model output for Alaskan stakeholders. Environ. Modell. Software , 110, 3851, https://doi.org/10.1016/j.envsoft.2018.03.021.

    • Search Google Scholar
    • Export Citation
  • Fig. 1.

    Current ESM development to climate resilience and adaptation decision-making transition “funnel” paradigm.

  • Fig. 2.

    Prototype workflow of an interactive and iterative ESM development to climate resilience and adaptation decision-making paradigm.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 749 147 8
PDF Downloads 403 84 7