• Armour, K. C., , C. M. Bitz, , and G. H. Roe, 2013: Time-varying climate sensitivity from regional feedbacks. J. Climate, 26, 45184534, doi:10.1175/JCLI-D-12-00544.1.

    • Search Google Scholar
    • Export Citation
  • Christensen, J. H., , E. Kjellström, , F. Giorgi, , G. Lenderink, , and M. Rummukainen, 2010: Weight assignment in regional climate models. Climate Res., 44, 179–194, doi:10.3354/cr00916.

    • Search Google Scholar
    • Export Citation
  • Dessler, A. E., 2013: Observations of climate feedbacks over 2000–10 and comparisons to climate models. J. Climate, 26, 333342, doi:10.1175/JCLI-D-11-00640.1.

    • Search Google Scholar
    • Export Citation
  • Doelling, D. R., and Coauthors, 2013: Geostationary enhanced temporal interpolation for CERES flux products. J. Atmos. Oceanic Technol., 30, 10721090, doi:10.1175/JTECH-D-12-00136.1.

    • Search Google Scholar
    • Export Citation
  • Giorgi, F., , and L. O. Mearns, 2002: Calculation of average, uncertainty range, and reliability of regional climate changes from AOGCM simulations via the “reliability ensemble averaging” (REA) method. J. Climate, 15, 11411158, doi:10.1175/1520-0442(2002)015<1141:COAURA>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Gleckler, P. J., , K. E. Taylor, , and C. Doutriaux, 2008: Performance metrics for climate models. J. Geophys. Res., 113, D06104, doi:10.1029/2007JD008972.

  • Hansen, J., , A. Lacis, , D. Rind, , G. Russel, , P. Stone, , I. Fung, , and R. Ruedy, 1984: Climate sensitivity: Analysis of feedback mechanisms. Climate Processes and Climate Sensitivity, Geophys. Monogr., Vol. 5, Amer. Geophys. Union, 130–163.

  • Hansen, J., , R. Ruedy, , M. Sato, , and K. Lo, 2010: Global surface temperature change. Rev. Geophys., 48, RG4004, doi:10.1029/2010RG000345.

  • Hartmann, D. L., , and P. Ceppi, 2014: Trends in the CERES dataset, 2000–13: The effects of sea ice and jet shifts and comparison to climate models. J. Climate, 27, 24442456, doi:10.1175/JCLI-D-13-00411.1.

    • Search Google Scholar
    • Export Citation
  • Kato, S., , N. G. Loeb, , F. G. Rose, , D. R. Doelling, , D. A. Rutan, , T. E. Caldwell, , L. Yu, , and R. A. Weller, 2013: Surface irradiances consistent with CERES-derived top-of-atmosphere shortwave and longwave irradiances. J. Climate, 26, 27192740, doi:10.1175/JCLI-D-12-00436.1.

    • Search Google Scholar
    • Export Citation
  • Knutti, R., 2010: The end of model democracy? Climatic Change, 102, 395–404, doi:10.1007/s10584-010-9800-2.

  • Knutti, R., , R. Furrer, , C. Tebaldi, , J. Cermak, , and G. A. Meehl, 2010: Challenges in combining projections from multiple climate models. J. Climate, 23, 27392758, doi:10.1175/2009JCLI3361.1.

    • Search Google Scholar
    • Export Citation
  • Lambert, F. H., , and P. C. Taylor, 2014: Regional variation of the tropical water vapor and lapse rate feedbacks. Geophys. Res. Lett., 41, 76347641, doi:10.1002/2014GL061987.

    • Search Google Scholar
    • Export Citation
  • Loeb, N. G., , B. A. Wielicki, , D. R. Doelling, , G. L. Smith, , D. F. Keyes, , S. Kato, , N. Manalo-Smith, , and T. Wong, 2009: Toward optimal closure of the earth’s top-of-atmosphere radiation budget. J. Climate, 22, 748766, doi:10.1175/2008JCLI2637.1.

    • Search Google Scholar
    • Export Citation
  • Loeb, N. G., , S. Kato, , W. Su, , T. Wong, , F. G. Rose, , D. R. Doelling, , J. R. Norris, , and X. Huang, 2012: Advances in understanding top-of-atmosphere radiation variability from satellite observations. Surv. Geophys., 33, 359–385, doi:10.1007/s10712-012-9175-1.

    • Search Google Scholar
    • Export Citation
  • Madison, G., 2001: Variability in isochronous tapping: Higher order dependencies as a function of intertap interval. J. Exp. Psychol., 27, 411422.

    • Search Google Scholar
    • Export Citation
  • Pincus, R., , C. P. Batstone, , R. J. P. Hofmann, , K. E. Taylor, , and P. J. Glecker, 2008: Evaluating the present-day simulation of clouds, precipitation, and radiation in climate models. J. Geophys. Res., 113, D14209, doi:10.1029/2007JD009334.

  • Qu, X., , and A. Hall, 2006: Assessing snow albedo feedback in simulated climate change. J. Climate, 19, 26172630, doi:10.1175/JCLI3750.1.

    • Search Google Scholar
    • Export Citation
  • Reichler, T., , and J. Kim, 2008: How well do coupled models simulate today’s climate? Bull. Amer. Meteor. Soc., 89, 303311, doi:10.1175/BAMS-89-3-303.

    • Search Google Scholar
    • Export Citation
  • Rubner, Y., , C. Tomasi, , and L. J. Guibas, 2000: The Earth Mover’s Distance as a metric for image retrieval. Int. J. Comput. Vision, 40, 99121, doi:10.1023/A:1026543900054.

    • Search Google Scholar
    • Export Citation
  • Taylor, K. E., 2001: Summarizing multiple aspects of model performance in a single diagram. J. Geophys. Res., 106, 71837192, doi:10.1029/2000JD900719.

    • Search Google Scholar
    • Export Citation
  • Taylor, K. E., , R. J. Stouffer, , and G. A. Meehl, 2012: An overview of CMIP5 and the experiment design. Bull. Amer. Meteor. Soc., 93, 485498, doi:10.1175/BAMS-D-11-00094.1.

    • Search Google Scholar
    • Export Citation
  • Taylor, P. C., , R. G. Ellingson, , and M. Cai, 2011a: Geographical distribution of climate feedbacks in the NCAR CCSM3.0. J. Climate, 24, 27372753, doi:10.1175/2010JCLI3788.1.

    • Search Google Scholar
    • Export Citation
  • Taylor, P. C., , R. G. Ellingson, , and M. Cai, 2011b: Seasonal contributions to climate feedbacks in the NCAR CCSM3.0. J. Climate, 24, 34333444, doi:10.1175/2011JCLI3862.1.

    • Search Google Scholar
    • Export Citation
  • Waugh, D. W., , and V. Eyring, 2008: Quantitative performance metrics for stratospheric-resolving chemistry–climate models. Atmos. Chem. Phys., 8, 56995713, doi:10.5194/acp-8-5699-2008.

    • Search Google Scholar
    • Export Citation
  • WCRP, 2014: CMIP5—Coupled Model Intercomparison Project. Accessed December 2014. [Available online at http://cmip-pcmdi.llnl.gov/cmip5/.]

  • Wetherald, R. T., , and S. Manabe, 1988: Cloud feedback processes in a general circulation model. J. Atmos. Sci., 45, 13971415, doi:10.1175/1520-0469(1988)045<1397:CFPIAG>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Williams, K. D., , and M. J. Webb, 2009: A quantitative performance assessment of cloud regimes in climate models. Climate Dyn., 33, 141157, doi:10.1007/s00382-008-0443-1.

    • Search Google Scholar
    • Export Citation
  • Willmott, C. J., 1982: Some comments on the evaluation of model performance. Bull. Amer. Meteor. Soc., 63, 13091313, doi:10.1175/1520-0477(1982)063<1309:SCOTEO>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Xu, K. M., , T. Wong, , B. A. Wielicki, , L. Parker, , and Z. A. Eitzen, 2005: Statistical analyses of satellite cloud object data from CERES. Part I: Methodology and preliminary results of the 1998 El Niño/2000 La Niña. J. Climate, 18, 24972514, doi:10.1175/JCLI3418.1.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 218 218 56
PDF Downloads 232 232 47

A Framework for Evaluating Climate Model Performance Metrics

View More View Less
  • 1 NASA Langley Research Center, Hampton, Virginia
© Get Permissions
Restricted access

Abstract

Given the large amount of climate model output generated from the series of simulations from phase 5 of the Coupled Model Intercomparison Project (CMIP5), a standard set of performance metrics would facilitate model intercomparison and tracking performance improvements. However, no framework exists for the evaluation of performance metrics. The proposed framework systematically integrates observations into metric assessment to quantitatively evaluate metrics. An optimal metric is defined in this framework as one that measures a behavior that is strongly linked to model quality in representing mean-state present-day climate. The goal of the framework is to objectively and quantitatively evaluate the ability of a performance metric to represent overall model quality. The framework is demonstrated, and the design principles are discussed using a novel set of performance metrics, which assess the simulation of top-of-atmosphere (TOA) and surface radiative flux variance and probability distributions within 34 CMIP5 models against Clouds and the Earth’s Radiant Energy System (CERES) observations and GISS Surface Temperature Analysis (GISTEMP). Of the 44 tested metrics, the optimal metrics are found to be those that evaluate global-mean TOA radiation flux variance.

Corresponding author address: Noel C. Baker, NASA Langley Research Center, 21 Langley Blvd., Mail Stop 420, Hampton, VA 23681-2199. E-mail: noel.c.baker@nasa.gov

Abstract

Given the large amount of climate model output generated from the series of simulations from phase 5 of the Coupled Model Intercomparison Project (CMIP5), a standard set of performance metrics would facilitate model intercomparison and tracking performance improvements. However, no framework exists for the evaluation of performance metrics. The proposed framework systematically integrates observations into metric assessment to quantitatively evaluate metrics. An optimal metric is defined in this framework as one that measures a behavior that is strongly linked to model quality in representing mean-state present-day climate. The goal of the framework is to objectively and quantitatively evaluate the ability of a performance metric to represent overall model quality. The framework is demonstrated, and the design principles are discussed using a novel set of performance metrics, which assess the simulation of top-of-atmosphere (TOA) and surface radiative flux variance and probability distributions within 34 CMIP5 models against Clouds and the Earth’s Radiant Energy System (CERES) observations and GISS Surface Temperature Analysis (GISTEMP). Of the 44 tested metrics, the optimal metrics are found to be those that evaluate global-mean TOA radiation flux variance.

Corresponding author address: Noel C. Baker, NASA Langley Research Center, 21 Langley Blvd., Mail Stop 420, Hampton, VA 23681-2199. E-mail: noel.c.baker@nasa.gov
Save