Do Statistical Pattern Corrections Improve Seasonal Climate Predictions in the North American Multimodel Ensemble Models?

Anthony G. Barnston International Research Institute for Climate and Society, Columbia University, Palisades, New York

Search for other papers by Anthony G. Barnston in
Current site
Google Scholar
PubMed
Close
and
Michael K. Tippett Department of Applied Physics and Applied Mathematics, Columbia University, New York, New York, and Center of Excellence for Climate Change Research, Department of Meteorology, King Abdulaziz University, Jeddah, Saudi Arabia

Search for other papers by Michael K. Tippett in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Canonical correlation analysis (CCA)-based statistical corrections are applied to seasonal mean precipitation and temperature hindcasts of the individual models from the North American Multimodel Ensemble project to correct biases in the positions and amplitudes of the predicted large-scale anomaly patterns. Corrections are applied in 15 individual regions and then merged into globally corrected forecasts. The CCA correction dramatically improves the RMS error skill score, demonstrating that model predictions contain correctable systematic biases in mean and amplitude. However, the corrections do not materially improve the anomaly correlation skills of the individual models for most regions, seasons, and lead times, with the exception of October–December precipitation in Indonesia and eastern Africa. Models with lower uncorrected correlation skill tend to benefit more from the correction, suggesting that their lower skills may be due to correctable systematic errors. Unexpectedly, corrections for the globe as a single region tend to improve the anomaly correlation at least as much as the merged corrections to the individual regions for temperature, and more so for precipitation, perhaps due to better noise filtering. The lack of overall improvement in correlation may imply relatively mild errors in large-scale anomaly patterns. Alternatively, there may be such errors, but the period of record is too short to identify them effectively but long enough to find local biases in mean and amplitude. Therefore, statistical correction methods treating individual locations (e.g., multiple regression or principal component regression) may be recommended for today’s coupled climate model forecasts. The findings highlight that the performance of statistical postprocessing can be grossly overestimated without thorough cross validation or evaluation on independent data.

Denotes content that is immediately available upon publication as open access.

© 2017 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Anthony G. Barnston, tonyb@iri.columbia.edu

Abstract

Canonical correlation analysis (CCA)-based statistical corrections are applied to seasonal mean precipitation and temperature hindcasts of the individual models from the North American Multimodel Ensemble project to correct biases in the positions and amplitudes of the predicted large-scale anomaly patterns. Corrections are applied in 15 individual regions and then merged into globally corrected forecasts. The CCA correction dramatically improves the RMS error skill score, demonstrating that model predictions contain correctable systematic biases in mean and amplitude. However, the corrections do not materially improve the anomaly correlation skills of the individual models for most regions, seasons, and lead times, with the exception of October–December precipitation in Indonesia and eastern Africa. Models with lower uncorrected correlation skill tend to benefit more from the correction, suggesting that their lower skills may be due to correctable systematic errors. Unexpectedly, corrections for the globe as a single region tend to improve the anomaly correlation at least as much as the merged corrections to the individual regions for temperature, and more so for precipitation, perhaps due to better noise filtering. The lack of overall improvement in correlation may imply relatively mild errors in large-scale anomaly patterns. Alternatively, there may be such errors, but the period of record is too short to identify them effectively but long enough to find local biases in mean and amplitude. Therefore, statistical correction methods treating individual locations (e.g., multiple regression or principal component regression) may be recommended for today’s coupled climate model forecasts. The findings highlight that the performance of statistical postprocessing can be grossly overestimated without thorough cross validation or evaluation on independent data.

Denotes content that is immediately available upon publication as open access.

© 2017 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Anthony G. Barnston, tonyb@iri.columbia.edu
Save
  • Barnett, T. P., and R. Preisendorfer, 1987: Origins and levels of monthly and seasonal forecast skill for United States surface air temperatures determined by canonical correlation analysis. Mon. Wea. Rev., 115, 18251850, doi:10.1175/1520-0493(1987)115<1825:OALOMA>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Barnston, A. G., 1994: Linear statistical short-term climate predictive skill in the Northern Hemisphere. J. Climate, 7, 15131564, doi:10.1175/1520-0442(1994)007<1513:LSSTCP>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Barnston, A. G., and H. M. van den Dool, 1993: A degeneracy in cross-validated skill in regression-based forecasts. J. Climate, 6, 963977, doi:10.1175/1520-0442(1993)006<0963:ADICVS>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Barnston, A. G., M. K. Tippett, M. Ranganathan, and M. L. L’Heureux, 2017: Deterministic skill of ENSO predictions from the North American Multimodel Ensemble. Climate Dyn., doi:10.1007/s00382-017-3603-3, in press.

    • Search Google Scholar
    • Export Citation
  • Benjamini, Y., and Y. Hochberg, 1995: Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Stat. Soc., 57B, 289300.

    • Search Google Scholar
    • Export Citation
  • Bjerknes, J., 1969: Atmospheric teleconnections from the equatorial Pacific. J. Phys. Oceanogr., 97, 163172, doi:10.1175/1520-0493(1969)097<0163:ATFTEP>2.3.CO;2.

    • Search Google Scholar
    • Export Citation
  • Bretherton, C. S., C. Smith, and J. M. Wallace, 1992: An intercomparison of methods for finding coupled patterns in climate data. J. Climate, 5, 541560, doi:10.1175/1520-0442(1992)005<0541:AIOMFF>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • DelSole, T., and M. K. Tippett, 2014: Comparing forecast skill. Mon. Wea. Rev., 142, 46584678, doi:10.1175/MWR-D-14-00045.1.

  • Delworth, T. L., and Coauthors, 2006: GFDL’s CM2 global coupled climate models. Part I: Formulation and simulation characteristics. J. Climate, 19, 643674, doi:10.1175/JCLI3629.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fan, Y., and H. van den Dool, 2008: A global monthly land surface air temperature analysis for 1948–present. J. Geophys. Res., 113, D01103, doi:10.1029/2007JD008470.

    • Search Google Scholar
    • Export Citation
  • Feddersen, H., A. Navarra, and M. N. Ward, 1999: Reduction of model systematic error by statistical correction for dynamical seasonal predictions. J. Climate, 12, 19741989, doi:10.1175/1520-0442(1999)012<1974:ROMSEB>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gent, P. R., and Coauthors, 2011: The Community Climate System Model version 4. J. Climate, 24, 49734991, doi:10.1175/2011JCLI4083.1.

  • Goddard, L., and N. E. Graham, 1999: The importance of the Indian Ocean for simulating precipitation anomalies over eastern and southern Africa. J. Geophys. Res., 104, 19 09919 116, doi:10.1029/1999JD900326.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hays, W. L., 1973: Statistics for the Social Sciences. Holt, Rinehart and Winston, Inc., 954 pp.

  • Infanti, J. M., and B. P. Kirtman, 2016: Prediction and predictability of land and atmosphere initialized CCSM4 climate forecasts over North America. J. Geophys. Res. Atmos., 121, 12 69012 701, doi:10.1002/2016JD024932.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Johansson, A., A. Barnston, S. Saha, and H. M. van den Dool, 1998: On the level and origin of seasonal forecast skill in northern Europe. J. Atmos. Sci., 55, 103127, doi:10.1175/1520-0469(1998)055<0103:OTLAOO>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kirtman, B. P., and Coauthors, 2014: The North American Multimodel Ensemble (NMME): Phase-1 seasonal-to-interannual prediction; phase-2 toward developing intraseasonal prediction. Bull. Amer. Meteor. Soc., 95, 585601, doi:10.1175/BAMS-D-12-00050.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Livezey, R. E., and W. Y. Chen, 1983: Statistical significance and its determination by Monte Carlo techniques. Mon. Wea. Rev., 111, 4659, doi:10.1175/1520-0493(1983)111<0046:SFSAID>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Merryfield, W. J., and Coauthors, 2013: The Canadian seasonal to interannual prediction system. Part I: Models and initialization. Mon. Wea. Rev., 141, 29102945, doi:10.1175/MWR-D-12-00216.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mo, R., and D. M. Straus, 2002: Statistical–dynamical seasonal prediction based on principal component regression of GCM ensemble integrations. Mon. Wea. Rev., 130, 21672187, doi:10.1175/1520-0493(2002)130<2167:SDSPBO>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Moron, V., A. W. Robertson, M. N. Ward, and P. Camberlin, 2007: Spatial coherence of tropical rainfall at the regional scale. J. Climate, 20, 52445263, doi:10.1175/2007JCLI1623.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., and E. S. Epstein, 1989: Skill scores and correlation coefficients in model verification. Mon. Wea. Rev., 117, 572581, doi:10.1175/1520-0493(1989)117<0572:SSACCI>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ropelewski, C. F., J. E. Janowiak, and M. S. Halpert, 1984: The Climate Anomaly Monitoring System (CAMS). Climate Analysis Center Tech. Rep., 39 pp.

  • Rukhovets, L. V., H. M. van den Dool, and A. G. Barnston, 1998: Forecast-observation pattern relationships in NCEP medium range forecasts of non-winter Northern Hemisphere 500-mb height fields. Atmos.–Ocean, 36, 5570, doi:10.1080/07055900.1998.9649606.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Saha, S., and Coauthors, 2014: The NCEP Climate Forecast System version 2. J. Climate, 27, 21852208, doi:10.1175/JCLI-D-12-00823.1.

  • Smith, T. M., and R. E. Livezey, 1999: GM systematic error correction and specification of the seasonal mean Pacific–North America region atmosphere from global SSTs. J. Climate, 12, 273288, doi:10.1175/1520-0442-12.1.273.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Tippett, M., M. Barlow, and B. Lyon, 2003: Statistical correction of central southwest Asia winter precipitation simulations. Int. J. Climatol., 23, 14211433, doi:10.1002/joc.947.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Tippett, M., T. DelSole, S. Mason, and A. G. Barnston, 2008: Regression-based methods for finding coupled patterns. J. Climate, 21, 43844398, doi:10.1175/2008JCLI2150.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Tippett, M., T. DelSole, and A. Barnston, 2014: Reliability of regression-corrected climate forecasts. J. Climate, 27, 33933404, doi:10.1175/JCLI-D-13-00565.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Van den Dool, H. M., and R. M. Chervin, 1986: A comparison of month-to-month persistence of anomalies in a general circulation model and in the Earth’s atmosphere. J. Atmos. Sci., 43, 14541466, doi:10.1175/1520-0469(1986)043<1454:ACOMTM>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Van den Dool, H. M., E. Becker, L.-C. Chen, and Q. Zhang, 2017: The probability anomaly correlation and calibration of probabilistic forecasts. Wea. Forecasting, 32, 199206, doi:10.1175/WAF-D-16-0115.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Vecchi, G. A., and Coauthors, 2014: On the seasonal forecasting of regional tropical cyclone activity. J. Climate, 27, 79948016, doi:10.1175/JCLI-D-14-00158.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Vernieres, G., M. M. Rienecker, R. Kovach, and C. L. Keppenne, 2012: The GEOS-iODAS: Description and evaluation. NASA Tech. Rep. NASA/TM-2012-104606, Vol. 30, 61 pp. [Available online at http://gmao.gsfc.nasa.gov/pubs/docs/Vernieres589.pdf.]

  • Walker, G., and T. Bliss, 1934: World weather V. Mem. Roy. Meteor. Soc., 4, 5384.

  • Ward, N. N., and A. Navarra, 1997: Pattern analysis of SST-forced variability in ensemble GCM simulations: Examples over Europe and the tropical Pacific. J. Climate, 11, 711743, doi:10.1175/1520-0442(1997)010<2210:PAOSFV>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2006: On “field significance” and the false discovery rate. J. Appl. Meteor. Climatol., 45, 11811189, doi:10.1175/JAM2404.1.

  • Wilks, D. S., 2016: “The stippling shows statistically significant grid points”: How research results are routinely overstated and overinterpreted, and what to do about it. Bull. Amer. Meteor. Soc., 97, 22632273, doi:10.1175/BAMS-D-15-00267.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Xie, P., and P. A. Arkin, 1997: Global precipitation: A 17-year monthly analysis based on gauge observations, satellite estimations, and numerical model outputs. Bull. Amer. Meteor. Soc., 78, 25392558, doi:10.1175/1520-0477(1997)078<2539:GPAYMA>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zhang, S., M. J. Harrison, A. Rosati, and A. Wittenberg, 2007: System design and evaluation of coupled ensemble data assimilation for global oceanic climate studies. Mon. Wea. Rev., 135, 35413564, doi:10.1175/MWR3466.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 476 131 15
PDF Downloads 368 83 6