• Alessandri, A., , A. Borrelli, , A. Navarra, , A. Arribas, , M. Déqué, , P. Rogel, , and A. Weisheimer, 2011: Evaluation of probabilistic quality and value of the ENSEMBLES multimodel seasonal forecasts: Comparison with DEMETER. Mon. Wea. Rev., 139, 581607, doi:10.1175/2010MWR3417.1.

    • Search Google Scholar
    • Export Citation
  • Brier, G. W., 1950: Verification of forecasts expressed in terms of probability. Mon. Wea. Rev., 78, 13, doi:10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Déqué, M., 1997: Ensemble size for numerical seasonal forecasts. Tellus, 49A, 7486, doi:10.1034/j.1600-0870.1997.00005.x.

  • Doblas-Reyes, F. J., , R. Hagedorn, , and T. N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting—II. Calibration and combination. Tellus, 57A, 234252, doi:10.1111/j.1600-0870.2005.00104.x.

    • Search Google Scholar
    • Export Citation
  • Doblas-Reyes, F. J., and et al. , 2009: Addressing model uncertainty in seasonal and annual dynamical ensemble forecasts. Quart. J. Roy. Meteor. Soc., 135, 15381559, doi:10.1002/qj.464.

    • Search Google Scholar
    • Export Citation
  • Fan, Y., , and H. van den Dool, 2008: A global monthly land surface air temperature analysis for 1948–present. J. Geophys. Res., 113, D01103, doi:10.1029/2007JD008470.

    • Search Google Scholar
    • Export Citation
  • Ferro, C., 2007: Comparing probabilistic forecasting systems with the Brier score. Wea. Forecasting, 22, 10761088, doi:10.1175/WAF1034.1.

    • Search Google Scholar
    • Export Citation
  • Hagedorn, R., , F. J. Doblas-Reyes, , and T. N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting—I. Basic concept. Tellus, 57A, 219–233, doi:10.1111/j.1600-0870.2005.00103.x.

    • Search Google Scholar
    • Export Citation
  • Jolliffe, I. T., , and D. B. Stephenson, 2012: Forecast Verification: A Practitioner’s Guide in Atmospheric Science. 2nd ed. John Wiley and Sons, 274 pp.

  • Kharin, V. V., , and F. W. Zwiers, 2003: On the ROC score of probability forecasts. J. Climate, 16, 41454150, doi:10.1175/1520-0442(2003)016<4145:OTRSOP>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Kharin, V. V., , Q. Teng, , F. W. Zwiers, , G. J. Boer, , J. Derome, , and J. S. Fontecilla, 2009: Skill assessment of seasonal hindcasts from the Canadian Historical Forecast Project. Atmos.–Ocean, 47, 204223, doi:10.3137/AO1101.2009.

    • Search Google Scholar
    • Export Citation
  • Kirtman, B. P., , and D. Min, 2009: Multimodel ensemble ENSO prediction with CCSM and CFS. Mon. Wea. Rev., 137, 29082930, doi:10.1175/2009MWR2672.1.

    • Search Google Scholar
    • Export Citation
  • Kirtman, B. P., and et al. , 2014: The North American Multimodel Ensemble: Phase-1 seasonal to interannual prediction; phase-2 toward developing intraseasonal prediction. Bull. Amer. Meteor. Soc., 95, 585601, doi:10.1175/BAMS-D-12-00050.1.

    • Search Google Scholar
    • Export Citation
  • Kumar, A., , A. Barnston, , and M. Hoerling, 2001: Seasonal predictions, probabilistic verifications, and ensemble size. J. Climate, 14, 16711676, doi:10.1175/1520-0442(2001)014<1671:SPPVAE>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Merryfield, W. J., and et al. , 2013: The Canadian Seasonal to Interannual Prediction System. Part I: Models and initialization. Mon. Wea. Rev., 141, 29102945, doi:10.1175/MWR-D-12-00216.1.

    • Search Google Scholar
    • Export Citation
  • Palmer, T. N., and et al. , 2004: Development of a European multimodel ensemble system for seasonal-to-interannual prediction (DEMETER). Bull. Amer. Meteor. Soc., 85, 853872, doi:10.1175/BAMS-85-6-853.

    • Search Google Scholar
    • Export Citation
  • Peña, M., , and H. van den Dool, 2008: Consolidation of multimodel forecasts by ridge regression: Application to Pacific sea surface temperature. J. Climate, 21, 65216538, doi:10.1175/2008JCLI2226.1.

    • Search Google Scholar
    • Export Citation
  • Reynolds, R. W., , N. A. Rayner, , T. M. Smith, , D. C. Stokes, , and W. Wang, 2002: An improved in situ and satellite SST analysis for climate. J. Climate, 15, 16091625, doi:10.1175/1520-0442(2002)015<1609:AIISAS>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Richardson, D. S., 2001: Measures of skill and value of ensemble prediction systems, their interrelationship and the effect of ensemble size. Quart. J. Roy. Meteor. Soc., 127, 24732489, doi:10.1002/qj.49712757715.

    • Search Google Scholar
    • Export Citation
  • Saha, S., and et al. , 2014: The NCEP Climate Forecast System version 2. J. Climate, 27, 21852208, doi:10.1175/JCLI-D-12-00823.1.

  • Tebaldi, C., , and R. Knutti, 2007: The use of the multi-model ensemble in probabilistic climate projections. Philos. Trans. Roy. Meteor. Soc., 365, 20532075, doi:10.1098/rsta.2007.2076.

    • Search Google Scholar
    • Export Citation
  • Unger, D. A., , H. van den Dool, , E. O’Lenic, , and D. C. Collins, 2008: Ensemble regression. Mon. Wea. Rev., 137, 2365–2379, doi:10.1175/2008MWR2605.1.

  • van den Dool, H. M., , and Z. Toth, 1991: Why do forecasts for near normal often fail? Wea. Forecasting, 6, 7685, doi:10.1175/1520-0434(1991)006<0076:WDFFNO>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Vernieres, G., , M. M. Rienecker, , R. Kovach, , and C. L. Keppenne, 2012: The GEOS-iODAS: Description and evaluation. NASA Tech. Rep. NASA/TM-2012-104606, Vol. 30, 61 pp. [Available online at http://gmao.gsfc.nasa.gov/pubs/docs/Vernieres589.pdf.]

  • Weisheimer, A., , and T. Palmer, 2014: On the reliability of seasonal climate forecasts. J. Roy. Soc. Interface, 11, 20131162, doi:10.1098/rsif.2013.1162.

    • Search Google Scholar
    • Export Citation
  • Weisheimer, A., and et al. , 2009: ENSEMBLES: A new multi-model ensemble for seasonal-to-annual predictions—Skill and progress beyond DEMETER in forecasting tropical Pacific SSTs. Geophys. Res. Lett., 36, L21711, doi:10.1029/2009GL040896.

    • Search Google Scholar
    • Export Citation
  • Wilks, D., 2006: Statistical Methods in the Atmospheric Sciences. Academic Press, 627 pp.

  • Xie, P., , and P. A. Arkin, 1997: Global precipitation: A 17-year monthly analysis based on gauge observations, satellite estimates, and numerical model outputs. Bull. Amer. Meteor. Soc., 78, 25392558, doi:10.1175/1520-0477(1997)078<2539:GPAYMA>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Zhang, S., , M. J. Harrison, , A. Rosati, , and A. Wittenberg, 2007: System design and evaluation of coupled ensemble data assimilation for global oceanic climate studies. Mon. Wea. Rev., 135, 35413564, doi:10.1175/MWR3466.1.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 76 76 6
PDF Downloads 67 67 6

Probabilistic Seasonal Forecasts in the North American Multimodel Ensemble: A Baseline Skill Assessment

View More View Less
  • 1 Climate Prediction Center, NCEP/NWS/NOAA, College Park, Maryland
© Get Permissions
Restricted access

Abstract

The North American Multimodel Ensemble (NMME) forecasting system has been continuously producing seasonal forecasts since August 2011. The NMME, with its suite of diverse models, provides a valuable opportunity for characterizing forecast confidence using probabilistic forecasts. The current experimental probabilistic forecast product (in map format) presents the most likely tercile for the seasonal mean value, chosen out of above normal, near normal, or below normal categories, using a nonparametric counting method to determine the probability of each class. The skill of the 3-month-mean probabilistic forecasts of 2-m surface temperature (T2m), precipitation rate, and sea surface temperature is assessed using forecasts from the 29-yr (1982–2010) NMME hindcast database. Three forecast configurations are considered: a full six-model NMME; a “mini-NMME” with 24 members, four each from six models; and the 24-member CFSv2 alone. Skill is assessed on the cross-validated hindcasts using the Brier skill score (BSS); forecast reliability and resolution are also assessed. This study provides a baseline skill assessment of the current method of creating probabilistic forecasts from the NMME system.

For forecasts in the above- and below-normal terciles for all variables and geographical regions examined in this study, BSS for NMME forecasts is higher than BSS for CFSv2 forecasts. Niño-3.4 forecasts from the full NMME and the mini-NMME receive nearly identical BSS that are higher than BSS for CFSv2 forecasts. Even systems with modest BSS, such as T2m in the Northern Hemisphere, have generally high reliability, as shown in reliability diagrams.

Corresponding author address: Emily Becker, NOAA Center for Weather and Climate Prediction, 5830 University Research Court, College Park, MD 20740. E-mail: emily.becker@noaa.gov

Abstract

The North American Multimodel Ensemble (NMME) forecasting system has been continuously producing seasonal forecasts since August 2011. The NMME, with its suite of diverse models, provides a valuable opportunity for characterizing forecast confidence using probabilistic forecasts. The current experimental probabilistic forecast product (in map format) presents the most likely tercile for the seasonal mean value, chosen out of above normal, near normal, or below normal categories, using a nonparametric counting method to determine the probability of each class. The skill of the 3-month-mean probabilistic forecasts of 2-m surface temperature (T2m), precipitation rate, and sea surface temperature is assessed using forecasts from the 29-yr (1982–2010) NMME hindcast database. Three forecast configurations are considered: a full six-model NMME; a “mini-NMME” with 24 members, four each from six models; and the 24-member CFSv2 alone. Skill is assessed on the cross-validated hindcasts using the Brier skill score (BSS); forecast reliability and resolution are also assessed. This study provides a baseline skill assessment of the current method of creating probabilistic forecasts from the NMME system.

For forecasts in the above- and below-normal terciles for all variables and geographical regions examined in this study, BSS for NMME forecasts is higher than BSS for CFSv2 forecasts. Niño-3.4 forecasts from the full NMME and the mini-NMME receive nearly identical BSS that are higher than BSS for CFSv2 forecasts. Even systems with modest BSS, such as T2m in the Northern Hemisphere, have generally high reliability, as shown in reliability diagrams.

Corresponding author address: Emily Becker, NOAA Center for Weather and Climate Prediction, 5830 University Research Court, College Park, MD 20740. E-mail: emily.becker@noaa.gov
Save