Combination of Multimodel Probabilistic Forecasts Using an Optimal Weighting System

Li-Chuan Gwen Chen Earth System Science Interdisciplinary Center/Cooperative Institute for Climate and Satellites, University of Maryland, College Park, and NOAA/NWS/NCEP Climate Prediction Center, College Park, Maryland

Search for other papers by Li-Chuan Gwen Chen in
Current site
Google Scholar
PubMed
Close
and
Huug van den Dool NOAA/NWS/NCEP Climate Prediction Center, College Park, and Innovim, LLC, Greenbelt, Maryland

Search for other papers by Huug van den Dool in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

In this study, an optimal weighting system is developed that combines multiple seasonal probabilistic forecasts in the North American Multimodel Ensemble (NMME). The system is applied to predict temperature and precipitation over the North American continent, and the analysis is conducted using the 1982–2010 hindcasts from eight NMME models, including the CFSv2, CanCM3, CanCM4, GFDL CM2.1, Forecast-Oriented Low Ocean Resolution (FLOR), GEOS5, CCSM4, and CESM models, with weights determined by minimizing the Brier score using ridge regression. Strategies to improve the performance of ridge regression are explored, such as eliminating a priori models with negative skill and increasing the effective sample size by pooling information from neighboring grids. A set of constraints is put in place to confine the weights within a reasonable range or restrict the weights from departing wildly from equal weights. So when the predictor–predictand relationship is weak, the multimodel ensemble forecast returns to an equal-weight combination. The new weighting system improves the predictive skill from the baseline, equally weighted forecasts. All models contribute to the weighted forecasts differently based upon location and forecast start and lead times. The amount of improvement varies across space and corresponds to the average model elimination percentage. The areas with higher elimination rates tend to show larger improvement in cross-validated verification scores. Some local improvements can be as large as 0.6 in temporal probability anomaly correlation (TPAC). On average, the results are about 0.02–0.05 in TPAC for temperature probabilistic forecasts and 0.03–0.05 for precipitation probabilistic forecasts over North America. The skill improvement is generally greater for precipitation probabilistic forecasts than for temperature probabilistic forecasts.

© 2017 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: L. Gwen Chen, lichuan.chen@noaa.gov

Abstract

In this study, an optimal weighting system is developed that combines multiple seasonal probabilistic forecasts in the North American Multimodel Ensemble (NMME). The system is applied to predict temperature and precipitation over the North American continent, and the analysis is conducted using the 1982–2010 hindcasts from eight NMME models, including the CFSv2, CanCM3, CanCM4, GFDL CM2.1, Forecast-Oriented Low Ocean Resolution (FLOR), GEOS5, CCSM4, and CESM models, with weights determined by minimizing the Brier score using ridge regression. Strategies to improve the performance of ridge regression are explored, such as eliminating a priori models with negative skill and increasing the effective sample size by pooling information from neighboring grids. A set of constraints is put in place to confine the weights within a reasonable range or restrict the weights from departing wildly from equal weights. So when the predictor–predictand relationship is weak, the multimodel ensemble forecast returns to an equal-weight combination. The new weighting system improves the predictive skill from the baseline, equally weighted forecasts. All models contribute to the weighted forecasts differently based upon location and forecast start and lead times. The amount of improvement varies across space and corresponds to the average model elimination percentage. The areas with higher elimination rates tend to show larger improvement in cross-validated verification scores. Some local improvements can be as large as 0.6 in temporal probability anomaly correlation (TPAC). On average, the results are about 0.02–0.05 in TPAC for temperature probabilistic forecasts and 0.03–0.05 for precipitation probabilistic forecasts over North America. The skill improvement is generally greater for precipitation probabilistic forecasts than for temperature probabilistic forecasts.

© 2017 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: L. Gwen Chen, lichuan.chen@noaa.gov
Save
  • Barnston, A. G., and H. M. Van den Dool, 1993: A degeneracy in cross-validated skill in regression-based forecasts. J. Climate, 6, 963977, doi:10.1175/1520-0442(1993)006<0963:ADICVS>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Becker, E., and H. Van den Dool, 2016: Probabilistic seasonal forecasts in the North American Multimodel Ensemble: A baseline skill assessment. J. Climate, 29, 30153026, doi:10.1175/JCLI-D-14-00862.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Becker, E., H. Van den Dool, and Q. Zhang, 2014: Predictability and forecast skill in NMME. J. Climate, 27, 58915906, doi:10.1175/JCLI-D-13-00597.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Brier, G. W., 1950: Verification of forecasts expressed in terms of probability. Mon. Wea. Rev., 78, 13, doi:10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Chen, L.-C., H. Van den Dool, E. Becker, and Q. Zhang, 2017: ENSO precipitation and temperature forecasts in the North American Multimodel Ensemble: Composite analysis and validation. J. Climate, 30, 11031125, doi:10.1175/JCLI-D-15-0903.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Chen, M., P. Xie, J. E. Janowiak, and P. A. Arkin, 2002: Global land precipitation: A 50-yr monthly analysis based on gauge observations. J. Hydrometeor., 3, 249266, doi:10.1175/1525-7541(2002)003<0249:GLPAYM>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Danabasoglu, G., S. C. Bates, B. P. Briegleb, S. R. Jayne, M. Jochum, W. G. Large, S. Peacock, and S. G. Yeager, 2012: The CCSM4 ocean component. J. Climate, 25, 13611389, doi:10.1175/JCLI-D-11-00091.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • DelSole, T., 2007: A Bayesian framework for multimodel regression. J. Climate, 20, 28102826, doi:10.1175/JCLI4179.1.

  • DelSole, T., X. Yang, and M. K. Tippett, 2013: Is unequal weighting significantly better than equal weighting for multi-model forecasting? Quart. J. Roy. Meteor. Soc., 139, 176183, doi:10.1002/qj.1961.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Delworth, T. L., and Coauthors, 2006: GFDL’s CM2 global coupled climate models. Part I: Formulation and simulation characteristics. J. Climate, 19, 643674, doi:10.1175/JCLI3629.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Doblas-Reyes, F. J., R. Hagedorn, and T. N. Palmer, 2005: The rationale behind the success of multi-model ensemble in seasonal forecasting—II. Calibration and combination. Tellus, 57A, 234252, doi:10.1111/j.1600-0870.2005.00104.x.

    • Search Google Scholar
    • Export Citation
  • Fan, Y., and H. M. Van den Dool, 2008: A global monthly land surface air temperature analysis for 1948–present. J. Geophys. Res., 113, D01103, doi:10.1029/2007JD008470.

    • Search Google Scholar
    • Export Citation
  • Gent, P. R., and Coauthors, 2011: The Community Climate System Model version 4. J. Climate, 24, 49734991, doi:10.1175/2011JCLI4083.1.

  • Giorgi, F., and R. Francisco, 2000: Uncertainties in regional climate change prediction: A regional analysis of ensemble simulations with the HADCM2 coupled AOGCM. Climate Dyn., 16, 169182, doi:10.1007/PL00013733.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gnanadesikan, A., and Coauthors, 2006: GFDL’s CM2 global coupled climate models. Part II: The baseline ocean simulation. J. Climate, 19, 675697, doi:10.1175/JCLI3630.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hagedorn, R., F. J. Doblas-Reyes, and T. N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting – I. Basic concept. Tellus, 57A, 219233, doi:10.1111/j.1600-0870.2005.00103.x.

    • Search Google Scholar
    • Export Citation
  • Hurrell, J. W., and Coauthors, 2013: The Community Earth System Model: A framework for collaborative research. Bull. Amer. Meteor. Soc., 94, 13391360, doi:10.1175/BAMS-D-12-00121.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jia, L., and Coauthors, 2015: Improved seasonal prediction of temperature and precipitation over land in a high-resolution GFDL climate model. J. Climate, 28, 20442064, doi:10.1175/JCLI-D-14-00112.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kharin, V. V., and F. W. Zwiers, 2002: Climate predictions with multimodel ensembles. J. Climate, 15, 793799, doi:10.1175/1520-0442(2002)015<0793:CPWME>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kirtman, B. P., and Coauthors, 2014: The North American Multi-Model Ensemble (NMME): Phase-1 seasonal to interannual prediction; phase-2 toward developing intra-seasonal prediction. Bull. Amer. Meteor. Soc., 95, 585601, doi:10.1175/BAMS-D-12-00050.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Krishnamurti, T. N., C. M. Kishtawal, T. E. LaRow, D. R. Bachiochi, Z. Zhang, C. E. Williford, S. Gadgil, and S. Surendran, 1999: Improved weather and seasonal climate forecasts from multimodel superensemble. Science, 285, 15481550, doi:10.1126/science.285.5433.1548.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Krishnamurti, T. N., C. M. Kishtawal, Z. Zhang, T. LaRow, D. Bachiochi, E. Williford, S. Gadgil, and S. Surendran, 2000: Multimodel ensemble forecasts for weather and seasonal climate. J. Climate, 13, 41964216, doi:10.1175/1520-0442(2000)013<4196:MEFFWA>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Merryfield, W. J., and Coauthors, 2013: The Canadian Seasonal to Interannual Prediction System. Part I: Models and initialization. Mon. Wea. Rev., 141, 29102945, doi:10.1175/MWR-D-12-00216.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Peña, M., and H. M. Van den Dool, 2008: Consolidation of multimodel forecasts by ridge regression: Application to Pacific sea surface temperature. J. Climate, 21, 65216538, doi:10.1175/2008JCLI2226.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Peng, P., A. Kumar, H. M. van den Dool, and A. G. Barnston, 2002: An analysis of multimodel ensemble predictions for seasonal climate anomalies. J. Geophys. Res., 107, 4710, doi:10.1029/2002JD002712.

    • Search Google Scholar
    • Export Citation
  • Phillips, D. L., 1962: A technique for the numerical solution of certain integral equations of the first kind. J. Assoc. Comput. Mach., 9, 8497, doi:10.1145/321105.321114.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Saha, S., and Coauthors, 2006: The NCEP Climate Forecast System. J. Climate, 19, 34833517, doi:10.1175/JCLI3812.1.

  • Saha, S., and Coauthors, 2014: The NCEP Climate Forecast System version 2. J. Climate, 27, 21852208, doi:10.1175/JCLI-D-12-00823.1.

  • Tikhonov, A., 1963: Solution of incorrectly formulated problems and the regularization method. Sov. Math. Dokl., 4, 651667.

  • Tikhonov, A., and V. Y. Arsenin, 1977: Solutions of Ill-Posed Problems. Winston, 258 pp.

  • Van den Dool, H. M., 2007: Empirical Methods in Short-Term Climate Prediction. Oxford University Press, 215 pp.

    • Crossref
    • Export Citation
  • Van den Dool, H. M., and Z. Toth, 1991: Why do forecasts for “near normal” often fail? Wea. Forecasting, 6, 7685, doi:10.1175/1520-0434(1991)006<0076:WDFFNO>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Van den Dool, H. M., and L. Rukhovets, 1994: On the weights for an ensemble-averaged 6–10-day forecast. Wea. Forecasting, 9, 457465, doi:10.1175/1520-0434(1994)009<0457:OTWFAE>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Van den Dool, H. M., E. Becker, L.-C. Chen, and Q. Zhang, 2017: The probability anomaly correlation and calibration of probabilistic forecasts. Wea. Forecasting, 32, 199206, doi:10.1175/WAF-D-16-0115.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Vecchi, G. A., and Coauthors, 2014: On the seasonal forecasting of regional tropical cyclone activity. J. Climate, 27, 79948016, doi:10.1175/JCLI-D-14-00158.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Vernieres, G., C. Keppenne, M. M. Rienecker, J. Jacob, and R. Kovach, 2012: The GEOS-ODAS, description and evaluation. NASA Tech. Rep. NASA/TM-2012-104606, NASA Tech. Rep. Series on Global Modeling and Data Assimilation, Vol. 30, 61 pp., https://gmao.gsfc.nasa.gov/pubs/docs/Vernieres589.pdf.

  • Wanders, N., and E. F. Wood, 2016: Improved sub-seasonal meteorological forecast skill using weighted multi-model ensemble simulations. Environ. Res. Lett., 11, 094007, doi:10.1088/1748-9326/11/9/094007.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Weigel, A. P., M. A. Liniger, and C. Appenzeller, 2008: Can multi-model combination really enhance the prediction skill of probabilistic ensemble forecasts? Quart. J. Roy. Meteor. Soc., 134, 241260, doi:10.1002/qj.210.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Weisheimer, A., and Coauthors, 2009: ENSEMBLES: A new multi-model ensemble for seasonal-to-annual predictions—Skill and progress beyond DEMETER in forecasting tropical Pacific SSTs. Geophys. Res. Lett., 36, L21711, doi:10.1029/2009GL040896.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Yun, W., L. Stefanova, and T. Krishnamurti, 2003: Improvement of the multimodel superensemble technique for seasonal forecasts. J. Climate, 16, 38343840, doi:10.1175/1520-0442(2003)016<3834:IOTMST>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 1082 690 62
PDF Downloads 339 45 9