Editorial Type:
Article
Article Type:
Research Article

Verification of TIGGE Multimodel and ECMWF Reforecast-Calibrated Probabilistic Precipitation Forecasts over the Contiguous United States*

Thomas M. Hamill NOAA/Earth System Research Laboratory/Physical Sciences Division, Boulder, Colorado

Search for other papers by Thomas M. Hamill in
Current site
Google Scholar
PubMed Close
Print Publication:
01 Jul 2012
DOI:
https://doi.org/10.1175/MWR-D-11-00220.1
Page(s):
2232–2252
Restricted access

Abstract

Probabilistic quantitative precipitation forecasts (PQPFs) were generated from The Observing System Research and Predictability Experiment (THORPEX) Interactive Grand Global Ensemble (TIGGE) database from July to October 2010 using data from Europe (ECMWF), the United Kingdom [Met Office (UKMO)], the United States (NCEP), and Canada [Canadian Meteorological Centre (CMC)]. Forecasts of 24-h accumulated precipitation were evaluated at 1° grid spacing within the contiguous United States against analysis data based on gauges and bias-corrected radar data.

PQPFs from ECMWF’s ensembles generally had the highest skill of the raw ensemble forecasts, followed by CMC. Those of UKMO and NCEP were less skillful. PQPFs from CMC forecasts were the most reliable but the least sharp, and PQPFs from NCEP and UKMO ensembles were the least reliable but sharper.

Multimodel PQPFs were more reliable and skillful than individual ensemble prediction system forecasts. The improvement was larger for heavier precipitation events [e.g., >10 mm (24 h)−1] than for smaller events [e.g., >1 mm (24 h)−1].

ECMWF ensembles were statistically postprocessed using extended logistic regression and the five-member weekly reforecasts for the June–November period of 2002–09, the period where precipitation analyses were also available. Multimodel ensembles were also postprocessed using logistic regression and the last 30 days of prior forecasts and analyses. The reforecast-calibrated ECMWF PQPFs were much more skillful and reliable for the heavier precipitation events than ECMWF raw forecasts but much less sharp. Raw multimodel PQPFs were generally more skillful than reforecast-calibrated ECMWF PQPFs for the light precipitation events but had about the same skill for the higher-precipitation events; also, they were sharper but somewhat less reliable than ECMWF reforecast-based PQPFs. Postprocessed multimodel PQPFs did not provide as much improvement to the raw multimodel PQPF as the reforecast-based processing did to the ECMWF forecast.

The evidence presented here suggests that all operational centers, even ECMWF, would benefit from the open, real-time sharing of precipitation forecast data and the use of reforecasts.

Supplemental information related to this paper is available at the Journals Online website: http://dx.doi.org/10.1175/MWR-D-11-00220.s1.

Corresponding author address: Dr. Thomas M. Hamill, NOAA/ESRL, Physical Sciences Division, R/PSD 1, 325 Broadway, Boulder, CO 80305-3328. E-mail: tom.hamill@noaa.gov

Abstract

Probabilistic quantitative precipitation forecasts (PQPFs) were generated from The Observing System Research and Predictability Experiment (THORPEX) Interactive Grand Global Ensemble (TIGGE) database from July to October 2010 using data from Europe (ECMWF), the United Kingdom [Met Office (UKMO)], the United States (NCEP), and Canada [Canadian Meteorological Centre (CMC)]. Forecasts of 24-h accumulated precipitation were evaluated at 1° grid spacing within the contiguous United States against analysis data based on gauges and bias-corrected radar data.

PQPFs from ECMWF’s ensembles generally had the highest skill of the raw ensemble forecasts, followed by CMC. Those of UKMO and NCEP were less skillful. PQPFs from CMC forecasts were the most reliable but the least sharp, and PQPFs from NCEP and UKMO ensembles were the least reliable but sharper.

Multimodel PQPFs were more reliable and skillful than individual ensemble prediction system forecasts. The improvement was larger for heavier precipitation events [e.g., >10 mm (24 h)−1] than for smaller events [e.g., >1 mm (24 h)−1].

ECMWF ensembles were statistically postprocessed using extended logistic regression and the five-member weekly reforecasts for the June–November period of 2002–09, the period where precipitation analyses were also available. Multimodel ensembles were also postprocessed using logistic regression and the last 30 days of prior forecasts and analyses. The reforecast-calibrated ECMWF PQPFs were much more skillful and reliable for the heavier precipitation events than ECMWF raw forecasts but much less sharp. Raw multimodel PQPFs were generally more skillful than reforecast-calibrated ECMWF PQPFs for the light precipitation events but had about the same skill for the higher-precipitation events; also, they were sharper but somewhat less reliable than ECMWF reforecast-based PQPFs. Postprocessed multimodel PQPFs did not provide as much improvement to the raw multimodel PQPF as the reforecast-based processing did to the ECMWF forecast.

The evidence presented here suggests that all operational centers, even ECMWF, would benefit from the open, real-time sharing of precipitation forecast data and the use of reforecasts.

Supplemental information related to this paper is available at the Journals Online website: http://dx.doi.org/10.1175/MWR-D-11-00220.s1.

Corresponding author address: Dr. Thomas M. Hamill, NOAA/ESRL, Physical Sciences Division, R/PSD 1, 325 Broadway, Boulder, CO 80305-3328. E-mail: tom.hamill@noaa.gov

Supplementary Materials

    • Supplemental Materials (ZIP 2.84 MB)
Save
Cover Monthly Weather Review
Monthly Weather Review

  • Accadia , C. , S. Mariani , M. Casaioli , A. Lavagnini , and A. Speranza , 2003 : Sensitivity of precipitation forecast skill scores to bilinear interpolation and a simple nearest-neighbor average method on high-resolution verification grids . Wea. Forecasting , 18 , 918 –932 .

    • Search Google Scholar
    • Export Citation

  • Bao , L. , T. Gneiting , E. P. Grimit , P. Guttorp , and A. E. Raftery , 2010 : Bias correction and bayesian model averaging for ensemble forecasts of surface wind direction . Mon. Wea. Rev. , 138 , 1811 –1821 .

    • Search Google Scholar
    • Export Citation

  • Barkmeijer , J. , F. Bouttier , and M. Van Gijzen , 1998 : Singular vectors and estimates of the analysis-error covariance metric . Quart. J. Roy. Meteor. Soc. , 124 , 1695 –1713 .

    • Search Google Scholar
    • Export Citation

  • Barkmeijer , J. , R. Buizza , and T. N. Palmer , 1999 : 3D-Var Hessian singular vectors and their potential use in the ECMWF ensemble prediction system . Quart. J. Roy. Meteor. Soc. , 125 , 2333 –2351 .

    • Search Google Scholar
    • Export Citation

  • Berner , J. , G. J. Shutts , M. Leutbecher , and T. N. Palmer , 2009 : A spectral stochastic kinetic energy backscatter scheme and its impact on flow-dependent predictability in the ECMWF ensemble prediction system . J. Atmos. Sci. , 66 , 603 –626 .

    • Search Google Scholar
    • Export Citation

  • Berner , J. , S.-Y. Ha , J. P. Hacker , A. Fournier , and C. Snyder , 2011 : Model uncertainty in a mesoscale ensemble prediction system: Stochastic versus multiphysics representations . Mon. Wea. Rev. , 139 , 1972 –1995 .

    • Search Google Scholar
    • Export Citation

  • Bougeault , P. , and Coauthors , 2010 : The THORPEX Interactive Grand Global Ensemble . Bull. Amer. Meteor. Soc. , 91 , 1059 –1072 .

    • Search Google Scholar
    • Export Citation

  • Bowler , N. E. , A. Arribas , K. R. Mylne , K. B. Robertson , and S. E. Beare , 2008 : The MOGREPS short-range ensemble prediction system . Quart. J. Roy. Meteor. Soc. , 134 , 703 –722 .

    • Search Google Scholar
    • Export Citation

  • Bowler , N. E. , A. Arribas , S. E. Beare , K. R. Mylne , and G. J. Shutts , 2009 : The local ETKF and SKEB: Upgrades to the MOGREPS short-range ensemble prediction system . Quart. J. Roy. Meteor. Soc. , 135 , 767 –776 .

    • Search Google Scholar
    • Export Citation

  • Buizza , R. , and T. N. Palmer , 1995 : The singular-vector structure of the atmospheric global circulation . J. Atmos. Sci. , 52 , 1434 –1456 .

    • Search Google Scholar
    • Export Citation

  • Buizza , R. , M. Miller , and T. N. Palmer , 1999 : Stochastic representation of model uncertainties in the ECMWF ensemble prediction system . Quart. J. Roy. Meteor. Soc. , 125 , 2887 –2908 .

    • Search Google Scholar
    • Export Citation

  • Buizza , R. , J.-R. Bidlot , N. Wedi , M. Fuentes , M. Hamrud , G. Holt , and F. Vitart , 2007 : The new ECMWF VAREPS (Variable Resolution Ensemble Prediction System) . Quart. J. Roy. Meteor. Soc. , 133 , 681 –695 .

    • Search Google Scholar
    • Export Citation

  • Candille , G. , 2009 : The multiensemble approach: The NAEFS example . Mon. Wea. Rev. , 137 , 1655 –1665 .

  • Charron , M. , G. Pellerin , L. Spacek , P. L. Houtekamer , N. Gagnon , H. L. Mitchell , and L. Michelin , 2010 : Toward random sampling of model error in the Canadian ensemble prediction system . Mon. Wea. Rev. , 138 , 1877 –1901 .

    • Search Google Scholar
    • Export Citation

  • Dee , D. P. , and Coauthors , 2011 : The ERA-Interim reanalysis: Configuration and performance of the data assimilation system . Quart. J. Roy. Meteor. Soc. , 137 , 553 –597 .

    • Search Google Scholar
    • Export Citation

  • Doblas-Reyes , F. J. , R. Hagedorn , and T. N. Palmer , 2005 : The rationale behind the success of multi-model ensembles in seasonal forecasting – II. Calibration and combination . Tellus , 57A , 234 –252 .

    • Search Google Scholar
    • Export Citation

  • Glahn , B. , M. Peroutka , J. Wiedenfeld , J. Wagner , G. Zylstra , B. Schuknecht , and B. Jackson , 2009 : MOS uncertainty estimates in an ensemble framework . Mon. Wea. Rev. , 137 , 246 –268 .

    • Search Google Scholar
    • Export Citation

  • Gneiting , T. , A. E. Raftery , A. H. Westveld , and T. Goldman , 2005 : Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation . Mon. Wea. Rev. , 133 , 1098 –1118 .

    • Search Google Scholar
    • Export Citation

  • Hagedorn , R. , 2008 : Using the ECMWF reforecast data set to calibrate EPS reforecasts. ECMWF Newsletter, No. 117, ECMWF, Reading, United Kingdom, 8–13 .

  • Hagedorn , R. , F. J. Doblas-Reyes , and T. N. Palmer , 2005 : The rationale behind the success of multi-model ensembles in seasonal forecasting – I. Basic concept . Tellus , 57A , 219 –233 .

    • Search Google Scholar
    • Export Citation

  • Hagedorn , R. , T. M. Hamill , and J. S. Whitaker , 2008 : Probabilistic forecast calibration using ECMWF and GFS ensemble reforecasts. Part I: Two-meter temperatures . Mon. Wea. Rev. , 136 , 2608 –2619 .

    • Search Google Scholar
    • Export Citation

  • Hagedorn , R. , R. Buizza , T. M. Hamill , M. Leutbecher , and T. N. Palmer , 2012 : Comparing TIGGE multi-model forecasts with reforecast-calibrated ECMWF ensemble forecasts . Quart. J. Roy. Meteor. Soc. , in press .

    • Search Google Scholar
    • Export Citation

  • Hamill , T. M. , 1999 : Hypothesis tests for evaluating numerical precipitation forecasts . Wea. Forecasting , 14 , 155 –167 .

  • Hamill , T. M. , and J. Juras , 2006 : Measuring forecast skill: Is it real skill or is it the varying climatology? Quart. J. Roy. Meteor. Soc. , 132 , 2905 –2923 .

    • Search Google Scholar
    • Export Citation

  • Hamill , T. M. , and J. S. Whitaker , 2006 : Probabilistic quantitative precipitation forecasts based on reforecast analogs: Theory and application . Mon. Wea. Rev. , 134 , 3209 –3229 .

    • Search Google Scholar
    • Export Citation

  • Hamill , T. M. , and J. S. Whitaker , 2007 : Ensemble calibration of 500-hPa geopotential height and 850-hPa and 2-m temperatures using reforecasts . Mon. Wea. Rev. , 135 , 3273 –3280 .

    • Search Google Scholar
    • Export Citation

  • Hamill , T. M. , J. S. Whitaker , and X. Wei , 2004 : Ensemble reforecasting: Improving medium-range forecast skill using retrospective forecasts . Mon. Wea. Rev. , 132 , 1434 –1447 .

    • Search Google Scholar
    • Export Citation

  • Hamill , T. M. , J. S. Whitaker , and S. L. Mullen , 2006 : Reforecasts: An important dataset for improving weather predictions . Bull. Amer. Meteor. Soc. , 87 , 33 –46 .

    • Search Google Scholar
    • Export Citation

  • Hamill , T. M. , R. Hagedorn , and J. S. Whitaker , 2008 : Probabilistic forecast calibration using ECMWF and GFS ensemble reforecasts. Part II: Precipitation . Mon. Wea. Rev. , 136 , 2620 –2632 .

    • Search Google Scholar
    • Export Citation

  • Hamill , T. M. , J. S. Whitaker , M. Fiorino , and S. G. Benjamin , 2011 : Global ensemble predictions of 2009’s tropical cyclones initialized with an ensemble Kalman filter . Mon. Wea. Rev. , 139 , 668 –688 .

    • Search Google Scholar
    • Export Citation

  • Higgins , R. W. , J. E. Janowiak , and Y.-P. Yao , 1996 : A gridded hourly precipitation data base for the United States (1963-1993). NCEP/Climate Prediction Center ATLAS 1, U.S. Department of Commerce, NOAA/NWS, 47 pp .

  • Hou , D. , Z. Toth , Y. Zhu , and W. Yang , 2008 : Impact of a stochastic perturbation scheme on NCEP Global Ensemble Forecast System. Proc. 19th Conf. on Probability and Statistics, New Orleans, LA, Amer. Meteor. Soc., 1.1. [Available online at http://ams.confex.com/ams/pdfpapers/134165.pdf.]

  • Houtekamer , P. L. , and H. L. Mitchell , 2005 : Ensemble Kalman filtering . Quart. J. Roy. Meteor. Soc. , 131 , 3269 –3289 .

  • Houtekamer , P. L. , H. L. Mitchell , and X. Deng , 2009 : Model error representation in an operational ensemble Kalman filter . Mon. Wea. Rev. , 137 , 2126 –2143 .

    • Search Google Scholar
    • Export Citation

  • Hunt , B. , E. Kostelich , and I. Szunyogh , 2006 : Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filter. Physica D, 230, 112–126.

  • Isaksen , L. , M. Bonavita , R. Buizza , M. Fisher , J. Haseler , M. Leutbecher , and L. Raynaud , 2010 : Ensemble of data assimilations at ECMWF. ECMWF Tech. Memo., Vol. 636, 46 pp .

  • Iversen , T. , A. Deckmyn , C. Santos , K. A. I. Sattler , J. B. Bremnes , H. Feddersen , and I.-L. Frogner , 2011 : Evaluation of ‘GLAMEPS’—A proposed multimodel EPS for short range forecasting . Tellus , 63A , 513 –530 .

    • Search Google Scholar
    • Export Citation

  • Johnson , C. , and R. Swinbank , 2009 : Medium-range multimodel ensemble combination and calibration . Quart. J. Roy. Meteor. Soc. , 135 , 777 –794 .

    • Search Google Scholar
    • Export Citation

  • Kleist , D. T. , D. F. Parrish , J. C. Derber , R. Treadon , W.-S. Wu , and S. Lord , 2009 : Introduction of the GSI into the NCEP Global Data Assimilation System . Wea. Forecasting , 24 , 1691 –1705 .

    • Search Google Scholar
    • Export Citation

  • Krishnamurti , T. N. , C. M. Kishtawal , Z. Zhang , T. LaRow , D. Bachiochi , E. Williford , S. Gadgil , and S. Surendran , 2000 : Multimodel ensemble forecasts for weather and seasonal climate . J. Climate , 13 , 4196 –4216 .

    • Search Google Scholar
    • Export Citation

  • Leutbecher , M. , 2005 : On ensemble prediction using singular vectors started from forecasts . Mon. Wea. Rev. , 133 , 3038 –3046 .

    • Search Google Scholar
    • Export Citation

  • Lin , Y. , and K. E. Mitchell , 2005 : The NCEP stage II/IV hourly precipitation analyses: Development and applications. Preprints, 19th Conf. on Hydrology, San Diego, CA, Amer. Meteor. Soc., 1.2. [Available online at http://ams.confex.com/ams/Annual2005/webprogram/Paper83847.html.]

  • Lorenz , J. , H. Rauhut , F. Schweitzer , and D. Helbing , 2011 : How social influence can undermine the wisdom of crowd effect . Proc. Natl. Acad. Sci. USA , 108 , 920 –925 .

    • Search Google Scholar
    • Export Citation

  • Lowry , D. A. , and H. R. Glahn , 1976 : An operational model for forecasting probability of precipitation—PEATMOS PoP . Mon. Wea. Rev. , 104 , 221 –232 .

    • Search Google Scholar
    • Export Citation

  • Molteni , F. , R. Buizza , T. N. Palmer , and T. Petroliagis , 1996 : The ECMWF Ensemble Prediction System: Methodology and validation . Quart. J. Roy. Meteor. Soc. , 122 , 73 –119 .

    • Search Google Scholar
    • Export Citation

  • Mylne , K. R. , R. E. Evans , and R. T. Clark , 2002 : Multi-model multi-analysis ensembles in quasi-operational medium-range forecasting . Quart. J. Roy. Meteor. Soc. , 128 , 361 –384 .

    • Search Google Scholar
    • Export Citation

  • Palmer , T. N. , R. Buizza , F. J. Doblas-Reyes , T. Jung , M. Leutbecher , G. J. Shutts , M. Steinheimer , and A. Weisheimer , 2009 : Stochastic parametrization and model uncertainty. ECMWF Tech. Memo. 598, 42 pp .

  • Raftery , A. E. , T. Gneiting , F. Balabdaoui , and M. Polakowski , 2005 : Using Bayesian model averaging to calibrate forecast ensembles . Mon. Wea. Rev. , 133 , 1155 –1174 .

    • Search Google Scholar
    • Export Citation

  • Rawlins , F. , S. P. Ballard , K. J. Bovis , A. M. Clayton , D. Li , G. W. Inverarity , A. C. Lorenc , and T. J. Payne , 2007 : The Met Office global four-dimensional variational data assimilation scheme . Quart. J. Roy. Meteor. Soc. , 133 , 347 –362 .

    • Search Google Scholar
    • Export Citation

  • Roulin , E. , and S. Vannitsem , 2012 : Postprocessing of ensemble precipitation predictions with extended logistic regression based on hindcasts . Mon. Wea. Rev. , 140 , 874 –888 .

    • Search Google Scholar
    • Export Citation

  • Schmeits , M. J. , and K. J. Kok , 2010 : A comparison between raw ensemble output, (modified) Bayesian model averaging, and extended logistic regression using ECMWF ensemble precipitation reforecasts . Mon. Wea. Rev. , 138 , 4199 –4211 .

    • Search Google Scholar
    • Export Citation

  • Shutts , G. , 2005 : A kinetic energy backscatter algorithm for use in ensemble prediction systems . Quart. J. Roy. Meteor. Soc. , 131 , 3079 –3102 .

    • Search Google Scholar
    • Export Citation

  • Sloughter , J. M. L. , A. E. Raftery , T. Gneiting , and C. Fraley , 2007 : Probabilistic quantitative precipitation forecasting using Bayesian model averaging . Mon. Wea. Rev. , 135 , 3209 –3220 .

    • Search Google Scholar
    • Export Citation

  • Stensrud , D. J. , and N. Yussouf , 2003 : Short-range ensemble predictions of 2-m temperature and dewpoint temperature over New England . Mon. Wea. Rev. , 131 , 2510 –2524 .

    • Search Google Scholar
    • Export Citation

  • Tennant , W. J. , G. J. Shutts , A. Arribas , and S. A. Thompson , 2011 : Using a stochastic kinetic energy backscatter scheme to improve MOGREPS probabilistic forecast skill . Mon. Wea. Rev. , 139 , 1190 –1206 .

    • Search Google Scholar
    • Export Citation

  • Vannitsem , S. , and C. Nicolis , 2008 : Dynamical properties of model output statistics forecasts . Mon. Wea. Rev. , 136 , 405 –419 .

    • Search Google Scholar
    • Export Citation

  • Vislocky , R. L. , and J. M. Fritsch , 1995 : Improved model output statistics forecasts through model consensus . Bull. Amer. Meteor. Soc. , 76 , 1157 –1164 .

    • Search Google Scholar
    • Export Citation

  • Wandishin , M. S. , S. L. Mullen , D. J. Stensrud , and H. E. Brooks , 2001 : Evaluation of a short-range multimodel ensemble system . Mon. Wea. Rev. , 129 , 729 –747 .

    • Search Google Scholar
    • Export Citation

  • Wei , M. , Z. Toth , R. Wobus , and Y. Zhu , 2008 : Initial perturbations based on the ensemble transform (ET) technique in the NCEP global operational forecast system . Tellus , 60A , 62 –79 .

    • Search Google Scholar
    • Export Citation

  • Weigel , A. P. , M. A. Liniger , and C. Appenzeller , 2008 : Can multi-model combination really enhance the prediction skill of probabilistic ensemble forecasts? Quart. J. Roy. Meteor. Soc. , 134 , 241 –260 .

    • Search Google Scholar
    • Export Citation

  • Whitaker , J. S. , X. Wei , and F. Vitart , 2006 : Improving week-2 forecasts with multimodel reforecast ensembles . Mon. Wea. Rev. , 134 , 2279 –2284 .

    • Search Google Scholar
    • Export Citation

  • Wilks , D. S. , 2006 : Statistical Methods in the Atmospheric Sciences. 2nd ed. Academic Press, 627 pp .

  • Wilks , D. S. , 2009 : Extending logistic regression to provide full-probability-distribution MOS forecasts . Meteor. Appl. , 16 , 361 –368 .

    • Search Google Scholar
    • Export Citation

  • Wilks , D. S. , and T. M. Hamill , 2007 : Comparison of ensemble-MOS methods using GFS reforecasts . Mon. Wea. Rev. , 135 , 2379 –2390 .

    • Search Google Scholar
    • Export Citation

  • Wilson , L. J. , S. Beauregard , A. E. Raftery , and R. Verret , 2007 : Calibrated surface temperature forecasts from the Canadian ensemble prediction system using Bayesian model averaging . Mon. Wea. Rev. , 135 , 1364 –1385 .

    • Search Google Scholar
    • Export Citation

  • Yussouf , N. , and D. J. Stensrud , 2007 : Bias-corrected short-range ensemble forecasts of near-surface variables during the 2005/06 cool season . Wea. Forecasting , 22 , 1274 –1286 .

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 1091 572 366
PDF Downloads 468 136 21