Editorial Type:
Article
Article Type:
Research Article

Increasing the Skill of Probabilistic Forecasts: Understanding Performance Improvements from Model-Error Representations

J. Berner National Center for Atmospheric Research,* Boulder, Colorado

Search for other papers by J. Berner in
Current site
Google Scholar
PubMed Close
,
K. R. Fossell National Center for Atmospheric Research,* Boulder, Colorado

Search for other papers by K. R. Fossell in
Current site
Google Scholar
PubMed Close
,
S.-Y. Ha National Center for Atmospheric Research,* Boulder, Colorado

Search for other papers by S.-Y. Ha in
Current site
Google Scholar
PubMed Close
,
J. P. Hacker National Center for Atmospheric Research,* Boulder, Colorado

Search for other papers by J. P. Hacker in
Current site
Google Scholar
PubMed Close
, and
C. Snyder National Center for Atmospheric Research,* Boulder, Colorado

Search for other papers by C. Snyder in
Current site
Google Scholar
PubMed Close
Print Publication:
01 Apr 2015
DOI:
https://doi.org/10.1175/MWR-D-14-00091.1
Page(s):
1295–1320
Restricted access

Abstract

Four model-error schemes for probabilistic forecasts over the contiguous United States with the WRF-ARW mesoscale ensemble system are evaluated in regard to performance. Including a model-error representation leads to significant increases in forecast skill near the surface as measured by the Brier score. Combining multiple model-error schemes results in the best-performing ensemble systems, indicating that current model error is still too complex to be represented by a single scheme alone.

To understand the reasons for the improved performance, it is examined whether model-error representations increase skill merely by increasing the reliability and reducing the bias—which could also be achieved by postprocessing—or if they have additional benefits.

Removing the bias results overall in the largest skill improvement. Forecasts with model-error schemes continue to have better skill than without, indicating that their benefit goes beyond bias reduction.

Decomposing the Brier score into its components reveals that, in addition to the spread-sensitive reliability, the resolution component is significantly improved. This indicates that the benefits of including a model-error representation go beyond increasing reliability. This is further substantiated when all forecasts are calibrated to have similar spread. The calibrated ensembles with model-error schemes consistently outperform the calibrated control ensemble.

Including a model-error representation remains beneficial even if the ensemble systems are calibrated and/or debiased. This suggests that the merits of model-error representations go beyond increasing spread and removing the mean error and can account for certain aspects of structural model uncertainty.

The National Center for Atmospheric Research is sponsored by the National Science Foundation.

Corresponding author address: Judith Berner, NCAR, P.O. Box 3000, Boulder, CO 80303. E-mail: berner@ucar.edu

Abstract

Four model-error schemes for probabilistic forecasts over the contiguous United States with the WRF-ARW mesoscale ensemble system are evaluated in regard to performance. Including a model-error representation leads to significant increases in forecast skill near the surface as measured by the Brier score. Combining multiple model-error schemes results in the best-performing ensemble systems, indicating that current model error is still too complex to be represented by a single scheme alone.

To understand the reasons for the improved performance, it is examined whether model-error representations increase skill merely by increasing the reliability and reducing the bias—which could also be achieved by postprocessing—or if they have additional benefits.

Removing the bias results overall in the largest skill improvement. Forecasts with model-error schemes continue to have better skill than without, indicating that their benefit goes beyond bias reduction.

Decomposing the Brier score into its components reveals that, in addition to the spread-sensitive reliability, the resolution component is significantly improved. This indicates that the benefits of including a model-error representation go beyond increasing reliability. This is further substantiated when all forecasts are calibrated to have similar spread. The calibrated ensembles with model-error schemes consistently outperform the calibrated control ensemble.

Including a model-error representation remains beneficial even if the ensemble systems are calibrated and/or debiased. This suggests that the merits of model-error representations go beyond increasing spread and removing the mean error and can account for certain aspects of structural model uncertainty.

The National Center for Atmospheric Research is sponsored by the National Science Foundation.

Corresponding author address: Judith Berner, NCAR, P.O. Box 3000, Boulder, CO 80303. E-mail: berner@ucar.edu
Save
Cover Monthly Weather Review
Monthly Weather Review

  • Atger, F., 2003: Spatial and interannual variability of the reliability of ensemble-based probabilistic forecasts: Consequences for calibration. Mon. Wea. Rev., 131, 1509–1523, doi:10.1175/1520-0493(2003)131<1509:SAIVOT>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Berner, J., F. J. Doblas-Reyes, T. N. Palmer, G. Shutts, and A. Weisheimer, 2008: Impact of a quasi-stochastic cellular automaton backscatter scheme on the systematic error and seasonal prediction skill of a global climate model. Philos. Trans. Roy. Soc. London, 366A, 2559–2579, doi:10.1098/rsta.2008.0033.

    • Search Google Scholar
    • Export Citation

  • Berner, J., G. Shutts, M. Leutbecher, and T. Palmer, 2009: A spectral stochastic kinetic energy backscatter scheme and its impact on flow-dependent predictability in the ECMWF ensemble prediction system. J. Atmos. Sci., 66, 603–626, doi:10.1175/2008JAS2677.1.

    • Search Google Scholar
    • Export Citation

  • Berner, J., S.-Y. Ha, J. P. Hacker, A. Fournier, and C. Snyder, 2011: Model uncertainty in a mesoscale ensemble prediction system: Stochastic versus multiphysics representations. Mon. Wea. Rev., 139, 1972–1995, doi:10.1175/2010MWR3595.1.

    • Search Google Scholar
    • Export Citation

  • Berner, J., T. Jung, and T. N. Palmer, 2012: Systematic model error: The impact of increased horizontal resolution versus improved stochastic and deterministic parameterizations. J. Climate, 25, 4946–4962, doi:10.1175/JCLI-D-11-00297.1.

    • Search Google Scholar
    • Export Citation

  • Bouttier, F., B. Vié, O. Nuissier, and L. Raynaud, 2012: Impact of stochastic physics in a convection-permitting ensemble. Mon. Wea. Rev., 140, 3706–3721, doi:10.1175/MWR-D-12-00031.1.

    • Search Google Scholar
    • Export Citation

  • Bowler, N. E., A. Arribas, K. R. Mylne, K. B. Robertson, and S. E. Beare, 2008: The MOGREPS short-range ensemble prediction system. Quart. J. Roy. Meteor. Soc., 134, 703–722, doi:10.1002/qj.234.

    • Search Google Scholar
    • Export Citation

  • Bowler, N. E., A. Arribas, S. E. Beare, K. R. Mylne, and G. J. Shutts, 2009: The local ETKF and SKEB: Upgrades to the MOGREPS short-range ensemble prediction system. Quart. J. Roy. Meteor. Soc., 135, 767–776, doi:10.1002/qj.394.

    • Search Google Scholar
    • Export Citation

  • Buizza, R., M. Milleer, and T. N. Palmer, 1999: Stochastic representation of model uncertainties in the ECMWF ensemble prediction system. Quart. J. Roy. Meteor. Soc., 125, 2887–2908, doi:10.1002/qj.49712556006.

    • Search Google Scholar
    • Export Citation

  • Buizza, R., P. L. Houtekamer, Z. Toth, G. Pellerin, M. Wei, and Y. Zhu, 2005: A comparison of the ECMWF, MSC, and NCEP Global Ensemble Prediction Systems. Mon. Wea. Rev., 133, 1076–1097, doi:10.1175/MWR2905.1.

    • Search Google Scholar
    • Export Citation

  • Charron, M., G. Pellerin, L. Spacek, P. L. Houtekamer, N. Gagnon, H. L. Mitchell, and L. Michelin, 2010: Toward random sampling of model error in the Canadian ensemble prediction system. Mon. Wea. Rev., 138, 1877–1901, doi:10.1175/2009MWR3187.1.

    • Search Google Scholar
    • Export Citation

  • Desroziers, G., L. Berre, B. Chapnik, and P. Poli, 2005: Diagnosis of observation, background and analysis-error statistics in observation space. Quart. J. Roy. Meteor. Soc., 131, 3385–3396, doi:10.1256/qj.05.108.

    • Search Google Scholar
    • Export Citation

  • Doblas-Reyes, F. J., R. Hagedorn, and T. N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting— II. Calibration and combination. Tellus, 57A, 234–252, doi:10.1111/j.1600-0870.2005.00104.x.

    • Search Google Scholar
    • Export Citation

  • Doblas-Reyes, F. J., and Coauthors, 2009: Addressing model uncertainty in seasonal and annual dynamical seasonal forecasts. Quart. J. Roy. Meteor. Soc., 135, 1538–1559, doi:10.1002/qj.464.

    • Search Google Scholar
    • Export Citation

  • Eckel, F. A., and C. F. Mass, 2005: Aspects of effective mesoscale, short-range, ensemble forecasting. Wea. Forecasting, 20, 328–350, doi:10.1175/WAF843.1.

    • Search Google Scholar
    • Export Citation

  • Fortin, V., A.-C. Favre, and M. Said, 2006: Probabilistic forecasting from ensemble prediction systems: Improving upon the best-member method by using a different weight and dressing kernel for each member. Quart. J. Roy. Meteor. Soc., 132, 1349–1369, doi:10.1256/qj.05.167.

    • Search Google Scholar
    • Export Citation

  • Gneiting, T., A. E. Raftery, A. H. Westveld III, and T. Goldman, 2005: Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation. Mon. Wea. Rev., 133, 1098–1118, doi:10.1175/MWR2904.1.

    • Search Google Scholar
    • Export Citation

  • Hacker, J. P., and Coauthors, 2011a: The U.S. Air Force Weather Agency’s mesoscale ensemble: Scientific description and performance results. Tellus, 63A, 625–641, doi:10.1111/j.1600-0870.2010.00497.x.

    • Search Google Scholar
    • Export Citation

  • Hacker, J. P., C. Snyder, S.-Y. Ha, and M. Pocernich, 2011b: Linear and nonlinear response to parameter variations in a mesoscale model. Tellus, 63A, 429–444, doi:10.1111/j.1600-0870.2010.00505.x.

    • Search Google Scholar
    • Export Citation

  • Hagedorn, R., T. M. Hamill, and J. S. Whitaker, 2008: Probabilistic forecast calibration using ECMWF and GFS ensemble reforecasts. Part I: Two-meter temperatures. Mon. Wea. Rev., 136, 2608–2619, doi:10.1175/2007MWR2410.1.

    • Search Google Scholar
    • Export Citation

  • Hagedorn, R., R. Buizza, T. M. Hamill, M. Leutbecher, and T. Palmer, 2012: Comparing TIGGE multimodel forecasts with reforecast-calibrated ECMWF ensemble forecasts. Quart. J. Roy. Meteor. Soc., 138, 1814–1827, doi:10.1002/qj.1895.

    • Search Google Scholar
    • Export Citation

  • Hamill, T. M., 2001: Interpretation of rank histograms for verifying ensemble forecasts. Mon. Wea. Rev., 129, 550–560, doi:10.1175/1520-0493(2001)129<0550:IORHFV>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Hamill, T. M., and S. J. Colucci, 1998: Evaluation of Eta–RSM ensemble probabilistic precipitation forecasts. Mon. Wea. Rev., 126, 711–724, doi:10.1175/1520-0493(1998)126<0711:EOEREP>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Hamill, T. M., and J. S. Whitaker, 2007: Ensemble calibration of 500-hPa geopotential height and 850-hPa and 2-m temperatures using reforecasts. Mon. Wea. Rev., 135, 3273–3280, doi:10.1175/MWR3468.1.

    • Search Google Scholar
    • Export Citation

  • Hamill, T. M., J. S. Whitaker, and X. Wei, 2004: Ensemble reforecasting: Improving medium-range forecast skill using retrospective forecasts. Mon. Wea. Rev., 132, 1434–1447, doi:10.1175/1520-0493(2004)132<1434:ERIMFS>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Hong, Y.-Y., Y. Noh, and J. Dudhia, 2006: A new vertical diffusion package with explicit treatment of entrainment processes. Mon. Wea. Rev., 134, 2318–2341, doi:10.1175/MWR3199.1.

    • Search Google Scholar
    • Export Citation

  • Jolliffe, I., and D. Stephenson, 2003: Forecast Verification: A Practitioners Guide in Atmospheric Science.Wiley and Sons, 240 pp.

  • Kharin, V. V., and F. W. Zwiers, 2003: Improved seasonal probability forecasts. J. Climate, 16, 1684–1701, doi:10.1175/1520-0442(2003)016<1684:ISPF>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Li, X., M. Charron, L. Spacek, and G. Candille, 2008: A regional ensemble prediction system based on moist targeted singular vectors and stochastic parameter perturbations. Mon. Wea. Rev., 136, 443–462, doi:10.1175/2007MWR2109.1.

    • Search Google Scholar
    • Export Citation

  • Mass, C. F., 2003: IFPS and the future of the National Weather Service. Wea. Forecasting, 18, 75–79, doi:10.1175/1520-0434(2003)018<0075:IATFOT>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Murphy, A. H., 1973: A new vector partition of the probability score. J. Appl. Meteor., 12, 595–600, doi:10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Palmer, T. N., R. Buizza, F. Doblas-Reyes, T. Jung, M. Leutbecher, G. Shutts, M. Steinheimer, and A. Weisheimer, 2009: Stochastic parametrization and model uncertainty. ECMWF Tech. Memo. 598, 42 pp [Available at https://www2.physics.ox.ac.uk/sites/default/files/2011-08-15/techmemo598_stochphys_2009_pdf_50419.pdf.]

  • Plant, R. S., and G. C. Craig, 2008: A stochastic parameterization for deep convection based on equilibrium statistics. J. Atmos. Sci., 65, 87–105, doi:10.1175/2007JAS2263.1.

    • Search Google Scholar
    • Export Citation

  • Raftery, A. E., T. Gneiting, F. Balabdaoui, and M. Polakowski, 2005: Using Bayesian model averaging to calibrate forecast ensembles. Mon. Wea. Rev., 133, 1155–1174, doi:10.1175/MWR2906.1.

    • Search Google Scholar
    • Export Citation

  • Reynolds, C. A., J. G. McLay, J. S. Goerss, E. A. Serra, D. Hodyss, and C. R. Sampson, 2011: Impact of resolution and design on the U.S. Navy global ensemble performance in the tropics. Mon. Wea. Rev., 139, 2145–2155, doi:10.1175/2011MWR3546.1.

    • Search Google Scholar
    • Export Citation

  • Romine, G. S., C. S. Schwartz, J. Berner, K. R. Smith, C. Snyder, J. L. Anderson, and M. L. Weisman, 2014: Representing forecast error in a convection-permitting ensemble system. Mon. Wea. Rev., 142, 4519–4541, doi:10.1175/MWR-D-14-00100.1.

    • Search Google Scholar
    • Export Citation

  • Roulston, M. S., and L. A. Smith, 2002: Evaluating probabilistic forecasts using information theory. Mon. Wea. Rev., 130, 1653–1660, doi:10.1175/1520-0493(2002)130<1653:EPFUIT>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Saetra, O., H. Hersbach, J.-R. Bidlot, and D. Richardson, 2004: Effects of observation errors on the statistics for ensemble spread and reliability. Mon. Wea. Rev., 132, 1487–1501, doi:10.1175/1520-0493(2004)132<1487:EOOEOT>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Shutts, G. J., 2005: A kinetic energy backscatter algorithm for use in ensemble prediction systems. Quart. J. Roy. Meteor. Soc., 131, 3079–3102, doi:10.1256/qj.04.106.

    • Search Google Scholar
    • Export Citation

  • Shutts, G. J., and A. Callado Pallarès, 2014: Assessing parametrization uncertainty associated with horizontal resolution in numerical weather prediction models. Philos. Trans. Roy. Soc. London., 372A, 20130284, doi:10.1098/rsta.2013.0284.

    • Search Google Scholar
    • Export Citation

  • Skamarock, W. C., and Coauthors, 2008: A description of the Advanced Research WRF version 3. NCAR Tech. Note NCAR/TN-475+STR, 113 pp. [Available online at http://www.mmm.ucar.edu/wrf/users/docs/arw_v3_bw.pdf.]

  • Stensrud, D., , J.-W. Bao, and T. T. Warner, 2000: Using initial condition and model physics perturbations in short-range ensemble simulations of mesoscale convective systems. Mon. Wea. Rev., 128, 2077–2107, doi:10.1175/1520-0493(2000)128<2077:UICAMP>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Teixeira, L., and C. A. Reynolds, 2008: Stochastic nature of physical parameterizations in ensemble prediction: A stochastic convection approach. Mon. Wea. Rev., 136, 483–496, doi:10.1175/2007MWR1870.1.

    • Search Google Scholar
    • Export Citation

  • Tennant, W. J., G. J. Shutts, and S. A. T. A. Arribas, 2011: Using a stochastic kinetic energy backscatter scheme to improve MOGREPS probabilistic forecast skill. Mon. Wea. Rev., 139, 1190–1206, doi:10.1175/2010MWR3430.1.

    • Search Google Scholar
    • Export Citation

  • von Storch, H., 1999: On the use of inflation in statistical downscaling. J. Climate, 12, 3505–3506, doi:10.1175/1520-0442(1999)012<3505:OTUOII>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Wang, X., and C. H. Bishop, 2005: Improvement of ensemble reliability with a new dressing kernel. Quart. J. Roy. Meteor. Soc., 131, 965–986, doi:10.1256/qj.04.120.

    • Search Google Scholar
    • Export Citation

  • Wang, X., C. H. Bishop, and S. J. Julier, 2004: Which is better, an ensemble of positive–negative pairs or a centered spherical simplex ensemble? Mon. Wea. Rev., 132, 1590–1605, doi:10.1175/1520-0493(2004)132<1590:WIBAEO>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Weaver, A., and P. Courtier, 2001: Correlation modelling on the sphere using a generalized diffusion equation. Quart. J. Roy. Meteor. Soc., 127, 1815–1846, doi:10.1002/qj.49712757518.

    • Search Google Scholar
    • Export Citation

  • Wei, M., Z. Toth, R. Wobus, and Y. Zhu, 2008: Initial perturbations based on the ensemble transform (ET) technique in the NCEP global operational forecast system. Tellus, 60A, 62–79, doi:10.1111/j.1600-0870.2007.00273.x.

    • Search Google Scholar
    • Export Citation

  • Weisheimer, A., T. Palmer, and F. Doblas-Reyes, 2011: Assessment of representations of model uncertainty in monthly and seasonal forecast ensembles. Geophys. Res. Lett., 38, L16703, doi:10.1029/2011GL048123.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 2348 669 34
PDF Downloads 1329 308 25