Development of Verification Methodology for Extreme Weather Forecasts

Hong Guan NOAA/NWS/NCEP/Environmental Modeling Center, College Park, Maryland, and System Research Group Inc., Colorado Springs, Colorado

Search for other papers by Hong Guan in
Current site
Google Scholar
PubMed
Close
and
Yuejian Zhu NOAA/NWS/NCEP/Environmental Modeling Center, College Park, Maryland

Search for other papers by Yuejian Zhu in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

In 2006, the statistical postprocessing of the National Centers for Environmental Prediction (NCEP) Global Ensemble Forecast System (GEFS) and North American Ensemble Forecast System (NAEFS) was implemented to enhance probabilistic guidance. Anomaly forecasting (ANF) is one of the NAEFS products, generated from bias-corrected ensemble forecasts and reanalysis climatology. The extreme forecast index (EFI), based on a raw ensemble forecast and model-based climatology, is another way to build an extreme weather forecast. In this work, the ANF and EFI algorithms are applied to extreme cold temperature and extreme precipitation forecasts during the winter of 2013/14. A highly correlated relationship between the ANF and EFI allows the determination of two sets of thresholds to identify extreme cold and extreme precipitation events for the two algorithms. An EFI of −0.78 (0.687) is approximately equivalent to a −2σ (0.95) ANF for the extreme cold event (extreme precipitation) forecast. The performances of the two algorithms in forecasting extreme cold events are verified against analysis for different model versions, reference climatology, and forecasts. The verification results during the winter of 2013/14 indicate that ANF forecasts more extreme cold events with a slightly higher skill than EFI. The bias-corrected forecast performs much better than the raw forecast. The current upgrade of the GEFS has a beneficial effect on the extreme cold weather forecast. Using the NCEP Climate Forecast System Reanalysis and Reforecast (CFSRR) as a climate reference gives a slightly better score than the 40-yr reanalysis. The verification methodology is also extended to an extreme precipitation case, showing a broad potential use in the future.

© 2017 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author e-mail: Dr. Hong Guan, hong.guan@noaa.gov

Abstract

In 2006, the statistical postprocessing of the National Centers for Environmental Prediction (NCEP) Global Ensemble Forecast System (GEFS) and North American Ensemble Forecast System (NAEFS) was implemented to enhance probabilistic guidance. Anomaly forecasting (ANF) is one of the NAEFS products, generated from bias-corrected ensemble forecasts and reanalysis climatology. The extreme forecast index (EFI), based on a raw ensemble forecast and model-based climatology, is another way to build an extreme weather forecast. In this work, the ANF and EFI algorithms are applied to extreme cold temperature and extreme precipitation forecasts during the winter of 2013/14. A highly correlated relationship between the ANF and EFI allows the determination of two sets of thresholds to identify extreme cold and extreme precipitation events for the two algorithms. An EFI of −0.78 (0.687) is approximately equivalent to a −2σ (0.95) ANF for the extreme cold event (extreme precipitation) forecast. The performances of the two algorithms in forecasting extreme cold events are verified against analysis for different model versions, reference climatology, and forecasts. The verification results during the winter of 2013/14 indicate that ANF forecasts more extreme cold events with a slightly higher skill than EFI. The bias-corrected forecast performs much better than the raw forecast. The current upgrade of the GEFS has a beneficial effect on the extreme cold weather forecast. Using the NCEP Climate Forecast System Reanalysis and Reforecast (CFSRR) as a climate reference gives a slightly better score than the 40-yr reanalysis. The verification methodology is also extended to an extreme precipitation case, showing a broad potential use in the future.

© 2017 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author e-mail: Dr. Hong Guan, hong.guan@noaa.gov
Save
  • Buizza, R., and T. N. Palmer, 1998: Impact of ensemble size on ensemble prediction. Mon. Wea. Rev., 126, 25032518, doi:10.1175/1520-0493(1998)126<2503:IOESOE>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cui, B., Z. Toth, Y. Zhu, and D. Hou, 2012: Bias correction for global ensemble forecast. Wea. Forecasting, 27, 396410, doi:10.1175/WAF-D-11-00011.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Doswell, C. A., III, R. Davies-Jones, and D. L. Keller, 1990: On summary measures of skill in rare event forecasting based on contingency tables. Wea. Forecasting, 5, 576585, doi:10.1175/1520-0434(1990)005<0576:OSMOSI>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Dutra, E., M. Diamantakis, I. Tsonevsky, E. Zsoter, F. Wetterhall, T. Stockdale, D. Richardson, and F. Pappenberger, 2013: The extreme forecast index at the seasonal scale. Atmos. Sci. Lett., 14, 256262, doi:10.1002/asl2.448.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Graham, R. A., and R. H. Grumm, 2010: Utilizing normalized anomalies to assess synoptic-scale weather events in the western United States. Wea. Forecasting, 25, 428445, doi:10.1175/2009WAF2222273.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Grumm, R. H., 2001: Standardized anomalies applied to significant cold season weather events: Preliminary findings. Wea. Forecasting, 16, 736754, doi:10.1175/1520-0434(2001)016<0736:SAATSC>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Guan, H., B. Cui, and Y. Zhu, 2015: Improvement of statistical postprocessing using GEFS reforecast information. Wea. Forecasting, 30, 841854, doi:10.1175/WAF-D-14-00126.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., G. T. Bates, J. S. Whitaker, D. R. Murray, M. Fiorino, T. J. Galarneau Jr., Y. Zhu, and W. Lapenta, 2013: NOAA’s second-generation global medium-range ensemble reforecast dataset. Bull. Amer. Meteor. Soc., 94, 15531565, doi:10.1175/BAMS-D-12-00014.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., and Coauthors, 2014: A recommended reforecast configuration for the NCEP Global Ensemble Forecast System. NOAA White Paper, 24 pp. [Available online at http://www.esrl.noaa.gov/psd/people/tom.hamill/White-paper-reforecast-configuration.pdf.]

  • Hosking, J. R. M., 1990: L-moments: Analysis and estimation of distributions using linear combinations of order statistics. J. Roy. Stat. Soc., 52B, 105124.

    • Search Google Scholar
    • Export Citation
  • Hosking, J. R. M., and J. R. Wallis, 1997: Regional Frequency Analysis: An Approach Based on L-Moments. Cambridge University Press, 244 pp.

    • Crossref
    • Export Citation
  • Hou, D., Z. Toth, Y. Zhu, and W. Yang, 2008: Evaluation of the impact of the stochastic perturbation schemes on global ensemble forecast. 19th Conf. on Probability and Statistics in the Atmospheric Sciences, New Orleans, LA, Amer. Meteor. Soc., 1.1. [Available online at https://ams.confex.com/ams/88Annual/techprogram/paper_134165.htm.]

  • Hou, D., and Coauthors, 2014: Climatology-calibrated precipitation analysis at fine scales: Statistical adjustment of stage IV toward CPC gauge-based analysis. J. Hydrometeor., 15, 25422557, doi:10.1175/JHM-D-11-0140.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • IPCC, 2011: Special Report on Renewable Energy Sources and Climate Change Mitigation. Cambridge University Press, 1075 pp.

  • Kalnay, E., and Coauthors, 1996: The NCEP/NCAR 40-Year Reanalysis Project. Bull. Amer. Meteor. Soc., 77, 437471, doi:10.1175/1520-0477(1996)077<0437:TNYRP>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kleist, D. T., and K. Ide, 2015: An OSSE-based evaluation of hybrid variational–ensemble data assimilation for the NCEP GFS. Part I: System description and 3D-hybrid results. Mon. Wea. Rev., 143, 433451, doi:10.1175/MWR-D-13-00351.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lalaurette, F., 2003: Early detection of abnormal weather conditions using a probabilistic extreme forecast index. Quart. J. Roy. Meteor. Soc., 129, 30373057, doi:10.1256/qj.02.152.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ma, J. H., Y. J. Zhu, R. Wobus, and P. X. Wang, 2012: An effective configuration of ensemble size and horizontal resolution for the NCEP GEFS. Adv. Atmos. Sci., 29, 782794, doi:10.1007/s00376-012-1249-y.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mason, I. B., 1982: A model for the assessment of weather forecasts. Aust. Meteor. Mag., 30, 291303.

  • Matsueda, S., and Y. Takaya, 2013: Verification of the Extreme Forecast Index in JMA’s operational one-month ensemble prediction system. Research Activities in Atmospheric and Oceanic Modelling, A. Zadra, Ed., WCRP Rep. 10/2013. [Available online at https://www.wcrp-climate.org/WGNE/BlueBook/2013/individual-articles/06_Matsueda_Satoko_EFI_Verification.pdf.]

  • Petroliagis, T. I., and P. Pinson, 2012: Early indication of extreme winds utilizing the extreme forecast index. ECMWF Newsletter, No. 132, ECMWF, Reading, United Kingdom, 13–19. [Available online at http://www.ecmwf.int/en/elibrary/14590-newsletter-no132-summer-2012.]

  • Roebber, P. J., 2009: Visualizing multiple measures of forecast quality. Wea. Forecasting, 24, 601608, doi:10.1175/2008WAF2222159.1.

  • Saha, S., and Coauthors, 2010: The NCEP Climate Forecast System reanalysis. Bull. Amer. Meteor. Soc., 91, 10151057, doi:10.1175/2010BAMS3001.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sela, J., 2010: The derivation of the sigma pressure hybrid coordinate Semi-Lagrangian model equations for the GFS. NCEP Office Note 462, 31 pp. [Available online at http://www.lib.ncep.noaa.gov/ncepofficenotes/files/on462.pdf.]

  • Schaefer, J. T., 1990: The critical success index as an indicator of forecasting skill. Wea. Forecasting, 5, 570575, doi:10.1175/1520-0434(1990)005<0570:TCSIAA>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Swets, J. A., 1988: Measuring the accuracy of diagnostic systems. Science, 240, 12851293, doi:10.1126/science.3287615.

  • Toth, Z., O. Talagrand, G. Candille, and Y. Zhu, 2003: Probability and ensemble forecasts. Forecast Verification: A Practitioner’s Guide in Atmospheric Science, I. T. Jolliffe and D. B. Stephenson, Eds., John Wiley and Sons, 137–163.

  • Van Oldenborgh, G. J., R. Haarsma, H. De Vries, and M. R. Allen, 2015: Cold extremes in North America vs. mild weather in Europe: The winter of 2013–14 in the context of a warming world. Bull. Amer. Meteor. Soc., 96, 707714, doi:10.1175/BAMS-D-14-00036.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wang, X., D. Parrish, D. Kleist, and J. Whitaker, 2013: GSI 3DVar-based ensemble–variational hybrid data assimilation for NCEP Global Forecast System: Single-resolution experiments. Mon. Wea. Rev., 141, 40984117, doi:10.1175/MWR-D-12-00141.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wei, M., Z. Toth, R. Wobus, and Y. Zhu, 2008: Initial perturbations based on the ensemble transform (ET) technique in the NCEP global operational forecast system. Tellus, 60A, 6279, doi:10.1111/j.1600-0870.2007.00273.x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Whitaker, J. S., and T. M. Hamill, 2012: Evaluating methods to account for system errors in ensemble data assimilation. Mon. Wea. Rev., 140, 30783089, doi:10.1175/MWR-D-11-00276.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Whitaker, J. S., T. M. Hamill, X. Wei, Y. Song, and Z. Toth, 2008: Ensemble data assimilation with the NCEP Global Forecast System. Mon. Wea. Rev., 136, 463482, doi:10.1175/2007MWR2018.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zhu, Y., 2015: GEFS upgrade (V11). NWS/Environmental Modeling Center. [Available online at http://www.emc.ncep.noaa.gov/gmb/yzhu/imp/i201412/GEFS_sci_briefing.pdf.]

  • Zhu, Y., and B. Cui, 2007: NAEFS mean, spread and probability forecasts. NOAA/NCEP Rep., 4 pp. http://www.emc.ncep.noaa.gov/gmb/yzhu/imp/i200711/3-Mean_spread.pdf.]

  • Zhu, Y., and Z. Toth, 2008: Ensemble based probabilistic forecast verification. 19th Conf. on Probability and Statistics in the Atmospheric Sciences, New Orleans, LA, Amer. Meteor. Soc., 2.2. [Available online at https://ams.confex.com/ams/pdfpapers/131645.pdf.]

  • Zhu Y.——, R. Wobus, M. Wei, B. Cui, and Z. Toth, 2007: March 2007 NAEFS upgrade. NWS/Environmental Modeling Center. [Available online at http://www.emc.ncep.noaa.gov/gmb/ens/ens_imp_news.html.]

  • Zhu, Y.——, D. Hou, M. Wei, R. Wobus, J. Ma, B. Cui, and S. Moorthi, 2012: GEFS upgrade—AOP plan—Major implementation. NWS/Environmental Modeling Center. [Available online at http://www.emc.ncep.noaa.gov/gmb/yzhu/html/imp/201109_imp.html.]

  • Zsótér, E., 2006: Recent developments in extreme weather forecasting. ECMWF Newsletter, No. 107, ECMWF, Reading, United Kingdom, 8–17. [Available online at http://www.ecmwf.int/sites/default/files/elibrary/2006/14618-newsletter-no107-spring-2006.pdf.]

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 932 171 22
PDF Downloads 722 149 15