Progress in Forecast Skill at Three Leading Global Operational NWP Centers during 2015–17 as Seen in Summary Assessment Metrics (SAMs)

Ross N. Hoffman NOAA/Atlantic Oceanographic and Meteorological Laboratory, Miami, Florida
Cooperative Institute for Marine and Atmospheric Studies, University of Miami, Miami, Florida

Search for other papers by Ross N. Hoffman in
Current site
Google Scholar
PubMed
Close
,
V. Krishna Kumar Riverside Technology Inc., College Park, Maryland
NOAA/NESDIS/STAR, College Park, Maryland

Search for other papers by V. Krishna Kumar in
Current site
Google Scholar
PubMed
Close
,
Sid-Ahmed Boukabara NOAA/NESDIS/STAR, College Park, Maryland

Search for other papers by Sid-Ahmed Boukabara in
Current site
Google Scholar
PubMed
Close
,
Kayo Ide University of Maryland, College Park, College Park, Maryland

Search for other papers by Kayo Ide in
Current site
Google Scholar
PubMed
Close
,
Fanglin Yang NOAA/NCEP/Environmental Modeling Center, College Park, Maryland

Search for other papers by Fanglin Yang in
Current site
Google Scholar
PubMed
Close
, and
Robert Atlas NOAA/Atlantic Oceanographic and Meteorological Laboratory, Miami, Florida

Search for other papers by Robert Atlas in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

The summary assessment metric (SAM) method is applied to an array of primary assessment metrics (PAMs) for the deterministic forecasts of three leading numerical weather prediction (NWP) centers for the years 2015–17. The PAMs include anomaly correlation, RMSE, and absolute mean error (i.e., the absolute value of bias) for different forecast times, vertical levels, geographic domains, and variables. SAMs indicate that in terms of forecast skill ECMWF is better than NCEP, which is better than but approximately the same as UKMO. The use of SAMs allows a number of interesting features of the evolution of forecast skill to be observed. All three centers improve over the 3-yr period. NCEP short-term forecast skill substantially increases during the period. Quantitatively, the effect of the 11 May 2016 NCEP upgrade to the four-dimensional ensemble variational data assimilation (4DEnVar) system is a 7.37% increase in the probability of improved skill relative to a randomly chosen forecast metric from 2015 to 2017. This is the largest SAM impact during the study period. However, the observed impacts are within the context of slowly improving forecast skill for operational global NWP as compared to earlier years. Clearly, the systems lagging ECMWF can improve, and there is evidence from SAMs in addition to the 4DEnVar example that improvements in forecast and data assimilation systems are still leading to forecast skill improvements.

© 2018 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Ross N. Hoffman, ross.n.hoffman@noaa.gov

Abstract

The summary assessment metric (SAM) method is applied to an array of primary assessment metrics (PAMs) for the deterministic forecasts of three leading numerical weather prediction (NWP) centers for the years 2015–17. The PAMs include anomaly correlation, RMSE, and absolute mean error (i.e., the absolute value of bias) for different forecast times, vertical levels, geographic domains, and variables. SAMs indicate that in terms of forecast skill ECMWF is better than NCEP, which is better than but approximately the same as UKMO. The use of SAMs allows a number of interesting features of the evolution of forecast skill to be observed. All three centers improve over the 3-yr period. NCEP short-term forecast skill substantially increases during the period. Quantitatively, the effect of the 11 May 2016 NCEP upgrade to the four-dimensional ensemble variational data assimilation (4DEnVar) system is a 7.37% increase in the probability of improved skill relative to a randomly chosen forecast metric from 2015 to 2017. This is the largest SAM impact during the study period. However, the observed impacts are within the context of slowly improving forecast skill for operational global NWP as compared to earlier years. Clearly, the systems lagging ECMWF can improve, and there is evidence from SAMs in addition to the 4DEnVar example that improvements in forecast and data assimilation systems are still leading to forecast skill improvements.

© 2018 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Ross N. Hoffman, ross.n.hoffman@noaa.gov
Save
  • Boukabara, S.-A., K. Garrett, and V. K. Kumar, 2016: Potential gaps in the satellite observing system coverage: Assessment of impact on NOAA’s numerical weather prediction overall skills. Mon. Wea. Rev., 144, 25472563, https://doi.org/10.1175/MWR-D-16-0013.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Boukabara, S.-A., and Coauthors, 2018: Community Global Observing System Simulation Experiment (OSSE) Package (CGOP): Assessment and validation of the OSSE system using an OSSE–OSE intercomparison of summary assessment metrics. J. Atmos. Oceanic Technol., 35, 20612078, https://doi.org/10.1175/JTECH-D-18-0061.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bretherton, C. S., M. Widmann, V. P. Dymnikov, J. M. Wallace, and I. Bladé, 1999: The effective number of spatial degrees of freedom of a time-varying field. J. Climate, 12, 19902009, https://doi.org/10.1175/1520-0442(1999)012<1990:TENOSD>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Buizza, R., G. Balsamo, and T. Haiden, 2018: IFS upgrade brings more seamless coupled forecasts. ECMWF Newsletter, No. 156, 18–22, ECMWF, Reading, United Kingdom, https://www.ecmwf.int/en/elibrary/14578-newsletter-no135-spring-2013.

  • Geer, A. J., 2016: Significance of changes in medium-range forecast scores. Tellus, 68A, 30229, https://doi.org/10.3402/tellusa.v68.30229.

  • Hoffman, R. N., S.-A. Boukabara, V. K. Kumar, K. Garrett, S. P. F. Casey, and R. Atlas, 2017a: An empirical cumulative density function approach to defining summary NWP forecast assessment metrics. Mon. Wea. Rev., 145, 14271435, https://doi.org/10.1175/MWR-D-16-0271.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hoffman, R. N., S.-A. Boukabara, V. K. Kumar, K. Garrett, S. P. F. Casey, and R. Atlas, 2017b: A non-parametric definition of summary NWP forecast assessment metrics. 28th Conf. on Weather Analysis and Forecasting/24th Conf. on Numerical Weather Prediction, Seattle, WA, Amer. Meteor. Soc., 618, https://ams.confex.com/ams/97Annual/webprogram/Paper309748.html.

  • Janousek, M., 2018: Score definitions and requirements. WMO Lead Centre for Deterministic NWP Verification, ECMWF, https://software.ecmwf.int/wiki/display/WLD/Score+definitions+and+requirements.

  • Lorenz, E. N., 1982: Atmospheric predictability experiments with a large numerical model. Tellus, 34, 505513, https://doi.org/10.3402/tellusa.v34i6.10836.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Newman, K. M., M. Hu, and H. Shao, 2013: Configuration testing of GSI within an operational environment. 17th Conf. on Integrated Observing and Assimilation Systems for the Atmosphere, Oceans, and Land Surface, Austin, TX, Amer. Meteor. Soc., 620, https://ams.confex.com/ams/93Annual/webprogram/Paper221922.html.

  • Rawlins, F., S. P. Ballard, K. J. Bovis, A. M. Clayton, D. Li, G. W. Inverarity, A. C. Lorenc, and T. J. Payne, 2007: The Met Office global four-dimensional variational data assimilation scheme. Quart. J. Roy. Meteor. Soc., 133, 347362, https://doi.org/10.1002/qj.32.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Shafran, P., T. L. Jensen, J. H. Gotway, B. Zhou, K. Nevins, Y. Lin, and G. DiMego, 2015: Web-based verification capability using NCEP’s verification database and DTC’s METviewer. 27th Conf. on Hurricanes and Tropical Meteorology, Chicago, IL, Amer. Meteor. Soc., 14A.7, http://ams.confex.com/ams/27WAF23NWP/webprogram/Paper273777.html.

  • Shao, H., and Coauthors, 2016: Bridging research to operations transitions: Status and plans of community GSI. Bull. Amer. Meteor. Soc., 97, 14271440, https://doi.org/10.1175/BAMS-D-13-00245.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zhou, B., and Coauthors, 2015: An overview of grid-to-grid verification at Environmental Modeling Center (EMC) of NCEP. 27th Conf. on Hurricanes and Tropical Meteorology, Chicago, IL, Amer. Meteor. Soc., 12A.4, http://ams.confex.com/ams/27WAF23NWP/webprogram/Paper273363.html.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 956 214 15
PDF Downloads 874 155 16