Precipitation and Temperature Forecast Performance at the Weather Prediction Center

David R. Novak NOAA/NWS/NCEP/Weather Prediction Center, College Park, Maryland

Search for other papers by David R. Novak in
Current site
Google Scholar
PubMed
Close
,
Christopher Bailey NOAA/NWS/NCEP/Weather Prediction Center, College Park, Maryland

Search for other papers by Christopher Bailey in
Current site
Google Scholar
PubMed
Close
,
Keith F. Brill NOAA/NWS/NCEP/Weather Prediction Center, College Park, Maryland

Search for other papers by Keith F. Brill in
Current site
Google Scholar
PubMed
Close
,
Patrick Burke NOAA/NWS/NCEP/Weather Prediction Center, College Park, Maryland

Search for other papers by Patrick Burke in
Current site
Google Scholar
PubMed
Close
,
Wallace A. Hogsett NOAA/NWS/NCEP/Weather Prediction Center, College Park, Maryland

Search for other papers by Wallace A. Hogsett in
Current site
Google Scholar
PubMed
Close
,
Robert Rausch NOAA/NWS/NCEP/Weather Prediction Center, College Park, Maryland

Search for other papers by Robert Rausch in
Current site
Google Scholar
PubMed
Close
, and
Michael Schichtel NOAA/NWS/NCEP/Weather Prediction Center, College Park, Maryland

Search for other papers by Michael Schichtel in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

The role of the human forecaster in improving upon the accuracy of numerical weather prediction is explored using multiyear verification of human-generated short-range precipitation forecasts and medium-range maximum temperature forecasts from the Weather Prediction Center (WPC). Results show that human-generated forecasts improve over raw deterministic model guidance. Over the past two decades, WPC human forecasters achieved a 20%–40% improvement over the North American Mesoscale (NAM) model and the Global Forecast System (GFS) for the 1 in. (25.4 mm) (24 h)−1 threshold for day 1 precipitation forecasts, with a smaller, but statistically significant, 5%–15% improvement over the deterministic ECMWF model. Medium-range maximum temperature forecasts also exhibit statistically significant improvement over GFS model output statistics (MOS), and the improvement has been increasing over the past 5 yr. The quality added by humans for forecasts of high-impact events varies by element and forecast projection, with generally large improvements when the forecaster makes changes ≥8°F (4.4°C) to MOS temperatures. Human improvement over guidance for extreme rainfall events [3 in. (76.2 mm) (24 h)−1] is largest in the short-range forecast. However, human-generated forecasts failed to outperform the most skillful downscaled, bias-corrected ensemble guidance for precipitation and maximum temperature available near the same time as the human-modified forecasts. Thus, as additional downscaled and bias-corrected sensible weather element guidance becomes operationally available, and with the support of near-real-time verification, forecaster training, and tools to guide forecaster interventions, a key test is whether forecasters can learn to make statistically significant improvements over the most skillful of this guidance. Such a test can inform to what degree, and just how quickly, the role of the forecaster changes.

Corresponding author address: David R. Novak, NOAA/NWS/Weather Prediction Center, 5830 University Research Ct., Rm. 4633, College Park, MD 20740. E-mail: david.novak@noaa.gov

Abstract

The role of the human forecaster in improving upon the accuracy of numerical weather prediction is explored using multiyear verification of human-generated short-range precipitation forecasts and medium-range maximum temperature forecasts from the Weather Prediction Center (WPC). Results show that human-generated forecasts improve over raw deterministic model guidance. Over the past two decades, WPC human forecasters achieved a 20%–40% improvement over the North American Mesoscale (NAM) model and the Global Forecast System (GFS) for the 1 in. (25.4 mm) (24 h)−1 threshold for day 1 precipitation forecasts, with a smaller, but statistically significant, 5%–15% improvement over the deterministic ECMWF model. Medium-range maximum temperature forecasts also exhibit statistically significant improvement over GFS model output statistics (MOS), and the improvement has been increasing over the past 5 yr. The quality added by humans for forecasts of high-impact events varies by element and forecast projection, with generally large improvements when the forecaster makes changes ≥8°F (4.4°C) to MOS temperatures. Human improvement over guidance for extreme rainfall events [3 in. (76.2 mm) (24 h)−1] is largest in the short-range forecast. However, human-generated forecasts failed to outperform the most skillful downscaled, bias-corrected ensemble guidance for precipitation and maximum temperature available near the same time as the human-modified forecasts. Thus, as additional downscaled and bias-corrected sensible weather element guidance becomes operationally available, and with the support of near-real-time verification, forecaster training, and tools to guide forecaster interventions, a key test is whether forecasters can learn to make statistically significant improvements over the most skillful of this guidance. Such a test can inform to what degree, and just how quickly, the role of the forecaster changes.

Corresponding author address: David R. Novak, NOAA/NWS/Weather Prediction Center, 5830 University Research Ct., Rm. 4633, College Park, MD 20740. E-mail: david.novak@noaa.gov
Save
  • Bakhshaii, A., and Stull R. , 2009: Deterministic ensemble forecasts using gene-expression programming. Wea. Forecasting, 24, 14311451, doi:10.1175/2009WAF2222192.1.

    • Search Google Scholar
    • Export Citation
  • Baldwin, M. E., Lakshmivarahan S. , and Kain J. S. , 2002: Development of an “events oriented” approach to forecast verification. Preprints, 19th Conf. on Weather Analysis and Forecasting/15th Conf. on Numerical Weather Prediction, San Antonio, TX, Amer. Meteor. Soc., 7B.3. [Available online at https://ams.confex.com/ams/SLS_WAF_NWP/techprogram/paper_47738.htm.]

  • Bélair, S., Roch M. , Leduc A.-M. , Vaillancourt P. A. , Laroche S. , and Mailhot J. , 2009: Medium-range quantitative precipitation forecasts from Canada’s new 33-km deterministic global operational system. Wea. Forecasting, 24, 690708, doi:10.1175/2008WAF2222175.1.

    • Search Google Scholar
    • Export Citation
  • Bosart, L. F., 2003: Whither the weather analysis and forecasting process? Wea. Forecasting, 18, 520529, doi:10.1175/1520-0434(2003)18<520:WTWAAF>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Brill, K. F., 2009: A general analytic method for assessing sensitivity to bias of performance measures for dichotomous forecasts. Wea. Forecasting, 24, 307318, doi:10.1175/2008WAF2222144.1.

    • Search Google Scholar
    • Export Citation
  • Brill, K. F., and Mesinger F. , 2009: Applying a general analytic method for assessing bias sensitivity to bias-adjusted threat and equitable threat scores. Wea. Forecasting, 24, 17481754, doi:10.1175/2009WAF2222272.1.

    • Search Google Scholar
    • Export Citation
  • Brown, J. D., and Seo D.-J. , 2010: A nonparametric postprocessor for bias correction of hydrometeorological and hydrologic ensemble forecasts. J. Hydrometeor., 11, 642665, doi:10.1175/2009JHM1188.1.

    • Search Google Scholar
    • Export Citation
  • Caplan, P., Derber J. , Gemmill W. , Hong S.-Y. , Pan H.-L. , and Parrish D. , 1997: Changes to the 1995 NCEP operational Medium-Range Forecast Model Analysis–Forecast System. Wea. Forecasting, 12, 581594, doi:10.1175/1520-0434(1997)012<0581:CTTNOM>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Clark, A. J., Gallus W. A. Jr., Xue M. , and Kong F. , 2009: A comparison of precipitation forecast skill between small convection-allowing and large convection-parameterizing ensembles. Wea. Forecasting, 24, 11211140, doi:10.1175/2009WAF2222222.1.

    • Search Google Scholar
    • Export Citation
  • Cui, B., Toth Z. , Zhu Y. , and Hou D. , 2012: Bias correction for global ensemble forecast. Wea. Forecasting, 27, 396410, doi:10.1175/WAF-D-11-00011.1.

    • Search Google Scholar
    • Export Citation
  • Daly, C., Neilson R. P. , and Phillips D. L. , 1994: A statistical-topographic model for mapping climatological precipitation over mountainous terrain. J. Appl. Meteor., 33, 140158, doi:10.1175/1520-0450(1994)033<0140:ASTMFM>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Daly, C., Halbleib M. , Smith J. I. , Gibson W. P. , Doggett M. K. , Taylor G. H. , Curtis J. , and Pasteris P. A. , 2008: Physiographically sensitive mapping of climatological temperature and precipitation across the conterminous United States. Int. J. Climatol., 28, 20312064, doi:10.1002/joc.1688.

    • Search Google Scholar
    • Export Citation
  • De Pondeca, M. S. F. V., and Coauthors, 2011: The Real-Time Mesoscale Analysis at NOAA’s National Centers for Environmental Prediction: Current status and development. Wea. Forecasting, 26, 593612, doi:10.1175/WAF-D-10-05037.1.

    • Search Google Scholar
    • Export Citation
  • Doswell, C. A., III, 2004: Weather forecasting by humans—Heuristics and decision making. Wea. Forecasting, 19, 11151126, doi:10.1175/WAF-821.1.

    • Search Google Scholar
    • Export Citation
  • Du, J., and Coauthors, 2006: New dimension of NCEP Short-Range Ensemble Forecasting (SREF) system: Inclusion of WRF members. Preprints, WMO Expert Team Meeting on the Ensemble Prediction System, Exeter, United Kingdom, World Meteorological Organization. [Available online at http://www.emc.ncep.noaa.gov/mmb/SREF/reference.html.]

  • Ebert, E. E., 2001: Ability of a poor man’s ensemble to predict the probability and distribution of precipitation. Mon. Wea. Rev., 129, 24612480, doi:10.1175/1520-0493(2001)129<2461:AOAPMS>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Etherton, B. J., 2007: Preemptive forecasts using an ensemble Kalman filter. Mon. Wea. Rev., 135, 34843495, doi:10.1175/MWR3480.1.

  • Fritsch, J. M., and Carbone R. E. , 2004: Improving quantitative precipitation forecasts in the warm season: A USWRP research and development strategy. Bull. Amer. Meteor. Soc., 85, 955965, doi:10.1175/BAMS-85-7-955.

    • Search Google Scholar
    • Export Citation
  • Giorgi, F., 1991: Sensitivity of simulated summertime precipitation over the western United States to different physics parameterizations. Mon. Wea. Rev., 119, 28702888, doi:10.1175/1520-0493(1991)119<2870:SOSSPO>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Glahn, H. R., and Ruth D. P. , 2003: The new digital forecast database of the National Weather Service. Bull. Amer. Meteor. Soc., 84, 195201, doi:10.1175/BAMS-84-2-195.

    • Search Google Scholar
    • Export Citation
  • Glahn, H. R., Gilbert K. , Cosgrove R. , Ruth D. P. , and Sheets K. , 2009: The gridding of MOS. Wea. Forecasting, 24, 520529, doi:10.1175/2008WAF2007080.1.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., 1999: Hypothesis tests for evaluating numerical precipitation forecasts. Wea. Forecasting, 14, 155167, doi:10.1175/1520-0434(1999)014<0155:HTFENP>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Henkel, A., and Peterson C. , 1996: Can deterministic quantitative precipitation forecasts in mountainous regions be specified in a rapid, climatologically consistent manner with Mountain Mapper functioning as the tool for mechanical specification, quality control, and verification? Extended Abstracts, Fifth National Heavy Precipitation Workshop, State College, PA, NWS/NOAA, 31 pp. [Available from Office of Climate, Water, and Weather Services, W/OS, 1325 East–West Hwy., Silver Spring, MD 20910.]

  • Higgins, R. W., Janowiak J. E. , and Yao Y.-P. , 1996: A gridded hourly precipitation data base for the United States (1963–1993). NCEP/Climate Prediction Center Atlas 1, NOAA/NWS, 47 pp.

  • Homar, V., Stensrud D. J. , Levit J. J. , and Bright D. R. , 2006: Value of human-generated perturbations in short-range ensemble forecasts of severe weather. Wea. Forecasting, 21, 347363, doi:10.1175/WAF920.1.

    • Search Google Scholar
    • Export Citation
  • Janjić, Z. I., 2003: A nonhydrostatic model based on a new approach. Meteor. Atmos. Phys.,82, 271–285, doi:10.1007/s00703-001-0587-6.

  • Klein, W. H., and Glahn H. R. , 1974: Forecasting local weather by means of model output statistics. Bull. Amer. Meteor. Soc., 55, 12171227, doi:10.1175/1520-0477(1974)055<1217:FLWBMO>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Lin, Y., and Mitchell K. , 2005: The NCEP stage II/IV hourly precipitation analyses: Development and applications. Preprints, 19th Conf. on Hydrology, San Diego, CA, Amer. Meteor. Soc., 1.2. [Available online at https://ams.confex.com/ams/pdfpapers/83847.pdf.]

  • Magnusson, L., and Kallen E. , 2013: Factors influencing skill improvements in the ECMWF forecast system. Mon. Wea. Rev., 141, 31423153, doi:10.1175/MWR-D-12-00318.1.

    • Search Google Scholar
    • Export Citation
  • Mass, C. F., 2003: IFPS and the future of the National Weather Service. Wea. Forecasting, 18, 7579, doi:10.1175/1520-0434(2003)018<0075:IATFOT>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • McCarthy, P. J., Ball D. , and Purcell W. , 2007: Project Phoenix: Optimizing the machine–person mix in high-impact weather forecasting. Preprints, 22nd Conf. on Weather Analysis and Forecasting/18th Conf. on Numerical Weather Prediction, Park City, UT, Amer. Meteor. Soc., 6A.5. [Available online at https://ams.confex.com/ams/pdfpapers/122657.pdf.]

  • Murphy, A. H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8, 281293, doi:10.1175/1520-0434(1993)008<0281:WIAGFA>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Novak, D. R., Bright D. R. , and Brennan M. J. , 2008: Operational forecaster uncertainty needs and future roles. Wea. Forecasting, 23, 10691084, doi:10.1175/2008WAF2222142.1.

    • Search Google Scholar
    • Export Citation
  • Olson, D. A., Junker N. W. , and Korty B. , 1995: Evaluation of 33 years of quantitative precipitation forecasting at the NMC. Wea. Forecasting, 10, 498511, doi:10.1175/1520-0434(1995)010<0498:EOYOQP>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Reynolds, D., 2003: Value-added quantitative precipitation forecasts: How valuable is the forecaster? Bull. Amer. Meteor. Soc., 84, 876878, doi:10.1175/BAMS-84-7-876.

    • Search Google Scholar
    • Export Citation
  • Roebber, P. J., 2010: Seeking consensus: A new approach. Mon. Wea. Rev., 138, 44024415, doi:10.1175/2010MWR3508.1.

  • Roebber, P. J., Schultz D. M. , Colle B. A. , and Stensrud D. J. , 2004: Toward improved prediction: High-resolution and ensemble modeling systems in operations. Wea. Forecasting, 19, 936949, doi:10.1175/1520-0434(2004)019<0936:TIPHAE>2.0.CO;2.

    • Search Google Scholar
    • Export Citation
  • Roebber, P. J., Westendorf M. , and Meadows G. R. , 2010: Innovative weather: A new strategy for student, university, and community relationships. Bull. Amer. Meteor. Soc., 91, 877888, doi:10.1175/2010BAMS2854.1.

    • Search Google Scholar
    • Export Citation
  • Ruth, D. P., Glahn B. , Dagostaro V. , and Gilbert K. , 2009: The performance of MOS in the digital age. Wea. Forecasting, 24, 504519, doi:10.1175/2008WAF2222158.1.

    • Search Google Scholar
    • Export Citation
  • Stuart, N. A., and Coauthors, 2006: The future of humans in an increasingly automated forecast process. Bull. Amer. Meteor. Soc., 87, 14971501, doi:10.1175/BAMS-87-11-1497.

    • Search Google Scholar
    • Export Citation
  • Stuart, N. A., Schultz D. M. , and Klein G. , 2007: Maintaining the role of humans in the forecast process: Analyzing the psyche of expert forecasters. Bull. Amer. Meteor. Soc., 88, 18931898, doi:10.1175/BAMS-88-12-1893.

    • Search Google Scholar
    • Export Citation
  • Voisin, N., Schaake J. C. , and Lettenmaier D. P. , 2010: Calibration and downscaling methods for quantitative ensemble precipitation forecasts. Wea. Forecasting, 25, 16031627, doi:10.1175/2010WAF2222367.1.

    • Search Google Scholar
    • Export Citation
  • Yussouf, N., and Stensrud D. J. , 2006: Prediction of near-surface variables at independent locations from a bias-corrected ensemble forecasting system. Mon. Wea. Rev., 134, 34153424, doi:10.1175/MWR3258.1.

    • Search Google Scholar
    • Export Citation
  • Zheng, M., Chang E. K. M. , and Colle B. A. , 2013: Ensemble sensitivity tools for assessing extratropical cyclone intensity and track predictability. Wea. Forecasting, 28, 1133–1156.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 4312 882 48
PDF Downloads 1628 326 24