• Black, T. L., 1994: The new NMC mesoscale Eta Model: Description and forecast examples. Wea. Forecasting, 9 , 265284.

  • Brooks, H. E., , Doswell C. A. , , and Maddox R. A. , 1992: On the use of mesoscale and cloud-scale models in operational forecasting. Wea. Forecasting, 7 , 120132.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Caplan, P., , Derber J. , , Gemmill W. , , Hong S-Y. , , Pan H-L. , , and Parrish D. , 1997: Changes to the 1995 NCEP operational medium-range forecast model analysis–forecast system. Wea. Forecasting, 12 , 581594.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Dalcher, A., , Kalnay E. , , and Hoffman R. N. , 1988: Medium-range lagged average forecasts. Mon. Wea. Rev., 116 , 402416.

  • Gilovich, T., 1993: How What We Know Isn't So: The Fallibility of Human Reason in Everyday Life. Free Press, 216 pp.

  • Grimit, E. P., , and Mass C. F. , 2002: Initial results of a mesoscale short-range ensemble forecasting system over the Pacific Northwest. Wea. Forecasting, 17 , 192205.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., , Mullen S. L. , , Snyder C. , , Toth Z. , , and Baumhefner D. P. , 2000: Ensemble forecasting in the short to medium range. Report from a workshop. Bull. Amer. Meteor. Soc., 81 , 26532664.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hoffman, R. N., , and Kalnay E. , 1983: Lagged average forecasting, an alternative to Monte-Carlo forecasting. Tellus, 35A , 100118.

  • Houtekamer, P. L., , Lefaivre L. , , and Derome J. , 1996: The RPN ensemble prediction system. Proc. ECMWF Seminar on Predictability, Vol. II, Reading, United Kingdom, ECMWF, 121–146. [Available from ECMWF, Shinfield Park, Reading, Berkshire RG2 9AX, United Kingdom.].

    • Search Google Scholar
    • Export Citation
  • Kalnay, E., and Coauthors. 1996: The NCEP/NCAR 40-Year Reanalysis Project. Bull. Amer. Meteor. Soc., 77 , 437472.

  • Kanamitsu, M., 1989: Description of the NMC global data assimilation and forecast system. Wea. Forecasting, 4 , 334342.

  • Kanamitsu, M., and Coauthors. 1991: Recent changes implemented into the global forecast system at NMC. Wea. Forecasting, 6 , 425435.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mesinger, F., 1996: Improvements in quantitative precipitation forecasts with the Eta regional model at the National Centers for Environmental Prediction: The 48-km upgrade. Bull. Amer. Meteor. Soc., 77 , 26372650.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Molteni, F., , Buizza R. , , Palmer T. N. , , and Petroliagis T. , 1996: The ECMWF ensemble prediction system: Methodology and validation. Quart. J. Roy. Meteor. Soc., 122 , 73119.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mullen, S. L., , and Buizza R. , 2001: Quantitative precipitation forecasts over the United States by the ECMWF ensemble prediction system. Mon. Wea. Rev., 129 , 638663.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mullen, S. L., , and Buizza R. , 2002: The impact of horizontal resolution and ensemble size on probabilistic forecasts of precipitation by the ECMWF ensemble prediction system. Wea. Forecasting, 17 , 173191.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rogers, E., , Deaven D. G. , , and Dimego G. S. , 1995: The regional analysis system for the operational “early” Eta Model: Original 80-km configuration and recent changes. Wea. Forecasting, 10 , 810825.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rogers, E., , Black T. L. , , Deaven D. G. , , DiMego G. J. , , Zhao Q. , , Baldwin M. , , Junker N. W. , , and Lin Y. , 1996: Changes to the operational “early” Eta analysis/forecast system at the National Centers for Environmental Prediction. Wea. Forecasting, 11 , 391416.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Toth, Z., , and Kalnay E. , 1993: Ensemble forecasting at NMC: The generation of perturbations. Bull. Amer. Meteor. Soc., 74 , 23172330.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Toth, Z., , and Kalnay E. , 1997: Ensemble forecasting at NCEP and the breeding method. Mon. Wea. Rev., 125 , 32973319.

  • Wandishin, M. S., , Mullen S. L. , , Stensrud D. J. , , and Brooks H. E. , 2001: Evaluation of a short-range multimodel ensemble system. Mon. Wea. Rev., 129 , 729747.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 1995: Statistical Methods in the Atmospheric Sciences: An Introduction. Academic Press, 467 pp.

  • Wu, W., , Iridell M. , , Saha S. , , and Caplan P. , 1997: Changes to the 1997 NCEP operational MRF model analysi/forecast system. NCEP Tech. Procedures Bull. 443, 301 pp. [Available online at http://www.nws.noaa.gov/om/tpb/indexb.htm; or from Programs and Plans Division, Office of Meteorology, National Weather Service, Silver Spring, MD 20910.].

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 222 222 5
PDF Downloads 78 78 5

Evaluating Forecasters' Rules of Thumb: A Study of d(prog)/dt

View More View Less
  • 1 University of Colorado and NOAA–CIRES Climate Diagnostics Center, Boulder, Colorado
© Get Permissions
Restricted access

Abstract

Forecasters often develop rules of thumb for adjusting model guidance. Ideally, before use, these rules of thumb should be validated through a careful comparison of model forecasts and observations over a large sample. Practically, such evaluation studies are difficult to perform because forecast models are continually being changed, and a hypothesized rule of thumb may only be applicable to a particular forecast model configuration.

A particular rule of thumb was examined here: dprog/dt. Given a set of lagged forecasts from the same model all verifying at the same time, this rule of thumb suggests that if the forecasts show a trend, this trend is more likely than not to continue and thus provide useful information for correcting the most recent forecast. Forecasters may also note the amount of continuity of forecasts to estimate the magnitude of the error in the most recent forecast.

Statistical evaluation of this rule of thumb was made possible here using a dataset of forecasts from a “frozen” model. A 23-yr record of forecasts was generated from a T62 version of the medium-range forecast model used at the National Centers for Environmental Prediction. Forecasts were initialized from reanalysis data, and January–March forecasts were examined for selected locations. The rule dprog/dt was evaluated with 850-hPa temperature forecasts. A total of 2070 sample days were used in the evaluation.

Extrapolation of forecast trends was shown to have little forecast value. Also, there was only a small amount of information on forecast accuracy from the amount of discrepancy between short-term lagged forecasts. The lack of validity of this rule of thumb suggest that others should also be carefully scrutinized before use.

Corresponding author address: Dr. Thomas M. Hamill, NOAA–CIRES CDC, R/CDC 1, 325 Broadway, Boulder, CO 80305-3328. Email: tom.hamill@noaa.gov

Abstract

Forecasters often develop rules of thumb for adjusting model guidance. Ideally, before use, these rules of thumb should be validated through a careful comparison of model forecasts and observations over a large sample. Practically, such evaluation studies are difficult to perform because forecast models are continually being changed, and a hypothesized rule of thumb may only be applicable to a particular forecast model configuration.

A particular rule of thumb was examined here: dprog/dt. Given a set of lagged forecasts from the same model all verifying at the same time, this rule of thumb suggests that if the forecasts show a trend, this trend is more likely than not to continue and thus provide useful information for correcting the most recent forecast. Forecasters may also note the amount of continuity of forecasts to estimate the magnitude of the error in the most recent forecast.

Statistical evaluation of this rule of thumb was made possible here using a dataset of forecasts from a “frozen” model. A 23-yr record of forecasts was generated from a T62 version of the medium-range forecast model used at the National Centers for Environmental Prediction. Forecasts were initialized from reanalysis data, and January–March forecasts were examined for selected locations. The rule dprog/dt was evaluated with 850-hPa temperature forecasts. A total of 2070 sample days were used in the evaluation.

Extrapolation of forecast trends was shown to have little forecast value. Also, there was only a small amount of information on forecast accuracy from the amount of discrepancy between short-term lagged forecasts. The lack of validity of this rule of thumb suggest that others should also be carefully scrutinized before use.

Corresponding author address: Dr. Thomas M. Hamill, NOAA–CIRES CDC, R/CDC 1, 325 Broadway, Boulder, CO 80305-3328. Email: tom.hamill@noaa.gov

Save