Evaluation of Probabilistic Medium-Range Temperature Forecasts from the North American Ensemble Forecast System

Doug McCollor University of British Columbia, and BC Hydro Corporation, Vancouver, British Columbia, Canada

Search for other papers by Doug McCollor in
Current site
Google Scholar
PubMed
Close
and
Roland Stull University of British Columbia, Vancouver, British Columbia, Canada

Search for other papers by Roland Stull in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Ensemble temperature forecasts from the North American Ensemble Forecast System were assessed for quality against observations for 10 cities in western North America, for a 7-month period beginning in February 2007. Medium-range probabilistic temperature forecasts can provide information for those economic sectors exposed to temperature-related business risk, such as agriculture, energy, transportation, and retail sales.

The raw ensemble forecasts were postprocessed, incorporating a 14-day moving-average forecast–observation difference, for each ensemble member. This postprocessing reduced the mean error in the sample to 0.6°C or less. It is important to note that the North American Ensemble Forecast System available to the public provides bias-corrected maximum and minimum temperature forecasts.

Root-mean-square-error and Pearson correlation skill scores, applied to the ensemble average forecast, indicate positive, but diminishing, forecast skill (compared to climatology) from 1 to 9 days into the future. The probabilistic forecasts were evaluated using the continuous ranked probability skill score, the relative operating characteristics skill score, and a value assessment incorporating cost–loss determination. The full suite of ensemble members provided skillful forecasts 10–12 days into the future.

A rank histogram analysis was performed to test ensemble spread relative to the observations. Forecasts are underdispersive early in the forecast period, for forecast days 1 and 2. Dispersion improves rapidly but remains somewhat underdispersive through forecast day 6. The forecasts show little or no dispersion beyond forecast day 6. A new skill versus spread diagram is presented that shows the trade-off between higher skill but low spread early in the forecast period and lower skill but better spread later in the forecast period.

Corresponding author address: Doug McCollor, Dept. of Earth and Ocean Sciences, University of British Columbia, 6339 Stores Rd., Vancouver, BC V6T 1Z4, Canada. Email: doug.mccollor@bchydro.com

Abstract

Ensemble temperature forecasts from the North American Ensemble Forecast System were assessed for quality against observations for 10 cities in western North America, for a 7-month period beginning in February 2007. Medium-range probabilistic temperature forecasts can provide information for those economic sectors exposed to temperature-related business risk, such as agriculture, energy, transportation, and retail sales.

The raw ensemble forecasts were postprocessed, incorporating a 14-day moving-average forecast–observation difference, for each ensemble member. This postprocessing reduced the mean error in the sample to 0.6°C or less. It is important to note that the North American Ensemble Forecast System available to the public provides bias-corrected maximum and minimum temperature forecasts.

Root-mean-square-error and Pearson correlation skill scores, applied to the ensemble average forecast, indicate positive, but diminishing, forecast skill (compared to climatology) from 1 to 9 days into the future. The probabilistic forecasts were evaluated using the continuous ranked probability skill score, the relative operating characteristics skill score, and a value assessment incorporating cost–loss determination. The full suite of ensemble members provided skillful forecasts 10–12 days into the future.

A rank histogram analysis was performed to test ensemble spread relative to the observations. Forecasts are underdispersive early in the forecast period, for forecast days 1 and 2. Dispersion improves rapidly but remains somewhat underdispersive through forecast day 6. The forecasts show little or no dispersion beyond forecast day 6. A new skill versus spread diagram is presented that shows the trade-off between higher skill but low spread early in the forecast period and lower skill but better spread later in the forecast period.

Corresponding author address: Doug McCollor, Dept. of Earth and Ocean Sciences, University of British Columbia, 6339 Stores Rd., Vancouver, BC V6T 1Z4, Canada. Email: doug.mccollor@bchydro.com

Save
  • Anderson, J. L., 1996: A method for producing and evaluating probabilistic precipitation forecasts from ensemble model integrations. J. Climate, 9 , 15181529.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Atger, F., 2003: Spatial and interannual variability of the reliability of ensemble-based probabilistic forecasts: Consequences for calibration. Mon. Wea. Rev., 131 , 15091523.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Brier, G. W., 1950: Verification of forecasts expressed in terms of probability. Mon. Wea. Rev., 78 , 13.

  • Buizza, R., Houtekamer P. L. , Toth Z. , Pellerin G. , Wei M. , and Zhu Y. , 2005: A comparison of the ECMWF, MSC, and NCEP global ensemble prediction systems. Mon. Wea. Rev., 133 , 10761097.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Candille, G., and Talagrand O. , 2005: Evaluation of probabilistic prediction systems for a scalar variable. Quart. J. Roy. Meteor. Soc., 131 , 21312150.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Côté, J., Desmarais J-G. , Gravel S. , Méthot A. , Patoine A. , Roch M. , and Staniforth A. , 1998a: The operational CMC–MRB Global Environmental Multiscale (GEM) model. Part I: Design considerations and formulation. Mon. Wea. Rev., 126 , 13731395.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Côté, J., Desmarais J-G. , Gravel S. , Méthot A. , Patoine A. , Roch M. , and Staniforth A. , 1998b: The operational CMC–MRB Global Environmental Multiscale (GEM) model. Part II: Results. Mon. Wea. Rev., 126 , 13971418.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Eckel, F. A., and Mass C. F. , 2005: Aspects of effective mesoscale, short-range ensemble forecasting. Wea. Forecasting, 20 , 328350.

  • Gallus W. A. Jr., , Baldwin M. E. , and Elmore K. L. , 2007: Evaluation of probabilistic precipitation forecasts determined from Eta and AVN forecasted amounts. Wea. Forecasting, 22 , 207215.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., Mullen S. L. , Snyder C. , Toth Z. , and Baumhefner D. P. , 2000: Ensemble forecasting in the short to medium range: Report from a workshop. Bull. Amer. Meteor. Soc., 81 , 26532664.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hersbach, H., 2000: Decomposition of the continuous ranked probability score for ensemble prediction systems. Wea. Forecasting, 15 , 559570.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hou, D., Kalnay E. , and Droegemeier K. K. , 2001: Objective verification of the SAMEX ‘98 ensemble forecasts. Mon. Wea. Rev., 129 , 7391.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., Lefaivre L. , Derome J. , Ritchie H. , and Mitchell H. L. , 1996: A system simulation approach to ensemble prediction. Mon. Wea. Rev., 124 , 12251242.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jones, M. S., Colle B. A. , and Tongue J. S. , 2007: Evaluation of a mesoscale short-range ensemble forecast system over the northeast United States. Wea. Forecasting, 22 , 3655.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Katz, R. W., and Murphy A. H. , 1997: Economic Value of Weather and Climate Forecasts. Cambridge University Press, 222 pp.

  • Mason, I., 1982: A model for assessment of weather forecasts. Aust. Meteor. Mag., 30 , 291303.

  • Mason, I., and Graham N. E. , 1999: Conditional probabilities, relative operating characteristics, and relative operating levels. Wea. Forecasting, 14 , 713725.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • McCollor, D., and Stull R. , 2008a: Hydrometeorological accuracy enhancement via postprocessing of numerical weather forecasts in complex terrain. Wea. Forecasting, 23 , 131144.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • McCollor, D., and Stull R. , 2008b: Hydrometeorological short-range ensemble forecasts in complex terrain. Part I: Meteorological evaluation. Wea. Forecasting, 23 , 533556.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • McCollor, D., and Stull R. , 2008c: Hydrometeorological short-range ensemble forecasts in complex terrain. Part II: Economic evaluation. Wea. Forecasting, 23 , 557574.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1969: On the “Ranked probability score.”. J. Appl. Meteor., 8 , 988989.

  • Murphy, A. H., 1973: A new vector partition of the probability score. J. Appl. Meteor., 12 , 595600.

  • Murphy, A. H., 1977: The value of climatological, categorical, and probabilistic forecasts in the cost–loss ratio situation. Mon. Wea. Rev., 105 , 803816.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1991: Forecast verification: Its complexity and dimensionality. Mon. Wea. Rev., 119 , 15901601.

  • Murphy, A. H., and Winkler R. L. , 1987: A general framework for forecast verification. Mon. Wea. Rev., 115 , 13301338.

  • Potts, J. M., 2003: Basic concepts. Forecast Verification: A Practitioner’s Guide in Atmospheric Science, I. Jolliffe and D. B. Stephenson, Eds., Wiley, 13–36.

    • Search Google Scholar
    • Export Citation
  • Richardson, D. S., 2000: Skill and relative economic value of the ECMWF Ensemble Prediction System. Quart. J. Roy. Meteor. Soc., 126 , 649667.

  • Richardson, D. S., 2003: Economic value and skill. Forecast Verification: A Practitioner’s Guide in Atmospheric Science, I. Jolliffe and D. B. Stephenson, Eds., Wiley, 164–187.

    • Search Google Scholar
    • Export Citation
  • Roulin, E., and Vannitsem S. , 2005: Skill of medium-range hydrological ensemble predictions. J. Hydrometeor., 6 , 729744.

  • Stensrud, D. J., Bao J. , and Warner T. T. , 2000: Using initial conditions and model physics perturbations in short-range ensemble simulations of mesoscale convective systems. Mon. Wea. Rev., 128 , 20772107.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Talagrand, O., Vautard R. , and Strauss B. , 1998: Evaluation of probabilistic prediction systems. Proc. Workshop on Predictability, Reading, United Kingdom, ECMWF, 1–25.

    • Search Google Scholar
    • Export Citation
  • Tennant, W. J., Toth Z. , and Rae K. J. , 2007: Application of the NCEP Ensemble Prediction System to medium-range forecasting in South Africa: New products, benefits, and challenges. Wea. Forecasting, 22 , 1835.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Toth, Z., and Kalnay E. , 1993: Ensemble forecasting at NMC: The generation of perturbations. Bull. Amer. Meteor. Soc., 74 , 23172330.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Toth, Z., Talagrand O. , Candille G. , and Zhu Y. , 2003: Probability and ensemble forecasts. Forecast Verification: A Practitioner’s Guide in Atmospheric Science, I. Jolliffe and D. B. Stephenson, Eds., Wiley, 137–163.

    • Search Google Scholar
    • Export Citation
  • Toth, Z., and Coauthors, 2006: The North American Ensemble Forecast System (NAEFS). Preprints, 18th Conf. on Probability and Statistics in the Atmospheric Sciences, Atlanta, GA, Amer. Meteor. Soc., 4.1. [Available online at http://ams.confex.com/ams/pdfpapers/102588.pdf.].

    • Search Google Scholar
    • Export Citation
  • Wei, M., Toth Z. , Wobus R. , and Zhu Y. , 2008: Initial perturbations based on the ensemble transform (ET) technique in the NCEP Global Operational Forecast System. Tellus, 60A , 6279.

    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2006: Statistical Methods in the Atmospheric Sciences. 2nd ed. Academic Press, 627 pp.

  • Yussouf, N., and Stensrud D. J. , 2007: Bias-corrected short-range ensemble forecasts of near-surface variables during the 2005/06 cool season. Wea. Forecasting, 22 , 12741286.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zhu, Y., Toth Z. , Wobus R. , Richardson D. , and Mylne K. , 2002: The economic value of ensemble-based weather forecasts. Bull. Amer. Meteor. Soc., 83 , 7383.

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 205 113 6
PDF Downloads 93 20 1