• Atger, F., 1999: The skill of ensemble prediction systems. Mon. Wea. Rev., 127 , 19411953.

  • Atger, F., 2003: Spatial and interannual variability of the reliability of ensemble-based probabilistic forecasts: Consequences for calibrations. Mon. Wea. Rev., 131 , 15091523.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Atger, F., 2004: Estimation of the reliability of ensemble-based probabilistic forecasts. Quart. J. Roy. Meteor. Soc., 130 , 627646.

  • Bonsal, B. R., and Wheaton E. E. , 2005: Atmospheric circulation comparisons between the 2001 and 2002 and the 1961 and 1988 Canadian prairie droughts. Atmos.–Ocean, 43 , 163172.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Brier, G. W., 1950: Verification of forecasts expressed in terms of probabilities. Mon. Wea. Rev., 78 , 13.

  • Bright, D. R., Weiss S. J. , Wandishin M. S. , Kain J. S. , and Stensrud D. J. , 2004: Evaluation of short-range ensemble forecasts during the 2003 SPC/NSSL Spring Program. Preprints, 22nd Conf. on Severe Local Storms, Hyannis, MA, Amer. Meteor. Soc., P15.5. [Available online at http://ams.confex.com/ams/pdfpapers/68921.pdf.].

  • Buizza, R., Hollingsworth A. , Lalaurette F. , and Ghelli A. , 1999a: Probabilistic predictions of precipitation using the ECMWF Ensemble Prediction System. Wea. Forecasting, 14 , 168189.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Buizza, R., Miller M. , and Palmer T. N. , 1999b: Stochastic simulation of model uncertainties. Quart. J. Roy. Meteor. Soc., 125 , 28872908.

    • Search Google Scholar
    • Export Citation
  • Buizza, R., Houtekamer P. L. , Toth Z. , Pellerin G. , Wei M. , and Zhu Y. , 2005: A Comparison of the ECMWF, MSC, and NCEP global ensemble prediction systems. Mon. Wea. Rev., 133 , 10761097.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Candille, G., and Talagrand O. , 2005: Evaluation of probabilistic prediction systems for a scalar variable. Quart. J. Roy. Meteor. Soc., 131 , 21312150.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cherubini, T., Ghelli A. , and Lalaurette F. , 2002: Verification of precipitation forecasts over the Alpine region using a high-density observing network. Wea. Forecasting, 17 , 238249.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Côté, J. S., Gravel S. , Méthot A. , Patoine A. , Roch M. , and Staniforth A. , 1998: The operational CMC/MRB Global Environmental Multiscale (GEM) model. Part I: Design considerations and formulation. Mon. Wea. Rev., 126 , 13731395.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Doswell C. A. III, , 1987: The distinction between large-scale and mesoscale contribution to severe convection: A case study example. Wea. Forecasting, 2 , 316.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Eckel, F. A., and Walters M. K. , 1998: Calibrated probabilistic quantitative precipitation forecasts based on the MRF ensemble. Wea. Forecasting, 13 , 11321147.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Feller, W., 1957: An Introduction to Probability Theory and its Applications. Vol. 1. Wiley, 461 pp.

  • Fritsch, J. M., Thomas I. J. , Hansen O. F. , and Hardy G. G. , 1998: Quantitative precipitation forecasting: Report of the Eighth Prospectus Development Team, U.S. Weather Research Program. Bull. Amer. Meteor. Soc., 79 , 285299.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gauthier, P., Charette C. , Fillion L. , Koclas P. , and Laroche S. , 1999: Implementation of a 3d variational data assimilation system at the Canadian Meteorological Centre. Part 1: The global analysis. Atmos.–Ocean, 37 , 103156.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ghelli, A., and Lalaurette F. , 2000: Verifying precipitation forecasts using upscaled observations. ECMWF Newsletter, No. 87, ECMWF, Reading, United Kingdom, 9–17.

  • Glahn, H., and Lowry D. A. , 1972: The use of model output statistics (MOS) in objective weather forecasting. J. Appl. Meteor., 11 , 12031211.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Goodison, B. E., 1978: Accuracy of Canadian snow gage measurements. J. Appl. Meteor., 17 , 15421548.

  • Hamill, T. M., and Colucci S. J. , 1998: Evaluation of Eta–RSM ensemble probabilistic precipitation forecasts. Mon. Wea. Rev., 126 , 711724.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., and Juras J. , 2006: Measuring forecast forecast skill: Is it real skill or is it the varying climatology? Quart. J. Roy. Meteor. Soc., 132 , 29052923.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Harvey L. O. Jr., , Hammond K. R. , Lusk C. M. , and Mross E. F. , 1992: The application of signal detection theory to weather forecasting behavior. Mon. Wea. Rev., 120 , 863883.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Houtekamer, P. L., Lefaivre L. , Derome J. , Ritchie H. , and Mitchell H. L. , 1996: A system simulation approach to ensemble prediction. Mon. Wea. Rev., 124 , 12251242.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jones, S. C., and Coauthors, 2003: Extratropical transition of tropical cyclones: Forecast challenges, current understanding, and future directions. Wea. Forecasting, 18 , 10521092.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lefaivre, L., Houtekamer P. L. , Bergeron A. , and Verret R. , 1997: The CMC Ensemble Prediction System. Proc. Sixth Workshop on Meteorological Operational Systems, Reading, United Kingdom, ECMWF, 31–44.

    • Search Google Scholar
    • Export Citation
  • Mason, I., 1982: A model for assessment of weather forecasts. Aust. Meteor. Mag., 30 , 291303.

  • Metcalfe, J. R., and Goodison B. E. , 1993: Correction of Canadian winter precipitation data. Proc. Eighth Symp. on Meteorological Observations and Instrumentation, Anaheim, CA, Amer. Meteor. Soc., 338–343.

    • Search Google Scholar
    • Export Citation
  • Metcalfe, J. R., Routledge B. , and Devine K. , 1997: Rainfall measurement in Canada: Changing observational methods and archive adjustment procedures. J. Climate, 10 , 92101.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mullen, S. L., and Buizza R. , 2001: Quantitative precipitation forecasts over the United States by the ECMWF Ensemble Prediction System. Mon. Wea. Rev., 129 , 638663.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mullen, S. L., and Buizza R. , 2002: The impact of horizontal resolution and ensemble size on probabilistic forecasts of precipitation by the ECMWF Ensemble Prediction System. Wea. Forecasting, 17 , 173191.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1973: A new vector partition of the probability score. J. Appl. Meteor., 12 , 595600.

  • Murphy, A. H., 1986: The attributes diagram: A geometric framework for assessing the quality of probability forecasts. Int. J. Forecasting, 2 , 285293.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8 , 281293.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., and Winkler R. L. , 1987: A general framework for forecast verification. Mon. Wea. Rev., 115 , 13301338.

  • Pellerin, G., Lefaivre L. , Houtekamer P. , and Girard C. , 2003: Increasing the horizontal resolution of ensemble forecasts at CMC. Nonlinear Processes Geophys., 10 , 463468.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Petterssen, S., 1956a: Motion and Motion Systems. Vol. 1, Weather Analysis and Forecasting, McGraw-Hill, 428 pp.

  • Petterssen, S., 1956b: Weather and Weather Systems. Vol. 2, Weather Analysis and Forecasting, McGraw-Hill, 266 pp.

  • Richardson, D. S., 2000: Skill and relative economic value of the ECMWF Ensemble Prediction System. Quart. J. Roy. Meteor. Soc., 126 , 649667.

  • Ritchie, H., and Beaudoin C. , 1994: Approximations and sensitivity experiments with a baroclinic semi-Lagrangian spectral model. Mon. Wea. Rev., 122 , 23912399.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sanders, F., 1986: Trends in skill of Boston forecasts made at MIT 1966–84. Bull. Amer. Meteor. Soc., 67 , 170176.

  • Sanders, F., and Gyakum J. R. , 1980: Synoptic–dynamic climatology of the “bomb”. Mon. Wea. Rev., 108 , 15891606.

  • Stanski, H. R., Wilson L. J. , and Burrows W. R. , 1990: Survey of common verification methods in meteorology. World Weather Watch Tech. Rep. 8., WMO, Geneva, Switzerland, 114 pp.

  • Swets, J. A., and Pickett R. M. , 1982: Evaluation of Diagnostic Systems: Methods from Signal Detection Theory. Academic Press, 253 pp.

  • Wilks, D. S., 2006: Statistical Methods in the Atmospheric Sciences. 2nd ed. Elsevier Academic, 627 pp.

  • Wilson, L. J., 2000: Comments on “Probabilistic predictions of precipitation using the ECMWF Ensemble Prediction System”. Wea. Forecasting, 15 , 361364.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Yuan, H., Mullen S. L. , Gao X. , Sorooshian S. , Du J. , and Juang H-M. H. , 2005: Verification of probabilistic quantitative precipitation forecasts over the southwest United States during winter 2002/03 by the RSM ensemble system. Mon. Wea. Rev., 133 , 279294.

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 48 33 0
PDF Downloads 35 30 0

A Diagnostic Verification of the Precipitation Forecasts Produced by the Canadian Ensemble Prediction System

View More View Less
  • 1 Meteorological Research Division, Environment Canada, Toronto, Ontario, Canada
  • | 2 Meteorological Research Division, Environment Canada, Dorval, Québec, Canada
Restricted access

Abstract

A comparatively long period of relative stability in the evolution of the Canadian Ensemble Forecast System was exploited to compile a large homogeneous set of precipitation forecasts. The probability of exceedance of a given threshold was computed as the fraction of ensemble member forecasts surpassing that threshold, and verified directly against observations from 36 stations across the country. These forecasts were stratified into warm and cool seasons and assessed against the observations through attributes diagrams, Brier skill scores, and areas under receiver operating characteristic curves. These measures were deemed sufficient to illuminate the salient features of a forecast system. Particular attention was paid to forecasts of 24-h accumulation, especially the exceedance of thresholds in the upper decile of station climates. The ability of the system to forecast extended dry periods was also explored.

Warm season forecasts for the 90th percentile threshold were found to be competitive with, even superior to, those for the cool season when verifying across the sample lumping together all of the stations. The relative skill of the forecasts in the two seasons depends strongly on station location, however. Moreover, the skill of the warm season forecasts rapidly drops below cool season values as the thresholds become more extreme. The verification, particularly of the cool season, is sensitive to the calibration of the gauge reports, which is complicated by the inclusion of snow events in the observational record.

Corresponding author address: Syd Peel, 4905 Dufferin St., Toronto, ON M3H 5T4, Canada. Email: syd.peel@ec.gc.ca

Abstract

A comparatively long period of relative stability in the evolution of the Canadian Ensemble Forecast System was exploited to compile a large homogeneous set of precipitation forecasts. The probability of exceedance of a given threshold was computed as the fraction of ensemble member forecasts surpassing that threshold, and verified directly against observations from 36 stations across the country. These forecasts were stratified into warm and cool seasons and assessed against the observations through attributes diagrams, Brier skill scores, and areas under receiver operating characteristic curves. These measures were deemed sufficient to illuminate the salient features of a forecast system. Particular attention was paid to forecasts of 24-h accumulation, especially the exceedance of thresholds in the upper decile of station climates. The ability of the system to forecast extended dry periods was also explored.

Warm season forecasts for the 90th percentile threshold were found to be competitive with, even superior to, those for the cool season when verifying across the sample lumping together all of the stations. The relative skill of the forecasts in the two seasons depends strongly on station location, however. Moreover, the skill of the warm season forecasts rapidly drops below cool season values as the thresholds become more extreme. The verification, particularly of the cool season, is sensitive to the calibration of the gauge reports, which is complicated by the inclusion of snow events in the observational record.

Corresponding author address: Syd Peel, 4905 Dufferin St., Toronto, ON M3H 5T4, Canada. Email: syd.peel@ec.gc.ca

Save