Editorial Type:
Article
Article Type:
Research Article

Verification of an Ensemble Prediction System against Observations

G. Candille Environment Canada, Dorval, Quebec, Canada

Search for other papers by G. Candille in
Current site
Google Scholar
PubMed Close
,
C. Côté Environment Canada, Dorval, Quebec, Canada

Search for other papers by C. Côté in
Current site
Google Scholar
PubMed Close
,
P. L. Houtekamer Environment Canada, Dorval, Quebec, Canada

Search for other papers by P. L. Houtekamer in
Current site
Google Scholar
PubMed Close
, and
G. Pellerin Environment Canada, Dorval, Quebec, Canada

Search for other papers by G. Pellerin in
Current site
Google Scholar
PubMed Close
Print Publication:
01 Jul 2007
DOI:
https://doi.org/10.1175/MWR3414.1
Page(s):
2688–2699
Restricted access

Abstract

A verification system has been developed for the ensemble prediction system (EPS) at the Canadian Meteorological Centre (CMC). This provides objective criteria for comparing two EPSs, necessary when deciding whether or not to implement a new or revised EPS. The proposed verification methodology is based on the continuous ranked probability score (CRPS), which provides an evaluation of the global skill of an EPS. Its reliability/resolution partition, proposed by Hersbach, is used to measure the two main attributes of a probabilistic system. Also, the characteristics of the reliability are obtained from the two first moments of the reduced centered random variable (RCRV), which define the bias and the dispersion of an EPS. Resampling bootstrap techniques have been applied to these scores. Confidence intervals are thus defined, expressing the uncertainty due to the finiteness of the number of realizations used to compute the scores. All verifications are performed against observations to provide more independent validations and to avoid any local systematic bias of an analysis. A revised EPS, which has been tested at the CMC in a parallel run during the autumn of 2005, is described in this paper. This EPS has been compared with the previously operational one with the verification system presented above. To illustrate the verification methodology, results are shown for the temperature at 850 hPa. The confidence intervals are computed by taking into account the spatial correlation of the data and the temporal autocorrelation of the forecast error. The revised EPS performs significantly better for all the forecast ranges, except for the resolution component of the CRPS where the improvement is no longer significant from day 7. The significant improvement of the reliability is mainly due to a better dispersion of the ensemble. Finally, the verification system correctly indicates that variations are not significant when two theoretically similar EPSs are compared.

Corresponding author address: G. Candille, Direction de la Recherche en Météorologie, 2121 Route Trans-Canadienne, Dorval, Québec H9P 1J3, Canada. Email: guillem.candille@ec.gc.ca

Abstract

A verification system has been developed for the ensemble prediction system (EPS) at the Canadian Meteorological Centre (CMC). This provides objective criteria for comparing two EPSs, necessary when deciding whether or not to implement a new or revised EPS. The proposed verification methodology is based on the continuous ranked probability score (CRPS), which provides an evaluation of the global skill of an EPS. Its reliability/resolution partition, proposed by Hersbach, is used to measure the two main attributes of a probabilistic system. Also, the characteristics of the reliability are obtained from the two first moments of the reduced centered random variable (RCRV), which define the bias and the dispersion of an EPS. Resampling bootstrap techniques have been applied to these scores. Confidence intervals are thus defined, expressing the uncertainty due to the finiteness of the number of realizations used to compute the scores. All verifications are performed against observations to provide more independent validations and to avoid any local systematic bias of an analysis. A revised EPS, which has been tested at the CMC in a parallel run during the autumn of 2005, is described in this paper. This EPS has been compared with the previously operational one with the verification system presented above. To illustrate the verification methodology, results are shown for the temperature at 850 hPa. The confidence intervals are computed by taking into account the spatial correlation of the data and the temporal autocorrelation of the forecast error. The revised EPS performs significantly better for all the forecast ranges, except for the resolution component of the CRPS where the improvement is no longer significant from day 7. The significant improvement of the reliability is mainly due to a better dispersion of the ensemble. Finally, the verification system correctly indicates that variations are not significant when two theoretically similar EPSs are compared.

Corresponding author address: G. Candille, Direction de la Recherche en Météorologie, 2121 Route Trans-Canadienne, Dorval, Québec H9P 1J3, Canada. Email: guillem.candille@ec.gc.ca

Save
Cover Monthly Weather Review
Monthly Weather Review

  • Anderson, J. L., 1996: A method for producing and evaluating probabilistic forecasts from ensemble model integrations. J. Climate, 9 , 1518–1530.

    • Search Google Scholar
    • Export Citation

  • Bélair, S., J. Mailhot, C. Girard, and P. Vaillancourt, 2005: Boundary layer and shallow cumulus clouds in a medium-range forecast of a large-scale weather system. Mon. Wea. Rev., 133 , 1938–1960.

    • Search Google Scholar
    • Export Citation

  • Brier, G. W., 1950: Verification of forecasts expressed in terms of probability. Mon. Wea. Rev., 78 , 1–3.

  • Buizza, R., 2001: Accuracy and potential economic value of categorical and probabilistic forecasts of discrete events. Mon. Wea. Rev., 129 , 2329–2345.

    • Search Google Scholar
    • Export Citation

  • Candille, G., and O. Talagrand, 2005: Evaluation of probabilistic prediction systems for a scalar variable. Quart. J. Roy. Meteor. Soc., 131 , 2131–2150.

    • Search Google Scholar
    • Export Citation

  • Côté, J., S. Gravel, A. Méthot, A. Patoine, M. Roch, and A. Staniforth, 1998a: The operational CMC-MRB Global Environmental Multiscale (GEM) model. Part I: Design considerations and formulation. Mon. Wea. Rev., 126 , 1373–1395.

    • Search Google Scholar
    • Export Citation

  • Côté, J., J-G. Desmarais, S. Gravel, A. Méthot, A. Patoine, M. Roch, and A. Staniforth, 1998b: The operational CMC-MRB Global Environmental Multiscale (GEM) model. Part II: Results. Mon. Wea. Rev., 126 , 1397–1418.

    • Search Google Scholar
    • Export Citation

  • Dee, D. P., and A. M. da Silva, 1998: Data assimilation in the presence of forecast bias. Quart. J. Roy. Meteor. Soc., 124 , 269–295.

  • Efron, B., and R. Tibshirani, 1993: An Introduction to the Bootstrap. Chapman & Hall, 436 pp.

  • Elms, J., 2003: WMO catalogue of radiosondes and upper-air wind systems in use by members in 2002 and compatibility of radiosonde geopotential measurements for period from 1998 to 2001. Instruments and Observing Methods Rep. 80, Tech. Doc. 1197, 219 pp.

  • Hamill, T. M., 2001: Interpretation of rank histograms for verifying ensemble forecasts. Mon. Wea. Rev., 129 , 550–560.

  • Hamill, T. M., and J. Juras, 2007: Measuring forecast skill: Is it real skill or is it the varying climatology? Quart. J. Roy. Meteor. Soc., in press.

    • Search Google Scholar
    • Export Citation

  • Hamill, T. M., J. S. Whitaker, and X. Wei, 2004: Ensemble reforecasting: Improving medium-range forecast skill using retrospective forecasts. Mon. Wea. Rev., 132 , 1434–1447.

    • Search Google Scholar
    • Export Citation

  • Hersbach, H., 2000: Decomposition of the continuous ranked probability score for ensemble prediction systems. Wea. Forecasting, 15 , 559–570.

    • Search Google Scholar
    • Export Citation

  • Hines, C. O., 1997a: Doppler-spread parameterization of gravity-wave momentum deposition in the middle atmosphere. Part 1: Basic formulation. J. Atmos. Sol. Terr. Phys., 59 , 371–386.

    • Search Google Scholar
    • Export Citation

  • Hines, C. O., 1997b: Doppler-spread parameterization of gravity-wave momentum deposition in the middle atmosphere. Part 2: Broad and quasi monochromatic spectra, and implementation. J. Atmos. Sol. Terr. Phys., 59 , 387–400.

    • Search Google Scholar
    • Export Citation

  • Houtekamer, P. L., and H. L. Mitchell, 2005: Ensemble Kalman filtering. Quart. J. Roy. Meteor. Soc., 131 , 3269–3289.

  • Houtekamer, P. L., L. Lefaivre, J. Derome, H. Ritchie, and H. L. Mitchell, 1996: A system simulation approach to ensemble prediction. Mon. Wea. Rev., 124 , 1225–1242.

    • Search Google Scholar
    • Export Citation

  • Houtekamer, P. L., H. L. Mitchell, G. Pellerin, M. Buehner, M. Charron, L. Spacek, and B. Hansen, 2005: Atmospheric data assimilation with an ensemble Kalman filter: Results with real observations. Mon. Wea. Rev., 133 , 604–620.

    • Search Google Scholar
    • Export Citation

  • Kuo, H. L., 1974: Further studies of the parameterization of the influence of cumulus convection on large-scale flow. J. Atmos. Sci., 31 , 1232–1240.

    • Search Google Scholar
    • Export Citation

  • Lefaivre, L., P. L. Houtekamer, A. Bergeron, and R. Verret, 1997: The CMC ensemble prediction system. Proc. Sixth Workshop on Meteorological Operational Systems, Reading, United Kingdom, ECMWF, 31–44.

  • Mailhot, J., and Coauthors, 1998: Scientific description of RPN physics library 3.6 version. Tech. Doc., Recherche en Prévision Numérique, Atmospheric Environment Service, 197 pp.

  • Mason, I., 1982: A model for assessment of weather forecasts. Aust. Meteor. Mag., 30 , 291–303.

  • Moorthi, S., and M. J. Suarez, 1992: Relaxed Arakawa-Schubert. A parameterization of moist convection for general circulation models. Mon. Wea. Rev., 120 , 978–1002.

    • Search Google Scholar
    • Export Citation

  • Murphy, A. H., 1973: A new vector partition of the probability score. J. Appl. Meteor., 12 , 595–600.

  • Noilhan, J., and S. Planton, 1989: A simple parameterization of land surface processes for meteorological models. Mon. Wea. Rev., 117 , 536–549.

    • Search Google Scholar
    • Export Citation

  • Pellerin, G., L. Lefaivre, P. Houtekamer, and C. Girard, 2003: Increasing the horizontal resolution of ensemble forecasts at CMC. Nonlinear Proc. Geophys., 10 , 463–488.

    • Search Google Scholar
    • Export Citation

  • Richardson, D. S., 2001: Measure of skill and value of ensemble prediction systems, their interrelationship and the effect of ensemble size. Quart. J. Roy. Meteor. Soc., 127 , 2473–2489.

    • Search Google Scholar
    • Export Citation

  • Ritchie, H., and C. Beaudoin, 1994: Approximations and sensitivity experiments with a baroclinic semi-Lagrangian spectral model. Mon. Wea. Rev., 122 , 2391–2399.

    • Search Google Scholar
    • Export Citation

  • Saetra, Ø, H. Hersbach, J-R. Bidlot, and D. S. Richardson, 2004: Effects of observation errors on the statistics for ensemble spread and reliability. Mon. Wea. Rev., 132 , 1487–1501.

    • Search Google Scholar
    • Export Citation

  • Simmons, A. J., and A. Hollingsworth, 2002: Some aspects of the improvement in skill of numerical weather prediction. Quart. J. Roy. Meteor. Soc., 128 , 647–677.

    • Search Google Scholar
    • Export Citation

  • Smith, L. A., and J. A. Hansen, 2004: Extending the limits of ensemble forecast verification with the minimum spanning tree. Mon. Wea. Rev., 132 , 1522–1528.

    • Search Google Scholar
    • Export Citation

  • Stanski, H. R., L. J. Wilson, and W. R. Burrows, 1989: Survey of common verification in meteorology. World Weather Watch Rep. 8, Tech. Doc. 358, World Meteorological Organization, 114 pp.

  • Talagrand, O., R. Vautard, and B. Strauss, 1999: Evaluation of probabilistic prediction systems. Proc. Workshop on Predictability, Reading, United Kingdom, ECMWF, 1–25.

  • Toth, Z., O. Talagrand, G. Candille, and Y. Zhu, 2003: Probability and ensemble forecasts. Forecast Verification: A Practitioner’s Guide in Atmospheric Science, I. T. Jolliffe and D. B. Stephenson, Eds., John Wiley and Sons, 137–163.

  • Tribbia, J. J., and D. P. Baumhefner, 2004: Scale interactions and atmospheric predictability: An updated perspective. Mon. Wea. Rev., 132 , 703–713.

    • Search Google Scholar
    • Export Citation

  • Zadra, A., M. Roch, S. Laroche, and M. Charron, 2003: The subgrid scale orographic blocking parametrization of the GEM model. Atmos.–Ocean, 41 , 155–170.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 727 273 6
PDF Downloads 577 180 1