Editorial Type:
Article
Article Type:
Research Article

Decomposition of a New Proper Score for Verification of Ensemble Forecasts

H. M. Christensen Atmospheric, Oceanic and Planetary Physics, University of Oxford, Oxford, United Kingdom

Search for other papers by H. M. Christensen in
Current site
Google Scholar
PubMed Close
Print Publication:
01 May 2015
DOI:
https://doi.org/10.1175/MWR-D-14-00150.1
Page(s):
1517–1532
Restricted access

Abstract

A new proper score, the error-spread score (ES), has recently been proposed for evaluation of ensemble forecasts of continuous variables. The ES is formulated with respect to the moments of the ensemble forecast. It is particularly sensitive to evaluating how well an ensemble forecast represents uncertainty: is the probabilistic forecast well calibrated? In this paper, it is shown that the ES can be decomposed into its reliability, resolution, and uncertainty components in a similar way to the Brier score. The first term evaluates the reliability of the forecast standard deviation and skewness, rewarding systems where the forecast moments reliably indicate the properties of the verification. The second term evaluates the resolution of the forecast standard deviation and skewness, and rewards systems where the forecast moments vary from the climatological moments according to the predictability of the atmospheric flow. The uncertainty term depends only on the observed error distribution and is independent of the forecast standard deviation or skewness. The decomposition was demonstrated using forecasts made with the European Centre for Medium-Range Weather Forecasts ensemble prediction system, and was able to identify the source of the skill in the forecasts at different latitudes.

Corresponding author address: H. M. Christensen, Atmospheric, Oceanic and Planetary Physics, Clarendon Laboratory, Parks Road, Oxford, OX1 3PU, United Kingdom. E-mail: h.m.christensen@atm.ox.ac.uk

Abstract

A new proper score, the error-spread score (ES), has recently been proposed for evaluation of ensemble forecasts of continuous variables. The ES is formulated with respect to the moments of the ensemble forecast. It is particularly sensitive to evaluating how well an ensemble forecast represents uncertainty: is the probabilistic forecast well calibrated? In this paper, it is shown that the ES can be decomposed into its reliability, resolution, and uncertainty components in a similar way to the Brier score. The first term evaluates the reliability of the forecast standard deviation and skewness, rewarding systems where the forecast moments reliably indicate the properties of the verification. The second term evaluates the resolution of the forecast standard deviation and skewness, and rewards systems where the forecast moments vary from the climatological moments according to the predictability of the atmospheric flow. The uncertainty term depends only on the observed error distribution and is independent of the forecast standard deviation or skewness. The decomposition was demonstrated using forecasts made with the European Centre for Medium-Range Weather Forecasts ensemble prediction system, and was able to identify the source of the skill in the forecasts at different latitudes.

Corresponding author address: H. M. Christensen, Atmospheric, Oceanic and Planetary Physics, Clarendon Laboratory, Parks Road, Oxford, OX1 3PU, United Kingdom. E-mail: h.m.christensen@atm.ox.ac.uk
Save
Cover Monthly Weather Review
Monthly Weather Review

  • Berner, J., G. J. Shutts, M. Leutbecher, and T. N. Palmer, 2009: A spectral stochastic kinetic energy backscatter scheme and its impact on flow-dependent predictability in the ECMWF ensemble prediction system. J. Atmos. Sci., 66, 603–626, doi:10.1175/2008JAS2677.1.

    • Search Google Scholar
    • Export Citation

  • Brier, G. W., 1950: Verification of forecasts expressed in terms of probability. Mon. Wea. Rev., 78, 1–3, doi:10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Bröcker, J., 2009: Reliability, sufficiency, and the decomposition of proper scores. Quart. J. Roy. Meteor. Soc., 135, 1512–1519, doi:10.1002/qj.456.

    • Search Google Scholar
    • Export Citation

  • Christensen, H. M., I. M. Moroz, and T. N. Palmer, 2015: Evaluation of ensemble forecast uncertainty using a new proper score: Application to medium-range and seasonal forecasts. Quart. J. Roy. Meteor. Soc., 141, 538–549, doi:10.1002/qj.2375.

    • Search Google Scholar
    • Export Citation

  • Ferro, C. A. T., and T. E. Fricker, 2012: A bias-corrected decomposition of the Brier score. Quart. J. Roy. Meteor. Soc., 138, 1954–1960, doi:10.1002/qj.1924.

    • Search Google Scholar
    • Export Citation

  • Gneiting, T., 2011: Making and evaluating point forecasts. J. Amer. Stat. Assoc., 106, 746–762, doi:10.1198/jasa.2011.r10138.

  • Gneiting, T., and A. E. Raftery, 2007: Strictly proper scoring rules, prediction, and estimation. J. Amer. Stat. Assoc., 102, 359–378, doi:10.1198/016214506000001437.

    • Search Google Scholar
    • Export Citation

  • Hersbach, H., 2000: Decomposition of the continuous ranked probability score for ensemble prediction systems. Wea. Forecasting, 15, 559–570, doi:10.1175/1520-0434(2000)015<0559:DOTCRP>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Isaksen, L., M. Bonavita, R. Buizza, M. Fisher, J. Haseler, M. Leutbecher, and L. Raynaud, 2010: Ensemble of data assimilations at ECMWF. Tech. Rep. 636, European Centre for Medium-Range Weather Forecasts, Reading, United Kingdom, 48 pp.

  • Leutbecher, M., 2010: Diagnosis of ensemble forecasting systems. Proc. Seminar on Diagnosis of Forecasting and Data Assimilation Systems, Reading, United Kingdom, ECMWF, 235–266.

  • Leutbecher, M., and T. N. Palmer, 2008: Ensemble forecasting. J. Comput. Phys., 227, 3515–3539, doi:10.1016/j.jcp.2007.02.014.

  • Murphy, A. H., 1973: A new vector partition of the probability score. J. Appl. Meteor., 12, 595–600, doi:10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Murphy, A. H., 1986: A new decomposition of the Brier score: Formulation and interpretation. Mon. Wea. Rev., 114, 2671–2673, doi:10.1175/1520-0493(1986)114<2671:ANDOTB>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Palmer, T. N., R. Buizza, F. Doblas-Reyes, T. Jung, M. Leutbecher, G. J. Shutts, M. Steinheimer, and A. Weisheimer, 2009: Stochastic parametrization and model uncertainty. Tech. Rep. 598, European Centre for Medium-Range Weather Forecasts, Reading, United Kingdom, 44 pp.

  • Sanders, F., 1963: On subjective probability forecasting. J. Appl. Meteor., 2, 191–201, doi:10.1175/1520-0450(1963)002<0191:OSPF>2.0.CO;2.

    • Search Google Scholar
    • Export Citation

  • Tödter, J., and B. Ahrens, 2012: Generalization of the ignorance score: Continuous ranked version and its decomposition. Mon. Wea. Rev., 140, 2005–2017, doi:10.1175/MWR-D-11-00266.1.

    • Search Google Scholar
    • Export Citation

  • Weijs, S. V., and N. van de Giesen, 2011: Accounting for observational uncertainty in forecast verification: An information-theoretical view on forecasts, observations, and truth. Mon. Wea. Rev., 139, 2156–2162, doi:10.1175/2011MWR3573.1.

    • Search Google Scholar
    • Export Citation

  • Weijs, S. V., R. van Nooijen, and N. van de Giesen, 2010: Kullback–Leibler divergence as a forecast skill score with classic reliability-resolution-uncertainty decomposition. Mon. Wea. Rev., 138, 3387–3399, doi:10.1175/2010MWR3229.1.

    • Search Google Scholar
    • Export Citation

  • Young, R. M. B., 2010: Decomposition of the Brier score for weighted forecast–verification pairs. Quart. J. Roy. Meteor. Soc., 136, 1364–1370, doi:10.1002/qj.641.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 871 509 47
PDF Downloads 269 51 8