Editorial Type:
Article
Article Type:
Research Article

Accounting for Observational Uncertainty in Forecast Verification: An Information-Theoretical View on Forecasts, Observations, and Truth

Steven V. Weijs Department of Water Resources Management, Delft University of Technology, Delft, Netherlands

Search for other papers by Steven V. Weijs in
Current site
Google Scholar
PubMed Close
and
Nick van de Giesen Department of Water Resources Management, Delft University of Technology, Delft, Netherlands

Search for other papers by Nick van de Giesen in
Current site
Google Scholar
PubMed Close
Print Publication:
01 Jul 2011
DOI:
https://doi.org/10.1175/2011MWR3573.1
Page(s):
2156–2162
Restricted access

Abstract

Recently, an information-theoretical decomposition of Kullback–Leibler divergence into uncertainty, reliability, and resolution was introduced. In this article, this decomposition is generalized to the case where the observation is uncertain. Along with a modified decomposition of the divergence score, a second measure, the cross-entropy score, is presented, which measures the estimated information loss with respect to the truth instead of relative to the uncertain observations. The difference between the two scores is equal to the average observational uncertainty and vanishes when observations are assumed to be perfect. Not acknowledging for observation uncertainty can lead to both overestimation and underestimation of forecast skill, depending on the nature of the noise process.

Corresponding author address: Steven Weijs, Delft University of Technology, Stevinweg 1, P.O. Box 5048, 2600 GA Delft, Netherlands. E-mail: s.v.weijs@tudelft.nl

Abstract

Recently, an information-theoretical decomposition of Kullback–Leibler divergence into uncertainty, reliability, and resolution was introduced. In this article, this decomposition is generalized to the case where the observation is uncertain. Along with a modified decomposition of the divergence score, a second measure, the cross-entropy score, is presented, which measures the estimated information loss with respect to the truth instead of relative to the uncertain observations. The difference between the two scores is equal to the average observational uncertainty and vanishes when observations are assumed to be perfect. Not acknowledging for observation uncertainty can lead to both overestimation and underestimation of forecast skill, depending on the nature of the noise process.

Corresponding author address: Steven Weijs, Delft University of Technology, Stevinweg 1, P.O. Box 5048, 2600 GA Delft, Netherlands. E-mail: s.v.weijs@tudelft.nl
Save
Cover Monthly Weather Review
Monthly Weather Review

  • Benedetti, R., 2010: Scoring rules for forecast verification. Mon. Wea. Rev., 138, 203–211.

  • Brier, G. W., 1950: Verification of forecasts expressed in terms of probability. Mon. Wea. Rev., 78, 1–3.

  • Briggs, W., M. Pocernich, and D. Ruppert, 2005: Incorporating misclassification error in skill assessment. Mon. Wea. Rev., 133, 3382–3392.

    • Search Google Scholar
    • Export Citation

  • Bröcker, J., 2009: Reliability, sufficiency, and the decomposition of proper scores. Quart. J. Roy. Meteor. Soc., 135 (643), 1512–1519.

    • Search Google Scholar
    • Export Citation

  • Bröcker, J., and L. Smith, 2007: Scoring probabilistic forecasts: The importance of being proper. Wea. Forecasting, 22, 382–388.

  • Cover, T. M., and J. A. Thomas, 2006: Elements of Information Theory. 2nd ed. Wiley-Interscience, 776 pp.

  • Good, I. J., 1952: Rational decisions. J. Roy. Stat. Soc., 14B, 107–114.

  • Jaynes, E. T., 2003: Probability Theory: The Logic of Science. Cambridge University Press, 758 pp.

  • Kelly, J., 1956: A new interpretation of information rate. IEEE Trans. Info. Theory, 2 (3), 185–189.

  • Kullback, S., and R. A. Leibler, 1951: On information and sufficiency. Ann. Math. Stat., 22, 79–86.

  • Murphy, A. H., 1973: A new vector partition of the probability score. J. Appl. Meteor., 12, 595–600.

  • Peirolo, R., 2010: Information gain as a score for probabilistic forecasts. Meteor. Appl., 18, 9–17, doi:10.1002/met.188.

  • Roulston, M. S., and L. A. Smith, 2002: Evaluating probabilistic forecasts using information theory. Mon. Wea. Rev., 130, 1653–1660.

  • Shannon, C. E., 1948: A mathematical theory of communication. Bell Syst. Tech. J., 27 (3), 379–423.

  • Weijs, S., R. van Nooijen, and N. van de Giesen, 2010a: Kullback–Leibler divergence as a forecast skill score with classic reliability–resolution–uncertainty decomposition. Mon. Wea. Rev., 138, 3387–3399.

    • Search Google Scholar
    • Export Citation

  • Weijs, S., G. Schoups, and N. van de Giesen, 2010b: Why hydrological predictions should be evaluated using information theory. Hydrol. Earth Syst. Sci., 14 (12), 2545–2558.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 506 117 7
PDF Downloads 339 77 7