A Probabilistic Derivation of Heidke Skill Score

Otto Hyvärinen Finnish Meteorological Institute, Helsinki, Finland

Search for other papers by Otto Hyvärinen in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

An alternative derivation of Heidke skill score for 2 × 2 tables is presented, starting from the assumption that a categorical forecast is useful, if the probability of an occurrence of an event, given the forecast, is greater than the base rate of the event. A tentative measure of skill would then be the difference of these probabilities, normalized by the maximum value based on the base rate. For binary events, the Heidke skill score is then the harmonic mean of these differences for both the occurrence and the nonoccurrence of the event. This derivation differs from the usual derivation in that the concept of chance agreement is not used. It is Bayesian in nature with implied updating of prior probabilities to posterior probabilities.

Corresponding author address: Otto Hyvärinen, Finnish Meteorological Institute, Erik Palménin aukio 1, P.O. Box 503, FI-00101 Helsinki, Finland. E-mail: otto.hyvarinen@fmi.fi

Abstract

An alternative derivation of Heidke skill score for 2 × 2 tables is presented, starting from the assumption that a categorical forecast is useful, if the probability of an occurrence of an event, given the forecast, is greater than the base rate of the event. A tentative measure of skill would then be the difference of these probabilities, normalized by the maximum value based on the base rate. For binary events, the Heidke skill score is then the harmonic mean of these differences for both the occurrence and the nonoccurrence of the event. This derivation differs from the usual derivation in that the concept of chance agreement is not used. It is Bayesian in nature with implied updating of prior probabilities to posterior probabilities.

Corresponding author address: Otto Hyvärinen, Finnish Meteorological Institute, Erik Palménin aukio 1, P.O. Box 503, FI-00101 Helsinki, Finland. E-mail: otto.hyvarinen@fmi.fi
Save
  • Bishop, Y., Fienberg S. , and Holland P. , 1975: Discrete Multivariate Analysis: Theory and Practice. The MIT Press, 557 pp.

  • Cohen, J., 1960: A coefficient of agreement for nominal scales. Educ. Psychol. Meas., 20, 213220.

  • Ferger, W. F., 1931: The nature and use of the harmonic mean. J. Amer. Stat. Assoc., 26, 3640, doi:10.1080/01621459.1931.10503148.

  • Hartmann, S., and Sprenger J. , 2010: Bayesian epistemology. Routledge Companion to Epistemology, S. Bernecker and D. Pritchard, Eds., Routledge, 609–620.

  • Heidke, P., 1926: Berechnung des Erfolges und der Gute der Windstarkevorhersagen im Sturmwarnungsdienst (Measures of success and goodness of wind force forecasts by the gale-warning service)Geogr. Ann., 8, 301349.

    • Search Google Scholar
    • Export Citation
  • Jolliffe, I. T., 2007: Uncertainty and inference for verification measures. Wea. Forecasting, 22, 637650.

  • Jolliffe, I. T., and Stephenson D. B. , Eds., 2012: Forecast Verification: A Practitioners Guide in Atmospheric Science. 2nd ed. Wiley, 292 pp.

  • Kenney, J. F., and Keeping E. S. , 1962: Mathematics of Statistics. 3rd ed. Van Nostrand, 346 pp.

  • Komić, J., 2011: Harmonic mean. International Encyclopedia of Statistical Science, M. Lovric, Ed., Springer, 622–624, doi:10.1007/978-3-642-04898-2_645.

  • Manning, C. D., Raghavan P. , and Schütze H. , 2008: Introduction to Information Retrieval. Cambridge University Press, 482 pp.

  • Murphy, A. H., 1996: The Finley affair: A signal event in the history of forecast verification. Wea. Forecasting, 11, 320.

  • Peirce, C. S., 1884: The numerical measure of the success of predictions. Science, 4, 453454, doi:10.1126/science.ns-4.93.453-a.

  • Pepe, M. S., 2003: The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford University Press, 302 pp.

  • Stehman, S. V., 1997: Selecting and interpreting measures of thematic classification accuracy. Remote Sens. Environ., 62, 7789, doi:10.1016/S0034-4257(97)00083-7.

    • Search Google Scholar
    • Export Citation
  • Van Rijsbergen, C. J., 1974: Foundation of evaluation. J. Doc., 30, 365373.

  • Von Eye, A., and Mun E. Y. , 2005: Analyzing Rater Agreement: Manifest Variable Methods. Lawrence Erlbaum Associates, 190 pp.

  • Zhang, S., and Wu X. , 2011: Fundamentals of association rules in data mining and knowledge discovery. Wiley Interdiscip. Rev.: Data Min. Knowl. Discovery, 1, 97116, doi:10.1002/widm.10.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 3526 1148 131
PDF Downloads 2556 603 53