Editorial Type:
Article
Article Type:
Research Article

Incorporating Misclassification Error in Skill Assessment

William Briggs General Internal Medicine, Weill Cornell Medical College, New York, New York

Search for other papers by William Briggs in
Current site
Google Scholar
PubMed Close
,
Matt Pocernich Research Applications Laboratory, National Center for Atmospheric Research, Boulder, Colorado

Search for other papers by Matt Pocernich in
Current site
Google Scholar
PubMed Close
, and
David Ruppert School of Operations Research and Industrial Engineering, Cornell University, Ithaca, New York

Search for other papers by David Ruppert in
Current site
Google Scholar
PubMed Close
Print Publication:
01 Nov 2005
DOI:
https://doi.org/10.1175/MWR3032.1
Page(s):
3382–3392
Restricted access

Abstract

It is desirable to account for misclassification error of meteorological observations so that the true skill of the forecast can be assessed. Errors in observations can occur, among other places, in pilot reports of icing and in tornado spotting. Not accounting for misclassification error gives a misleading picture of the forecast’s true performance. An extension to the climate skill score test developed in Briggs and Ruppert is presented to account for possible misclassification error of the meteorological observation. This extension supposes a statistical misclassification-error model where “gold standard” data, or expert opinion, is available to characterize the misclassification-error characteristics of the observation. These model parameters are then inserted into the Briggs and Ruppert skill score for which a statistical test of significance can be performed.

Corresponding author address: William Briggs, GIM, Weill Cornell Medical College, 525 E. 68th, Box 46, New York, NY 10021. Email: wib2004@med.cornell.edu

Abstract

It is desirable to account for misclassification error of meteorological observations so that the true skill of the forecast can be assessed. Errors in observations can occur, among other places, in pilot reports of icing and in tornado spotting. Not accounting for misclassification error gives a misleading picture of the forecast’s true performance. An extension to the climate skill score test developed in Briggs and Ruppert is presented to account for possible misclassification error of the meteorological observation. This extension supposes a statistical misclassification-error model where “gold standard” data, or expert opinion, is available to characterize the misclassification-error characteristics of the observation. These model parameters are then inserted into the Briggs and Ruppert skill score for which a statistical test of significance can be performed.

Corresponding author address: William Briggs, GIM, Weill Cornell Medical College, 525 E. 68th, Box 46, New York, NY 10021. Email: wib2004@med.cornell.edu

Save
Cover Monthly Weather Review
Monthly Weather Review

  • Briggs, W., 2005: A general method of incorporating forecast cost and loss in value scores. Mon. Wea. Rev., 133 , 3393–3397.

  • Briggs, W. M., and D. Ruppert, 2004: Assessing the skill of yes/no forecasts for Markov observations. Preprints, 17th Conf. on Probability and Statistics in the Atmospheric Sciences, Seattle, WA, Amer. Meteor. Soc, CD-ROM, 2.2.

  • Briggs, W. M., and D. Ruppert, 2005: Assessing the skill of yes/no forecasts. Biometrics, 61 , 799–807.

  • Brown, B. G., G. Thompson, R. T. Bruintjes, R. Bullock, and T. Kane, 1997: Intercomparison of in-flight icing algorithms. Part II: Statistical verification results. Wea. Forecasting, 12 , 890–914.

    • Search Google Scholar
    • Export Citation

  • Finley, J. P., 1884: Tornado forecasts. Amer. Meteor. J., 1 , 85–88.

  • Geisler, F. H., J. J. Jelinek, J. N. Joslyn, and F. Gelland, 1988: Acute cervical spine trauma: Evaluation with 1.5-T MR imaging. Radiology, 166 , 807–816.

    • Search Google Scholar
    • Export Citation

  • Hamill, T. M., 1998: Hypothesis tests for evaluating numerical precipitation forecasts. Wea. Forecasting, 14 , 155–167.

  • Mozer, J. B., and W. M. Briggs, 2003: Skill in real-time solar wind shock forecasts. J. Geophys. Res., 108 .1262, doi:10.1029/2003JA009827.

    • Search Google Scholar
    • Export Citation

  • Murphy, A. H., 1996: The Finley affair: A signal event in the history of forecast verification. Wea. Forecasting, 11 , 3–20.

  • Murphy, A. H., and A. Ehrendorfer, 1987: One the relationship between the accuracy and value of forecasts in the cost–loss ratio situation. Wea. Forecasting, 2 , 243–251.

    • Search Google Scholar
    • Export Citation

  • Murphy, A. H., and R. L. Winkler, 1987: A general framework for forecast verification. Mon. Wea. Rev., 115 , 1330–1338.

  • Parker, R. A., and R. B. Davis, 1999: Evaluating whether a binary decision rule operates better than chance. Biom. J., 41 , 25–31.

  • Pocernich, M. J., C. Wolff, and T. Fowler, 2004: Statistical models of aircraft icing. Preprints, 17th Conf. on Probability and Statistics in the Atmospheric Sciences, Seattle, WA, Amer. Meteor. Soc., CD-ROM, 1.4.

  • Sand, W. R., and C. Biter, 1997: Pilot response to icing: It depends. Preprints, Seventh Conf. on Aviation, Range, and Aerospace Meteorology, Long Beach, CA, Amer. Meteor. Soc., 116–119.

  • Schervish, M. J., 1989: A general method for comparing probability assessors. Ann. Stat., 17 , 1856–1879.

  • Self, S. G., and K. Y. Liang, 1987: Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions. J. Amer. Stat. Assoc., 82 , 605–610.

    • Search Google Scholar
    • Export Citation

  • Speheger, D. A., C. A. Doswell III, and G. J. Stumpf, 2002: The tornadoes of 3 May 1999: Event verification in central Oklahoma and related issues. Wea. Forecasting, 17 , 362–381.

    • Search Google Scholar
    • Export Citation

  • Wilks, D. S., 1995: Statistical Methods in the Atmospheric Sciences. Academic Press, 467 pp.

  • Wolff, C. A., and B. C. Bernstein, 2004: Scales of aircraft icing: A comparison of icing PIREPs to liquid water measurements from research aircraft. Preprints, 11th Conf. on Aviation, Range, and Aerospace Meteorology, Hyannis, MA, Amer. Meteor. Soc., CD-ROM, 8.4.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 181 76 19
PDF Downloads 94 28 4