• American Meteorological Society, 2001: AMS statement: Seasonal to interannual climate prediction (adopted by AMS Council 14 January 2001). Bull. Amer. Meteor. Soc., 82 , 701704.

    • Search Google Scholar
    • Export Citation
  • Buizza, R., and T. N. Palmer, 1998: Impact of ensemble size on ensemble prediction. Mon. Wea. Rev., 126 , 25032518.

  • Buizza, R., T. Petroliagis, T. N. Palmer, J. Barkmeijer, M. Hamrud, A. Hollingsworth, A. Simmons, and N. Wedi, 1998: Impact of model resolution and ensemble size on the performance of an ensemble prediction system. Quart. J. Roy. Meteor. Soc., 124 , 19351960.

    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., and J. Juras, 2006: Measuring forecast skill: Is it real skill or is it the varying climatology? Quart. J. Roy. Meteor. Soc., in press.

    • Search Google Scholar
    • Export Citation
  • Hanley, J. A., and B. J. McNeil, 1982: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143 , 2936.

    • Search Google Scholar
    • Export Citation
  • Hanley, J. A., and B. J. McNeil, 1983: A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology, 148 , 839843.

    • Search Google Scholar
    • Export Citation
  • Harvey Jr., L. O., K. R. Hammond, C. M. Lusk, and E. F. Mross, 1992: The application of signal detection theory to weather forecasting behavior. Mon. Wea. Rev., 120 , 863883.

    • Search Google Scholar
    • Export Citation
  • Jalihal, D., and L. W. Nolte, 1994: Signal detection theory and reconstruction algorithms—Performance for images in noise. IEEE Trans. Biomed. Eng., 41 , 501504.

    • Search Google Scholar
    • Export Citation
  • Juras, J., 2000: Comments on “Probabilistic predictions of precipitation using the ECMWF ensemble prediction system.”. Wea. Forecasting, 15 , 365366.

    • Search Google Scholar
    • Export Citation
  • Kharin, V. V., and F. W. Zwiers, 2003: On the ROC score of probability forecasts. J. Climate, 16 , 41454150.

  • Kheshgi, H. S., and B. S. White, 2001: Testing distributed parameter hypotheses for the detection of climate change. J. Climate, 14 , 34643481.

    • Search Google Scholar
    • Export Citation
  • Laman, D. M., G. H. Wieneke, H. van Duijn, R. J. Veldhuizen, and A. C. van Huffelen, 2005: QEEG changes during carotid clamping in carotid endarterectomy: Spectral edge frequency parameters and relative band power parameters. J. Clin. Neurophysiol., 22 , 244252.

    • Search Google Scholar
    • Export Citation
  • Marzban, C., 2004: The ROC curve and its area under it as performance measures. Wea. Forecasting, 19 , 11061114.

  • Mason, I., 1982: A model for assessment of weather forecasts. Aust. Meteor. Mag., 30 , 291303.

  • Mason, S. J., and N. E. Graham, 2002: Areas beneath the relative operating characteristics (ROC) and levels (ROL) curves: Statistical significance and interpretation. Quart. J. Roy. Meteor. Soc., 128 , 21452166.

    • Search Google Scholar
    • Export Citation
  • Mason, S. J., L. Goddard, N. E. Graham, E. Yulaeva, L. Sun, and P. A. Arkin, 1999: The IRI seasonal climate prediction system and the 1997/98 El Niño event. Bull. Amer. Meteor. Soc., 80 , 18531873.

    • Search Google Scholar
    • Export Citation
  • McNeil, B. J., and S. J. Adelstein, 1976: Determining the value of diagnostic and screening tests. J. Nucl. Med., 17 , 439448.

  • Molesworth, A. M., L. E. Cuevas, S. J. Connor, A. More, and M. C. Thomson, 2003: Environmental risk and meningitis in Africa. Emerging Infect. Dis., 9 , 12871293.

    • Search Google Scholar
    • Export Citation
  • Mylne, K. R., 1999: The use of forecast value calculations for optimal decision making using probability forecasts. Preprints, 17th Conf. on Weather Analysis and Forecasting, Denver, CO, Amer. Meteor. Soc., 235–239.

  • Palmer, T. N., C. Brankovic, and D. S. Richardson, 2000: A PROVOST probability and decision-model analysis of PROVOST seasonal multi-model ensemble integrations. Quart. J. Roy. Meteor. Soc., 126 , 20132034.

    • Search Google Scholar
    • Export Citation
  • Richardson, D. S., 2000a: Skill and economic value of the ECMWF ensemble prediction system. Quart. J. Roy. Meteor. Soc., 126 , 649668.

    • Search Google Scholar
    • Export Citation
  • Richardson, D. S., 2000b: Applications of cost-loss models. Proc. Seventh ECMWF Workshop on Meteorological Operational Systems, Reading, United Kingdom, ECMWF, 209–213.

  • Richardson, D. S., 2001: Measures of skill and value of ensemble prediction systems, their interrelationship and the effect of ensemble size. Quart. J. Roy. Meteor. Soc., 127 , 24732489.

    • Search Google Scholar
    • Export Citation
  • Roebber, P. J., and L. F. Bosart, 1996: The complex relationship between forecast skill and forecast value: A real world analysis. Wea. Forecasting, 11 , 544559.

    • Search Google Scholar
    • Export Citation
  • Stanski, H. R., L. J. Wilson, and W. R. Burrows, 1989: Survey of common verification methods in meteorology. WMO World Weather Watch Tech. Rep. 8, WMO/TD 358, 114 pp.

  • Swets, J. A., 1973: The relative operating characteristic in psychology. Science, 182 , 990999.

  • Swets, J. A., and R. M. Pickett, 1982: Evaluation of Diagnostic Systems: Methods from Signal Detection Theory. Academic Press, 253 pp.

  • Walter, S. D., and L. M. Irwig, 1988: Estimation of test error rates, disease prevalence, and relative risk from misclassified data. J. Clin. Epidemiol., 41 , 923938.

    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2006: Statistical Methods in the Atmospheric Sciences. Cambridge University Press, 630 pp.

  • Wilson, L. J., 2000: Comments on “Probabilistic predictions of precipitation using the ECMWF ensemble prediction system.”. Wea. Forecasting, 15 , 361364.

    • Search Google Scholar
    • Export Citation
  • World Meteorological Organization, 1992: Section III, Attachment II.7 and II.8 (revised in 2002). Manual on the Global Data-Processing System, WMO No. 485, 217 pp. [Available online at http://www.wmo.int/web/www/DPS/Manual/WMO485.pdf.].

  • Zhu, Y., Z. Toth, R. Wobus, D. Richardson, and K. Mylne, 2002: The economic value of ensemble weather forecasts. Bull. Amer. Meteor. Soc., 83 , 7383.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 48 31 7
PDF Downloads 35 31 17

An Extended Procedure for Implementing the Relative Operating Characteristic Graphical Method

View More View Less
  • 1 Department of Marine, Earth and Atmospheric Sciences, and Department of Mathematics, North Carolina State University, Raleigh, North Carolina
  • | 2 Department of Marine, Earth and Atmospheric Sciences, North Carolina State University, Raleigh, North Carolina
Restricted access

Abstract

The functional relationship between the relative operating characteristic (ROC) and the economic value (EV) graphical methods have been exploited to develop a hybrid procedure called the extended ROC (EROC) method. The EROC retains the appealing simplicity of the traditional ROC method and the ability of the EV method to provide evaluation of the performance of an ensemble climate prediction system (EPS) for a hypothetical end user defined by the cost–loss ratio (μ = C/L). An inequality defining the lower and upper theoretical bounds of μ has been derived. Outside these limits, the EPS yields no added benefits for end user μ relative to the use of climatological persistence as an alternative prediction system. In the traditional ROC graphical method, the ROC skill (ROCS) is often expressed in terms of the area between the ROC graph and the diagonal baseline passing through the origin with slope m = 1. Thus, ROCS = 2A − 1, where A is the area under the ROC graph. In the proposed EROC approach, a more general procedure is recommended based on the construction of user-specific baselines that do not necessarily pass through the origin and, in general, have slope m ≠ 1. The skill of a particular EPS computed from the EROC method is proportional to the corresponding estimated value based on the EV graphical method. Therefore, the EROC geometry conveys the same basic information as the EV method. The Semazzi–Mera skill score (SMSS) is proposed as a convenient and compact way of expressing the combined verification based on the ROC and EV methods. The ROCS estimate is a special case of the SMSS. The near-horizontal trail-like geometry sometimes exhibited by EV graphs is also examined. It is shown to occur when either the hit-rate or false-alarm term dominates in the formula for EV, unlike the more typical situation in which both terms are comparable in magnitude.

Corresponding author address: Fredrick Semazzi, Department of Mathematics, North Carolina State University, Raleigh, NC 28694. Email: fred_semazzi@ncsu.edu

Abstract

The functional relationship between the relative operating characteristic (ROC) and the economic value (EV) graphical methods have been exploited to develop a hybrid procedure called the extended ROC (EROC) method. The EROC retains the appealing simplicity of the traditional ROC method and the ability of the EV method to provide evaluation of the performance of an ensemble climate prediction system (EPS) for a hypothetical end user defined by the cost–loss ratio (μ = C/L). An inequality defining the lower and upper theoretical bounds of μ has been derived. Outside these limits, the EPS yields no added benefits for end user μ relative to the use of climatological persistence as an alternative prediction system. In the traditional ROC graphical method, the ROC skill (ROCS) is often expressed in terms of the area between the ROC graph and the diagonal baseline passing through the origin with slope m = 1. Thus, ROCS = 2A − 1, where A is the area under the ROC graph. In the proposed EROC approach, a more general procedure is recommended based on the construction of user-specific baselines that do not necessarily pass through the origin and, in general, have slope m ≠ 1. The skill of a particular EPS computed from the EROC method is proportional to the corresponding estimated value based on the EV graphical method. Therefore, the EROC geometry conveys the same basic information as the EV method. The Semazzi–Mera skill score (SMSS) is proposed as a convenient and compact way of expressing the combined verification based on the ROC and EV methods. The ROCS estimate is a special case of the SMSS. The near-horizontal trail-like geometry sometimes exhibited by EV graphs is also examined. It is shown to occur when either the hit-rate or false-alarm term dominates in the formula for EV, unlike the more typical situation in which both terms are comparable in magnitude.

Corresponding author address: Fredrick Semazzi, Department of Mathematics, North Carolina State University, Raleigh, NC 28694. Email: fred_semazzi@ncsu.edu

Save