1. Introduction

The relative (or receiver) operating characteristic (ROC) is a representation of the skill of a forecasting system in which the hit rate and the false-alarm rate are compared (Swets 1973; Mason 1982). The related ROC score is often used to evaluate the quality of probability forecasts (Stanski et al. 1989; Buizza and Palmer 1998; Mason and Graham 1999).

The ROC has its origins in the signal detection theory [e.g., see the review by Swets (1973)]. Typically, the ROC is described in terms of the parameters of two hypothetical, often Gaussian, probability distributions (e.g., Mason 1982; Harvey et al. 1992). One distribution represents the evidence strength associated with the occurrence of the event, and the other represents the nonevent distribution. The two distributions are used to obtain the hit rate and the false-alarm rate as a function of the decision criterion. In the present study, the ROC is derived in the framework of a simple analog of a climate forecasting system (Kharin and Zwiers 2003, hereinafter KZ2003) and the link between the ROC score and deterministic potential predictability (Zwiers 1996; Rowell 1998) is established.

The ROC score is relatively independent of forecast calibration, that is, the correspondence between the forecast probability and observed relative frequency. It is qualitatively similar to resolution, that is, the ability of the forecast system to discriminate between event occurrences and nonoccurences [see, e.g., Mason (1982) and, more recently, Wilks (2001) and Mullen and Buizza (2001)]. The insensitivity to some types of forecast biases is demonstrated here in the context of a simple analog of a climate forecasting system. In particular, it is shown that under certain conditions the ROC score is a function of the properties of the observed system only.

The outline of this note is as follows. The ROC curve and score are derived and discussed in section 2. Some findings are illustrated with a collection of 24-member ensemble hindcasts of seasonal mean 700-hPa temperature in section 3, followed by a summary in section 4.

2. Relative operating characteristic

In this section, a simple analog of a climate forecasting system is introduced and expressions for the hit rate and false-alarm rate, the essential ingredients in the ROC curve definition, are derived. Examples of ROC curves are given for a “perfect” forecasting system in a Gaussian setting, and the relationship between the ROC score and potential predictability is illustrated.

a. Climate forecasting system analog

The simple climate forecasting system considered here has the form

View Expanded

where X is a scalar predictand and F is its forecast. The predictand X is assumed to be a sum of two independent components: a potentially predictable signal β and an unpredictable component that is represented by stochastic Gaussian noise ϵ. On seasonal and longer time scales, the primary predictability source is thought to be the atmospheric response to slowly varying lower boundary conditions such as sea surface temperature. The effect of the initial conditions is felt primarily in the first few forecast weeks. The noise term represents the unpredictable effects of day-to-day weather. The properties of the Gaussian noise are fully characterized by a single parameter σ²_ϵ, the noise variance. Similarly, the forecast F is assumed to be a sum of the model-simulated signal β′ and Gaussian noise ϵ′. In this simplified world, a perfect forecasting system has β′ = β and σ²_ϵ′ = σ²_ϵ.

b. The hit rate and false-alarm rate

We begin by considering a deterministic forecasting system in which a warning is issued when an event is predicted to occur. The operation of such a forecasting system can be summarized in a 2 × 2 contingency table (Table 1). Assume that there are a total of N forecasts and verifying observations. Let the total number of warnings issued be W and the number of nonwarnings be W′ = N − W. Similarly, let O be the number of events that occurred, and let O′ = N − O be the number of nonevents. Also let H be the number of hits, for which an event occurred and a warning was issued; let FA be the number of false alarms, for which a warning was issued but an event did not occur; let M be the number of misses, for which an event occurred but a warning was not issued; and let CR be the number of correct rejections, for which an event did not occur and a warning was not issued.

The hit rate (HR) and the false-alarm rate (FAR) are defined as (e.g., Mason 1982)

View Expanded

Let the random variable E denote a dichotomous predictand, defined as E = 1 when the event occurs and E = 0 otherwise. Asymptotically, for N → ∞, the ratios in (2) converge to conditional probabilities

View Expanded

The HRs and FARs are equal in a forecasting system with no skill. When the forecasting system has some skill, the hit rate exceeds the false-alarm rate. For perfect forecasts, HR = 1 and FAR = 0. Both HR and FAR are equal to 1 in a forecasting system that always issues warnings, and both are equal to 0 in a system that never issues a warning.

For probabilistic forecasts, a warning may be issued when the forecast probability P exceeds some critical threshold P_cr. Thus, the HR and FAR can be expressed as a function of P_cr as follows:

View Expanded

where Ω_P denotes forecast probabilities P > P_cr and f(P | E) is the conditional probability density function of probability forecasts. By using the calibration–refinement and likelihood–base rate factorizations of the join distribution of forecasts and predictands (Murphy and Winkler 1987), f(P | E) in (4) can be expressed as

View Expanded

where f(E | P) is the expectation of the predictand E conditional on the forecast probability P and f_P(P) and f_E(E) are the marginal probability density functions of probability forecasts and predictands, respectively. By substituting (5) into (4), the HR and FAR can be written as

View Expanded

If sampling variability can be neglected, for example, when the ensemble size of independent forecasts produced with the system (1) is sufficiently large, the forecast probability P may be assumed to be a function of the forecast signal β′ only. In this case, the condition P ≥ P_cr for issuing a warning can be expressed equivalently as a condition on β′. Given the probability density function f_β′ of β′, expressions in (6) may be rewritten as

View Expanded

where Ω_β′ denotes all values of β′ for which P > P_cr. The conditional expectation f(E | β′) depends explicitly on the forecast signal β′ and the properties of the observed system. Therefore, HR and FAR generally depend on the properties of both the observed and forecasting systems.

The expressions under the integrals in (7) become functions of the observed system only when there is a one-to-one correspondence between β′ and β. In this case, HR and FAR take the form

View Expanded

where Ω_β denotes all values of β for which P > P_cr. In this case, the HR and FAR depend on the forecast probability P only implicitly through the domain of integration Ω_β.

3. ROC scores of HFP hindcasts

In this section we describe the ROC scores of probability hindcasts of 700-hPa temperature (T₇₀₀) derived from a collection of 24-member ensemble hindcasts. The hindcasts were produced for 26 northern winters [December–January–February (DJF)] for the 1969–95 period with the second-generation general circulation model of the Canadian Centre for Climate Modelling and Analysis (McFarlane et al. 1992). These integrations were performed as the part of the Canadian Historical Forecast Project (HFP; Derome et al. 2001). Each HFP ensemble member is initialized from reanalyzed fields (Kalnay et al. 1996) lagged at 6-h intervals prior to the forecast season. The monthly mean sea surface temperature anomalies observed in the month prior to the forecast period are “persisted” throughout the forecast season. These anomalies are obtained from the Global Sea Ice and Sea Surface Temperature (GISST) dataset (version 2.2; Rayner et al. 1996). Sea ice extent is specified from climatological data. The initial snow line in the Northern Hemisphere is specified from National Centers for Environmental Prediction satellite observations for the week before the forecast period. The soil conditions are initialized from climatological data.

KZ2003 described a number of probability hindcasts of three equiprobable categories of below-normal, near-normal, and above-normal DJF T₇₀₀ in the Tropics (30°S–30°N) and the North American sector (20°–80°N, 150°–45°W). Two of those hindcasts are considered here, the unadjusted Gaussian hindcasts and statistically improved hindcasts. The unadjusted Gaussian probability hindcasts P̂_G are obtained from

View Expanded

where F_N is the distribution function of the standard normal distribution, x_a = σ̂_XF⁻¹_N(2/3) ≈ 0.43σ̂_X is the boundary between the N and A categories, and β̂^′_t and σ̂_;ge′ are estimated from N-member ensembles of seasonal anomaly hindcasts for T years {F_nt, n = 1, … , N; t = 1, … , T} as

View Expanded

KZ2003 demonstrated that the probability forecast obtained with (10)–(11) is superior to the standard nonparameteric probability forecast obtained as the relative number of the ensemble members in an event category for T₇₀₀ on seasonal time scales.

The improved hindcasts P̂_I are obtained by adjusting β̂^′_t and σ̂_ϵ′ in (10) by factors â and b̂, respectively, to minimize the sum of the Brier scores for each category. The adjusting factors are chosen to satisfy the relationship â²σ̂²_β′ + b̂²σ̂²_ϵ′ = σ̂²_X, that is, to match the observed variability estimate, and are obtained in a cross-validation procedure in which the data for the forecast year are excluded from the training dataset (see KZ2003 for details). KZ2003 showed that the unadjusted T₇₀₀ probability hindcasts, which are not very reliable (i.e., they are conditionally biased) are substantially improved by this statistical technique.

Figure 3 shows the ROC curves of the unadjusted Gaussian and improved probability hindcasts in the Tropics and in the North American sector. The ROC skill score for each category is indicated in the upper-left corner of each diagram. The ROC curves for the probability hindcasts in the Tropics deviate farther away from the no-skill diagonal toward the upper-left corner than those for the North American sector, indicating better skill in the Tropics. The hindcasts of the A category are somewhat more skillful than those of the B category, and the hindcasts of the N category are substantially less skillful than those of the other two categories. These results are consistent with the findings in KZ2003.

The statistically improved hindcasts have slightly smaller ROC skill scores than unadjusted hindcasts, a result that is not unexpected given the discussion of the ROC score properties in section 2 in which it was argued that the ROC score cannot be improved by adjusting the amplitude of the predictable signal. The degradation of the ROC skill scores of the “improved” hindcasts is apparently the result of sampling errors introduced by the statistical technique.

4. Summary

The properties of the ROC curve and ROC score have been discussed in the context of a simple analog of a seasonal forecasting system in which the total variability may be decomposed into two components, one that is associated with the potentially predictable signal and another that is unpredictable. The relationship between the ROC skill score and the potential predictability, defined as the ratio of the variance of the potentially predictable signal variance to the total variance, is established in the Gaussian setting. It is shown that there is a one-to-one correspondence between the deterministic potential predictability and the ROC scores of optimal probability forecasts. This correspondence depends on the definition of the forecast event of interest. In particular, the same level of potential predictability results in a smaller ROC score for the N category than that for the A or B category.

The insensitivity of the ROC score to certain types of forecast biases is demonstrated in the framework of the simple forecasting system. In an idealized setting in which the model-simulated signal is a function of the observed potentially predictable signal only, the ROC score is shown to be a function of the properties of the observed system only. If sampling variability can be neglected, probability forecasts derived from a forecast system that systematically under- or oversimulates the amplitude of the potentially predictable signal have the same ROC score as that of the perfectly reliable probability forecasts. That is, the ROC score does not penalize forecasting biases. Thus, care is needed when interpreting and intercomparing the performance of model forecasts based on the ROC score.

The findings are illustrated with a collection of 24-member ensemble seasonal hindcasts of 700-hPa temperature. It was shown that although statistically improved temperature hindcasts are more reliable (less biased) than the unadjusted hindcasts, the ROC scores of the statistically improved hindcasts are slightly degraded as compared with those of unadjusted hindcasts.