1. Introduction

Signal detection theory (Green and Swets 1966) is a technique used to verify the goodness of a classifier and is widespread in many disciplines, such as signal processing, pattern recognition, medicine diagnosis, psychological testing, and also weather forecasting. In forecast verification a contingency table (e.g., Wilks 2006) is built for assessing the quality of a binary classifier, which is a tool to forecast the occurrence of an event. Let us introduce these concepts in more detail.

Given N samples, x₁, . . . , x_N, of a continuous predictor X (which can be a simple observed variable or the output of a complex model, which uses as inputs many observed variables), one can draw the histogram, normalized by N, which estimates the probability density function, that will be called p(x). For a particular event, the joint distribution of the predictor X and of the event observations—a binary variable—can be used to split p(x) in two conditional components: the function f_Y(x) when there is the event occurrence (e.g., for the N_YES cases of rain occurrence) and its complement f_N(x) (i.e., the N_NO = N − N_YES cases without rain). These two functions are not used in this form, because they are not probabilities. In fact, if we integrate them on the X domain we do not obtain 1, but obtain the sample estimate of the event (P_YES) and nonevent (P_NO) prior probabilities:

View Expanded

Instead, what is usually done is to consider the two conditional density probabilities, obtained by normalizing f_Y and f_N by the prior probabilities. In particular, the first conditional density probability, also called the event likelihood, is that of having a value x given that there is the event occurrence: p(x | YES) = f_Y(x)/P_YES. The latter conditional density probability, namely the nonevent likelihood, is that of having a value x given that there is nonoccurrence: p(x | NO) = f_N(x)/P_NO.

If a threshold, let us say x, is used to dichotomize the domain of the values of X, what is obtained is a binary event classifier based on the X predictor. Without any loss of generality, let us suppose that this classifier forecasts event occurrence if x > x and nonoccurrence in the opposite situation. For example, there will be a false alarm in all the N_NO · P(x > x | NO) = b cases. On the other hand, the classifier will miss N_YES · P(x ≤ x | YES) = c events.

In this way, it is finally possible to build the contingency table and derive many statistical measures of the classifier skill, as is done for the example in Table 1. In this case the event studied is the occurrence of at least 20 mm of rain during a 6-h period in the Friuli Venezia Giulia region (notheast Italy, hereafter FVG) using as a simple predictor the mid- to-low-level relative humidity (MRH) derived from the lowest 500 hPa of the Udine radiosounding, released at the beginning of the 6-h period. These data will be used in section 3.

2. The maximum Peirce skill score

The Peirce skill score has been rediscovered many times since its first formulation given by Peirce (1884). For that reason it has been renamed over the years, examples include the Kuipers skill score or the true skill score. Following the suggestion of Mason (2003), it will be abbreviated as PSS, even if in previous works of the same author it was called KSS. As defined in Table 1, its value is computed by

View Expanded

where POD is the probability of detection, POFD is the probability of false detection, and the dependence on the threshold x used to build the contingency table from the continuous X variable has been emphasized.

The relative operating characteristic (ROC) curve (Swets 1973) is the line connecting all the points [POD(x), POFD(x)] obtained for x varying on the entire X domain. In Manzato (2005b, hereafter ORP), it is stated that PSS(x) is the vertical line connecting a point on the ROC curve to the bisector line POD = POFD (called the no-skill line).¹ In addition, for the properties of a bisector line, it is obvious that PSS is also the horizontal line connecting the point on the ROC curve to the bisector line. Then, using only PSS, it is possible to compute the value of the distance (H, perpendicular to the bisector) of a ROC point from the bisector, which is simply given by

View Expanded

In ORP it has also been shown that the threshold that maximizes PSS, let us say x_P, corresponds to a ROC point with a 45° tangent, because (dPOD/dPOFD)|_x=xP = 1. Here, it is added that maximizing the PSS(x) maximizes the distance H between the ROC point and the diagonal bisector, which is considered the zero- skill level of a binary classifier. So, the ROC point [POD(x_P), POFD(x_P)] is the point farthest from the no-skill line and, in that sense, it can be stated that it is the point with the maximum skill of a classifier.²

Richardson (2000) has already shown how the classifier forecast relative value—relative to a climatological forecast—is equal to its PSS and then is maximized at x_P. Moreover, Woodcock (1976) has also shown that PSS does not vary for unequal trials (as explained in the next section) and is the slope of the least squares linear regression line between the—binary—observed and forecast occurrences.

Finally, in ORP it was shown that PSS is also equitable, in the sense introduced by Gandin and Murphy (1992), for an asymmetrical scoring matrix. So, it was suggested that all of these properties make the maximum Peirce skill score, PSS(x_P), a good scalar measure of the whole classifier quality. In this section it is shown how other properties relate the threshold that maximizes PSS to the prior probabilities.

a. The x_P and the likelihood ratio

First, it is noted how the POD and POFD can be written in an integral form, similar to 1:

View Expanded

and

View Expanded

Then, it is possible to rewrite PSS(x) as follows:

View Expanded

PSS(x) depends on the shape of the two conditional distributions only above the thresholds x.

The likelihood ratio, Λ(x), is usually defined as follows (e.g., Jolliffe and Stephenson 2003):

View Expanded

It is well known that Λ(x) is the ROC slope at the point corresponding to the threshold x (e.g., Green and Swets 1966, Choi 1998):

View Expanded

So, what was stated in ORP about the slope at x_P means that the threshold that maximizes PSS has a likelihood ratio equal to 1. In fact, if one is interested in maximizing PSS, then one has to derive PSS(x) in (6) and set the derivative to zero:

View Expanded

If the two conditional probabilities overlap in part of their domain and are unimodal, then there is only one threshold that maximizes PSS: the point where they intersect. At this threshold the ratio of the components f_N(x_P) and f_Y(x_P) is equal to the prior probability ratio, α = P_NO/P_YES, so x_P corresponds to the intersection of the conditional components only when the event has a prior probability of P_YES = 0.5, that is α = 1.³

Finally, note that even if the threshold that maximizes PSS varies as the event climatology changes, as stated before, the maximum PSS value itself theoretically does not. Woodcock (1976) has shown that PSS does not change if one uses a database with N_YES ≠ N_NO or a random subsection of it that is “equalized” with respect to the event and nonevent frequency, so that N^sub_YES = N^sub_NO. In that case, the underlying likelihood distributions, that is, the shapes of p(x | YES) and p(x | NO), are the same for the original and the equalized datasets.

Also in ORP it was shown, for Gaussian likelihood, how the maximum PSS does not change with varying α if the likelihood mean and standard deviation are the same. In any case, it seems more likely to obtain higher maximum PSS for rare events than for near-equiprobable event problems,⁴ because the likelihood shapes are more likely to change. Thus, the event climatology must be shown when reporting the classifier skill, because the maximum PSS can vary for events with different frequencies.

b. The x_P and the event posterior probability

It is also possible to use the property in (9) to determine the value of the event posterior probability associated with the predictor X in x_P. First, Bayes’s theorem (Bayes 1763) states that

View Expanded

If we put (9) in this equation, we obtain the event posterior probability value associated with the threshold x_P, given by

View Expanded

So, even the event posterior probability in x_P is known a priori, because it is equal to the event prior probability. In the rare-event case (α ≫ 1), the event posterior probability will tend to zero (α⁻¹) for the threshold that maximizes PSS.

The result in (11) is consistent with that originally found by Mason (1979) for probabilistic forecasts. In that case, he found that the threshold on the event probability that maximizes PSS for the N + 1 forecast is given by

View Expanded

which approximates P_YES for large N.⁵

Equation (11) can be particularly useful when X is the output of a complex model that correctly estimates the event posterior probability. For example, if an artificial neural network is used to predict the event posterior probability from the value of many different predictors (inputs), then one can simply choose as an output threshold the event prior, P_YES, to dichotomize the forecast in yes–no classes maximizing the PSS of the model.

In Manzato (2005a) a method for transforming a predictor X into its event posterior probability was shown. It is interesting to note that (10) shows how the event posterior probability is a monotonic transformation of the likelihood ratio Λ (Kupinski et al. 2001). It is straightforward to show that the ROC curve is invariant for any monotonic transformation of the thresholded variable, because it is just a relabeling of the threshold (e.g., Green and Swets 1966). This means that converting the original X variable into its posterior probability or into its likelihood ratio will produce the same ROC curve for the new variable.

The fact that the same ROC is obtained from either the posterior probability or from the likelihood ratio transformation is important because the ROC curve obtained using the likelihood ratio Λ(x) as a mapping function is the optimal ROC, that is, a curve that will always lie on or above the ROC curve made with the original X values, or with any other transformation of X (Green and Swets 1966; Zhang and Mueller 2005). This is a consequence of the Neyman–Pearson criterion (Neyman and Pearson 1928). Then, it can be said that transforming the predictor X into its p(YES | x), as was done in Manzato (2005a), is an optimal preprocessing choice for classification problems.

3. A practical example

Let us show how the previous properties can be applied in a concrete example. Figure 1a shows the two estimated conditional components (normalized by N) of the sounding-derived mean relative humidity in the lowest 500 hPa. These two histograms are built by splitting the normalized histogram of all the MRH values for the case when there was an occurrence of rainfall greater than 20 mm (in the 6 h after the sounding release) in the FVG plain, [ f_Y(MRH)], and for the case when there was not occurrence, [ f_N(MRH)]. The conditional density probabilities, or likelihoods, could be obtained by dividing these components by the prior probabilities estimated from the data sample with (1).

Varying the threshold in all of the MRH domain, one can compute many contingency tables and their derived PSSs. The vertical dashed line in Fig. 1a indicates the threshold (MRH_P ≅ 71%) that gives the maximum PSS in this empirical way and sets the four coefficients of the contingency table shown in Table 1. The corresponding ratio of the two conditional components is 23, while the sample estimate of α is 21, which is quite close. The ratio of the two conditional probabilities (figure not shown) is 1.1, which is very close to the theoretical value (Λ = 1). These small errors are because continuous density functions have been approximated with discrete histograms.

Figure 1b shows the sample estimate posterior probability fit derived for the same dataset. The small circles are obtained by splitting the MRH domain into 21 equal bins and then, for each single bin, dividing the number of event occurrences by the number of total cases populating the bin. The continuous black line is a two-parameter exponential fit of these points, weighted with the population of each bin. Other details on how to interpret this kind of figure and the fitting line can be found in Manzato (2005a).

The gray dashed horizontal line shows the event prior probability, P_YES = 1/(1 + α) ≅ 0.045, while the gray dashed vertical line shows the threshold MRH_P, which empirically maximizes PSS. These two lines intersect very close to the fitted posterior probability line. So, instead of empirically computing the threshold that maximizes PSS, one can transform the original values into their posterior probability (which means issuing a calibrated event probability forecast) and then use the event prior probability as the threshold that maximizes PSS. Because of the optimality of the posterior probability-derived ROC, this method has the advantage that the PSS will be greater than the original one when the posterior probability fit is not monotonic.

Figure 2 shows the ROC curve and the point corresponding to PSS(MRH_P). The value of the Peirce skill score is given by the vertical and the horizontal gray segments, while the third steep segment shows the maximum distance H from the no-skill bisector. In this particular case, one obtains the same ROC curve if the sample estimate event posterior probability of MRH is used as the thresholded variable, because p(YES | MRH) is monotonic.

For an example of a nonmonotonic transformation, if we consider the vertical component of the water vapor flux in the lowest 3 km (VFlux), then we obtain a f_Y (VFlux) function with a minimum around the mean VFlux value and two maxima around the range extremes. This leads to a u-shaped posterior probability fit. The threshold that maximizes PSS on VFlux leads to a PSS of 0.56, while using the posterior probability transformed data leads to a higher ROC, with a maximum PSS of 0.57.

4. Conclusions

The threshold that maximizes the Peirce skill score identifies the point on the ROC curve that has the maximum distance H from the no-skill line. In this sense, it is the ROC point that maximizes the skill of the classifier. As shown by Richardson (2000), this threshold maximizes the forecast relative value. It has been shown how the likelihood ratio and the event posterior probability are known a priori for that particular threshold. In particular, at the threshold that maximizes PSS, the two likelihoods have the same value (Λ = 1) and the event posterior probability is equal to the event prior probability.

These results support the conclusion made in ORP that it seems reasonable to use the maximum PSS as a scalar measure of the absolute classifier skill, together with an estimate of the event climatology (like P_YES or α). This can be particularly useful when comparing different classifiers, especially when applied to different datasets. Of course, this does not mean that the threshold that maximizes PSS is always the best one for all end-user purposes, which can be differently sensitive to false alarms or missing events. This is because of the complex relationship between forecast skill and forecast value (e.g., Roebber and Bosart 1996; Semazzi and Mera 2006).

To clarify this point, let us review a simple example of customer economic cost. Suppose that the user will suffer a loss L when there is an event occurrence, but that the user can completely avoid any loss by taking a protective action, which costs C (with a “loss–cost” ratio λ = L/C > 1). If this user will make the decision to take the protective action based on the binary classifier obtained from the threshold x, and its associated contingency table, then the total expense, E(x), will be given by (Richardson 2003)

View Expanded

To minimize this cost function, we can write the contingency coefficients in integral form, similarly to (4) and (5), and then set the derivative of E(x) to zero, as was done previously for PSS. It is straightforward to show how this leads to the following properties for the threshold, x_E, which minimizes this cost problem:

View Expanded

So, the threshold that minimizes the user expense is found on the posterior probability curve at p(YES | x_E) = C/L. Hence, x_E will be the same value that maximizes PSS only if the loss–cost ratio λ is equal to P⁻¹_YES = α + 1 (maximum value). Similar properties can be found in the same way for a generic cost function, F(x) = s₁₁a(x) + s₁₂b(x) + s₂₁c(x) + s₂₂d(x), minimized at x_m such that

View Expanded

where the last equation is the same result found by Mason (1979), while the previous one was already shown in Eq. (1.20) of Green and Swets (1966).