1. Introduction

Forecasts are intended to provide information to the user. Forecast verification is the assessment of the quality of a single forecast or forecasting scheme (Jolliffe and Stephenson 2008). Verification should therefore assess the quality of the information provided by the forecast. It is important here to note the distinction between quality, which depends on the correspondence between forecasts and observations, and value, which depends on the benefits of forecasts to users (Murphy 1993). In this paper, we assume that the verification is intended to quantitatively measure quality. Several scores and visualization techniques have been developed that measure certain desirable properties of forecasts with the purpose of assessing their quality. One of the most commonly used skill scores (Stephenson et al. 2008) is the Brier score (BS) (Brier 1950), which is applicable to probabilistic forecasts of binary events. The Brier skill score (BSS) measures the BS relative to some reference forecast, which is usually climatology. Murphy (1973) showed that the BS can be decomposed into three components: uncertainty, resolution, and reliability. These components give insight into some different aspects of forecast quality. The first component, uncertainty, measures the inherent uncertainty in the process that is forecast. Resolution measures how much of this uncertainty is explained by the forecast. Reliability measures the bias in the probability estimates of the probabilistic forecasts. A perfect forecast has a resolution that is equal to (fully explains) the uncertainty and a perfect reliability.

Information theory provides a framework for measuring information and uncertainty (see Cover and Thomas 2006 for a good introduction). As forecast verification should assess the information that the forecaster provides to the user, using information theory for forecast verification appears to be a logical choice. A concept central to information theory is the measure of uncertainty named entropy. However, consulting two standard works about forecast verification, we noted that the word entropy is mentioned only thrice in Jolliffe and Stephenson (2003) and not one single time in Wilks (1995). This indicates that the use of information–theoretical measures for forecast verification is not yet widespread, although some important work has been done by Roulston and Smith (2002), Ahrens and Walser (2008), Leung and North (1990), and Kleeman (2002).

Leung and North (1990) used information–theoretical measures like entropy and transinformation in relation to predictability. Kleeman (2002) proposed to use the relative entropy between the climatic and the forecast distribution to measure predictability. The applications of information theory in the framework of predictability are mostly concerned with modeled distributions of states and how uncertainty evolves over time. Forecast verification, however, is concerned with comparing observed values with the forecast probability distributions. Roulston and Smith (2002) introduced the ignorance score, a logarithmic score for forecast verification, reinterpreting the logarithmic score (Good 1952) from an information–theoretical point of view. They related their score to relative entropy between the forecast distribution and the “true” probability distribution function (PDF), which they defined as “the PDF of consistent initial conditions evolved forward in time under the dynamics of the real atmosphere.” Ahrens and Walser (2008) proposed information–theoretical skill scores to be applied to cumulative probabilities of multicategory forecasts. Very recently, Benedetti (2009) showed that the logarithmic score is a unique measure of forecast goodness. He showed that the logarithmic score is the only score that simultaneously satisfies three basic requirements for such a measure. These requirements are additivity, locality (which he interprets as exclusive dependence on physical observations), and strictly proper behavior. For a discussion of these requirements, see Benedetti (2009). Furthermore, Benedetti (2009) analyzed the Brier score and showed that it is equivalent to a second-order approximation of the logarithmic score. He concludes that lasting success of the Brier score can be explained by the fact that it is an approximation of the logarithmic score. Benedetti also mentions the well-known and useful decomposition of the Brier score into uncertainty, resolution, and reliability as a possible reason for its popularity.

In this paper, we follow a route similar to Benedetti’s, but from a different direction. From an analogy with the Brier score, we propose to use the Kullback–Leibler divergence (or relative entropy) of the observation from the forecast distribution as a measure for forecast verification. The score is named “divergence score.” When assuming perfect observations, our measure is equal to the ignorance score or logarithmic score, and can be seen as a new reinterpretation of ignorance as the Kullback–Leibler divergence from the observation to the forecast distribution. Presenting a new decomposition into uncertainty, resolution, and reliability, analogous to the well-known decomposition of the Brier score (Murphy 1973), yields insight into the way the divergence score (DS) measures the information content of probabilistic binary forecasts. The decomposition can help acceptance and wider application of the logarithmic score in meteorology.

Section 2 of this paper presents the mathematical formulation of the DS and its components. Section 2 also shows the analogy with the Brier score components. Section 3 compares the divergence score with existing information–theoretical scores. It is shown that the DS is actually a reinterpretation of the ignorance score (Roulston and Smith 2002) and that one of the ranked mutual information scores defined by Ahrens and Walser (2008) is equal to the skill score version of DS, when the reliability component is neglected (perfect calibration assumed). A generalization to multicategory forecasts is presented in section 4. The inherent difficulty found in formulating skill scores for ordinal category forecasts is also analyzed and leads to the idea that this can be explained by explicitly distinguishing between information and useful information for some specific user. This distinction provides some insights in the roles of the forecaster and the user of the forecast. Section 5 presents an application to a real dataset of precipitation forecasts. Section 6 summarizes the conclusions and restates the main arguments for adopting the divergence score.

2. Definition of the divergence score

b. Definitions

Consider a binary event, like a day without rainfall or with rainfall. This can be seen as a stochastic process with two possible outcomes. The outcome of the event can be represented in a probability mass function (PMF). For the case of binary events, the empirical PMF of the event after the outcome has been observed is a 2D vector, denoted by o = (1 − o, o)^T. Assuming certainty in the observations, o {0, 1}; therefore, o = (0, 1)^T if it rained and (1, 0)^T otherwise. Now suppose a probabilistic forecast of the outcome of the binary event is issued in the form of a probability of occurrence f. This can also be written as a forecast PMF f = (1 − f, f )^T, with f [0, 1]. If for example an 80% chance of rainfall is forecast, this is denoted as f = (0.2, 0.8)^T.

The Brier score for a single forecast at time t measures distance between observation and forecast PMFs by the square Euclidean distance:

View Expanded

For a series of forecasts and observations, the Brier score is simply the average of the Brier scores for the individual forecasts:

View Expanded

Note that this is the original definition by Brier (1950). Nowadays, the Brier score is almost always defined as half the value of (2) (Ahrens and Walser 2008).

In information theory, the difference between two probability distributions is measured by relative entropy or Kullback–Leibler divergence (D_KL). The D_KL is not symmetric, so as not to be confused with a true distance like the quadratic score. In this paper D_KL(x‖y) will be referred to as the divergence from x to y (or of y from x). It measures the missing information in case one assumes the distribution is y while the true distribution is x.

c. The divergence score

We now define the divergence score (DS), replacing the quadratic distance from the BS with the Kullback–Leibler divergence. For one single forecast, the DS functions as a scoring rule. It is the Kullback–Leibler divergence of the forecast distribution from the observation distribution over the n = 2 possible events i:

View Expanded

The DS over a series of forecast–observation pairs measures the average divergence of the forecast distribution from the observation:

View Expanded

The divergence score can be interpreted as the information gain when one moves from the prior forecast distribution to the observation distribution. When this information gain is zero, the forecast already contained all the information that is in the observation and therefore is perfect. If the information gain from the forecast to the certain observation is equal to the climatological uncertainty, the forecast did not contain more information than the climate, and therefore was useless. Another way to view the divergence score is the remaining uncertainty about the true outcome, after having received the forecast (see Fig. 1).

d. Decomposition

The new score can be decomposed in a similar way as the Brier score. The classic decomposition of the BS into reliability (REL), resolution (RES), and uncertainty (UNC) is

View Expanded

View Expanded

where N is the total number of forecasts issued, K is the number of unique forecasts issued, is the observed climatological base rate for the event to occur, n_k is the number of forecasts with the same probability category, and o_k is the observed frequency, given forecasts of probability f_k. The reliability and resolution terms in (6) are summations of some distance measure between two binary probability distributions, while the last term measures the uncertainty in the climatic distribution, using a polynomial of degree two. We now present information–theoretical twins for each of the three quadratic components of the BS, using entropy and relative entropy. It is shown that they add up to the divergence score proposed earlier.

The first component, reliability, measures the conditional bias in the forecast probabilities. In the DS, it is the expected divergence of the observed probability distribution from the forecast probability distribution, both stratified (conditioned) on all issued forecast probabilities. In the ideal case, the observed frequency is equal to the forecast probability for all of the issued forecast probabilities. Only in this case is the reliability 0. This is referred to as a perfectly calibrated forecast. Note that reliability is defined in the opposite direction to the meaning of the word in the English language. A perfectly reliable forecast has a reliability of 0. The reliability can be calculated with

View Expanded

where N is the total number and K the number of unique forecasts issued, n_k is the number of forecasts with the same probability category, o_k is the observed frequency distribution for forecasts in group k, and f_k is the forecast PMF for group k.

The second component, resolution, measures the reduction in climatic uncertainty. It can be seen as the amount of information in the forecast. In the DS, it is defined as the expected divergence of the conditional frequencies from the marginal frequency of occurrence. The minimum resolution is 0, which occurs when the climatological probability is always forecast or the forecasts are completely random. The resolution measures the amount of uncertainty in the observation explained by the forecast. In the ideal case the resolution is equal to the uncertainty, which means all uncertainty is explained. This is only the case for a deterministic forecast that is either always right or always wrong. In the last case, the forecast needs to be recalibrated:

View Expanded

From (8) it becomes clear that the resolution term is the expectation over all forecast probabilities of the divergence from the conditional probability of occurrence to the marginal probability of occurrence. In information theory, this quantity is known as the mutual information I between the forecasts and the observation ( denotes the expectation over k):

View Expanded

View Expanded

The third component, uncertainty, measures the initial uncertainty about the event. This observational uncertainty is measured by the entropy of the climatological distribution H(o). It is a function of the climatological base rate only and does not depend on the forecast. The uncertainty is maximum if the probability of occurrence is 0.5 and 0 if the probability is either 0 or 1:

View Expanded

Like for the BS, for a single forecast–observation pair, uncertainty and resolution are 0 and the total score is equal to the reliability, which acts as a scoring rule. Over a larger number of forecasts uncertainty approaches climatic uncertainty and reliability should go to zero if the forecast is well calibrated. In the appendix it is shown that, just like in the Brier score, the relation DS = REL − RES + UNC holds. The relation between the components and the total score (DS) is

View Expanded

Note that this decomposition is valid for forecasts of events with an arbitrary number of categories and is not restricted to the binary case.

f. Normalization to a skill score

Because the Brier score depends on the climatological probability, which is independent from the forecast quality, it is common practice to normalize it to the BSS with the climatology forecast as a reference. A perfect forecast is taken as a second reference. For the DS it is possible to use the same normalization procedure, yielding the divergence skill score (DSS):

View Expanded

The score for a perfect forecast DS_perf is 0. In the climatological forecast DS_ref, both resolution and reliability are 0 (perfect reliability, no resolution). The DSS therefore reduces to

View Expanded

This leads to a positively oriented skill score that becomes one for a perfect forecast and zero for a forecast of always the climatological probability. Also, a completely random forecast that has a marginal distribution equal to climatology gets a zero score. Negative (i.e., “worse than climate”) skill scores are possible if the reliability is larger (worse) than the resolution. In case the resolution is significant, calibration of the forecast can yield a positive skill score, meaning that a decision maker using the recalibrated forecast is better off than a decision maker using climatology or a random strategy. This shows the importance of looking at the individual components when diagnosing a forecast system’s performance.

Summarizing, the divergence score and its components combine two types of measures to replace the quadratic components in the Brier score decomposition. First, the quadratic distances between probability distributions are replaced by Kullback–Leibler divergences, which are asymmetric. Care should therefore be taken in which direction the divergence is calculated. Second, the polynomial uncertainty term is replaced by the entropy of the climatology distribution. The total scores and components are visualized in Fig. 1.

3. Relation to existing information–theoretical scores

a. Relation to the ranked mutual information skill scores

Ahrens and Walser (2008) proposed the ranked mutual information skill score (RMIS). The score is intended for use with multicategory forecasts, which will be treated later in this paper. For the special case of forecasts of binary events, the RMIS_O can be written as the mutual information between forecasts and observations divided by the entropy of the observations:

View Expanded

When comparing (15) with (9), (11), and (14), it becomes clear that

View Expanded

This means that for the case of a binary forecast, RMIS_O equals the DSS in case the reliability is perfect (zero). In case the forecast is not well calibrated, RMIS_O neglects the reliability component and measures the amount of information that would be available to a user after calibration. The DSS measures the information conveyed to a user taking the forecasts at face value. The individual components of the DSS also indicate the potentially extractable information as measured by the RMIS_O.

b. Equivalence to the ignorance score

Roulston and Smith (2002) defined the ignorance score from the viewpoint of using the forecast probability distribution as basis for a data compression scheme. The scoring rule measures the ignorance or information deficit of a forecaster, compared to a person knowing the true outcome of the event (j). The ignorance scoring rule is defined as

View Expanded

where f_j is the probability that the forecaster had assigned to the event that actually occurred. The ignorance score is a reinterpretation of the logarithmic score by Good (1952).

By expanding the relative entropy measure that we use as a scoring rule, it becomes clear that divergence from the certain observation PMF (o) to the forecast PMF (f) is actually the same as the ignorance (IGN) or the logarithmic score:

View Expanded

Because o_i≠j = 0 and o_i=j = 1, this reduces to

View Expanded

This means that the divergence scoring rule presented in this paper (DS) is actually equal to the ignorance (IGN). The ignorance is therefore not only “A scoring rule … closely related to relative entropy” as defined by Roulston and Smith (2002), but actually is also a relative entropy. The difference is in the distributions that are used to calculate the relative entropy. Roulston and Smith (2002) refer to a relation to the divergence between the unknown “true” distribution p and the forecast distribution f (see 20):

View Expanded

The divergence that is used in the divergence score is calculated from the PMF after the observation o instead of p. That makes the second term of the rhs vanish and IGN equal to the divergence.

Using the decomposition presented in (12), the ignorance score for a series of forecasts can now also be decomposed into a reliability, a resolution, and an uncertainty component. This decomposition, until now only applied to the Brier score, has proven very useful to gain insight into the aspects of forecast quality. Furthermore, the new interpretation of the ignorance score as the average divergence of observation PMFs from forecast PMFs links to results from information theory more straightforwardly.

4. Generalization to multicategory forecasts

b. Ordinal category forecasts

When dealing with forecasts of events in ordinal categories, there is a natural ordering in the categories. This means that the cumulative distribution function (CDF) starts to become meaningful. There are now two possible questions that can be relevant for verification of the probabilistic forecast.

1. How well did the forecaster predict the category to which the outcome of the event belongs?

2. How well did the forecaster predict the exceedence probabilities of the thresholds that define the category boundaries?

The first question is equal to the one that is of interest for nominal forecasts. However, in the ordinal case there is a difference between the categories in which the observed event did not fall. Forecasts of categories close to the one observed are preferred over categories more distant from the one observed. Therefore, skill scores for ordinal category forecasts are often required to be “sensitive to distance” (Epstein 1969; Laio and Tamea 2007; Murphy 1971, 1970). This requirement has led to the introduction of the ranked probability score (RPS; Epstein 1969), which is now widely used. The DS is not sensitive to distance in the sense of the RPS, because DS is insensitive to the forecasts for the nonoccurring events. However, there still is an apparent sensitivity to distance introduced through the forecast PMF. A forecaster will usually attach more probability to the categories adjacent to the one that is considered most likely, simply because they are also likely. Therefore, missing the exact category of the observation with the most likely forecast still leads to a relatively low penalty in the score, if the uncertainty estimation of the forecaster was correct and significant probability was forecast for the other likely (often neighboring) categories. Over a series of forecasts, the apparent distance sensitivity of the penalty given by the DS is therefore defined by the PMF of the forecaster alone, and independent of what the categories represent. In verification literature, the property of only being dependent on the probability assigned to the event that actually occurred is known as locality, which is often seen as a nondesirable property of a score. Whether or not locality is desirable can be questioned (Mason 2008). We argue that in absence of a context of the users of the forecast there is no justification for using nonlocal scores, which require some sensible distance measure to be specified apart from the natural distance sensitivity introduced by the forecast PMF. When assessing value or utility of forecasts, as opposed to quality, nonlocal scores can be used. However, in that case the distance measure should depend on the users and associated decision processes, as these determine the consequences of missing the true event by a certain number of categories distance. In nonlocal verification scores, the distance measure is often not explicitly specified in terms of utility, making it unclear what is actually measured. In those cases, the utility function of the users becomes more like an emerging property of the skill score instead of the other way around. Benedetti (2009) also presents locality as a basic requirement for a measure of forecast goodness, interpreting locality as “exclusive dependence on physical observations.” He correctly states that it is a violation of scientific logic if two series of forecasts that assign the same probabilities to a series of observed events gain different scores, based on probabilities assigned to events that have never been observed. For a more elaborate treatment of this view on the fundamental discussion about locality, the reader is referred to Benedetti (2009) and Mason (2008).

The second question, regarding the forecast quality of exceedence probabilities, differs from the first because all the thresholds are considered at once. Therefore, the quality of a single forecast depends on the entire PMF of the forecast and not only on the probability forecast for the event that occurs. Therefore, scores that are formulated for cumulative distributions can never be local. This means that apart from the physical observations, the importance attached to the events influences the score. So some assumption about value is added and the score is not a pure measure of quality alone. The RPS evaluates the sum of squared differences in CDF values between the forecast and the observation of the event:

View Expanded

The RPS can be seen as a summation of the binary Brier scores over all n − 1 thresholds defined by the category boundaries. The summation implies that the Brier scores for all thresholds are weighted equally. Whether the BS for all thresholds should be considered equally important depends on the users. It has been shown that the RPS is a strictly proper scoring rule in case the cost–loss ratio is uniformly distributed over the users (Murphy 1970). In that case the RPS is a linear function of the expected utility.

c. The ranked divergence score

Now an information–theoretical score is presented for ordinal category forecasts, which are defined in terms of cumulative probabilities. An equivalent to the RPS would be the ranked divergence score (RDS), averaging of the DS over all n − 1 category thresholds m:

View Expanded

where DS_m denotes the divergence score for the forecast of the binary event j ≤ m. This assumes equally important thresholds. The RDS, just like the DS, is dependent on the climatological uncertainty. To make the score comparable between forecasts, the RDS can be converted into a skill score. Now, two intuitive options exist to do the normalization. The first is to normalize the individual DS_m scores for each threshold m to a skill score for that threshold, like (14), using the climatic uncertainty for the binary event defined by that threshold:

View Expanded

and then averaging the resulting skill score over all thresholds:

View Expanded

This means that the relative contributions to the reduction of climatic uncertainty about each threshold m are considered equally important. In other words, all skills of forecasts about the exceedence of the n − 1 thresholds are equally weighted.

The second option for normalization is the first to sum the DS_m and then normalizing with the climatic score for the sum:

View Expanded

The formulation of the RDSS₂ according to (25) does not normalize the scores for the different thresholds individually, but applies the same normalization to every DS_m. This means that the merits of the forecaster for all thresholds are implicitly weighted according to the inherent uncertainties in the climate. In this way, the forecast of extreme (nearly certain) events are hardly contributing to the total score, while they could in fact be most important for the users.

d. Relation to ranked mutual information

An alternative skill score defined in terms of cumulative probabilities is the RMIS_O in (15) as defined by Ahrens and Walser (2008). The version of the RMIS_O for multiple category forecasts can be written as

View Expanded

where f_m denotes the series forecast probabilities of exceedence of threshold m, o_m denotes the corresponding series of observations, and o_m is the average observed occurrence. For a perfectly reliable forecast, the RMIS_O is therefore equal to the RDSS₂ formulated in (25). For forecasts that are not well calibrated, the RMIS_O measures the amount of information that would be available after calibration, while the RDSS measures the information as presented by the forecaster. By using the decomposition presented in (14) it is possible to write

View Expanded

By presenting both the resolution and the reliability terms of (27) separately, the potential information and the loss of information due to imperfect calibration are visible.

Apart from including the reliability or not, another question is how to weight the scores for the different thresholds, to come to one aggregated score. As every binary decision by some user with a certain cost–loss ratio can be associated with some threshold, the weighting reflects the importance of the forecast to the various users. No matter what aggregation method is chosen, there will always be an implicit assumption about the user’s importance and stakes in a decision-making process. This is inherent to summarizing forecast performance in one score. A diagram that plots the two skill score components against the thresholds contains the relevant information characteristics for different users. In this way each user can look up the score on the individual threshold, that is relevant for his decision problem, and compare it with the performance of some other forecasting system on that threshold.

5. An example: Rainfall forecasts in the Netherlands

As an illustration of the practical application of the divergence score and its decomposition, it was applied to a series of probabilistic rainfall forecasts and corresponding observations for the measurement location De Bilt, Netherlands. The forecast series consist of daily forecasts of the probability of a precipitation amount equal or larger than 0.3 mm, for 0–5 days ahead. They are derived using output of both the European Centre for Medium-Range Weather Forecasts (ECMWF) deterministic numerical weather prediction model and the ECMWF ensemble prediction model. Predictors from both models are used in a logistic regression to produce probability forecasts for precipitation. The data cover the period from December 2005 to November 2008, in which the average probability was 0.4613, leading to an initial uncertainty of 0.9957 bits.

Figure 5 confirms the expectation that both the Brier skill score and the divergence skill score show a decline with increasing lead time. It also shows that the forecasts possess skill over annual climatology up to a lead time of at least 5 days. The dashed lines show the potential skill that could be attainable after recalibration. The estimation of this potential skill, however, is dependent on the correct decomposition. The decompositions of both the Brier and the divergence score need enough data (large enough n_k) to calculate the conditional distributions f_k for all unique forecasts k (1, K). To be able to calculate the contribution to reliability of all the 99% forecasts, for example, at least 100 of such forecasts are necessary to not surely overestimate reliability. Also for a larger number of forecasts, there is a bias toward overestimation of the reliability, which decreases with the amount of data available per conditional to be estimated.

A solution for estimating the components with limited data is rounding the forecast probabilities to a limited set of values. In this way, less conditional distributions f_k need to be estimated and more data per distribution are available.

Figure 6 shows that for these 3 yr of data, using finer-grained probabilities as forecasts leads to an increasing overestimation of reliability. The skill scores themselves are not sensitive to this overestimation because the lack of data causes a compensating overestimation of resolution. The potential skill, however, should be interpreted with caution, as solving reliability issues by calibration on the basis of biased estimates of reliability does not lead to a real increase in skill. From the figure it can also be noted that too coarse-grained probabilities lead to a real loss of skill. In this case, giving the forecasts in 5% intervals seems the minimum needed to convey the full information that is in the raw forecast probabilities.

Figure 7 sheds more light on the relation between the Brier skill score and the divergence skill score, based on 5-day-ahead forecasts from a second dataset, which covers February 2008 to December 2009. For this set, the forecast probabilities ranged from 1% to 99%. The black dots indicate the scores that were attained for single forecast observation pairs. The dots show that the BSS and DSS have a monotonic relation as scoring rules. The limits of this relation are at (1, 1) for perfect forecasts and, in this case, (−∞, −3.095) for certain but wrong forecasts. The worst forecast was 98%, while no rain fell.

The total scores for different weeks of forecasts are plotted as gray dots. They are averages of sets of seven black dots. Because the relation of the single forecast scores is not a straight line, a scatter occurs in the relation of the weekly average scores, which is therefore no longer monotonic. The scatter implies that two series of forecasts can be ranked differently by the Brier score and the divergence score. In this example, the scatter is relatively small (r ² = 0.9938) and will probably have no significant implications, but it would be larger if many overconfident forecasts were issued. An interesting example of differently ranked forecast series are the two weeks indicated by the triangles, where the scores disagree on which of the two weeks was better forecast than climate. The downward-pointing triangle marks the score for forecasts in week A, where performance according to the divergence score was worse than the climatological forecast (DSS = −0.0758), but according to the Brier score was slightly better than climate (BSS = 0.0230). Conversely, the upward-pointing triangle marks week B, where the forecasts according to Brier were worse than climate (=−0.0355) but still contained more information than climate according to the DSS (=0.0066).

Given that the scatter in the practical example is small, the Brier score appears to be a reasonable approximation of the divergence score and is useful to get an idea about the quality of forecasts. More practical comparisons are needed to determine if the approximation can lead to significantly different results in practice. These are mostly expected in case extreme probabilities are forecast.

The severe penalty the divergence gives for errors in the extreme probabilities, which is sometimes seen as a weakness, should actually be viewed as one of its strengths. As the saying goes, “you have to be cruel to be kind.” It is constructive to give an infinite penalty to a forecaster who issues a wrong forecast that was supposed to be certain. This is fair because the value that a user would be willing to risk when trusting such a forecast is also infinite.

6. Conclusions

Analogous to the Brier score, which measures the squared Euclidean distance between the distributions of observation and forecast, we formulated an information–theoretical verification score, measuring the Kullback–Leibler divergence between those distributions. More precisely, our score measures the divergence from the distribution of the event after the observation to the distribution that is the probability forecast. Our divergence score is a reinterpretation of the ignorance score or logarithmic score, which was previously not defined as a Kullback–Leibler divergence. Extending the analogy to the useful and well-known decomposition of the Brier score, the divergence score can be decomposed into uncertainty − resolution + reliability. For binary events, Brier score and its components are second-order approximations of the divergence score and its components.

The divergence score and its decomposition generalize to multicategory forecasts. A distinction can be made between nominal and ordinal category forecasts. Scores based on the cumulative distribution over ordinal categories can be seen as combinations of binary scores on multiple thresholds. How the scores for all thresholds should be weighted relative to each other depends on the user of the forecast. Scores on cumulative distribution are therefore not exclusively dependent on physical observations, but contain subjective weights for the different thresholds. Two possible formulations of a ranked divergence skill score have been formulated. The first equally weighs the skill scores relative to climate, while the second equally weights the absolute scores. The second-ranked divergence skill score is equal to the existing ranked mutual information skill score for the case of perfectly calibrated forecasts, but additionally includes a reliability component, measuring miscalibration.

In forecasting, a distinction can be made between information and useful information in a forecast. The latter cannot be evaluated without a statement about context in which the forecast will be used. The first is only dependent on how the forecasts relate to the observations and is objective. Therefore, in the authors’ opinion, information should be the measure for forecast quality. It can be measured using the logarithmic score, which now can be interpreted as the Kullback–Leibler divergence of the forecast from the observation. Useful information or forecast value, on the other hand, is a different aspect of forecast “goodness” (Murphy 1993), which should be evaluated while explicitly considering the decision problems of the users of the forecast.

The Brier score can be used as an approximation for quality or as an exact measure of value under the assumption of a group of users with uniformly distributed cost–loss ratios. In our opinion, these two applications should be clearly separated. In case one wants to assess quality, information–theoretical scores should be preferred. If an approximation is sufficient, the Brier score could still be used, with the advantage that it is well understood and extensive experience exists with the use of it. However, when extreme probabilities have to be forecast, the differences might become significant and the divergence score is to be preferred on theoretical grounds.

In case value is to be measured, an inventory of the users of the forecasts should be made to asses the total utility. When explicitly investigating the user base, a better estimator for utility than the Brier score can probably be defined. Using the Brier score as a surrogate for forecast value, implicitly assuming the emergent utility function is appropriate for a specific type of forecasts, is clearly unsatisfactory. In this respect, it is important to also stress that the divergence score does not measure value, but quality. Only in a very unrealistic case (a bookmaker offering fair odds) does a clear relation exist between the two. It might be argued that for practitioners in meteorology, quality is most likely of more concern than value, because the latter is in fact evaluating decisions rather than forecasts. (To facilitate the use of the score and its decomposition, scripts that can be used in Matlab and Octave are available online at http://divergence.wrm.tudelft.nl.)