1. Introduction

Information theory became a field of research with the mathematical theory of communication of Shannon (1948). Next to applications in data communication, there was a strong impact on computer science, ranging from data compression and cryptography to error correcting codes [e.g., the fact that a scratch on a compact disk (CD) is inaudible]. Other fields affected by information theory are statistics, gambling (Kelly 1956), and financial portfolio theory. Cover and Thomas (2006) give an extensive introduction into information theory and its applications. The basis of the theory is a measure of uncertainty represented by a probability distribution named entropy because of the mathematical similarity with the formulation of entropy in statistical thermodynamics.¹ The measure uniquely follows from a set of elementary desiderata for a measure of uncertainty to be useful (see Shannon 1948). Apart from entropy, information theory also defines measures like relative entropy (also Kullback–Leibler divergence), cross entropy, and mutual information. The theory defines information as the reduction in uncertainty. Various inequalities form certain “laws of information,” such as the data processing inequality, which says that we cannot increase the amount of information in data by processing it. In other words, we cannot produce information out of thin air, except when a meteorologist observes the atmosphere with measurement equipment and generates data. Although these measurements give some information, they also leave some uncertainty about the variables of interest, which is the topic of this paper.

Additive relations between various measures of information and uncertainty provide a useful basis for a framework for forecast verification. In Weijs et al. (2010a), it was argued that the Kullback–Leibler divergence from the observation to the forecast is a measure for forecast quality with a number of desirable properties (see also Benedetti 2010; Weijs et al. 2010b). The view was presented that forecasting can be seen as a communication problem, in which information is given by the forecaster to reduce the uncertainty of the user. The quality of such forecasts can therefore be evaluated using the information-theoretical measures relating to information and uncertainty. Examples of such scores are the average information gain from the climatological distribution to the forecast (Peirolo 2010) and the ignorance score (Roulston and Smith 2002). In Weijs et al. (2010a), it was found that the remaining uncertainty relative to the observations, as measured by this Kullback–Leibler divergence from the observations to the forecasts, is an appropriate scoring rule, which we refer to as the divergence score.

Moreover, a decomposition was presented, analogous to the decomposition of the Brier score (Brier 1950; Murphy 1973), which decomposes the Brier score into (climatological) “uncertainty” minus “resolution” plus “reliability” (actually unreliability). The decomposition in Weijs et al. (2010a) is equal to the logarithmic case of a general decomposition for the proper scoring rules presented in Bröcker (2009). Note that in the latter, the asymmetric divergence measure was defined with a reverse order of the arguments compared to the Kullback–Leibler divergence common in information theory. A possible interpretation of the information-theoretical decomposition in Weijs et al. (2010a) is that “the remaining uncertainty is equal to the missing information minus the correct information plus the wrong information” (see Fig. 1). A forecast that is reliable but has no perfect resolution does not give complete information, but the information it does give is correct. In the decompositions of both the Brier score (BS) and the divergence score (DS), it was assumed that the observations are certain and correspond to the truth about the forecast variable.

View Full Size

The relations between the components and the scores presented in this article are additive. The bars give the average remaining uncertainty about the truth (measured in bits) for various (hypothetical) stages in the forecasting process. The naive forecast is always assigning 50% probability of precipitation (complete absence of information), the climatological forecast takes into account observed frequencies. This climatological uncertainty UNC_XES can be reduced to XES by believing the forecasts f. If these are not completely reliable, the uncertainty can be further reduced with REL by recalibration. After observation, there is still some uncertainty (obs unc) about the hypothetical true outcome, given that observations are not perfect. Only for an all knowing observer, the uncertainty is reduced to 0. The resolution RES is the information that could maximally be extracted from the forecasts by perfect calibration. The divergence score DS and the new uncertainty component UNC_DS measure the uncertainty after forecast and in the climate, relative to the observations.

Citation: Monthly Weather Review 139, 7; 10.1175/2011MWR3573.1

• Download Figure
• Download figure as PowerPoint slide

In reality, no observation can be assumed to correspond to the true outcome with certainty. For example, in the evaluation of binary probabilistic precipitation forecasts of the Royal Netherlands Meteorological Office (KNMI), the observation that corresponds to “no precipitation” is defined as an observed precipitation of less than 0.3 mm on a given day. Given the observational errors in the exact precipitation amount, measured values close to the threshold would be best represented by a probabilistic observation, accounting for the uncertainties (see Fig. 2). Briggs et al. (2005) also noted that uncertainty in the observation must be taken into account to assess the true skill of forecasts. This requires either “gold standard” observations or subjective estimates of the observation errors. Bröcker and Smith (2007) proposed to use a noise model for the observation to transform the forecast, using this to define a generalized score.

View Full Size

The measurement uncertainty in the precipitation measurement leads to a probabilistic binary observation. In the example, a simple Gaussian measurement uncertainty is assumed. The measurement distributions are centered around the measurements and have a constant standard deviation.

Citation: Monthly Weather Review 139, 7; 10.1175/2011MWR3573.1

• Download Figure
• Download figure as PowerPoint slide

We propose to define an “uncertain observation” as the retrospective conditional distribution of the true outcome of event or quantity that was forecast, given the reading on one or more measurement instruments. For example, when the spatial scale or location of the measurements and the forecasts differs, the distribution can be based on spatial statistics of various instruments. In another case, the distribution may be derived from a model of the observational noise (e.g., due to wind around a rain gauge, noise in the electronics, etc.). Note that the correctness and reliability of such uncertainty models cannot be verified, because the “true” value cannot be observed directly. Although the term verification suggests a comparison between forecasts and truth (Latin: veritas = truth), both the divergence and the Brier score are actually comparing the forecasts with observations, which are an estimate of the unknown truth about the forecast variable. An uncertain observation acknowledges this by representing the uncertainty explicitly by a probabilistic best estimate after the event has taken place.

When the uncertainty in the observations is accounted for by representing them with probability distributions with nonzero entropy (i.e., the observation assigns probability to more than one outcome), the decomposition that was presented in Weijs et al. (2010a) does not hold. The divergence score as a whole, however, is still a useful measure of correspondence between forecasts and observations. It would therefore be interesting to define a meaningful decomposition of the divergence score that is applicable in the case of uncertain observations. A second point is whether the quality of forecasts should be measured with respect to the known probabilistic observations or estimated with respect to the unknown truth.

In this article, we present a modification to the decomposition presented in Weijs et al. (2010a), which generalizes the case of uncertain observations. We discuss the interpretation of the new decomposition in terms of uncertainty and information. We furthermore present a second, related measure for forecast quality, in information theory often referred to as cross entropy, which in this case estimates the uncertainty relative to the truth instead of relative to the observation. A decomposition for this score is also presented. The scores are applied to a real dataset for illustration.

2. The concepts of surprise, uncertainty, and relative uncertainty

In information theory, uncertainty is related to surprise. Surprise is defined as minus the log of the prior probability assigned to the true outcome, S = −logP. This forms the basis for the measure of uncertainty associated with a discrete probability distribution, entropy H, which is the expected surprise upon hearing the truth. For example, the uncertainty associated with a binary probabilistic forecast of 70% chance of precipitation, f = (0.3, 0.7)^T, is

View Expanded

where H(f) is the entropy of probability mass function (PMF) f, calculated in the unit “bits” because the logarithm is taken to the base 2 (throughout this article). Here E_f denotes the expectation operator with respect to f, and n is the number of categories in which the outcome can fall, in this case 2. The notation [f]_i will be used throughout this paper to denote the element i of vector f.

Next to this entropy measure, introduced by Shannon (1948), there is also a definition of relative entropy, or Kullback–Leibler divergence D_KL (Kullback and Leibler 1951). This is a measure for the expected amount of additional surprise a person is expected to experience, compared to another person having a more accurate and reliable probability estimate. Therefore, it is a relative uncertainty. The divergence score is based on this idea and measures the expected extra surprise that a person having the forecast f_t for an instance t will experience, compared to a person knowing the observation o_t, from the perspective of the latter:

View Expanded

3. Decomposition of the divergence score for uncertain observations

The decomposition of the divergence score DS for a series of N forecasts that was presented in Weijs et al. (2010a) was analogous to the decomposition of the Brier score by Murphy (1973) into uncertainty, resolution, and reliability [cf. Eqs. (3) and (6)]:

View Expanded

in which N is the total number of forecasts issued, K is the number of unique forecasts issued, o_t is the observed outcome at time t, is the observed climatological frequency of the event, n_k is the number of forecasts with the same forecast probability, and is the observed frequency given forecasts of probability f_k.

The uncertain observation o_t is the probability mass function (PMF) of the true outcome of the uncertain event that is forecast, given the available information after it occurred. Note that when perfect data assimilation is performed, this also includes all information from f_t. Because measurements are usually indirect, we can regard the observation as a (usually subjective) conditional distribution of the true outcome, given the information from the measurement equipment.

The decomposition as formulated in Eq. (3) relies on the assumption that observations are certain [i.e., o_t = (0, 1)^T or o_t = (1, 0)^T]. In the appendix of Weijs et al. (2010a) this assumption was used to rewrite the closing term of the decomposition as the uncertainty component . When we want the decomposition to be valid for uncertain observations, the last step of the derivation in Weijs et al. (2010a) can be omitted. We thus replace the uncertainty component, the last term in Eq. (3), by the average Kullback–Leibler divergence from the uncertain observations to the average observation (i.e., the observed climatic distribution), the last term in Eq. (4):

View Expanded

This represents the expected climatological uncertainty relative to the observation, which is depicted in Fig. 1 as UNC_DS. By writing the uncertainty term of the divergence score decomposition in this way, it remains valid for uncertain observations. The original uncertainty term, the entropy , can be seen as representing the estimated climatological uncertainty relative to the truth, which we from now on will denote as UNC_XES, because it is part of a decomposition of XES we will introduce shortly.

Likewise, DS represents the average remaining uncertainty relative to the observations, which in the case of uncertain observations can become different from the estimated remaining uncertainty relative to the truth. This latter measure, XES, will be introduced in section 4.

Analogy for the Brier score decomposition

Analogously to the new decomposition of the DS, the Brier score decomposition introduced by Murphy (1973) can be modified in a similar manner to remain valid for uncertain observations. This can be achieved by replacing the uncertainty term in the original decomposition by the average squared Euclidean distance from the observations to the average observation. The original decomposition and the modified one are shown in Eqs. (6) and (7), respectively. For perfect observations, Eqs. (6) and (7) are the same. When observational uncertainty is considered, Eq. (6) does not hold, but Eq. (8) does. For ease of notation, we write (f_t − o_t)² for (f_t − o_t)^T(f_t − o_t):

View Expanded

View Expanded

View Expanded

4. Expected remaining uncertainty about the truth: The cross-entropy score

We now present the cross-entropy score XES. The expected uncertainty relative to the unknown truth can be expressed by taking the expectation, with respect to the PMF that represents the uncertain observation, of the Kullback–Leibler divergence from the hypothetical truth to the forecast distribution:

View Expanded

In which n = 2 is the number of categories in which the event can fall. Here v_t denotes the hypothetical distribution of the truth at instance t, which, like a perfect observation, is either (1, 0)^T if the event in fact did not occur or (0, 1)^T if the event truly occurred. The is the expectation operator with respect to the probability distribution o_t. In this case, the Kullback–Leibler divergence D_KL(v_t||f_t) reduces to the logarithmic score (Good 1952), which is also known as the ignorance score (Roulston and Smith 2002). These scores are simply minus the logarithm of the probability attached to the event that truly occurred:

View Expanded

where k(t) is the category in which the true outcome of the event fell at instance t. Because o_t is the best estimate of the unknown true outcome, we can use the expectation to evaluate the forecast, which can also be written as the right-hand expression in Eq. (10). In information theory, this expression is often defined as the cross entropy between o_t and f_t, hence we refer to it as cross-entropy score XES_t:

View Expanded

This measure can be interpreted as the expected remaining uncertainty relative to the truth, for a single forecast f_t that is evaluated in the light of the observation o_t. In the following interpretations, it is reasonable to assume that this observation is a reliable probability estimate of the truth, because there is no way to establish its unreliability without having access to the truth. The difference with the divergence score becomes clear from Fig. 1. For a series of forecasts, the cross-entropy score is defined as .

Decomposition of cross entropy

From the figure, we can see that the relation between all the components allows for several decompositions. The relation between DS and XES can be written as

View Expanded

The estimated remaining uncertainty in the forecasts relative to the truth (XES) is equal to the average uncertainty relative to the observations (DS) plus the average uncertainty that the observations represent, relative to the estimated truth [the second term on the right-hand side of Eq. (11)].

Another natural decomposition for the XES is the original decomposition of DS for perfect observations as presented in Weijs et al. (2010a). For uncertain observations, the three components presented there add up to the XES instead of to the DS (see also Fig. 1). The decomposition of the cross-entropy score XES therefore reads as

View Expanded

View Expanded

Note that the resolution and reliability components are equal to those of the DS decomposition in Eq. (4).

5. Example application

As an illustration of the new term in the decomposition, the scores were calculated for a real dataset of binary probabilistic rainfall forecasts of the Royal Netherlands Meteorological Office (KNMI). The observed rainfall amounts were transformed into probabilistic uncertain observations using a very simple uncertainty model. The purpose of this exercise is merely to illustrate the concepts in this article. The forecasts that are evaluated are the forecast probabilities of a daily precipitation of 0.3 mm or more. This is the same dataset that was used in Weijs et al. (2010a). In that paper the rainfall amounts x_t, which were given with a precision of 0.1 mm, were converted to binary observations with a simple threshold filter: if x_t ≥ 0.3 → o_t = (0, 1), if x < 0.3 → o_t = (1, 0). In this article, we assume random measurement error to make o_t probabilistic and account for the uncertainty in the observation.

The model of the uncertainty in the observation is Gaussian. The observed rainfall amount becomes a random variable with a normal probability density function:

View Expanded

with mean μ_obs and standard deviation σ_obs. Because in this case we deal with a binary predictand, the probability distribution function (pdf) of the observation can be converted to a binary probability mass function o_t = (1 − o_t, o_t) by using

View Expanded

in which G_obs(T) is the cumulative distribution function of the observation, evaluated at threshold T. This conversion is illustrated in Fig. 2.

The decompositions of the DS and XES scores for the entire dataset were calculated for a range of different standard deviations σ_obs for the measurement uncertainty. In Fig. 3 it can be seen that while the average observation uncertainty grows, the divergence score improves (decreases), but the cross-entropy score XES deteriorates. This indicates that if the observation uncertainty is reflected by this model, the best estimate for the information loss compared to the truth is higher than would be assumed neglecting observation uncertainty. Not taking the observation uncertainty in account would lead to an overestimation of the forecast quality in this specific case. A closer analysis reveals that most of the deterioration is caused by observations of 0 mm, which start to give significant probability to rain when the standard deviation grows beyond 0.1 mm. As many of the 0-mm observations are during cloudless days and in fact almost certain, we might reconsider the simple Gaussian uncertainty model.

View Full Size

The resulting decompositions (reliability not shown) as a function of standard deviation measurement. The growing XES with standard deviation, indicates that for the homoscedastic Gaussian observation uncertainty model, forecast quality is lower than would be estimated assuming perfect observations. The dashed lines show the decomposition for the same observation uncertainty model, with the exception that measurements of 0 mm are assumed to be certain. The almost constant XES in that case indicates that the estimation of forecast quality is robust against those observation uncertainties.

Citation: Monthly Weather Review 139, 7; 10.1175/2011MWR3573.1

• Download Figure
• Download figure as PowerPoint slide

When the uncertainty model is changed to have no uncertainty for 0-mm observations, the decomposition changes significantly (see dashed lines in Fig. 3), leading to a cross-entropy score that is almost constant (sometimes even slightly decreasing) with increasing standard deviation. For this particular error model, the uncertainty in the observations thus hardly affects the estimation of the forecast quality. This gives us confidence that as long as the 0-mm observations are certain, the estimate of forecast quality is robust against Gaussian observation errors with standard deviation up to 0.3 mm. Although in that case there is significant observation uncertainty (bottom dashed line, Fig. 3) that lowers the divergence score, the changes in the cross-entropy score for individual forecasts cancel each other out. Not surprisingly, the robustness of forecast quality estimates depends very much on the characteristics of the observation uncertainty. Further experiments are necessary to determine how to formulate realistic observation uncertainty models and how this can benefit verification practice.

6. Discussion and conclusions