1. Introduction

Rank histograms are used extensively to evaluate ensemble forecast system performance (e.g., Hamill and Colucci 1997, 1998; Hou et al. 2001; Stensrud and Yussouf 2003). Rank histograms were introduced into the climate field by Anderson (1996), and Anderson and Stern (1996) used the Kolmogorov–Smirnov test along with the Anderson–Darling test for comparing samples in seasonal simulation cases. Hamill (2001) shows how to appropriately use rank histograms for evaluating ensemble forecasts. Once biases in individual members are removed and observational error is accounted for (Hamill 2001), the ideal ensemble produces flat, or uniform, rank histograms. Certain deviations from the uniform distribution are bellwether indicators that the ensemble forecasting system is deficient. Depending on the nature of these deviations, the nature of the deficiency may be better defined. For example, a U-shaped distribution indicates the ensemble is underdispersive, a peaked distribution suggests that the ensemble is overdispersive, and a sloped rank histogram indicates that the ensemble remains biased in some way.

Due to random variations, even ideal ensembles will not produce perfectly uniform rank histograms. Hence, one wishes to test the assumption that, within sampling error, the rank histogram is derived from a discrete uniform distribution. Such tests are derived from the general family of goodness-of-fit (GOF) tests, which test the null hypothesis, H₀: the rank histogram is indistinguishable from a discrete uniform distribution. A common test for evaluating whether the resulting rank histograms come from a discrete uniform distribution is the χ² test. But the χ² test is far from ideal and lacks power for small sample sizes. More powerful tests come from the Cramér–von Mises (CvM) family of statistics, specifically the Watson test and the Anderson–Darling test, which are described in section 2. Section 3 discusses the results of applying the different tests to both large and small datasets generated by sampling at random from a uniform distribution. Section 4 provides conclusions and recommendations.

2. GOF tests

The most common GOF test is the χ² test. This is a natural test for rank histograms, which represent data binned by rank. The χ² test is defined as follows by the test statistic, T:

View Expanded

where O_i is the observed frequency in bin i, and F_i is the expected frequency in bin i under the null distribution with k cells. The T statistic for the null distribution is approximately distributed as χ² with k − 1 degrees of freedom.

Sample size is always an issue with GOF tests. In practice, GOF tests have limited value for both very large, and very small, sample sizes, though what constitutes “very large” and “very small” is not usually clear and differs from test to test. If the sample size is large enough, almost any GOF test will reject the null hypothesis because real data are never distributed according to any theoretical distribution (Millard 2002). As the sample size decreases, the power, or ability to detect a difference between the sample distribution and the hypothesized or null distribution (uniform, in this case), of any test suffers, though certain tests are more sensitive than others against particular alternative hypotheses for any given sample size.

For the χ² test, the conservatively defined required sample size is that which would provide an expected count of at least 5 for each bin. Thus, for a 15-member ensemble, the resulting rank histogram has 16 bins and 90 cases are required. However, the χ² approximation for the T statistic remains valid for surprisingly small samples. If N is the number of samples, c is the number of bins, and E_i is the expected frequency in bin i under the null hypothesis, the T statistic is still well approximated by the χ² distribution with c − 1 degrees of freedom if N ≥ 10, c ≥ 3, N²/c ≥ 10, and E_i ≥ 0.25, which means that for a 15-member ensemble the minimum number of cases must be no less than 13. Conover (1999) gives a good treatment of how to compute the required sample size for a χ² test.

Other useful GOF tests exist. A notable example is the Cramér–von Mises (CvM) family of test, which has forms for the discrete uniform distribution (Choulakian, et al., 1994). This family consists of the Cramér–von Mises (Cramér 1928; von Mises 1931; Smirnov 1936), the Watson (Watson 1961), and the Anderson–Darling (Anderson and Darling 1952) tests. In general, the CvM family of GOF tests has more power than does the χ² test for small sample sizes. Unlike the χ² test, the CvM test statistics are nonparametric. The CvM provides nearly identical results as the Kolmogorov–Smirnov (KS) test, though some find its formulation more appealing because the CvM tests use an integrated departure of the empirical distribution function (EDF) from the null distribution, instead of the largest departure (Conover 1999).

Consider a discrete distribution with k cells with probability p_i of an observation landing in any cell. Let o_i be the observed number of counts in bin i, and let, Np_i = e_i be the expected number of counts in bin i under the null distribution. Then, let S_j = Σ^j_i=1o_i, and T_j = Σ^j_i=1e_i. Thus, S_j/N and H_j = T_j/N correspond to the EDF F_N(x). Finally, let Z_j = S_j – T_j, j = 1, 2, . . . , k. Then the discrete form of the CvM statistic is given by

View Expanded

the discrete form of the Watson statistic is given by

View Expanded

and the discrete form of the Anderson–Darling statistic is given by

View Expanded

where Z = Σ^k_j=1Z_jp_j.

By definition, Z_k = 0, so the last term in W² = 0, and the last term in A² = 0/0, which is set to 0. An alternative notation is to extend the index over which the sums operate to only k − 1 in (2) and (4) (Choulakian et al. 1994).

Because the CvM family uses an integrated departure from the EDF, it is order dependent, which means the way the bins are indexed affects the value of the computed statistic. For example, the discrete CvM test statistic has the following form: W² ∼ (O₁ − F₁)² + (O₁ + O₂ − F₁ + F₂)² + (O₁ + O₂ + O₃ − F₁ + F₂ + F₃)² + . . . , and so the order in which the binned values appear affects the statistic’s value. This is also true for the Anderson–Darling statistic. This is only partially true for the Watson statistic, which has a circular dependence. The Watson statistic differs from the other two in that it is invariant with regard to the “starting” cell. Regardless of the start index in (3), as long as all indices are addressed in order thereafter, the Watson statistic is invariant for any given dataset. Hence, the Watson statistic is particularly useful for testing the uniformity of counts around a circle (Choulakian et al. 1994), such as calendar months or wind direction. All three, however, apply to linear data. The distribution theory, and so methods for constructing p values for the CvM, Watson, and Anderson–Darling statistics, are all given in Choulakian et al. (1994) and will not be elaborated upon here.

In contrast, the χ² test is insensitive to the nature of the departure from the null distribution because it uses only the sum of the individual deviations at each bin over all bins. Hence, the χ² test cannot distinguish between noisy departures from the null distribution and U-shaped, peaked, or sloped departures, which are ordered departures. The CvM family is relatively insensitive to random departures but more sensitive to ordered departures from the null distribution, and retains more power against these departures than does the χ² test. This can lead to profound differences between GOF test results based on the χ² test and results based on the CvM statistics. Table 1 shows critical values for the various significance levels for the three CvM statistics (from Choulakian et al. 1994).

3. Examples

Differences between the χ² and CvM test behavior are illustrated using a Monte Carlo simulation that draws random samples from a uniform distribution. Assume an ensemble forecasting system with 15 members, which yields a rank histogram containing 16 bins. Define a small-sample rank histogram as consisting of 60 cases. Thus, for the small-sample simulation, each of 1000 Monte Carlo trials draws 60 numbers uniformly distributed between 1 and 16 (Fig. 1). Define a large-sample rank histogram as consisting of 540 cases. So, the large-sample Monte Carlo simulation uses 1000 sets of 540 numbers uniformly distributed between 1 and 16 (Fig. 2). In the small-sample case, the expected number of counts, E_i, in each cell is 3.75, large enough for both the χ² test and the CvM family of tests to be valid. For the large sample, E_i = 33.75. Each sample is then reordered pathologically to produce a U-shaped, a peaked-shaped, and a sloped bias trend. Hence, each sample results in four possible distributions: random, U shaped, peaked, and sloped. In each case, the χ² test p value remains invariant, but the CvM tests vary widely depending on the nature of the reordering.

For a test at p = 0.05, the expectation is that close to 5% of the samples will result in a p value less than 0.05 for all of these tests simply by random chance before the samples are reordered. By definition, the χ² test p value is independent of order. However, the CvM family yields different p values, depending on how the data are ordered. Table 2 shows the proportion of cases that are associated with p values ≤ 0.05 for each ordering for the small sample size, while Table 3 shows the same results for the large sample size (boldface values are associated with the most sensitive tests).

Clearly, for both sample sizes, the expectation for the random rank histograms is met by all tests. However, for the pathologically reordered rank histograms, the statistics show marked differences. While the χ² statistic is unaffected, the Watson statistic is clearly very sensitive to U-shaped and peaked rank histograms (yielding identical values for both due to circular symmetry), while both the CvM and Anderson–Darling tests are most sensitive to the bias slope rank histograms. These results hold for both the small and large samples.

More insight may be gained by examining how the CvM tests behave relative to the χ² test (Fig. 3). For the random data, the two GOF tests are uncorrelated for 16-bin rank histograms constructed from both 60 cases (Fig. 3a) and 540 cases (Fig. 3c), which means that the individual samples with p values <0.05 differ from the two tests in nearly random ways. However, the χ² test is by definition insensitive for the data reordered into a U shape, while the Watson statistic is clearly very sensitive to this reordering for both the 60- (Fig. 3b) and 540-case samples (Fig. 3d). The χ² test p values are clearly discrete for the small-sample (60 values) cases (Figs. 3a and 3b), which is a result of both the small sample size and the insensitivity to order: with only 60 values, the number of different T-statistic values that can be generated is significantly limited. The Watson test p value rises as the χ² test p value approaches 1 (Figs. 3b and 3d). This is true for all of the CvM family of tests, and indicates that as the χ² test p value approaches 1, the rank histogram approaches exact uniformity. Because the sample size used in the simulations is not a multiple of the number of bins, such a “perfect” rank histogram cannot occur within these data. However, were all bins to contain the same number, the rank histogram itself is invariant under bin-order permutations. The sensitivity of the Watson statistic to U-shaped distributions is demonstrated by how high the χ² test p value may become before the Watson test p value begins to increase (Figs. 3b and 3d). The higher the χ² test p value at which the Watson p value starts to rise, the more sensitive the Watson test statistic is to the particular deviation from the null distribution. Simulations that use sample sizes as large as 5000 and as small as 30 show similar results.

4. Conclusions

The most common problems associated with ensemble forecasting systems are underdispersion, overdispersion, and bias. Rank histograms depict these problems with either a U shape, a peaked center, or one end of the rank histogram being higher than the other (slope), respectively. The ubiquitous χ² test possesses certain characteristics that may make it unsuitable for assessing the quality of rank histograms derived from ensemble forecasting systems. Specifically, the χ² test statistic is order invariant, which means rank histograms that clearly display a problem within the ensemble forecasting system may go undetected using the χ² test.

Better tests for the specific problems encountered with ensemble forecasting systems are available from the discrete form of the Cramér–von Mises family of GOF tests. These tests are based on the Cramér–von Mises, Watson, and Anderson–Darling statistics. Of these three, the Watson test statistic is considerably more sensitive to either U-shaped or peaked rank histograms than are the other two. The other two tests are almost equally sensitive to a slope/bias within the rank histograms. All of the CvM tests retain considerable power for relatively small samples.

Like any statistical test, the CvM tests are not infallible. Yet, better results will be obtained by using a combination of either the Watson and CvM, or Watson and Anderson–Darling tests for evaluating rank histograms, instead of the χ² test. If the rank histogram in question passes either combination of CvM tests at an appropriate p value, then that rank histograms may be considered statistically free from either U-shaped/peaked or biased/sloped deficiencies.