1. Introduction

There is an increasing interest in decadal climate predictions. Successful predictions on daily and seasonal time scales depend crucially on correct modeling of internal variability and therefore require that the model system is initialized with realistic conditions. In contrast, climate model scenarios depend on external forcings such as the atmospheric greenhouse gas concentrations and natural (e.g., from volcanic eruptions) and anthropogenic aerosol loadings. Predictions on decadal time scales may depend on both sources of predictability (see, e.g., Kushnir et al. 2019; Merryfield et al. 2020; Meehl et al. 2021; and references therein) and therefore require inclusion of both initializations and external forcings. See DelSole and Tippett (2018) for a more rigorous discussion of the different kinds of predictability.

Both phase 5 and phase 6 of the Coupled Model Intercomparison Project (CMIP5 and CMIP6) contain initialized decadal predictions for the periods since 1960 in addition to the uninitialized historical experiments. Skill in decadal forecasts is related to slowly evolving parts of the climate system and is mainly found in the North Atlantic region and has been reported for surface air temperature, precipitation, and for the frequency of some extreme events. For details and references, see the summaries in, for example, Kirtman et al. (2014), Kushnir et al. (2019), Simpson et al. (2019), Meehl et al. (2021), and Hermanson et al. (2022). Recently, it has been reported that the predictable signal might be underestimated in models and that the average of very large ensembles might be needed to isolate this signal (Scaife and Smith 2018; Smith et al. 2019, 2020).

For some atmospheric variables, such as temperature, there is a strong influence from greenhouse gases in the last century. Thus, both the initialized and uninitialized experiments contain this forcing and show strong skill related to the general warming. It is often found (see, e.g., Borchert et al. 2021b; Bilbao et al. 2021) that correlations between observations and the ensemble mean of the historical experiments are large and significant in many geographical regions. The same holds for correlations between observations and the ensemble means of the initialized experiments even for decadal lead times. An example is discussed in section 4 (Fig. 8).

To estimate the effect of the initialized experiments, it is therefore important to either remove the forced response from the initialized experiments or to more carefully directly compare the initialized and noninitialized experiments. If the only forcing was the greenhouse gases one could try to remove this part from the prediction by expressing it as low-order polynomial fit or by removing the part linearly congruent to the amount of greenhouse gases. However, the value of this approach is limited by other faster forcings such as volcanic eruptions, the effect of which is harder to identify and remove (Trenberth and Shea 2006; Borchert et al. 2021a).

Attempts to demonstrate significant added skill from initializations for surface temperature on decadal time scales have shown mixed success outside the North Atlantic polar gyre region (Kirtman et al. 2014; Smith et al. 2010). See, for example, the discussions in Doblas-Reyes et al. (2013), Borchert et al. (2021a), and Sospedra-Alfonso and Boer (2020). However, recently Smith et al. (2019) have reported significant added skill from initializations in near-surface temperature, surface pressure, and precipitation using a new statistical approach to estimate the statistical significance.

In this paper, we compare different methods to estimate the statistical significance of the effect of initializations. An important concept is the power of the test, which here is defined as the probability of correctly rejecting the null hypothesis that initialized and noninitialized predictions have the same skill. The methods are all based on the generation of surrogate time series that preserve the temporal characteristics (power spectrum and autocorrelations) of the original series. The methods differ by the statistic considered, that is, the measure used to represent the difference in skill between initialized and noninitialized predictions. While the paper is cast in the context of decadal predictions, we believe that the results will also be useful for other situations where different forecast systems are compared, for example, for seasonal predictions.

In section 2, we describe the method. In section 2a, we define the different statistics, and in section 2b, we discuss the statistical test. In section 3, we look at data generated from simple, idealized models. In section 3a, we take a first look at the behavior of the statistics, while we in section 3b study the power of the tests. In section 4, we use—as an example of a realistic situation—the statistical tests on historical and initialized ensembles from a single climate model. The paper is closed with the conclusions in section 5.

2. The method

In section 2a, we introduce the measures—the statistics—of the added skill from initializations. These statistics are all based on correlations. To estimate the statistical significance, we need to compare an observed statistic to the value it would have had if there were no added skill. However, the measures are stochastic variables and are not characterized by single numbers but by distributions. The statistical significance is therefore estimated by comparing the observed statistic to its distribution when there is no added skill: if the observed statistic falls in the extreme 5% of the distribution (two-sided test) we reject the null hypothesis of no added skill. Note that it is not guaranteed that the actual risk of rejecting a true null hypothesis is equal to this nominal 5%. In section 2b, we discuss how to obtain these distributions.

3. Comparing the tests on idealized data

Here, we compare the different tests using simple numerically generated data.

a. Simple examples

We first briefly look at some simple examples to illustrate the difference between the statistics.

In the first example, we let the contribution of the observations to f and h be identical, ${\sigma }_{{\mathit{\xi }}_{\mathbf{h}}}={\sigma }_{{\mathit{\xi }}_{\mathbf{f}}}=1$ and λ_h = λ_f = 1, thus fulfilling the null hypothesis. We also choose ${\sigma }_{{\mathit{\xi }}_{\mathbf{o}}}=0$ and N = 100. The model then reduces to h = o + ξ_h, and f = o + ξ_f, where o is a normalized first-order autoregression (AR1) series. For the realization shown in Fig. 1, we have cor(h, o) = 0.72 and cor(f, o) = 0.71, so the difference statistic is 0.01. The residual statistic is 0.55, and the split statistic is 0.01 (the statistics are shown with cyan vertical lines in Fig. 1). Thus, while the difference and split statistics are close to zero, this is certainly not the case for the residual statistic. A simple test based on the residual statistic assuming that o|h and f|h are independent will wrongly suggest that the null hypothesis can be rejected.

View Full Size

Cyan vertical lines indicate the values of the statistics for an example where o is a normalized AR1 series with coefficient 0.5, h = o + ξ_h, and f = o + ξ_f, where ξ_h and ξ_f are normalized, Gaussian white noise. This example fulfills the null hypothesis. (left) Difference statistic. (center) Residual statistic. (right) Split statistic. The histograms show the distributions of the statistics under the null hypothesis that h and f are identically distributed as calculated with the surrogate method. The sample size is N = 100. The red vertical lines show the values of the statistics from Eqs. (5) to (7) valid for N → ∞.

Citation: Journal of Climate 36, 9; 10.1175/JCLI-D-22-0605.1

• Download Figure
• Download figure as PowerPoint slide

The distributions of the statistics under the null hypothesis are also shown in Fig. 1. We see that the original values of the statistics fall well inside these distributions, and we can therefore not reject the null hypothesis in any of the three tests.

We now look at another simple example (Fig. 2) where the forecast is superior. We take o as before, but now h = o + ξ_h and f = o + 0.5ξ_f. For this realization we have the difference statistic = 0.17, the residual statistic = 0.81, and the split statistic = −0.03. Now the values for the difference and the residual statistics fall clearly outside the distributions under the null hypothesis as calculated with the surrogate method. However, here the split statistic disagrees, and we will not reject the null hypothesis according to this statistic.

View Full Size

As in Fig. 1, but now h = o + ξ_h, and f = o + 0.5ξ_f, where ξ_h and ξ_f are normalized white noise. This example does not fulfill the null hypothesis.

Citation: Journal of Climate 36, 9; 10.1175/JCLI-D-22-0605.1

• Download Figure
• Download figure as PowerPoint slide

Equations (5)–(7), which are valid for large ensemble sizes, agree well with the value of the statistics from the realizations as is shown with the red vertical lines in Figs. 1 and 2.

From Eqs. (5)–(7), we see that under the null hypotheses $\sigma ={\sigma }_{{\mathit{\xi }}_{\mathbf{f}}}={\sigma }_{{\mathit{\xi }}_{\mathbf{h}}}$ and λ = λ_f = λ_h, the difference and the split statistics are 0 while the residual statistic becomes ${\{(2+{\sigma }^2/{\lambda }^2)[1+(1+{\lambda }^2/{\sigma }^2){\sigma }_{{\mathit{\xi }}_{\mathbf{o}}}^2/{\lambda }_{\mathbf{o}}^2]\}}^{-1/2}$. This will depend nonmonotonously on the signal-to-noise ratio λ/σ when ${\sigma }_{{\mathit{\xi }}_{\mathbf{o}}}$ is different from zero. When estimating the skill of forecasts and historical experiments we often consider ensemble means. As the effective noise amplitude scales as $1/\sqrt{K}$, where K is the ensemble size, this nontrivial behavior is also seen as function of ensemble size.

As a simple example we extend the model in Eq. (4). Now ${\mathbf{h}}_i=\mathbf{s}+\sigma {\mathit{\xi }}_{\mathbf{h}}$ and ${\mathbf{f}}_i=\mathbf{s}+\sigma {\mathit{\xi }}_{\mathbf{f}}$, for i = 1, …, K, where K is the ensemble size, and o = s + ξ_o. In this example, the null hypothesis holds. We consider the statistic $\text{cor}(\mathbf{o}\left|\overline{\mathbf{h}},\overline{\mathbf{f}}\right|\overline{\mathbf{h}})$, where the overline denotes the ensemble mean. Figure 3 shows the residual statistic as function of ensemble size and the noise amplitude σ. The nonmonotonous structure means that we cannot use the statistic to meaningfully estimate the effect of ensemble size. Likewise, we cannot directly compare the statistics in different geographical regions, as these regions may have different amounts of noise. These results are robust and hold for other choices of the signal s.

View Full Size

The residual statistic $\text{cor}(\mathbf{o}\left|\overline{\mathbf{h}},\overline{\mathbf{f}}\right|\overline{\mathbf{h}})$ as function of ensemble size K and noise amplitude σ. The model is ${\mathbf{h}}_i=\mathbf{s}+\sigma {\mathit{\xi }}_{\mathbf{h}}$ and ${\mathbf{f}}_i=\mathbf{s}+\sigma {\mathit{\xi }}_{\mathbf{f}}$, i = 1, …, K, and o = s + ξ_o. Here, s is a normalized AR1 series, and N = 100. Note the nonlinear scales on the axes.

Citation: Journal of Climate 36, 9; 10.1175/JCLI-D-22-0605.1

• Download Figure
• Download figure as PowerPoint slide

b. Power of the tests

The power of a test is its probability of correctly rejecting the null hypothesis. The power depends both on the design of the test and the length of the time series (Sienz et al. 2016; Totaro et al. 2020)—or more precisely on the independent degrees of freedom. A power curve is the power shown as a function of a parameter. The well-known structure of the power curve and its relation to type I and II errors are shown schematically in Fig. 4. A perfect test would have the power close to one when the null hypothesis does not hold, that is, almost always correctly rejecting the null hypothesis and thereby minimizing the risk of type II errors. Likewise, the power curve would have small values when the null hypothesis holds, that is, having only a small risk of erroneously rejecting a true null hypothesis and thereby minimizing the risk of type I errors.

View Full Size

Schematic view of a power curve. The power p is shown as function of a parameter (full black curve). The null hypothesis is fulfilled only when the parameter is zero. Away from zero, p is the probability that the false null hypothesis is correctly rejected (indicated by green hatching). The probability that the false null hypothesis is incorrectly not rejected (type II error) is 1 − p (indicated by red hatching). At zero—where the null hypothesis is fulfilled—the length of the green line, 1 − p, indicates the probability to correctly not reject the true null hypothesis, while the length of the red line p indicates the probability to incorrectly reject the true null hypothesis (type I error).

Citation: Journal of Climate 36, 9; 10.1175/JCLI-D-22-0605.1

• Download Figure
• Download figure as PowerPoint slide

1) The simple idealized model

We now fix ${\sigma }_{{\mathit{\xi }}_{\mathbf{h}}}$, ${\sigma }_{{\mathit{\xi }}_{\mathbf{o}}}$, λ_h, and λ_o, while we let λ_f (or ${\sigma }_{{\mathit{\xi }}_{\mathbf{f}}}$) vary, and for each λ_f (or ${\sigma }_{{\mathit{\xi }}_{\mathbf{f}}}$) we consider many realizations (2500) of o, h, and f. For each realization we calculate the statistical significance with the phase-scrambling method and note if the null hypothesis is rejected (at the 95% level). From the 2500 realizations, we then calculate the power as the rate of rejections.

Power curves—giving the power as function of λ_f—are shown in Fig. 5 for ${\sigma }_{{\xi }_{\mathbf{h}}}={\sigma }_{{\mathit{\xi }}_{\mathbf{f}}}={\lambda }_{\mathbf{h}}=1$, ${\sigma }_{\xi }{}_{{}_o}=0$, and sample sizes N = 25, 100, and 1000. The power is shown with the full curves for all three statistics: difference (black), residual (blue), and split (orange). As expected, we find that for all statistics that the power increases when λ_f moves away from 1. We also see that the power increases with N and in the limit of large N the power converges to 1 when λ_f is different from 1.

View Full Size

The power—the probability of rejecting the null hypothesis—of the statistical tests as function of λ_f for different values of the sample size N: 25, 100, and 1000. Here, h = o + ξ_h and f = λ_fo + ξ_f, where o is a normalized AR1 process, and ξ_h and ξ_f are normalized white noise. This corresponds to Eq. (4) with ${\sigma }_{{\mathit{\xi }}_{\mathbf{h}}}={\sigma }_{{\mathit{\xi }}_{\mathbf{f}}}={\lambda }_{\mathbf{h}}=1$ and ${\sigma }_{{\mathit{\xi }}_{\mathbf{o}}}=0$. Black, blue, and orange full curves are the power for the difference statistic cor(f, o) − cor(h, o), the residual statistic cor(o|h, f|h), and the split statistic cor(f − h, o)σ_f−h/σ_h, respectively. Dashed lines are power calculated with the optimal benchmark method.

Citation: Journal of Climate 36, 9; 10.1175/JCLI-D-22-0605.1

• Download Figure
• Download figure as PowerPoint slide

More interesting in our context, we find that the largest power is found when using the split statistic. Also, the power of the difference statistic is in general a bit larger than the power of the residual statistic. These differences are largest for small N (sample size) and decreases with N. In particular, the difference and residual statistics show almost the same power for N ≥ 100, while they still differ considerably from the split statistic.

Unfortunately, a high power is here connected to a high risk of falsely rejecting a true null hypothesis (type I error). For λ_f = 1, where the null hypothesis holds, the rejection rates for the residual and difference statistics are around 0.2 for N = 25 and decreases toward the nominal value of 0.05 for large N. The situation is worse for the split statistic, which even for N = 1000 is still around 0.3.

Above we calculated the surrogate time series by the phase-scrambling procedure based on given realizations of h and o. This corresponds to realistic situations. In the simple tests we know the process $\mathbf{h}={\lambda }_{\mathbf{h}}\mathbf{s}+{\sigma }_{{\mathit{\xi }}_{\mathbf{h}}}{\mathit{\xi }}_{\mathbf{h}}$ and can produce surrogates directly from this process, as discussed in section 2b. The power calculated this way is shown as dashed curves in Fig. 5. Here, the probability for rejecting a true null hypothesis is the nominal 5% for all N and for all statistics. For small N the difference statistic still has a bit more power than the residual and the split statistics. Compared to this optimal benchmark, the surrogate method based on the phase-scrambling procedure has to estimate the parameter a in Eq. (2) and in general underestimates the power.

In Fig. 6, we show some other examples of the power. In Fig. 6a, we vary ${\sigma }_{{\mathit{\xi }}_{\mathbf{f}}}$ and keep the other parameters constant. While the difference and residual statistics behave very much as when we varied λ_f (Fig. 5), the split statistic totally fails. This is due to the missing sensitivity of ${\sigma }_{{\mathit{\xi }}_{\mathbf{h}}}$, as seen in Eq. (7), and holds also for the benchmark. Figures 6b and 6c show situations when there is also noise in the observations, that is, when ${\sigma }_{{\mathit{\xi }}_{\mathbf{o}}}\ne 0$. Here, we see that the difference statistic behaves very sensibly, while the residual statistic does not have the minimum in power when the null hypothesis holds. As this is not seen for the optimal benchmark, although the difference statistic still has a little more power than the residual statistic, it must be related to a combination of the phase-scrambling method and the statistics.

View Full Size

(a) Power as function of ${\sigma }_{{\mathit{\xi }}_{\mathbf{f}}}$ with h = o + ξ_h and $\mathbf{f}=\mathbf{o}+{\sigma }_{{\mathit{\xi }}_{\mathbf{f}}}{\mathit{\xi }}_{\mathbf{f}}$. Here, o is a normalized AR1 process. (b) Power as function of λ_f with f = λ_fs + ξ_f. (c) Power as function of ${\sigma }_{{\mathit{\xi }}_{\mathbf{f}}}$ with $\mathbf{f}=\mathbf{s}+{\sigma }_{{\mathit{\xi }}_{\mathbf{f}}}{\mathit{\xi }}_{\mathbf{f}}$. In (b) and (c), h = s + ξ_h, o = s + 0.5ξ_o, and s is a normalized AR1 process. The terms ξ_h, ξ_f, and ξ_o are normalized white noise. N = 100 in all panels.

Citation: Journal of Climate 36, 9; 10.1175/JCLI-D-22-0605.1

• Download Figure
• Download figure as PowerPoint slide

2) Including trends

When applying the surrogate method to climate models—and in particular to temperature—we have to consider the trends. Trends are not treated well by the phase-scrambling method (Schreiber and Schmitz 2000) and should not be included in the stochastic part of the process. When calculating the surrogate ${\mathbf{f}}^{*}$, we therefore first detrend o and h and use the detrended values in Eqs. (2) and (3). The trend from h is then added to ${\mathbf{f}}^{*}$. We find that the method in general gives good results when applied to the idealized models. The trend is estimated as a third-order polynomial, but the results are robust to changes in the order. We also note that the results in section 4 are robust to the changes in the order of the polynomial detrending.

The left panels in Figs. 7a and 7c show examples where the time series are generated as in Fig. 5b but now s includes a nonlinear trend. Likewise, the right panels in Figs. 7b and 7d are generated as in Fig. 6b but now with the trend. The trend has the form of a fourth-order polynomial. Not taking special care of the trends in the phase-scrambling gives rather wide power curves (Figs. 7a,b) compared to the situation without trend (Figs. 5a and 6b). Using the detrending procedure described in the previous paragraph improves the results and gives power curves (Figs. 7c,d) closely resembling those of Figs. 5b and 6b. We have confirmed that similar results are obtained with other powers of the true trend.

View Full Size

Including trend in s of the form 6t⁴, with t = (0, 1, … N − 1)/N. Power as function of λ_f. N = 100 in all panels. (left) Parameters are as in Fig. 5b, including ${\sigma }_{{\mathit{\xi }}_{\mathbf{o}}}=0$. (right) As in Fig. 6b, including ${\sigma }_{{\mathit{\xi }}_{\mathbf{o}}}=0.5$. (top) No detrending in significance estimation. (bottom) The third-order polynomial is removed. Legend as in Figs. 5 and 6.

Citation: Journal of Climate 36, 9; 10.1175/JCLI-D-22-0605.1

• Download Figure
• Download figure as PowerPoint slide

4. Climate models

In this section we consider the near-surface temperature from the historical single-model ensemble performed in the Community Earth System Model (CESM) Large Ensemble Simulation (LENS; Kay et al. 2015). We also use the corresponding ensemble of initialized decadal forecasts from the CESM Decadal Prediction Large Ensemble (DPLE; Yeager et al. 2018). Both ensembles have 40 members and use the same code and external forcing datasets. As observations, we use NCEP–NCAR data (Kalnay et al. 1996). Prior to the analysis, model data are interpolated to the horizontal NCEP global grid, 2.5° × 2.5°, using a simple nearest neighbor procedure. Our analyses are based on annual means for the period 1970–2017. As we use annual means we do not expect the choice of reanalysis to be important. To be sure we have confirmed that we get similar results using the JRA-55 (Kalnay et al. 1996). The DPLE experiments are initialized in November every year and a lead time of 1 year will refer to the mean over the first full calendar year. When calculating the statistical significance of the statistics we use the phase-scrambling method described in section 2b amended with the approach described in section 3b(2) dealing with trends.

Figure 8 shows the correlations between the ensemble mean of the historical ensemble and observations and the correlations between the ensemble mean of the initialized decadal prediction ensemble and observations for lead times of 1 and 10 years. There are strong positive and significant correlations between the historical ensemble and observations (top-left panel). The correlations between the forecast ensemble and observations are comparable for a lead time of 10 years and somewhat larger for a lead time of 1 year (bottom panels). The top-right panel in Fig. 8 demonstrates the challenges of isolating the effect of initializations by showing that large and significant correlations between observations and the historical ensemble mean persist even after the linear trends have been removed.

View Full Size

Results for LENS annual-mean near-surface temperature. (top left) Correlations between observations and historical ensemble mean. (top right) As in the top left, but now time series have been detrended. (bottom) Correlations between observations and the initialized ensemble mean for lead times of (left) 1 year and (right) 10 years. The period is 1970–2017. Dots indicate where correlations are significantly different from zero at the 95% level.

Citation: Journal of Climate 36, 9; 10.1175/JCLI-D-22-0605.1

• Download Figure
• Download figure as PowerPoint slide

In Fig. 9, we show the three statistics for a lead time of 1 year. The difference statistic shows large values only in the eastern Pacific and in the Atlantic subpolar gyre region. In contrast, the residual statistic is large in more extended areas. However, the two statistics rejects the null hypothesis—that historical and initialized experiments have the same skill—in many of the same regions: the eastern Pacific [the region of the El Niño–Southern Oscillation (ENSO)] and the subpolar gyre region. The split statistic is statistically significant in even larger areas. Note that there are many areas where the statistics are negative and significant, indicating that the historical ensemble has more skill than the initialized ensemble. Figure 10 shows the situation for a lead time of 10 years. Again the values for the residual statistic is larger than values of the difference statistic, but the two statistics mainly agree on the regions where the null hypothesis can be rejected. This is basically in the subpolar gyre region, as also found in other studies, but also in the polar regions, although here the significant correlations may be of different sign. Again, the split statistic is significant in larger areas.

View Full Size

The three statistics measuring the difference in skill between the initialized ensemble-mean forecast and the uninitialized historical ensemble mean for annual-mean near-surface temperature. (top left) The difference statistic, (top right) the residual statistic, and (bottom left) the split statistic. Based on LENS ensemble means for 1970–2017. A lead time of 1 year is used for the initialized forecast. Dots indicate where the null hypothesis—that historical and initialized experiments have the same skill—can be rejected at the 95% level.

Citation: Journal of Climate 36, 9; 10.1175/JCLI-D-22-0605.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

As in Fig. 9, but for a lead time of 10 years.

Citation: Journal of Climate 36, 9; 10.1175/JCLI-D-22-0605.1

• Download Figure
• Download figure as PowerPoint slide

The left panel in Fig. 11 shows the fractional area of significant correlations for the three statistics as function of lead time. For a lead time of 1 year the difference statistic is significant in 60% of the globe and for a lead time of 10 years this fraction is reduced to 15%. The similar fractions for the residual statistic are 50% and 20% and for the split statistic 78% and 33%. While these fractions even for a lead time of 10 years are larger than the nominal 5%, they are not impressive when compared to the results in Fig. 5. There we found that the risk of rejecting a true null hypothesis is around 10%–20% for time series of length 25–100. Furthermore, the area of significant positive values of the statistic is only larger than the area of significant negative values for lead times less than 3–4 years (right panel in Fig. 11). This holds for both the difference and the residual statistics and indicates that the significance for larger lead times is due to chance.

View Full Size

(left) The fraction of the global area where the skills in forecast and historical experiments are estimated to be significantly different. The fraction is shown as a function of lead time for each of the three statistics. (right) The fraction of the global area where the skills are significantly different calculated for grid points with positive and negative correlations, respectively.

Citation: Journal of Climate 36, 9; 10.1175/JCLI-D-22-0605.1

• Download Figure
• Download figure as PowerPoint slide

5. Conclusions

Attempts to find out if initialization improves forecasts as compared to uninitialized forecasts are hampered by the skill coming from the shared forcings. The skill from the forcings can be large, in particular for the last decades, due to the increased levels of greenhouse gases and the added skill from initialization is often relatively small. It is therefore important to carefully estimate the statistical significance of the added skill.

Previous literature has used different correlation-based statistics measuring the added skill from initialization. The simplest statistic is the difference in correlations between observations and the initialized experiment and the correlation between observations and the uninitialized experiment. Another statistic is the partial correlation between observations and the initialized forecast given the uninitialized forecast, that is, the correlation of the residuals after the contributions linearly congruent to the uninitialized forecast have been removed from the observations and the initialized forecast. The third statistic is based on the correlation between the difference of the initialized and uninitialized experiments and the observations.

We have studied the properties of the three statistics and used a Monte Carlo procedure to test the statistical significance. The Monte Carlo procedure is based on surrogate data with the same temporal characteristics as the original time series and the same relation to the observations. We first considered examples based on a simple model to study the power of the tests and then considered decadal forecasts with the LENS climate model. For the simple model, we derived analytical expressions of the statistics in the limit of large ensemble size.

Our main findings from the idealized model are as follows:

• The residual statistic is in general—also in the limit of large samples—not zero under the null hypothesis that initialized and uninitialized forecasts are drawn from the same process. Furthermore, the value of the residual statistic does not vary monotonously with signal-to-noise ratio. And when we consider ensemble-mean forecasts, it therefore does not vary monotonously with the ensemble size (Fig. 3).

• The split statistic does not recognize differences in the signal-to-noise ratio but only differences in the absolute level of the signal. This statistic therefore fails in some circumstances, but it is worth noting that if h and f are normalized, then the split statistic becomes identical to the difference statistic [Eq. (1)].

• The power of the difference and residual statistics are comparable when observations are noise free. When observations are noisy, the difference statistic is superior.

• We also calculated a benchmark for the power based on the full information from the idealized model. The problems for the split statistic still remain. The difference and the residual statistics are still comparable when observations are noise free. However, when observations are noisy the difference between these statistics has been reduced, although the difference statistic is still behaving somewhat better.

• The rejection rate when the null hypothesis is true is larger than the nominal value for small lengths of the time series but gets closer to the nominal value for long time series. This holds for all three statistics and the problem is quite severe for realistic sample lengths up to 100.

• Secular trends cannot be directly dealt with by the phase-scrambling (or bootstrap) procedure. We find that generating surrogates from detrended data and then adding the trend back to these surrogates gives good results.

Note that the first bullet point means that the residual statistic cannot be compared across different geographical regions with different signal-to-noise ratios.

From the climate model analysis, we find the following:

• When comparing the near-surface temperature in initialized decadal forecasts with uninitialized historical simulations, we recognize the larger values of the residual statistic over the difference statistic. However, the two statistics show mainly statistical significance in the same regions. These results are not unexpected considering the results from the simple examples.

• For the near-surface temperature, the initialized forecasts add significant skill for lead times of a few years, in particular in the regions influenced by the ENSO. For decadal lead times, the area of statistical significance is consistent with the null hypothesis (no added skills from the initializations) when the results from the simple examples are taken into consideration. This interpretation is supported by the fact that for large parts of the significant areas, the statistics are negative, indicating that initialization has deteriorated the predictions.

Based on our results we recommend using the difference statistic while being aware of the risk of the overestimation of the significance that can come from the considerable risk of falsely rejecting a true null hypothesis.

The discussion in this paper has mainly been in the context of decadal forecasts but is also relevant for other situations where different forecast systems are compared. In particular, seasonal forecasts share many of the same challenges as decadal forecasts as the skill depends both on forcings and initial conditions.

Data availability statement.

The CESM Large Ensemble Project (http://www.cesm.ucar.edu/projects/community-projects/LENS/) was downloaded via EFGS from http://www.cesm.ucar.edu/projects/community-projects/LENS/data-sets.html. The CESM Decadal Prediction Large Ensemble Project (http://www.cesm.ucar.edu/projects/community-projects/DPLE/) was downloaded via EFGS from http://www.cesm.ucar.edu/projects/community-projects/DPLE/data-sets.html. The NCEP–NCAR Reanalysis data were provided by the NOAA–CIRES Climate Diagnostics Center, Boulder, Colorado, from their website at http://www.cdc.noaa.gov/.