1. Introduction
Predictability is a measure that refers to the extent the future evolution of a dynamical system can be anticipated. High predictability refers to the scenario when the future state of the dynamical system can be well anticipated. Conversely, low predictability refers to the scenario where the future state of the dynamical system cannot be pinned down among a wide range of possibilities. Predictability is an inherent property of the dynamical system.
Predictability in the context of the earth system refers to either (i) the extent its future evolution can be anticipated as an initial value problem, or (ii) the extent external forcings affect various statistical properties (e.g., mean or variance) of the probability density function that characterize various components of the earth system. In the context of weather and climate predictions, the components of the earth system are ocean, atmosphere, land, etc. The predictability as an initial value problem is generally referred to as “predictability of the first kind” while predictability due to the influence of boundary conditions is referred to as “predictability of the second kind” (Lorenz 1965).
In practice, predictions of various components of the earth system are based on general circulation models (GCMs) (Charney et al. 1950; Phillips 1956; Shuman 1989; Ji et al. 1994; Lynch 2008; Meehl et al. 2014). GCM-based ensemble prediction techniques (Molteni et al. 1996; Toth and Kalnay 1997) aim to realize the inherent predictability in the earth system. The realization of predictability associated with the delivery of forecasts to the users can be quantified using various measures of prediction skill [for different measures skill see Wilks (1995)] that are based on comparisons of forecast and the observed states done over a sample of forecasts. The ultimate goal for the forecasts, and forecast systems is to fully realize inherent predictability as prediction skill.
In the context of our discussion, it is important to distinguish between the inherent predictability and prediction skill. Inherent predictability is the property of the observable system. Prediction skill, on the other hand, (i) relies on a forecast system’s (e.g., GCM-based ensemble prediction) attempts to anticipate the future evolution of the components of the earth system to be predicted, and (ii) is an assessment of the correspondence between the anticipated future state and what is actually observed. Because of errors in the analysis from where the forecast begins (Kumar and Murtugudde 2013), and because of biases in forecast systems, prediction skill is generally lower than the corresponding predictability. As a consequence, the inherent predictability may not be fully realized as prediction skill. In the subsequent discussion, to differentiate between predictability and prediction skill by not using two similar sounding words, prediction skill is referred to as forecastability (Hamill and Kiladis 2014).
For long-range predictions—seasonal, decadal—it is often the case that forecastability is marginal (Peng et al. 2000, 2012; Teng et al. 2011; Kim et al. 2012; Goddard et al. 2013; Doblas-Reyes et al. 2013). Faced with marginal forecastability, a possibility that it may be a consequence of an observable system with low inherent predictability is sometimes overlooked, and it is stated that an increase in ensemble size will lead to improvements in forecastability. Although it is a well-known analytical result that with increasing ensemble size, and for predictions based either on deterministic or probabilistic information, the forecastability increases and eventually asymptotes to inherent predictability (Kumar and Hoerling 2000; Kumar et al. 2001), the general scope of the supposition about the need for larger ensemble size is generally not clear. For example, whether the statement merely refers to taking advantage of the known relationship between larger ensemble sizes and an increase in forecastability, or refers to a sentiment that no matter what the inherent predictability limit may be, a general increase in forecastability can be achieved by the use of larger ensemble, cannot be readily discerned. For example, Beraki et al. (2014, p. 1729) state that “…This suggests that there is still room for further improvement by simply increasing the ensemble size of the OAGCM integrations.” In reading through their discussion leading to this assertion, the precise context of the statement is unclear. Other instances of statements along a similar line (i.e., for a desire to increase in the ensemble to improve forecastability) can also be found in Daron and Stainforth (2013) and Scaife et al. (2014).
Another context where a desire for an increase in ensemble size is often stated is for establishing statistical significance of responses to external forcings (e.g., atmospheric response to interannual variability of sea surface temperature or to increases in CO2). Although by increasing ensemble size an arbitrarily low-amplitude atmospheric response can be made to pass statistical significance tests, utility of small-amplitude response for anticipating evolution of the earth system, and its usefulness for making skillful prediction still remains marginal. Examples for a desire for an increase in ensemble size to establish statistical significance of a response can also be found: “…Large numbers of model runs or ensembles are likely required to achieve statistically significant responses to forced sea-ice changes…”(Cohen et al. 2014, p. 631), or Suckling and Smith (2013) and Materia et al. (2014). In summary, a call for larger ensembles, either with the desire to improve forecastability or to establish statistical significance, implicitly ignores the possibility that the very need for large ensembles may also point to low predictability.
The problem with statements about a desire for a larger ensemble size when confronted with low forecastability likely stems from the fact that the concepts related to the interplay between ensemble size and increase in forecastability, or inherent predictability being the upper bound of forecastability, or requirements for ensemble size necessary to capitalize inherent predictability as forecastability, have not been integrated (or have not been internalized) as conventional wisdom. The goal of this paper is to reemphasize and discuss some known aspects on the requirements for ensemble size, its relationship with inherent predictability, and more importantly, if the upper limit of forecastability is constrained by predictability then what one may stand to gain by an increase in ensemble size of predictions.
Another concept that has not been appreciated well enough is that the ensemble size (required for capitalizing predictability as forecastability) and predictability are complementary variables. As will be discussed, the concept of complementarity alludes to the fact that a perceived requirement (or an aspiration) for predictions based on large ensembles also implies smaller inherent predictability. In this paper, we framed our discussion in the context of seasonal predictions, however, similar concepts also extend to predictions for other time scales.
2. Complementarity of ensemble size and predictability
We begin our discussion with highlighting some key concepts that underlie the practice of seasonal predictions:
Seasonal climate prediction requires estimating the future state of the probability density function (PDF) for the variable that is to be predicted;
Characterization of the quantity to be predicted by its PDF is a fundamental feature of seasonal predictions, and is necessitated by the fact that starting from a narrow distribution of initial states, widely different outcomes of future climate states are likely (Kumar et al. 2013). This spread in prediction comes from increasing divergence of initial conditions with increasing lead time of prediction (Kumar and Murtugudde 2013).
The methodology of ensemble prediction, whereby individual prediction in the ensemble starts from the perturbed initial conditions, is an attempt to infer the future characteristics of the PDF for the variable to be predicted.
For the particular event that is being predicted (e.g., seasonal mean height anomaly for the upcoming season), it is the differences between the climatological PDF and the PDF inferred from the ensemble of predictions, or lack thereof, that determine the limits of predictability. If the climatological PDF and the predicted PDF are the same, predictability is zero, and no information about the future state different from climatology can be inferred.
A convenient way to quantify predictability for a particular forecast event is the signal-to-noise ratio (SNR). SNR quantifies the difference in the first moment between the climatological and the predicted PDF relative to the spread of the predicted PDF (Kumar and Hoerling 1995). The SNR can be mapped on the expected value of skill measures, for example, anomaly correlation (Kumar and Hoerling 2000; Sardeshmukh et al. 2000) or rank probability skill score (Kumar et al. 2001), among others. It can be shown analytically that low (high) SNRs correspond to low (high) expected value of prediction skill. This particular definition of the SNR quantifying difference between the forecast and climatological PDF keys on the event-to-event changes in the first two moments of the PDF—mean and its spread—and is used to frame the subsequent discussion. More general definitions of predictability that take into account the full PDF can also be formulated (Kleeman 2002; Abramov et al. 2005), but lose the conceptual simplicity of the SNR.
Based on the knowledge of the forecast PDF and its differences relative to the climatological PDF, predictions are cast in terms of the probability of various possible outcomes.
The concept of ensemble size required to capitalize on inherent predictability, and the notion of complementarity between the two arises from how large an ensemble is required to realize predictability as forecastability (or prediction skill; i.e., for different SNRs what ensemble sizes bring the forecastability close to the limits imposed by the predictability). This concept can be understood with examples of PDFs of two contrasting scenarios: one for which predictability is high and the other for which predictability is low.
The case of a meteorological variable with high predictability is illustrated in Fig. 1. One of the PDFs is the climatological PDF (blue curve) relative to which prediction for the same variable (red curve) for a specific event is made. The term “specific event” refers to a season in a particular year (e.g., seasonal mean height anomaly for December–January–February 2013/14). In section 4, using data from state-of-the-art GCM simulations, we will demonstrate that such examples of well separated PDFs indeed occur in the analysis of seasonal climate variability.
PDFs for a prediction scenario with high inherent predictability. The blue curve is the PDF of a variable relative to which the prediction for a particular event is made (climatological PDF). The red curve is the predicted PDF. Red dots are outcomes of individual members in the ensemble; the green dot is the observed outcome for which the prediction is made. The relative difference between two PDFs determines the inherent predictability. Prediction of seasonal mean tropical atmospheric variability due to year-to-year variations in sea surface temperatures is an example of a high-predictability regime.
Citation: Monthly Weather Review 143, 8; 10.1175/MWR-D-15-0022.1
In the prediction problem illustrated in Fig. 1, the climatological and the predicted PDFs are well separated, the relative difference is large, and the SNR and the corresponding inherent predictability are high. As will be demonstrated in section 4, the seasonal mean variability of tropical upper-level heights forced by year-to-year changes in sea surface temperatures (SST) is one such example (Kumar and Hoerling 1995; Peng et al. 2000).
The ensemble prediction approach is used to estimate the PDF of the variable for the season to be predicted (the red curve), and realization of the inherent predictability then depends on how well relative differences between the two PDFs can be inferred. We note that various points on the PDF in Fig. 1 are all viable outcomes for the future state that the seasonal mean value of the variable can have.
For two well separated PDFs, as in Fig. 1, a small ensemble is sufficient to infer that relative to the climatological PDF the forecast PDF is different. This is not to say that a small ensemble will resolve the forecast PDF in an absolute sense (i.e., its mean and spread can be estimated well), but a small ensemble can still provide ample indication that it is well separated from the climatological PDF. As a consequence, a prediction with high confidence can be made that the (future) observed value will be toward the right of the climatological mean. Therefore, although in an absolute sense a small size ensemble may not adequately resolve the predicted PDF and provide accurate estimates of various characteristics of the PDF, the prediction based on a small ensemble can still have forecastability approaching the limits of inherent predictability. Kumar and Hoerling (2000) demonstrated that with the SNR exceeding 1, an ensemble size of ~10 is sufficient for forecastability to approach its maximum expected value.
For high SNRs and for forecasts based on small ensemble sizes, the physical reason that forecastability approaches its maximum value is the following: even with the information based on a small ensemble, a high confidence can be placed that the observed outcome will be toward the right of the climatological mean. This is so because almost all ensemble members have high probability for being on the right of the climatological mean. Further, the chances are also good that the corresponding observation (which is also a point on the predicted PDF and is denoted by a green dot) will also be toward the right of the climatological mean. If the forecast is repeated over a large sample of events with high SNR scenarios and verified against observations, the forecastability will be high, and will be close to its upper limit (Kumar and Hoerling 2000).
In Fig. 1, for the sake of convenience, we chose the spread of the climatological PDF to be the same as for the forecast PDF. For seasonal climate variability this assumption, however, is not far from reality. There is evidence that for seasonal climate variability the spread of the PDF changes little from year to year (Tang et al. 2005, 2007; Kumar and Hu 2014; Chen and Kumar 2015). The spread of climatological PDF, which is a weighted average of PDFs of individual events, is not influenced much by the events that have high SNRs (but at the same time, also are less likely to occur and thus contribute little to the spread of the climatological PDF). Real case examples discussed in section 4 will conform to the validity of the constancy of spread approximation.
The contrasting case of the low-predictability scenario is shown in Fig. 2 and corresponds to when there is a considerable overlap between the climatological PDF (blue curve) and the predicted PDF (red curve). For this, the relative difference between these two PDFs is small, and the SNR and the corresponding inherent predictability is low. Again, as will be demonstrated in section 4, seasonal mean variability in extratropical upper-level heights is an example of this (Kumar and Hoerling 1995; Peng et al. 2000).
As in Fig. 1, but PDFs for a prediction scenario with low inherent predictability. Prediction of seasonal mean extratropical atmospheric variability due to year-to-year variations in sea surface temperatures is an example of a low-predictability regime.
Citation: Monthly Weather Review 143, 8; 10.1175/MWR-D-15-0022.1
In the case of Fig. 2, a small ensemble is insufficient to assess the relative difference between the climatological and the predicted PDFs. If the sampling of a PDF is limited by a small ensemble size, one may not be sure if the expectation for the future observed value of the variable is to be toward the right or is to the left of the climatological distribution. By increasing the ensemble size, one will get a better estimate for the probability for the variable to be on the right of the climatological mean. For prediction of a small SNR event, a large ensemble is required; however, as forecastability reaches its maximum value with increasing ensemble size, it is still constrained to be low at the start.
For small SNRs the physical reason for low forecastability even with the use of large ensembles is the following: even though using large ensembles, one is able to assign a low probability for the observed outcome to be on the right of the climatological PDF; however, an individual observed value for which the forecast is made and that would be a particular point on the PDF, does not have to be on the right side of climatological mean on an event-to-event basis. If the forecast is repeated over a large sample of low SNR events, the forecastability, even with the use of large ensembles will still be low and will be close to its (low) upper limit (Kumar and Hoerling 2000). These concepts of the expected value of forecastability as a function of relative differences in the PDFs have been quantified for deterministic and probabilistic climate predictions (Kumar and Hoerling 2000; Kumar et al. 2001; Kumar 2009).
Two scenarios of high and low predictability lead to contrasting requirements for the ensemble size to capitalize the inherent predictability. It has been shown (Kumar and Hoerling 2000; Kumar et al. 2001) that for high SNRs and high-predictability scenarios, predictions based on a small ensemble size are sufficient to realize inherent predictability, while events with low SNRs and low predictability require a larger ensemble size. An implication is that when confronted with low prediction skill if a need for a large ensemble is perceived, it is very likely that low skill was due to a low-predictability limit. If the predictability were to be high, one would not have required large ensemble sizes to realize that as forecastability. The inherent predictability and requirements on ensemble size to capitalize inherent predictability into forecastability, therefore, are complementary variables—when one is low, the other is high and vice versa. Similar conclusions have been reported in several studies (Brankovic and Palmer 1997; Déqué 1997; Taschetto and England 2008); however, as mentioned earlier, the significance of these results has not been well appreciated or has not become accepted as conventional wisdom.
3. Discussion and implications of simple model results
The analysis of Kumar and Hoerling (2000) and Kumar et al. (2001) also quantified the gain in forecastability with increasing ensemble size and how it depends on the level of predictability. For systems with high predictability, forecastability based on small ensembles is close to the predictability limit, and further gains in forecastability with ensemble size are small. The authors also demonstrated that for systems with low predictability, although an ever increasing ensemble size leads to incremental increase in forecastability, it still asymptotes to a low value. This is because the inherent predictability is low to begin with, and an increase in forecastability cannot be gained by an increase in ensemble size. Indeed, Kumar and Hoerling (2000, p. 261) stated that “Although GCM ensemble methods are needed to correctly estimate the observed composite,” (or the PDF of the forecast event) “the very necessity for larger ensemble sizes to do so is a testimony for diminishing predictability in the system.” In an extreme case, when a system has zero predictability (and climatological and predicted PDF are identical), an ensemble of any size cannot add to the expected value of zero forecastability.
What then are the advantages to predictions based on large ensembles? The answer resides with the decision-making perspective of the use of probabilistic prediction information. Given predicted probabilities for various climate outcomes, decision-makers use the probability information. The decision process involves considering cost–benefit scenarios associated with various possible climate outcomes and knowing the probability associated with them (Hammer 2000; Msangi et al. 2006; Kumar 2010).
A key for the decision-making process is a trust by the decision-maker that the forecast probabilities are also a reflection of the corresponding observed outcomes, that is, forecast probabilities are reliable in that they are consistent with the observed frequency. For forecast systems with small biases, forecast probabilities are indeed reliable (Peng et al. 2012). If the forecast system has large biases, it could result in unreliable forecasts in that forecast probabilities could either be an over or an under prediction of the observed frequency. Unreliable forecasts, however, can be calibrated to ascertain reliability (Doblas-Reyes et al. 2005; Johnson and Bowler 2009; Dutton et al. 2013; Schepen et al. 2014) and are not considered further in the discussion below.
For prediction based on small ensembles, the following issue needs to be confronted: for small ensembles, and for low-predictability scenarios, although in the long run, one is constrained to have low forecastability, the forecast probabilities may have large fluctuations from their true values. On an event-to-event basis this may lead to decision-making that is based on inaccurate information about the probabilities of different outcomes. As the ensemble size increases, although the forecastability still remains low due to the constraint of low inherent predictability, forecast probabilities get closer to their true probability. We illustrate this following a simple Monte Carlo approach of sampling forecasts states from PDFs that correspond to different SNR scenarios. This approach allows us to quantify the influence of ensemble size to the extent forecast probabilities can differ from the corresponding correct probability.
The first step in illustrating how small ensemble sizes influence forecast probabilities to depart from their actual value, and what is the dependence on predictability, is to outline the methodology. A general practice for seasonal prediction is to divide climatological PDFs into three categories: above normal, normal, and below normal (Peng et al. 2012; Graham et al. 2011). Assuming a climatological PDF for a variable to be a normal distribution with zero mean and unit standard deviation (corresponding to the blue curve in Figs. 1 and 2), the category boundaries are defined as those that divide the entire PDF into three regions with each region having a probability of occurrence for the variable to fall in as 33.33%. The forecast is to assign probabilities for forecast variable to fall into below-normal, normal, and above-normal categories.
Based an ensemble of forecasts, forecast probabilities for each category can be obtained as the fraction of forecasts that fell into each category. This can be simulated using the Monte Carlo approach of drawing samples from the forecast PDF (Fig. 1, red curve).
The following provides a summary of the Monte Carlo approach:
Boundaries for above-normal, normal, and below-normal categories are defined based on a climatological PDF (Figs. 1 and 2; blue curve) such that the probability of occurrence for each category is 33.3%. The climatological distribution is assumed to be normal with zero mean, and unit standard deviation.
We also assume that the forecast PDF (red curve) also has a normal distribution with unit standard deviation. The mean of the forecast PDF, however, is different from that for the climatological distribution.
The shift in the forecast PDF is proportional to the SNR with larger (smaller) shifts corresponding to higher (lower) SNRs. The shift in the forecast PDF relative to the climatological PDF also determines the true forecast probabilities for each category.
To mimic the ensemble forecasting approach using GCMs, a sample of n is randomly drawn from the forecast PDF. Conceptually, this sample corresponds to an ensemble of n forecasts.
Forecast probabilities for each category are computed based on what fraction of samples fell into respective categories.
Next, one can draw repeated samples of size n, and for each sample compute the forecast probabilities.
Finally, above procedure can then be repeated for different SNRs that correspond to different magnitudes of shift in forecast PDF from the climatological PDF.
Because of the randomness associated with samples drawn from the forecast PDF, forecast probabilities change from one forecast ensemble to another. The degree of variation in forecast probabilities depends on the ensemble size n. As the ensemble size n increases, forecast probabilities for each sample of forecast ensemble converges to the true forecast probability (see step 3). The Monte Carlo approach to understand dependence of the SNR on limits of predictability, and the influence of ensemble size on forecastability, etc. has been used earlier (Kumar and Hoerling 2000; Kumar et al. 2001).
For finite forecast ensemble of size n, we compute the difference between true forecast probability and the forecast probability based on an ensemble of size n for the above-normal category. There is nothing special about our choice of the above-normal category, and a similar analysis could be done either for the normal or the below-normal category. For different SNRs, the expected value of the square of (i) the difference between true probability and forecast probability and (ii) normalized by the true probability as a function of ensemble size n is shown in Fig. 3. This quantity, referred to as the root-mean-square probability error (RMSPE), could be considered a relative measure of error in the forecast in that it is larger (smaller) if the forecast probabilities deviate farther (or stay closer) to the true forecast probability.
The root-mean-square probability error (RMSPE, y axis) of the forecast probability for different ensemble sizes (x axis). RMSPE is computed as the root-mean-square of the difference between forecast probability based on a sample of ensemble size n, and the true forecast probability, and is normalized by the true probability. The four different curves are for different values of true probability for the seasonal mean to be in the above-normal category, with higher values of true probability indicating larger inherent predictability.
Citation: Monthly Weather Review 143, 8; 10.1175/MWR-D-15-0022.1
For small SNRs (corresponding to small shift in the PDF or smaller values of true probability to be in the above-normal category), and for small forecast ensemble size, the expected value of RMSPE can be large. For example, for the true probability P = 0.4 when the forecast PDF is shifted slightly to the right of the climatological PDF, and further, climatological and forecast PDFs have considerable overlap, the RMSPE is as large as the true probability of 0.4. For a larger size ensemble the RMSPE decreases. This indicates that for small SNRs and for a small ensemble size, variations in forecast probability for the above-normal category, on a case-by-case basis can differ appreciably from their true probability of 0.4. Such variations have the potential to adversely affect the decision-making process due to false probabilities assigned to different categories. For a larger shift in the PDF corresponding to larger SNRs and larger value of true probability (e.g., P = 0.9), the RMSPE, even for a small ensemble size is much lower, indicating that forecast probabilities are much closer to their true probability. This behavior is consistent with an implicit expectation that for large separation in PDFs, even for predictions based on small ensembles, forecast probabilities will reach their true value, and RMSPE will be close to zero.
The analysis of RMSPE indicates that for smaller SNRs, predictions based on larger ensembles have the advantage of forecast probability being close to the true value, even though a large ensemble does not lead to improvements in forecastability beyond the low inherent predictability limits. With forecasts based on large ensembles, a decision-maker, therefore, need not be concerned that forecast probabilities for various outcomes may be false. For low-predictability scenarios, even though the difference in the average level of forecastability based on small and large ensemble size may not be much (Kumar and Hoerling 2000), prediction probabilities based on large ensembles do convey the right expectations for predicted events. As discussed by Kumar (2010) false expectations about the probabilities of outcomes in decision-making is one of the foremost impediments in realizing the value of long-range predictions.
4. Validation based on GCM simulations
In this section using GCM simulations, we provide a validation of results based on the conceptual model discussed in sections 2 and 3. The GCM used is the atmospheric component of the National Centers for Environmental Prediction (NCEP) Climate Forecast System version 2 (CFSv2) (Kumar et al. 2012; Saha et al. 2014). For the GCM simulations described here the GCM was run at T62 spectral resolution corresponding to approximately 2° longitude–latitude resolution and with 64 levels in the vertical. The GCM simulations were forced by the observed evolution of SSTs (the so-called AMIP simulations) and were initiated on 1 January 1957 and integrated up to 31 August 2014, and therefore, span a 57-yr period. The analysis is based on an ensemble of 101 simulations starting from slightly different atmospheric initial conditions. Instead of using data from seasonal hindcasts (Kumar et al. 2012), we used the AGCM simulations because of a large ensemble size of 101. For the sake of illustration, we focus our analysis on the December–January–February (DJF) seasonal mean. Over the analysis period extending from 1957 to 2013 a sample of 5757 DJFs (57 years × 101 simulations) was available.
It is well known that sea surface temperature variability in the tropical equatorial Pacific associated with El Niño–Southern Oscillation (ENSO) is the largest factor in influencing seasonal atmospheric and terrestrial variability and the source of prediction skill (Trenberth et al. 1998; Hoerling and Kumar 2002). The influence of ENSO SST variability comes via its influence on the PDF of seasonal means of the atmospheric state. To illustrate the validity of the concepts discussed in section 3, our analysis is presented in the context of ENSO SST variability and its influence on seasonal atmospheric means. Shown in Fig. 4 is the spatial pattern of 200-mb (1 mb = 1 hPa) DJF seasonal mean height variability regressed against the Niño-3.4 SST index. The regression is based on all 5757 simulated DJFs and the spatial structure of the regression is the familiar ENSO teleconnection pattern (Trenberth et al. 1998; Hoerling and Kumar 2002).
Spatial patterns of regression between 200-mb seasonal mean DJF heights (m) and Niño-3.4 SST index. Regression is computed based on atmospheric general circulation model simulations over the period 1957–2014. The boxes in the tropical and extratropical Northern Hemisphere are the regions over which the PDFs of seasonal mean DJF height for two extreme El Niño events of 1982/83 and 1997/98 are analyzed and compared with corresponding climatological PDFs.
Citation: Monthly Weather Review 143, 8; 10.1175/MWR-D-15-0022.1
To illustrate the validity of PDFs shown in Figs. 1 and 2 we select two extreme El Niño years: 1982/83 and 1997/98 DJFs. For these two years there is a sample of 202 DJFs from model simulations. Based on 202 samples, we compare the PDFs for a composite of two El Niño events against the climatological PDF of 200-mb seasonal mean heights. The climatological PDF is estimated based on the entire sample of 5757 DJFs. The climatological PDF and PDFs for the two El Niños are estimated for a set of points over contrasting geographical regions. The first region is over the tropics covering 20°S–20°N, 180°–240°E and contains grid 425 points. This region is located in the tropics where regression with the Niño-3.4 SST index is highly positive (Fig. 4). To construct the PDFs we pool all the DJFs over the 425 grid points in the domain and estimate the respective PDFs. For El Niño this results in a sample of 425 × 2 × 101 DJFs while for climatological PDF there is a sample of 425 × 57 × 101 DJFs.
The second choice of geographical region covers 35°–50°N, 190°–220°E and has 91 grid points. This region is located in the Northern Hemisphere extratropics and also corresponds to the region where ENSO regression is relatively large (but is negative) (Fig. 4). As will be evident from the PDFs, the contrasting feature between the two regions is that while for the tropical region the spread in the PDFs is low and climatological and El Niño PDFs are well separated, for the extratropical region, spread in PDFs is high, and there is considerable overlap between the climatological and the PDF for the two El Niño events.
Shown in Fig. 5 are the climatological PDF (blue) and El Niño PDF (red) for 200-mb seasonal mean heights over the tropical region. The two PDFs are well separated, and the separation can be attributed to the influence of El Niño. Further, the characteristics of two PDFs are similar to the conceptual PDFs in Fig. 1. The spread of the El Niño PDFs is only slightly smaller than for the climatological PDF and is consistent with the earlier discussion that differences in the spread of seasonal means from one year to another is generally small, and climatological spread is a good approximation of spread for individual events.
PDFs for DJF seasonal mean 200-mb height (m) for climatology (blue) and for El Niño years (red). These PDFs are computed over the points within the tropical box in Fig. 4, and correspond to a composite of El Niño events of 1982/83 and 1997/98. Because the regression between 200-mb and Niño-3.4 SST anomaly over this region is positive, the PDF for the El Niño years is shifted to the right. The estimated p value for the above-normal category is 0.99.
Citation: Monthly Weather Review 143, 8; 10.1175/MWR-D-15-0022.1
Shown in Fig. 6 are the climatological PDF (blue) and El Niño PDF (red) for 200-mb heights over the extratropical region. Consistent with the negative regression, the PDF for El Niño is shifted to the left of the climatological PDF. In contrast to the PDFs for the tropical region in Fig. 5, however, there is a considerable overlap between the climatological and El Niño PDFs, and this feature is very similar to that in Fig. 2. Further, the spread of two PDFs also has similar magnitude. To summarize, Figs. 5 and 6 illustrate that idealizations in Figs. 1 and 2 indeed have counterparts in state-of-art GCM simulations. The results are also consistent with an array of previous studies that seasonal predictability and forecastability is high (low) in tropical (extratropical) latitudes and is a consequence of well separated (intermingled) climatological and ENSO PDFs (Peng et al. 2000; Min et al. 2014).
As in Fig. 5, but for the Northern Hemisphere extratropical box in Fig. 4. The estimated p value for the below-normal category is 0.52.
Citation: Monthly Weather Review 143, 8; 10.1175/MWR-D-15-0022.1
In the final analysis we establish the validity of Fig. 3 but based on the GCM simulations. In Fig. 7, the true probability for the 200-mb heights to be in the above-normal category for the 1982/83 and 1997/98 El Niño composite is shown. The procedure for computing the true probability is similar to that for the conceptual model: at each grid point, based on the climatological PDF estimated from a sample of 5757 DJFs, we first estimate the boundary that defines the above-normal category for 200-mb seasonal mean height. From the sample of 202 DJFs for two El Niño events we next estimate the true probability for the 200-mb height to be in the above-normal category based on the seasonal mean height anomaly for how many forecast members were in the above-normal category. This estimate of true probability assumes that a sample size of 202 DJFs is large enough.
Anomalous probability for seasonal mean 200-mb DJF height to be in the above-normal category for the composite of El Niño events of 1982/83 and 1997/98. Regions with only positive anomalies larger than 0.4 are shaded and generally correspond to regions with positive regressions in Fig. 4.
Citation: Monthly Weather Review 143, 8; 10.1175/MWR-D-15-0022.1
The spatial structure of the true probability (Fig. 7) broadly mimics the ENSO regression pattern for the height in Fig. 4, with a larger probability for 200-mb heights to be in the upper tercile in tropical latitudes and also at certain locations in extratropical latitudes. We note that while the ENSO regression pattern in Fig. 4 represents the mean shift associated with the PDF due to ENSO SSTs, the estimate of true probability is also modulated by spread in the PDFs. Because the variability of the seasonal mean in tropical latitudes is small (Kumar and Hoerling 1995; Peng et al. 2000), the true probability for the seasonal mean height to be in the upper tercile gets disproportionally amplified compared to extratropical latitudes where seasonal mean variability is larger. We also note that the estimate of true probability is not constrained by any assumptions on the shape or the spread of the PDFs, however, the analysis discussed next replicates that based on the simple model.
From the sample of 202 DJFs, next we repeatedly, and randomly, sample forecasts with ensemble sizes varying from 1 to 50, and compute the RMSPE relative to the estimated true probability. RMSPE based on this procedure for different values of true probability is shown in Fig. 8. The dependence of the RMSPE with ensemble size is indeed very similar to that in Fig. 3 confirming that departures in forecast probabilities from their true probability in Fig. 3 were not an artifact of assumptions implicit in the construction of the simple model but also hold for GCMs. In fact, results inferred based on the simple model are also analytical outcomes of signal-to-noise considerations, and dynamical systems ultimately are constrained to follow the same results. In summary, we demonstrated that conceptual diagrams illustrating changes in the PDF for high (Fig. 1) and low (Fig. 2) signal-to-noise regimes indeed are realistic examples taken from seasonal climate variability, and further, the consequences for the errors in the forecast probability, and its dependence on the ensemble size, is also replicated in GCM-based analysis.
As in Fig. 3, but for results based on atmospheric general circulation model simulations.
Citation: Monthly Weather Review 143, 8; 10.1175/MWR-D-15-0022.1
5. Summary
Predictability is a property inherent to the climate system. The ensemble approach, by attempting to sample the PDF of the possible climate outcomes, is a method to realize the inherent predictability in the observable system as forecastability. The realization of the inherent predictability comes from our ability to assess the relative difference between predicted and climatological PDFs. The ensemble size required to glean relative differences between two PDFs depends on the degree of difference itself, which, in turn, also governs the limit of inherent predictability.
The maximum value of forecastability that can be achieved, however, is ultimately constrained by the level of predictability. The ensemble size required to realize inherent predictability, however, has a complementary relationship with the level of predictability. The complementary aspect is the following:
For systems with high predictability, ensembles with small size are sufficient to realize predictability, and average forecastability, even with small ensemble size, is close to the predictability limit.
For systems with low predictability, ensembles with larger size are required to realize predictability; however, as dictated by low predictability to begin with, the average forecastability remains low.
There is a subtle point that underlies a need for a large ensemble to resolve the forecast PDF in an absolute sense versus the ensemble size required to assess differences for the same PDF in a relative sense (i.e., relative to the climatological PDF). Resolving the forecast PDF in an absolute sense refers to having a good estimate of different moments of the PDF, for example, for the red curve in Fig. 1 what is the mean and spread? This requires availability of forecasts with large ensemble size. Further, this requirement is not dependent on the SNR and is the same for estimating different moments for any PDF. On the other hand, resolving the PDF in a relative sense refers to assessing differences in the forecast PDF relative to the climatological PDF, and ensemble size requirements for this does depend on the SNR. For low-predictability scenarios large ensemble sizes are required to resolve the forecast PDFs both in an absolute and relative sense. For high-predictability scenarios, although larger ensembles are still required to resolve the PDF in an absolute sense, small ensembles can still recognize differences in a relative sense. A need for large ensembles to resolve the PDF in an absolute sense is not the same requirement as the size of ensemble required to realize inherent predictability, because the latter only needs quantifying the relative difference between two PDFs, while the former does not.
A corollary of the notion of complementarity is that the necessity of an ensemble approach for forecasting also ensures that the expected forecastability will never be perfect. Further, a perceived need for larger ensembles for improving skill also carries the kernel that one is confronting an inherently low-predictability scenario.
Acknowledgments
We thank the editor and two anonymous reviewers for their constructive comments.
REFERENCES
Abramov, R., A. Majda, and R. Kleeman, 2005: Information theory and predictability for low-frequency variability. J. Atmos. Sci., 62, 65–87, doi:10.1175/JAS-3373.1.
Beraki, A. F., D. G. Dewitt, W. A. Landman, and C. Olivier, 2014: Dynamical seasonal climate prediction using an ocean–atmosphere coupled climate model developed in partnership between South Africa and the IRI. J. Climate, 27, 1719–1741, doi:10.1175/JCLI-D-13-00275.1.
Brankovic, C., and T. N. Palmer, 1997: Atmospheric seasonal predictability and estimates of ensemble size. Mon. Wea. Rev., 125, 859–874, doi:10.1175/1520-0493(1997)125<0859:ASPAEO>2.0.CO;2.
Charney, J. G., R. Fjørtoft, and J. von Neuman, 1950: Numerical integration of the barotropic vorticity equation. Tellus, 2A, 237–254, doi:10.1111/j.2153-3490.1950.tb00336.x.
Chen, M., and A. Kumar, 2015: Influence of ENSO SSTs on the spread of probability density function for precipitation and land surface temperature. Climate Dyn., doi:10.1007/s00382-014-2336-9, in press.
Cohen, J., and Coauthors, 2014: Recent Arctic amplification and extreme mid-latitude weather. Nature Geosci.,7, 627–637, doi:10.1038/ngeo2234.
Daron, J. D., and D. A. Stainforth, 2013: On predicting climate under climate change. Environ. Res. Lett., 8, 034021, doi:10.1088/1748-9326/8/3/034021.
Déqué, M., 1997: Ensemble size for numerical seasonal forecasts. Tellus, 49A, 74–86, doi:10.1034/j.1600-0870.1997.00005.x.
Doblas-Reyes, F. J., R. Hagedorn, and T. N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting. Part II: Calibration and combination. Tellus,57A, 234–252, doi:10.1111/j.1600-0870.2005.00104.x.
Doblas-Reyes, F. J., and Coauthors, 2013: Initialized near-term regional climate change prediction. Nature Commun., 4, 1715, doi:10.1038/ncomms2704.
Dutton, J. A., R. P. James, and J. D. Ross, 2013: Calibration and combination of dynamical seasonal forecasts to enhance the value of predicted probabilities for managing risk. Climate Dyn., 40, 3089–3105, doi:10.1007/s00382-013-1764-2.
Goddard, L., and Coauthors, 2013: A verification framework for interannual-to-decadal prediction experiments. Climate Dyn., 40, 245–272, doi:10.1007/s00382-012-1481-2.
Graham, R., and Coauthors, 2011: Long-range forecasting and the global framework for climate services. Climate Res., 47, 47–55, doi:10.3354/cr00963.
Hamill, T. M., and G. N. Kiladis, 2014: Skill of the MJO and Northern Hemisphere blocking in GEFS medium-range reforecasts. Mon. Wea. Rev., 142, 868–885, doi:10.1175/MWR-D-13-00199.1.
Hammer, G. L., 2000: A general systems approach to applying seasonal climate forecasts. Applications of Seasonal Climate Forecasting in Agricultural and Natural Ecosystems: An Australian Experience, G. L. Hammer, N. Nicholls, and C. Mitchell, Eds., Kluwer Academic Publishers, 51–65.
Hoerling, M. P., and A. Kumar, 2002: Atmospheric response patterns associated with tropical forcing. J. Climate, 15, 2184–2203, doi:10.1175/1520-0442(2002)015<2184:ARPAWT>2.0.CO;2.
Ji, M., A. Kumar, and A. Leetmaa, 1994: A multiseason climate forecast system at the National Meteorological Center. Bull. Amer. Meteor. Soc., 75, 569–577, doi:10.1175/1520-0477(1994)075<0569:AMCFSA>2.0.CO;2.
Johnson, C., and N. Bowler, 2009: On the reliability and calibration of ensemble forecasts. Mon. Wea. Rev., 137, 1717–1720, doi:10.1175/2009MWR2715.1.
Kim, H.-M., P. J. Webster, and J. A. Curry, 2012: Evaluation of short-term climate change prediction in multi-model CMIP5 decadal hindcasts. Geophys. Res. Lett., 39, L10701, doi:10.1029/2012GL051644.
Kleeman, R., 2002: Measuring dynamical prediction utility using relative entropy. J. Atmos. Sci., 59, 2057–2072, doi:10.1175/1520-0469(2002)059<2057:MDPUUR>2.0.CO;2.
Kumar, A., 2009: Finite samples and uncertainty estimates for skill measures for seasonal predictions. Mon. Wea. Rev., 137, 2622–2631, doi:10.1175/2009MWR2814.1.
Kumar, A., 2010: On the assessment of the value of the seasonal forecast information. Meteor. Appl., 17, 385–392, doi:10.1002/met.167.
Kumar, A., and M. P. Hoerling, 1995: Prospects and limitations of seasonal atmospheric GCM predictions. Bull. Amer. Meteor. Soc., 76, 335–345, doi:10.1175/1520-0477(1995)076<0335:PALOSA>2.0.CO;2.
Kumar, A., and M. P. Hoerling, 2000: Analysis of a conceptual model of seasonal climate variability and implications for seasonal predictions. Bull. Amer. Meteor. Soc., 81, 255–264, doi:10.1175/1520-0477(2000)081<0255:AOACMO>2.3.CO;2.
Kumar, A., and R. Murtugudde, 2013: Predictability and uncertainty: A unified perspective to build a bridge from weather to climate. Curr. Opin. Environ. Sustainability, 5, 327–333, doi:10.1016/j.cosust.2013.05.009.
Kumar, A., and Z.-Z. Hu, 2014: How variable is the uncertainty in ENSO sea surface temperature prediction? J. Climate, 27, 2779–2788, doi:10.1175/JCLI-D-13-00576.1.
Kumar, A., A. G. Barnston, and M. P. Hoerling, 2001: Seasonal predictions, probabilistic verifications, and ensemble size. J. Climate, 14, 1671–1676, doi:10.1175/1520-0442(2001)014<1671:SPPVAE>2.0.CO;2.
Kumar, A., M. Chen, L. Zhang, W. Wang, Y. Xue, C. Wen, L. Marx, and B. Huang, 2012: An analysis of the nonstationarity in the bias of seas surface temperature forecasts for the NCEP Climate Forecast System (CFS) version 2. Mon. Wea. Rev., 140, 3003–3016, doi:10.1175/MWR-D-11-00335.1.
Kumar, A., M. Chen, M. P. Hoerling, and J. Eischeid, 2013: Do extreme climate events require extreme forcings? Geophys. Res. Lett., 40, 3440–3445, doi:10.1002/grl.50657.
Lorenz, E. N., 1965: A study of the predictability of a 28-variable atmospheric model. Tellus, 17A, 321–333, doi:10.1111/j.2153-3490.1965.tb01424.x.
Lynch, P., 2008: The origins of computer weather prediction and climate modeling. J. Comput. Phys., 227, 3431–3444, doi:10.1016/j.jcp.2007.02.034.
Materia, S., A. Borrelli, A. Bellucci, A. Alessandri, P. Di Pietro, P. Athanasiadis, A. Navarra, and S. Gualdi, 2014: Impact of atmosphere and land surface initial conditions on seasonal forecasts of global surface temperature. J. Climate, 27, 9253–9271, doi:10.1175/JCLI-D-14-00163.1.
Meehl, G. A., H. Teng, and J. M. Arblaster, 2014: Climate model simulations of the observed early-2000s hiatus of global warming. Nat. Climate Change, 4, 898–902, doi:10.1038/nclimate2357.
Min, Y.-M., V. N. Kryjov, and S. M. Oh, 2014: Assessment of APCC multimodel ensemble prediction in seasonal climate forecasting: Retrospective (1983–2003) and real-time forecasts (2008–2013). J. Geophys. Res. Atmos., 119, 12 132–12 150, doi:10.1002/2014JD022230.
Molteni, F., R. Buizza, T. N. Palmer, and T. Petroliagis, 1996: The ECMWF ensemble prediction system: Methodology and validation. Quart. J. Roy. Meteor. Soc., 122, 73–119, doi:10.1002/qj.49712252905.
Msangi, S., M. W. Rosegrant, and L. You, 2006: Ex post assessment methods of climate forecast impacts. Climate Res., 33, 67–79, doi:10.3354/cr033067.
Peng, P., A. Kumar, A. G. Barnston, and L. Goddard, 2000: Simulation skills of the SST-forced global climate variability of the NCEP-MRF9 and Scripps/MPI ECHAM3 models. J. Climate, 13, 3657–3679, doi:10.1175/1520-0442(2000)013<3657:SSOTSF>2.0.CO;2.
Peng, P., A. Kumar, M. S. Halpert, and A. Barnston, 2012: An analysis of CPC’s operational 0.5 month lead seasonal outlooks. Wea. Forecasting, 27, 898–917, doi:10.1175/WAF-D-11-00143.1.
Phillips, N. A., 1956: The general circulation of the atmosphere: A numerical experiment. Quart. J. Roy. Meteor. Soc., 82, 123–164, doi:10.1002/qj.49708235202.
Saha, S., and Coauthors, 2014: The NCEP Climate Forecast System version 2. J. Climate,27, 2185–2208, doi:10.1175/JCLI-D-12-00823.1.
Sardeshmukh, P. D., G. P. Compo, and C. Penland, 2000: Changes of probability associated with El Niño. J. Climate, 13, 4268–4286, doi:10.1175/1520-0442(2000)013<4268:COPAWE>2.0.CO;2.
Scaife, A. A., and Coauthors, 2014: Skillful long-range prediction of European and North American winters. Geophys. Res. Lett.,41, 2514–2519, doi:10.1002/2014GL059637.
Schepen, A., Q. J. Wang, and D. E. Robertson, 2014: Seasonal forecasts of Australian rainfall through calibration and bridging of coupled GCM outputs. Mon. Wea. Rev., 142, 1758–1770, doi:10.1175/MWR-D-13-00248.1.
Shuman, F. G., 1989: History of numerical weather prediction at the National Meteorological Center. Wea. Forecasting, 4, 286–296, doi:10.1175/1520-0434(1989)004<0286:HONWPA>2.0.CO;2.
Suckling, E. B., and L. A. Smith, 2013: An evaluation of decadal probability forecasts from state-of the-art climate models. J. Climate, 26, 9334–9347, doi:10.1175/JCLI-D-12-00485.1.
Tang, Y., R. Kleeman, and A. M. Moore, 2005: Reliability of ENSO dynamical predictions. J. Atmos. Sci., 62, 1770–1791, doi:10.1175/JAS3445.1.
Tang, Y., H. Lin, J. Derome, and M. K. Tippett, 2007: A predictability measure applied to seasonal prediction of the Arctic Oscillation. J. Climate, 20, 4733–4750, doi:10.1175/JCLI4276.1.
Taschetto, A. S., and M. H. England, 2008: Estimating ensemble size requirements of AGCM simulations. Meteor. Atmos. Phys., 100, 23–36, doi:10.1007/s00703-008-0293-8.
Teng, H., G. Branstator, and G. A. Meehl, 2011: Predictability of the Atlantic Overturning Circulation and associated surface patterns in two CCSM3 climate change ensemble experiments. J. Climate, 24, 6054–6076, doi:10.1175/2011JCLI4207.1.
Toth, Z., and E. Kalnay, 1997: Ensemble forecasting at NCP and breeding method. Mon. Wea. Rev., 125, 3297–3319, doi:10.1175/1520-0493(1997)125<3297:EFANAT>2.0.CO;2.
Trenberth, K. E., G. W. Branstator, D. Karoly, A. Kumar, N.-C. Lau, and C. Ropelewski, 1998: Progress during TOGA in understanding and modeling global teleconnections associated with tropical sea surface temperatures. J. Geophys. Res., 103, 14 291–14 324, doi:10.1029/97JC01444.
Wilks, D. S., 1995: Statistical Methods in the Atmospheric Sciences: An Introduction. Academic Press, 464 pp.
