a. Regression-corrected forecasts and sample size

Often SST forecast models have short forecast histories, on the order of 20 yr. When linear regression is used to correct systematic errors, the small sample size leads to errors in the estimation of linear regression coefficients. We investigate the issue of how expected improvement in rms error depends on sample size and forecast skill by considering the idealized univariate situation where all distributions are normal and forecast errors are independent of the forecast. Suppose that a forecast anomaly x and the verifying observation anomaly y are related by the equation

View Expanded

where the forecast and observation have zero mean, and the state-independent error e is a mean-zero Gaussian random variable with variance σ²; the departure of a from unity is a measure of linear systematic forecast error while the size of σ is a measure of the random forecast error. The coefficient a can be estimated from a historical sample of n forecasts {x₁, . . . , x_n}, and observations {y₁, . . . , y_n} using the usual least squares estimate a_est ≡ S_xy/S_xx, where S_xy ≡ Σⁿ_i=1x_iy_i and S_xx ≡ Σⁿ_i=1x²_i. The estimated regression coefficient can be used to make a corrected forecast y^pred ≡ a_estx. The least squares estimate of the regression coefficient minimizes the in-sample sum of squared errors

View Expanded

The size of the in-sample error S_e is related to the variance of the error term e that appears in (1). However, S_e underestimates the population (out-of-sample) variance of e in the sense that an unbiased estimate σ̂² of the population error variance σ² is (Montgomery and Peck 1992)

View Expanded

This means that the expected population rms error due to e is larger than the in-sample rms error by a factor of n/(n − 2), which for large n is approximately 1 + 1/n.

When the record length n is large enough to estimate the regression coefficient perfectly (i.e., when n → ∞), the error of the corrected forecast variance is due only to the random error term e, whose variance σ² can be estimated from (3). However, in general, the error in the estimate a_est of the coefficient a also contributes to the population error variance of the corrected forecast y^pred. The estimated regression coefficient a_est is a function of the sample data and is a Gaussian random variable whose mean is the true coefficient a and whose variance is given by the relation (Montgomery and Peck 1992)

View Expanded

where the notation 〈·〉 denotes expectation. Since, to leading order S_xx is proportional to n, the relation in (4) means that the variance of the regression coefficient estimate goes to zero like 1/n as the sample size is increased. The impact of the regression coefficient estimate error on the corrected forecast error variance is shown by

View Expanded

to enhance error variance by the factor (1 + 1/n).

In practice, the variance σ² is not known, but it can be estimated from the relation in (3) and the in-sample error S_e. Combining (3) and (5), the expected error variance of the corrected forecast is related to the in-sample error variance by

View Expanded

This means that the corrected forecast rms error exceeds the in-sample rms error by a factor of (n + 1)/(n − 2), which for large n is approximately 1 + 3/(2n) and for sample size n = 20 represents an 8% increase. Therefore under some circumstances, the expected out-of-sample error of the corrected forecasts may be larger than that of the uncorrected forecasts; this is in contrast to the in-sample error, which is always smaller. Intuitively, one expects that this could occur when the forecast is sufficiently skillful that the in-sample improvement due to regression is modest, or when the sample size is too small for a stable estimate of the coefficient a.

We now identify the factors and discuss quantitatively their roles in determining when linear regression corrections results in lower expected error variance. The error variance δ² of the uncorrected forecast is defined as

View Expanded

where σ²_x ≡ 〈x²〉, and we use (1) and the assumption that the error e is state independent. The expected error variance of the corrected forecast is less than the error of the uncorrected forecast when

View Expanded

or equivalently when

View Expanded

where we define σ²_y ≡ 〈y²〉 and use the relations σ² = 〈y²〉 (1 − r ²) and r = aσ_x/σ_y; r is the correlation between forecasts and observation; the linear correction does not change the correlation of the forecast with observations. Although four parameters, a, r, n and the variance ratio, appear in (9), only three can be specified independently, for instance, the ratio σ_x/σ_y, the coefficient a, and the sample size n. For a given record length n and forecast–observation correlation r, the difference of the regression coefficient from unity is the factor that determines whether the error variance of the corrected forecast is less than that of the uncorrected forecast. The corrected forecast has lower error variance on average when the regression coefficient is far enough from unity. When the coefficient a is 1 or sufficiently close to 1, the systematic forecast error is small and correction generally does not improve the uncorrected forecasts. The coefficient a equals 1 when σ_x/σ_y = r. The degradation of the corrected forecast skill in situations where the coefficient a is unity makes qualitative sense, since in that case any sample estimate of the coefficient will generally be different from unity, in either direction with equal likelihood, and will cause the corrected forecast to have larger error.

To understand better how the condition in (9) depends on the correlation between forecast and observations, the ratio of forecast to observation variance and sample size, Fig. la shows curves of correlation and standard deviation ratio values leading to an expected improvement of zero for various sample sizes n. The curves are roughly parabolic and centered about the line r = 〈x²〉/〈y²〉, that is, a = 1. The expected improvement is negative inside the curves. The region of negative expected improvement narrows as the sample size increases. Forecasts that require inflation (a > 1 in the region to the left of the line a = 1) benefit from correction if the correlation is stronger than would be suggested by the uncorrected forecast amplitude. Forecasts that require damping (a < 1 in the region to the right of the line a = 1) benefit from correction if the correlation is weaker than would be suggested by the uncorrected forecast amplitude. When the forecast variance is larger than that of observations, the correction is beneficial for nearly all parameter values. Figure 1b shows contours of the expected degradation values [the difference of the right and left side of inequality (8) normalized by σ²_y] for n = 20, which is presently typical for SST forecasts. The values are fairly small, on the order of a couple of percent, with the maximum (with respect to standard deviation ratio) degradation decreasing as the correlation increases. Figure lc shows the 10th percentiles of the degradation (i.e., values so that 90% of the time the degradation is no worse) due to correction computed by using the variance from (4).

b. Cross validation

Another consequence of the small sample size is the impracticality of dividing the data into large enough independent sets to estimate reliably the regression coefficients and the out-of-sample error. Performance could be estimated using the expression in (6) relating out-of-sample and in-sample error. However, for real problems such as correcting forecast SST, the underlying assumptions do not hold precisely, and robust, nonparametric methods like cross validation are desirable.

Cross validation is a nonparametric method of estimating out-of-sample performance that makes no particular assumptions on the distributions (Michaelsen 1987). In cross-validation methods, one to a few years are excluded from the calculation of the regression model parameters and the performance of the regression model is evaluated on this independent dataset of withheld years, and then the procedure is repeated using different sets of withheld years. If a single year is being left out, the number of iterations is at most the length of the dataset. If more years are left out, more iterations are possible. Leaving out more than a single data point is especially useful in model selection when the dimension and predictors of the regression model is being decided (Shao 1993) or when serial correlation is a concern. However, in the univariate case where there is no predictor selection, leaving out more than one in the cross-validation procedure degrades the error estimate. The rms error estimated by cross validation has a positive bias that is larger when the sample size is small and when more years are left out of the model parameter estimation. Here we use only leave-one-out cross validation.

To understand better the performance of cross validation, we perform Monte Carlo simulations of the correction scheme in (1), focusing on the issue of how well cross validation estimates the corrected forecast error, and in particular how well cross validation identifies situations where linear regression leads to increased error. We generate 20 yr of observations and forecasts with normally distributed errors. First, we compute the regression coefficient from the 20-yr sample and compare the rms error of the corrected and uncorrected forecasts in the population. Figure 2a shows, as a function of correlation and variance ratio, the probability of the sample-estimated regression coefficient producing a corrected forecast with lower rms error. Then we use cross validation to estimate the rms error of the corrected forecast, and Fig. 2b shows the expected improvement as indicated by cross validation. The cross-validation estimate indicates an increase in rms error over a larger region of phase space (correlation r and variance ratio) than actually occurs (cf. with Fig. lb). This result is not surprising since the expected error variance estimated by cross validation is larger than the population error variance (Bunke and Droge 1984).

Comparing the cross-validation estimate of corrected forecast error with the in-sample estimate of uncorrected forecast error provides the likelihood of cross validation indicating an improved forecast (Fig. 2c). Comparison of Figs. 2a and 2c shows that the likelihood of cross validation indicating improvement is too large near the line where a = 1 and too small away from it. This result suggests that one cannot characterize cross validation as being uniformly either too generous or too strict in deciding whether or not correction gives an improvement since it may err in either direction depending on the characteristics of the problem. The Monte Carlo simulation allows us to compute the expected change in rms error when cross validation is used to decide whether or not to do linear regression relative to the strategy of always doing linear regression; linear regression is used to correct the forecast when cross validation indicates that it reduces rms error (Fig. 2d). In regions of phase space where the benefit of correction is large, the impact of using cross validation to decide whether or not to correct is slightly negative. This finding is not unreasonable since while the likelihood of cross validation not indicating a reduction in rms error is slight (see Fig. 2a), the negative impact of not correcting is large. A positive impact is seen around the line a = 1, where most of the time cross validation will correctly indicate that linear regression increases rms error.

While correction on a per-gridpoint basis does not change the correlation, there is a negative correlation bias associated with leave-one-out cross validation that we expect to be a function of sample size and correlation level to leading order (Barnston and Van den Dool 1993). Following Barnston and Van den Dool (1993), we construct a synthetic dataset with 20 samples and compute the cross-validation bias as a function of correlation (Fig. 3). For a correlation value of 0.6, the bias is around 0.1. However, if spatially averaged correlation is the skill metric, contributions from regions where the skill is essentially zero (and bias is greater than 0.6) may be significant. Thus, the average bias may be more severe than that corresponding to the average correlation.

c. Pattern and local correction

The performance of univariate corrections is limited by not using information about the spatial relations that underlie large-scale patterns like those observed in the tropical Pacific SST. Here we use CCA (see appendix) to identify forecast patterns whose time series are correlated with those of observation patterns. EOF prefiltering is used to reduce the number of spatial degrees of freedom (Barnett and Preisendorfer 1987); forecast and observed SST are projected onto some small number of EOFs that describe large-scale structures.

The observed SST initial condition for the coupled model contains small-scale information that may not project onto the generally large-scale EOF patterns. It is plausible that some of this small-scale information persists in the coupled model forecast and in nature, and that because of EOF prefiltering, it is neglected in the CCA correction. It is desirable to develop a correction that retains useful information lost in EOF prefiltering. Here we use two regressions to produce a forecast correction that behaves like a pattern correction on the structures explained by the leading EOFs and like a gridpoint correction otherwise. First we use CCA to develop a few sets of regression forecast and observation patterns. The gridpoint regression is performed between the parts of the observation and forecast that are spatially orthogonal and temporally uncorrelated to the large-scale patterns defined by the EOFs (see appendix). Since the data used for the CCA and for the gridpoint regression are mutually independent, the procedure remains a least squares estimate and there is no “double counting” of the data. A limitation of the procedure is that it assumes no correlation between large- and small-scale errors.

a. Domain-averaged results

We examine the performance of three correction methods: grid point (univariate), CCA, and CCA plus grid point (CCA+) applied to ECM, ECM-Ens, and ICM SST predictions. Forecast starts are limited to January and July with leads of up to 6 months. For example, the six leads for the 1 January start are forecasts for January, February, . . . , June. Benchmarks are provided by two reference forecasts: (i) a persistence forecast, which consists of the observed SST anomaly from the month preceding the forecast start added to the climatology of the verification month and (ii) a purely statistical CCA forecast whose predictor is the observed SST anomaly of the month preceding the forecast start. This choice of predictor for the statistical forecast is certainly not optimal since it contains only partial information about the state and recent direction of evolution of the coupled atmosphere–ocean system (Xue et al. 2000). The statistical forecasts starting in January use seven EOF modes and five CCA modes for all leads; the forecasts starting in July use three EOF modes and three CCA modes. The number of modes is chosen to give good cross-validated skill. The statistical forecasts are made in a leave-one-out cross-validation manner, and the monthly SST climatology used in the persistence and statistical forecasts does not contain any of the data being forecast. Systematic bias is removed from the uncorrected forecasts in a similar manner. The forecast SST is the forecast anomaly with respect to the forecast climatology added to the observed climatology where the climatologies do not include the year of the current forecast. This method gives slightly lower skill levels compared to using all the data to compute the climatologies.

A general picture of model skill and correction performance is given by the domain-averaged correlation and rms error shown in Figs. 4 –6 for the for the ECM, ECM-Ens, and ICM predictions, respectively. These basinwide skill metrics are similar to, but distinct from, ones based on Niño indices. The skill of forecasts made in January tends to decrease more quickly as lead time increases than do forecasts made in July, a reflection of the difficulty of making forecasts through the so-called “spring barrier.” The average correlation of the gridpoint-corrected forecasts is lower than that of the uncorrected forecasts in all cases. To examine the extent to which this reduction in average correlation can be explained by the bias due to cross validation, we use the bias data shown in Fig. 3 to construct an empirical relation between correlation level and cross-validation bias. When we add this cross-validation bias to the average correlation of the uncorrected forecasts, the sum agrees well with the cross-validation-estimated correlation in most cases; the agreement is poorest for ICM forecasts starting in January.

Turning to the rms error of the gridpoint corrections, we see that the gridpoint-corrected ECM forecasts have lower domain-averaged rms error than the uncorrected forecasts for all leads and both start months (Figs. 4b,d). The same is true for most of the ECM-Ens forecasts, with January starts at lead 6 and July starts at lead 1 being exceptions (Figs. 5b,d). In contrast, the gridpoint-corrected ICM forecasts (Figs. 6b,d) have reduced domain-averaged rms error only with leads greater than 4 (January starts) or 3 (July starts). That is, linear regression increases the average rms error estimated by cross validation of ICM forecasts for shorter leads. To see if this increase in rms error can be explained by the impact of sampling error on the regression coefficient, we plot the rms error estimate based on sample size and in-sample error in (5). This in-sample estimate also indicates that linear regression increases rms error, although to a lesser degree. The in-sample estimate of the average rms error after gridpoint correction generally agrees with that given by cross validation.

Now looking at the CCA corrections we note that at the first lead, the CCA corrections have less skill (lower domain-averaged correlation and higher domain-averaged rms error) than the gridpoint-corrected forecasts, in line with our expectation that information in initial conditions may provide useful information that does not project onto patterns that can be robustly estimated. Perhaps for similar reasons, the persistence forecasts have more skill (particularly average correlation) than the purely statistical ones at the initial lead. The hybrid CCA+ method, designed to incorporate both large- and small-scale information, performs comparably or better than either the grid point or CCA, particularly at the initial leads. To illustrate the character of the skill of the CCA+ corrections relative to the CCA ones, Fig. 7 shows the anomaly correlation skill of the January ECM-Ens forecasts at the first lead. The CCA+ corrections shows modest, though fairly consistent, improvement in anomaly correlation relative to the CCA corrections. We expect the CCA+ corrections to have limited benefit compared to the CCA corrections at longer leads since there is little reason to expect that the prediction models are either able to retain or create useful information that does not project onto the leading modes of variability of the system.

Comparison of Figs. 4 and 5 indicates the benefit that ensemble averaging has on skill, particularly rms error. We expect that using the ensemble average tends to remove SST responses associated with unpredictable components of the coupled model, such as atmospheric noise or “weather.” The ICM by design has a simpler atmosphere that eliminates much of this kind of internal variability. It is interesting to compare the qualitative behavior of ensemble averaging and linear correction. In regions where there is little SST predictability derived from initial conditions, we expect both the ensemble average and the single-member gridpoint-corrected forecast to be approximately zero. The ensemble average is zero because the SST is unconstrained by the initial conditions, and the single-member gridpoint-corrected forecast is zero because it is uncorrelated with observations. In this circumstance, the ensemble averaged forecast and regression-corrected forecast have the same rms error, which is determined by climatology. However, ensemble averaging is able to enhance small predictable signals by reducing the noise due to internal variability. The regression methods discussed here are unable to do this since the noise in a single realization of the model (as well as noise in observations) may overwhelm the predicable component. We notice that the rms error of the ECM-Ens forecasts starting in January is lower than that of the gridpoint-corrected ECM forecasts, suggesting that ensemble averaging may be doing more than simply damping forecasts where there is little skill. On the other hand, the rms error of the ECM-Ens forecasts starting in July is roughly comparable to that of the gridpoint-corrected ECM forecasts. Domain-averaged correlation skill also increases with ensemble averaging with the change being largest for January starts, behavior consistent with the speculation above regarding the different effects of ensemble averaging and gridpoint regression on rms error.

The persistence and purely statistical forecasts have skill levels that are not easily surpassed, at least with the skill metrics used here that include off-equatorial regions. Uncorrected ECM forecasts starting in January do not equal the average correlation of the persistence forecasts at any lead time, and July forecasts match the average correlation of the persistence forecasts only after lead 4. Uncorrected ECM forecasts have lower average rms error than persistence after leads 3 and 4 for January and July starts, respectively, but do not match the rms error of the purely statistical forecasts. Gridpoint- and pattern-corrected ECM forecasts have lower rms error than persistence after the first lead but higher rms error than the purely statistical forecast for January starts; corrected and purely statistical forecast have similar rms error for July starts. Average correlation of the uncorrected ECM-Ens forecasts matches that of persistence after leads 5 and 2 for the January and July starts, respectively. The rms error of the uncorrected ECM-Ens forecasts is lower than that of persistence for almost all starts and leads (the lead-1 forecast starting in January being the exception). Gridpoint and pattern corrections of the ECM-Ens forecasts have slight impact on rms error for January starts; the rms error of the corrected ECM-Ens forecasts is comparable for the July starts and matches that of the purely statistical forecast. ICM SST forecasts made in January match the average correlation of persistence up to lead 4 but do not match the skill of the statistical forecasts for leads 3–6. ICM forecasts starting in July surpass the average correlation of the benchmark forecasts at all leads though the rms error is slightly higher than that of the purely statistical forecast for leads greater than 2.

b. July forecasts of December SST

We look in more detail at the forecasts of December SST made the previous July. These forecasts are quite skillful, and statistical corrections can use this skill to correct systematic deficiencies. Figure 8 shows the homogeneous covariance patterns of the leading CCA mode for the ECM forecasts. The model pattern extends farther west, is more restricted to the equator, and does not extend far enough south along the coast of South America. Differences in the model and observation patterns are an indication of systematic model errors. The western extension of the coupled model ENSO pattern is likely related to the tendency of coupled models to suffer from a double ITCZ and too much western extension of the cold tongue, and hence too much variability in the west. The effect of the CCA correction is to replace the model pattern in Fig. 8a with the observation pattern in Fig. 8b. The CCA mode for the ECM-Ens forecast in Fig. 9 has smoother structure. Westward extension of variability is reduced but deficiencies in the structure along the South American coast remain. Figure 10 shows the homogeneous covariance patterns of the leading CCA mode for the ICM forecasts. Here the differences between the model and observation patterns are smaller, reflecting perhaps the use of an empirical atmosphere and the empirical relation between the subsurface and SST. Variability along the equator does not quite extend far enough west. Similar to the ECM and ECM-Ens forecasts, the ICM forecasts lack structure extending south along the South American coast.

The spatial pattern of the rms error of the ECM forecasts (Fig. 11a) is consistent with the systematic differences between model and observation CCA patterns with rms errors that extend across the domain. Errors in the west are associated with excessive variance caused by an ENSO pattern that extends too far west; the natural climate system appears to be sensitive to SST forcing in this region (Barlow et al. 2002). Both the gridpoint- and pattern-based CCA correction are able to fix this deficiency; recall that regression is effective and relatively insensitive to sample size when damping variance that is too large. The gridpoint correction has less success in reducing error levels in the eastern part of the domain; in fact, rms error levels there rise slightly. On the other hand, pattern-based CCA corrections (not shown) use spatial relationships and are able to reduce the rms level of systematic errors in the east. The CCA+ correction (four EOFs and four CCA modes; Fig. 11c) has similar overall performance and reduces error variance in both regions. Highest correlations of the ECM forecasts with observations are found in the central part of the domain and are mostly restricted to the equator (Fig. 12a). The cross-validation estimate of the anomaly correlation of the gridpoint correction has lower values because of the bias of the procedure, though the pattern of correlation is similar. The pattern-based CCA+ method increases the spatial extent and the values of high anomaly correlation skill.

Ensemble averaging reduces the western extent of rms error in ECM-Ens forecasts (Fig. 13a), and overall error levels of the uncorrected ECM-Ens forecast are comparable to those of the gridpoint-corrected ECM forecast. Gridpoint and pattern corrections slightly reduce rms error further with some benefit in the eastern part of the domain. The ECM-Ens forecasts have correlation exceeding 0.8 in a large domain, and correction does not lead to improvement (Fig. 14).

Gridpoint correction reduces the overall error level of the ICM forecasts, although there is an increase in rms error in the extreme eastern part of the domain (Figs. 15a,b). The CCA+ correction gives additional reduction in domain-averaged rms error, including and in the east (five EOFs and three CCA modes; Fig. l5c). Figure 16 shows that the anomaly correlation of the ICM forecasts with observations are above 0.7 for much of the domain. The cross-validated estimate of anomaly correlation for the gridpoint-corrected forecasts has slightly lower values. The CCA+ correction has a positive impact on the basin-averaged correlation and increases correlation values in the central Pacific to over 0.8.