1. Introduction

A general prediction problem is to find the best estimate of a quantity y given a related quantity x. We refer to vectors y and x as the predictand and predictor, respectively. Examples of typical earth science prediction problems are as follows: x is the current sea surface temperature and y is its future state (Penland and Magorian 1993); x is a prescribed CO₂ concentration and y is global surface temperature (Krueger and Von Storch 2011); x is a large-scale climate feature and y is an associated small-scale climate feature (Robertson et al. 2012). In principle, the probability distribution of y for a particular value of predictor x = x₀ (the conditional distribution) can be computed from physical laws or estimated from data. In either case, the mean of that distribution (the conditional mean) is the best forecast in the sense of minimizing the expected squared error. When x and y have a joint Gaussian distribution, the best forecast, as well as its uncertainty, is given by linear regression (LR).

The idea of conditional averaging is also found in the constructed analog (CA) method (Van den Dool 1994, 2006), a statistical forecast method that has been applied in a variety of geophysical problems (e.g., Van den Dool et al. 2003; Maurer and Hidalgo 2008; Hawkins et al. 2011). A prediction y_CA is made for a particular value of the predictor x = x₀ by searching through historical data for values of y corresponding to values of x that are close to x₀, so-called analogs. The CA method expresses the current predictor state x₀ as a weighted linear combination of past states and makes a prediction by applying those same weights to the corresponding values of y, an averaging procedure reminiscent of the conditional mean. The CA has previously been described as differing from LR in two fundamental ways. First, it has been claimed that by making no assumption of a linear relation between predictor and predictand, CA captures nonlinearity. Second, it has been claimed that since CA is not based on minimizing the mean squared error of the predictions, there is no danger of overfitting. Here we show that typical implementations of CA do not have these properties, and, in fact, CA forecasts are identical to LR forecasts.

The paper is organized as follows. In section 2 we review the least squares problems that arise in the formulations of LR and CA, and use the matrix pseudoinverse to show that simple (without predictor truncation or regularization) implementations of the two methods give identical forecasts. In section 3, we identify situations where the simple implementation overfits the data and show that a recommended CA implementation is the same as principal component regression. In section 4, we show that another common CA implementation corresponds to ridge regression. In section 5, we show that LR and CA predictions of the Niño-3.4 index are identical and may have large variance even at long leads. In section 6, we present and illustrate some nonlinear regression methods that follow naturally from modifications to CA. A summary and discussion are given in section 7.

2. Linear regression, constructed analogs, and pseudoinverses

We use the following matrix notation for the training data. Let be the N_x × N_t matrix of predictor data; N_x is the number of predictor variables and N_t is the number of time samples. Each column of contains the predictor variables (x) at a particular time; each row of contains the time series of a particular predictor variable. Likewise, let be the N_y × N_t matrix of predictand data; N_y is the number of predictand variables. Let x₀ be the N_x × 1 column vector of predictor variables to be used in a forecast. We assume that predictors and predictands are expressed as anomalies. More generally, a row of ones can be included in to account for an intercept term.

Linear regression finds the N_y × N_x matrix of regression coefficients such that the norm of the residuals

View Expanded

is minimized. The notation ‖ · ‖² denotes the square of the Frobenius norm, which is the sum of the squares of the entries of the matrix or vector to which it is applied. The linear regression forecast y_LR is

View Expanded

Practically, computing the matrix of regression coefficients by direct minimization of (1) may be ill posed (there is no unique solution when N_x > N_t) or ill advised (overfitting can lead to poor performance in independent data when N_x is comparable to N_t).

The CA method also involves a linear least squares minimization problem. In the CA method, x₀ is expressed as a weighted sum of past states (columns of ), and a prediction is formed by applying those same weights to the columns of . Specifically, CA finds the N_t × 1 column-vector a of weights that minimizes

View Expanded

and then makes a prediction y_CA by applying those weights to the columns of :

View Expanded

The linear least squares problems appearing in the formulations of LR and CA appear quite different. For instance, the matrix of LR coefficients multiplies the data on the left to combine different predictors, and the vector a of CA weights multiplies the data on the right to combine different times. Also, CA involves fitting x₀ while LR fits . One of the least squares problems is always underdetermined and the other overdetermined unless N_x = N_t. We will use the pseudoinverse of the data matrix to solve both linear least squares problems and show that the resulting LR and CA forecasts are identical. In particular, (1) is minimized by

View Expanded

where ⁺ is the pseudoinverse of , a quantity that we will define and discuss later (Hansen 1998). When N_x > N_t, (1) is underdetermined, its minimizer is not unique, and = ⁺ is the minimizer with minimum value of ‖‖². The pseudoinverse commutes with the transpose in that, (⁺)^T = (^T)⁺. For this reason, the minimizer of (3) can also be expressed using the pseudoinverse, and

View Expanded

When N_x < N_t, (3) is underdetermined, and a = ⁺x₀ is the minimizer with minimum norm. We refer to these direct minimizing solutions as providing “simple” implementations of LR and CA. Substituting the simple minimizers of (5) and (6) into the definitions of the LR and CA predictions, (2) and (4), respectively, we see that

View Expanded

Remarkably, the simple linear regression and constructed analog predictions are identical.

3. Connection to principal component regression

While the simple LR implementation does solve the least squares problem and find the best fit to the data, it does so using all of the predictors. Such an approach is ill advised when the number of predictors is comparable to the number of samples since overfitting may result in poor predictions on independent data. To see this point more clearly, let us return to the matter of actually defining the pseudoinverse. The pseudoinverse of is defined using its singular value decomposition (SVD):

View Expanded

where and are orthogonal square matrices of size N_x × N_x and N_t × N_t, respectively, and is a diagonal N_x × N_t matrix with nonnegative entries (Golub and Van Loan 1996). The so-called economical SVD is

View Expanded

where and retain the columns of and , respectively, corresponding to the nonzero diagonal elements of , and the elements of the square diagonal matrix are strictly positive; the number of positive diagonal entries of is at most min(N_x, N_t − 1) for anomaly data. The pseudoinverse of is defined to be

View Expanded

The matrix is square with positive diagonal entries and is thus invertible. Therefore, the simple LR and CA forecasts are

View Expanded

In the language of principal component analysis (PCA), the columns of the matrices and are the empirical orthogonal function (EOFs) and principal components (PCs), respectively, of the anomaly data . The factors of serve to normalize the PCs to have unit variance since the columns of are unit vectors with zero mean. Principal component regression (PCR) arises from taking the PCs as predictors rather than the original data in . If we were to use all of the PCs as predictors [simple PCR (SPCR)], we would find the matrix _SPCR of regression coefficients that minimizes

View Expanded

This linear least squares problem can be solved by finding the pseudoinverse of , which is . Therefore,

View Expanded

The simple PCR forecast y_SPCR is obtained by applying _SPCR to the PC amplitudes of x₀ which are . Therefore,

View Expanded

Therefore, the LR and CA forecasts with the simple minimizers are the same as the simple PCR forecast, which uses all of the PCs as predictors. Such an approach overfits the data and has poor prediction skill on independent data unless the number of samples is substantially larger than the number of predictors.

To obtain more robust CA weights in the case where the number N_x of predictors is comparable or exceeds the number N_t of samples, Van den Dool (2006) proposed projecting x₀ and on to a truncated set of EOFs. We use the tilde notation to denote such a truncation with and , and the double-dot notation to denote the truncation of the SVD. Computing the CA weights with the truncated data gives

View Expanded

and applying them to gives as prediction,

View Expanded

where and are the (truncated) PC amplitudes of x₀. From the previous discussion leading to (14) we recognize (16) as the PCR forecast based on the truncated set of PCs. Computing CA weights with data projected on to a truncated set of EOFs gives the same forecast as PCR using the same truncated set of PCs.

The choice of the number of PCs to use in the calculation of the CA weights has exactly the same effect on the forecast as the choice of the number of PCs to use in PCR. In both cases, using too many PCs leads to overfitting.

4. Connection to ridge regression

Another approach to the linear least squares problems in (1) and (3) that appear in the formulations of LR and CA is ridge regression, also known as Tikhonov regularization. The regularized solutions of (1) and (3) are

View Expanded

and

View Expanded

respectively, where is the appropriately sized identity matrix and the ridge parameter δ is a positive scalar (Hansen 1998). The regularized solutions are well-defined irrespective of the parameters N_x and N_t. The matrix _δ is precisely that used in ridge regression, and Van den Dool (2006) suggested using a_δ in CA. Remarkably, the resulting forecasts y_LA,δ and y_CA,δ are identical:

View Expanded

where we have used the push-through matrix identity (^T + δ)^−1T = ^T(^T + δ)⁻¹. Use of ridging in computing the CA weights or in computing the LR coefficients results in identical forecasts.

The ridge regression solution is directly related to the pseudoinverse-based solution since an equivalent definition of the pseudoinverse is

View Expanded

Consequently,

View Expanded

The ridge regression forecast in the limit of δ going to zero is the same as the LR or CA forecast with a simple minimizer. This result is consistent with the interpretation of ridge regression as solving the least squares problems subject to a constraint on the size of the solution (DelSole 2007).

5. Example: Niño-3.4 prediction

A typical application of CA and LR is the prediction of the Niño-3.4 index (Van den Dool 2006). We consider forecasts made in the beginning of July and take as predictors the gridded April–June sea surface temperature (SST) anomaly in the region from 40°S to 40°N from the extended reconstructed SST (ERSST) dataset, version 3b (Smith and Reynolds 2004). The historical data used to form come from the 49-yr period 1955–2003, and the anomalies are computed with respect to the same period. The predictand y is the 3-month average of Niño-3.4 anomaly with respect to the 1971–2000 period taken from the extended Kaplan dataset (Kaplan et al. 1998) at leads extending to lead 22; denoting the July–September 2005 as the zero-month lead forecast, lead 22 is April–June 2007. Here the initial condition x₀ is the April–June 2005 SST anomaly, and y consists of the Niño-3.4 index from April–June 2005 to April–June 2007, 25 leads. Forecasts are made based on varying number of area-weighted EOFs; no ridging is used.

Figure 1 shows that CA and PCR forecasts based on the same number of EOFs are identical. On the other hand, forecasts based on different numbers of EOFs can vary greatly. Forecasts using 10 EOFs show little variability, while those with 25 or more show considerable variability. This particular set of forecasts verifies well against observations out to a lead of nearly two years. The skill of forecasts made in July for the following March–May (lead 8) was computed for period 1955–2003 using the entire dataset and using leave-one-out cross validation (CV) applied to the LR coefficients and CA weights; the PCs were computed using the full dataset. The CV skill of the 10 EOF forecasts is the highest, and as the number of EOFs increases, the resulting forecasts have lower CV skill and greater variance (Table 1). On the other hand, the in-sample correlation increases as the number of EOFs increases, and the in-sample ratio of forecast to climatological variance is equal to the in-sample correlation. The variance of the cross-validated forecasts is greater than the climatological variance when 25 or more EOFs are used. The reason for this behavior is that the in-sample explained variance and the variance of the regression coefficient estimates, both of which are increasing functions of the number of predictors, contribute to the variance of the cross-validated forecasts. The behavior of the CV forecasts, especially those with more than 10 EOFs, is consistent with that of overfitting with in-sample skill being substantially greater than the CV skill, and the CV skill being inconsistent with the variance.

View Full Size

Constructed analog (CA) and principal component regression (PCR) forecasts along with observations (obs) of the three-month-average Niño-3.4 index. Forecasts are made at the beginning of July and extend through April–June of 2007. The numbers in the legend indicate the number of EOFs retained.

Citation: Monthly Weather Review 141, 7; 10.1175/MWR-D-12-00223.1

• Download Figure
• Download figure as PowerPoint slide

Table 1.

Skill and ratio of forecast to climatological variance of in-sample and leave-one-out cross-validated (CV) forecasts made in the beginning of July for the following March–May average (lead 8) of the Niño-3.4 index during the period 1955–2003.

View Table

6. Nonlinear CA

Our demonstration that CA forecasts are identical to LR forecasts depends on the weights being defined as the solution of the least squares problem in (3), and a particular solution being chosen in the underdetermined case. Other characterizations of the weights lead to quite different methods. Before considering other methods of computing weights, we examine the properties of the CA weights in more detail, focusing on the case when N_x < N_t. In this case, (3) does not have a unique solution and using the pseudoinverse or ridge regression selects a particular solution for the weights. The simple minimizer weights are

View Expanded

The form of (22) means that for any x₀, the vector a of weights is a linear combination of the columns of . Since the columns of span the same linear space as the rows of , the weights are a linear combination of the rows of . In other words, for some N_x × 1 vector b,

View Expanded

in particular, , and in the case that the predictors are PCs, b = x₀. Equation (23) means that the weights, viewed as a function of the data, lie on a hyperplane perpendicular to the (N_t + 1) × 1 vector [b^T, −1]. Because the weights are linear functions of the data, data with values near x₀ do not receive the largest weights, nor do data far from x₀ receive the smallest weights. The CA weights do not measure the distance of x₀ to the training data values. In particular, if x₀ is a “natural analog” and has the same values as a column of , the weights are not concentrated on that column of .

Modifying the definition of the CA weights, so that they are a function of the distance between the data and x₀, results in nonlinear statistical prediction algorithms with weights that depend nonlinearly on the data. Importantly, in the case when N_x < N_t, such a modification requires neither changing the least squares problem in (3) or the forecast equation in (4), but rather involves constructing alternative solutions to (3), that is, ones without the constraint that ‖a‖² be minimized. For instance, in the k-nearest neighbors (KNN) algorithm, the elements of the weight vector are all zero except for those corresponding to the k columns of that are closest to x₀, which have value 1/k (Hastie et al. 2009). Explicitly, the ith KNN weight is

View Expanded

where C_k(x₀) is the set of k columns of nearest to x₀. The KNN prediction is the average of the columns of corresponding to the k columns of nearest to x₀. Kernel methods generalize KNN by using weights that are a smoothly decreasing function of the distance between the columns of and x₀. In particular, the ith kernel smoother (KS) weight is

View Expanded

where the kernel function K(x, x₀, λ) is a smoothly decreasing, positive function of the distance between x and x₀, and λ is a parameter that determines how quickly the kernel function decreases to zero. Local linear regression (LLR) is another kernel method and computes the weights using generalized least squares with data close to x₀ receiving more emphasis. Specifically,

View Expanded

where the matrix is a N_t × N_t diagonal matrix that depends on x₀ and whose ith diagonal entry is

View Expanded

We applied these methods to 30 samples of univariate data generated by

View Expanded

where x and ε are Gaussian distributed with mean zero and unit variance. A row of ones is included in to account for a possible intercept term. Figure 2a shows that LR–CA fails to capture the nonlinear relation. The KKN fit with k = 5 is noisy and piecewise constant with discontinuities. A Gaussian kernel smoother (GKS) with a standard deviation of 0.35 and LLR (with the same Gaussian kernel) give similar results with LLR showing an advantage near the boundaries of the data. It is important to note that the performance of KNN depends on the choice of k, while the performance of the GKS and LRR depends on the kernel parameter λ. Here we have selected fairly arbitrary values for these parameters that give good performance. However, like the regression coefficients, these parameters should be chosen objectively in a way that avoids overfitting.

View Full Size

(a) Data (plus signs) generated by (28) fit by linear regression (LR)–constructed analog (CA), k-nearest neighbors (KNN), Gaussian kernel smoother (GKS), and local linear regression (LLR). The “truth” curve is the expected value of y given x. (b) The CA, KNN, GKS, and LLR weights for x₀ = −0.5. The LLR weights are divided by 4 for display purposes.

Citation: Monthly Weather Review 141, 7; 10.1175/MWR-D-12-00223.1

• Download Figure
• Download figure as PowerPoint slide

The CA, KNN, GKS, and LLR weights are quite different for x₀ = −0.5 as shown in Fig. 2b. The sum of the weights is one for all methods due to the intercept term. A clear feature of the CA weights is that they are a linear function of the data values and display no maximum near x₀. This behavior is general as discussed earlier. The KNN weights are zero except for the five data points nearest to x₀ where they are ⅕. The GKS weights have largest values near x₀ and decrease to zero as the distance to x₀ increases. The LLR weights are locally linear near x₀ with values that go to zero far from x₀.

7. Summary and discussion

While the constructed analog (CA) statistical forecast method has previously been described as having properties that are distinct from those of linear regression (LR; Van den Dool 2006), we have shown here that, with comparable treatment of the data, CA and LR produce identical forecasts, and therefore the properties of CA are the same as those of LR. In particular, CA forecasts are linear functions of the predictors and subject to overfitting. When EOF truncation is used in the CA calculation, the resulting forecast is the same as that given by principal component regression (PCR) based on the same EOFs. Likewise, using ridging in the calculation of CA weights results in the same forecast as does ridge regression.

These results were illustrated in an example where sea surface temperature was used to predict the Niño-3.4 index. The CA and PCR forecasts based on the same number of PCs are identical. When many PCs were used, the forecasts show high variance, even at long leads, but low cross-validated skill, a symptom of overfitting. The equivalence between LR and CA depends on the precise definition of the weights. Allowing the weights to depend nonlinearly on the data leads naturally to generalizations of CA such as kernel smoothers and local linear regression, which we have illustrated with an example.

In practice, LR forecasts are observed to differ from CA forecasts. Moreover, forecasts from different implementations of LR also differ. For instance, LR-based statistical forecasts of ENSO including CA have quite different properties (Barnston et al. 2012). Use of distinct datasets may explain some of these differences. However, it must be recognized that many linear regression forecasts, with significant variations in skill, can be constructed from a given dataset of predictors and predictands. There are two primary sources of variety. First, the predictors or predictands can be truncated, and the regression developed on the truncated data. Principal component analysis and canonical correlation analysis are commonly used methods for truncating the data that enter a LR. The resulting forecasts depend on the truncation choices as illustrated here in the Niño-3.4 example where the forecasts depend strongly on the number of principal components retained as predictors. Linear inverse models and autoregressive methods usually project both the predictors and predictands onto EOFs (DelSole and Chang 2003); CA generally only projects the predictors, thus leading to different forecasts. Second, there are a variety of methods for estimating the LR coefficients. In addition to the classic least squares method, there are shrinkage methods like ridge and lasso (Hastie et al. 2009). The CA often uses ridge; PCR does not, again leading to different forecasts. Appropriate choices of data truncation and coefficient estimation method are key to developing a skillful LR forecast.