1. Introduction

Regression patterns are useful diagnostics of multivariate variability associated with a prespecified time series. For instance, patterns of trend coefficients often are used to quantify the spatial structure of climate change (see Trenberth et al. 2007), and the relation between El Niño and seasonal climate has been investigated using regression patterns at least since Walker (1923). However, a major limitation of regression patterns is the absence of a rigorous significance test—while testing the significance of a single regression coefficient is standard, testing the significance of multiple coefficients simultaneously is not straightforward. First, the multiplicity of tests needs to be taken into account. If individual coefficients are tested at the 5% significance level and the tests are independent, then 5% of the coefficients, on average, would be expected to be found significant even if no relation existed. Second, the dependence of the tests needs to be taken into account. In general, variables in gridded datasets tend to be correlated with nearby variables, implying that if a particular variable is found to be significant, then nearby variables will tend to be deemed significant as well.

Walker (1910) appears to have been the first to recognize the problem of multiple testing with regression patterns, or equivalently correlation patterns, and to propose a solution for it. Specifically, Walker proposed adjusting the significance threshold of individual tests to ensure that the probability of at least one false rejection of the null hypothesis was a predefined threshold. The latter is often called the experimentwise error rate. However, this approach assumes the tests are independent, which is a dubious assumption for gridded data.

Livezey and Chen (1983) proposed testing the “field significance” of a regression pattern by Monte Carlo techniques. The core of this technique is to replace the prespecified time series by random numbers drawn from a similar distribution, calculate the regression coefficients between the resulting time series and the gridded data, and count the number of rejections of the null hypothesis. This procedure is then repeated many times and the number of rejections is recorded for each regression pattern. The original regression pattern is said to have field significance if the number of rejections exceeds a threshold derived from the upper tail of rejections based on random time series. However, this approach is based on counting the number of rejections regardless of the spatial location or degree of significance. These shortcomings could be important if the spatial correlation structure varies significantly in the domain or if some coefficients are highly significant.

Wilks (2006) proposed testing field significance based on the false discovery rate. The false discovery rate is defined as the expected proportion of rejected null hypotheses that are actually true. The procedure is to perform M hypothesis tests for M variables and record the corresponding p values. Then, the p values are ordered in increasing order. For a given significance level α, the largest value of k such that the kth p value is less than αk/M is determined. Then the k hypotheses corresponding to the k smallest p values are rejected. The main shortcomings of this approach are that it assumes the hypothesis tests are independent and ignores spatial structure in these dependencies, though Wilks argues that this approach is relatively insensitive to correlations among local tests.

In this paper, we argue that multivariate regression provides a natural framework for testing the significance of a regression pattern relative to the null hypothesis that the regression coefficients vanish simultaneously. A statistic for testing the null hypothesis can be derived from the likelihood ratio test. This statistic depends on the ratio of determinants of relevant covariance matrices and, as such, simultaneously accounts for the multiplicity and interdependence of the tests. The statistic depends only on the canonical correlations between predictors and predictands, thereby revealing a connection to canonical correlation analysis. Furthermore, the statistic and its distribution are the same when the predictors and predictands are reversed. This symmetry allows the field significance test to be reduced to a standard univariate significance test. These results are discussed in section 2. However, the proposed statistic cannot be evaluated when the number of coefficients exceeds the sample size, reflecting the fact that testing more hypotheses than data is ill conceived. To formulate a proper significance test and to mitigate overfitting that occurs in any optimization problem, we follow a standard approach of applying the test in a reduced space spanned by the leading principal components of the data. The number of principal components is chosen based on cross-validation experiments. In contrast to most studies, we do not select the model that minimizes the cross-validated error but instead estimate a confidence interval for the minimum cross-validated error, and then select the most parsimonious model (i.e., the model with the fewest principal components) whose error is within the selected confidence interval. This additional step avoids picking complex models whose error is close to those of much simpler models. Details of this procedure are discussed in section 3. In section 4, the framework is used to diagnose multivariate trends in annual mean SST data and winter mean 300-hPa zonal wind data. Conclusions are given in section 5.

2. Hypothesis testing in multivariate regression

We argue that field significance of a regression pattern can be framed as a hypothesis test in multivariate regression. To justify this argument, we review certain results of multivariate regression that lay a foundation for field significance. Proofs and further details can be found in standard multivariate texts (Mardia et al. 1979; Johnson and Wichern 2002; Press 2005).

a. Testing multivariate hypotheses

Multivariate regression is concerned with quantifying the relation between several y variables and one or more x variables. The y variables are called predictands and the x variables are called predictors. Let M be the number of predictands and K be the number of predictors. The N observed values of the predictors can be collected in the N × K matrix and, similarly, the predictands in the N × M matrix . We assume that each predictand is a linear function of the predictors plus a random error. This assumption can be expressed mathematically as

View Expanded

where the dimension of each matrix is indicated directly underneath the corresponding matrix, is a matrix of random errors, and specifies the regression coefficients for predicting given .

The regression coefficients can be estimated by the method of least squares. In multivariate contexts, the method of least squares requires defining a “distance” between and . A general distance measure is

View Expanded

where Σ_δ is a positive definite matrix that accounts for spatial correlations and variance structure in the data. The parameter that minimize this distance measure is found by differentiating δ with respect to and setting the result to zero, which gives

View Expanded

Since Σ_δ is positive definite, the derivative vanishes if and only if the term in parentheses vanishes, which requires that the coefficients be

View Expanded

We use the caret symbol to denote sample estimates. Note that is independent of the covariance matrix Σ_δ, which is fortunate since it is unknown. This independence occurs because the errors are independent of time. If the errors were correlated in time, then generalized least squares would be needed.

A convenient property of the least squares solution (4) is that each column of depends on y only through the corresponding column of . Thus, can be determined column by column by fitting each predictand separately to the available predictors. It follows that multivariate estimation is merely a repeated application of univariate estimation. However, this property of multivariate estimation does not carry over to multivariate hypothesis testing. In particular, the individual elements of generally are correlated with each other due to correlations between predictors and predictands. For uncorrelated predictors, the elements within a column of would be uncorrelated, but the elements of that column generally would be correlated with elements of another column due to the correlation between predictands. Consequently, testing a hypothesis concerning the simultaneous behavior of the columns of generally cannot be done by applying univariate F tests to each column of , because these tests do not account for correlations between columns or multiple testing across columns.

In this paper, we consider the particular hypothesis . For this hypothesis, it is understood that the time mean of the predictors and predictands have been removed—that is, the predictors and predictands have been centered. This assumption leads to simpler forms of the equations but requires care in specifying the proper significance test. It turns out that the final statistic used to test the hypothesis is the same whether the constant term is included in the model or not, but the degrees of freedom need to be adjusted to take into account the estimation of the mean. We will automatically take this adjustment into account.

The standard approach to testing hypotheses in multivariate regression is the likelihood ratio test (Anderson 1984; Mardia et al. 1979; Press 2005). If the columns of are drawn independently from a normal distribution with zero mean, then the likelihood ratio test of leads to the statistic

View Expanded

where is the residual of the least squares estimate

View Expanded

If the hypothesis is true, then Λ has a distribution known as the Wilk’ s U distribution with parameters M, K, N − K − 1 (where centering has been taken into account). The hypothesis is rejected for small values of Λ.

The statistic Λ provides a basis for testing a multivariate hypothesis in a manner that accounts for the multiplicity and interdependency of the tests. Note that the statistic depends on the determinant of matrices and therefore depends on correlations between predictands and correlations between residuals. Multiplicity of hypothesis tests also taken into account.

b. Remarks on the hypothesis text

There are several connections between the procedure discussed in the previous section and other hypothesis tests that clarify the test itself. Many of these connections have been discussed in Muller (1982) or follow from standard linear algebra. We start with the fact that the statistic Λ in (5) can be written in terms of sample covariance matrices as

View Expanded

where the sample covariance matrices are defined as

View Expanded

In writing (7), we have used the fact that and

View Expanded

Expression (7) reveals several interesting properties of Λ. First, by invoking standard properties of the determinant, this expression can be written as

View Expanded

The matrix appearing in (10) appears frequently in multivariate statistics. For instance, it is well known that the eigenvector problem

View Expanded

is fundamental to canonical correlation analysis (CCA). CCA is a procedure for finding a linear combination of one set of variables (say, x) that is maximally correlated with a linear combination of another set of variables (say, y). The maximum squared correlation is given by the leading eigenvalue ρ², called the squared canonical correlation. Since the determinant of a matrix equals the product of the eigenvalues, it follows that the statistic can be written equivalently as

View Expanded

where ρ_i is the ith canonical correlation. Thus, the statistic for testing depends only on the canonical correlations between the predictors and predictands. In particular, if the predictors and predictands are independent, then the canonical correlations vanish and Λ = 1. Conversely, if the predictors and predictands are highly dependent, then the squared canonical correlations approach one and Λ → 0. Thus, small values of Λ indicate a strong relation between the variables and rejection of the hypothesis of independence.

The fact that the statistic Λ depends only on the canonical correlations is not surprising. The hypothesis in the model (1) is tantamount to assuming that the predictors and predictands are independent, in which case no combination of x is correlated with any combination of y. That is, the hypothesis is equivalent to the hypothesis that the predictors and predictands are independent in a multivariate sense.

A second important property of Λ is that it is invariant to reversing the x and y variables. This follows from the fact that Λ defined in (12) depends only on the canonical correlations and canonical correlations are symmetric with respect to the predictors and predictands. A more direct demonstration is to note that reversing the x and y variables in (10) leads to

View Expanded

which has the same value because the eigenvalues of the matrix are the same, owing to the fact that and have the same eigenvalues for any two matrices and for which matrix multiplication is defined. Moreover, it turns out that Wilk’ s U distribution with parameters M, K, N − K − 1 is the same as that with parameters K, M, N − M − 1 (Anderson 1984, theorem 8.4.2). Thus, both the value of Λ and its distribution under the null hypothesis of independence are invariant to reversing x and y. It follows that testing the hypothesis in model (1) is equivalent to testing the hypothesis in the model

View Expanded

Thus, testing multivariate independence of x and y by multivariate regression or by canonical correlation analysis yields precisely the same decision rule, regardless of which variables are identified as predictors and predictands. These equivalences are somewhat surprising, given that the regression coefficients for predicting y given x differ from those for predicting x given y. Nevertheless, these equivalences are natural because the concept of independence does not depend on which variables are identified as predictors or predictands. A similar symmetry occurs in line fitting: fitting the line y = ax + ϵ yields a different slope than fitting the line x = a′y′ + ϵ, but testing the hypothesis a = 0 leads to the same decision rule as testing the hypothesis a′ = 0, which in turn is equivalent to testing the hypothesis that x and y are uncorrelated.

The fact that the test for in model (1) is equivalent to the test for in model (14) can be anticipated in a different way. The exact regression coefficients for model (1) are

View Expanded

while the exact regression coefficients for model (14) are

View Expanded

We see that while the regression coefficients and differ, they both depend linearly on the cross-covariance matrix Σ_yx. In particular, the hypothesis implies Σ_xy = 0, and the hypothesis implies Σ_yx = 0. These two hypotheses are equivalent because . As a result of these relations, the vanishing of any one of , Σ_xy, and Σ_yx implies the vanishing of the other three.

c. Application to testing field significance

We now specialize the above procedure to the case of one predictor, that is, K = 1 in model (1). In this case, X is a column vector and B is a row vector. The column vector X is identified with a prespecified time series, for example, a linear trend or a climate index. Both the predictor and predictands are assumed to be centered. In this context, the least squares estimate in (4) often is called a regression pattern because each element of the vector equals the regression coefficient for predicting y given x. We are interested in testing the hypothesis that all regression coefficients vanish simultaneously,

View Expanded

which can be written concisely as . As reviewed in the previous section, testing this hypothesis is equivalent to testing the model with x and y reversed. Accordingly, we consider the model (14), but with only one predictand x, and test the hypothesis . This test is a well-known hypothesis test in regression analysis with statistic

View Expanded

where R² is the multiple correlation between x and y, which is a scalar defined as

View Expanded

Under the null hypothesis , the statistic F has an F distribution with M and N − M − 1 degrees of freedom. In this way, standard results from univariate regression can be invoked to solve the multivariate hypothesis associated with field significance. A direct algebraic proof that the statistic Λ in (7) reduces to (18) when K = 1 is given in appendix A.

The appearance of R² in the significance test is not surprising. As discussed in section 2 2b, the multivariate hypothesis is equivalent to testing whether all linear combinations of y are independent of x. The multiple correlation is in fact defined as the largest possible correlation between x and a linear combination of y (Anderson 1984, definition 2.5.4),

View Expanded

The optimal weighting coefficients for y are proportional to the least squares coefficients (16). For one predictor, the canonical correlation reduces to the multiple correlation.

A direct demonstration that CCA leads to the above results is given in appendix B.

3. Solution based on PCs

The field significance test proposed in the previous section cannot be applied when the number of regression coefficients exceeds the sample size because the test depends on the inverse of the covariance matrix , which is singular in this case. This singularity is not merely a practical limitation but rather is fundamental. Field significance involves testing that all regression coefficients vanish simultaneously. If the number of coefficients exceeds the sample size, then we are testing more hypotheses than data, which is ill conceived. To pose a proper field significance test, some constraints need to be imposed. The constraints need to be more stringent than merely allowing the evaluation of the significance test, as multivariate regression is sensitive to overfitting when the number of spatial elements M is not a small fraction of the sample size N. To mitigate these issues, we reduce the dimension of the data by projecting the data onto a few leading PCs.

a. Calculation details

Let the singular value decomposition (SVD) of the centered data be denoted as

View Expanded

where and are unitary matrices and is a diagonal matrix with nonnegative diagonal elements. Let denote the first K columns of and similarly for and . It turns out to be convenient to define normalized principal components as

View Expanded

which satisfy that is, the normalized PCs are uncorrelated and have unit variance. Similarly, the normalized EOFs would be

View Expanded

so that recovers the data matrix to the extent that it can be represented by K principal components. With this definition, we make the substitution in the previous sections. Accordingly, the regression “pattern” is

View Expanded

where the tilde indicates a quantity in PC space. The spatial structure of the regression pattern is found by multiplying by the EOFs:

View Expanded

This regression pattern is the only nontrivial canonical pattern between x and the PCs of y. The associated time series is found by fitting the model

View Expanded

The least squares estimate of , called the projection pattern or canonical weights, is

View Expanded

Because the PCs are uncorrelated, the projection pattern is merely the regression coefficient between x and each PC. The time series for predicting x, called the canonical time series, is

View Expanded

To derive the spatial structure of the projection pattern, we use the trivial identities

View Expanded

which implies that the projection pattern transformed back to the original data space is

View Expanded

The squared canonical correlation, or equivalently the squared multiple correlation, between x and the PCs is

View Expanded

b. Selecting the PCs to include in the regression model

The main difficulty with applying the above methodology is choosing the appropriate number of PCs. To gain insight into this problem, recall that testing the significance of the model for predicting y given x is equivalent to testing the significance of the model for predicting x given y. In the latter case, however, the choice of the number of PCs is a model selection problem, for which a variety of strategies have been proposed (Hastie et al. 1995). Therefore, we consider the model

View Expanded

where we include the mean μ for completeness. Here, there is one predictand but K predictors. The question then is which and how many PCs to include in .

A relevant and interesting fact is that the squared canonical correlation (31) is merely the sum square correlations between the individual PCs and the time series . To see this, note that the squared correlation between the ith PC and is

View Expanded

where (24) and has been used. Comparison with (31) shows

View Expanded

which proves the statement. This result has important implications for model selection. Specifically, if the PCs are ordered in terms of their correlation with x, then selecting the first K PCs gives the greatest canonical correlation of any K PCs. That is, the best subset of PCs can be obtained without searching all possible subsets, because the contribution of each PC to the canonical correlation is uncorrelated with the other PCs. However, selecting PCs in this way is a form of best subset regression, also called screening or “data fishing.” Such selection methods invalidate standard hypothesis tests and lead to poor prediction models (Rencher and Pun 1980; Freedman 1983; Flack and Chang 1987; Pinault 1988; DelSole and Shukla 2009). Using Monte Carlo techniques, we have confirmed that screening substantially inflates the canonical correlations, and hence inflates the F statistic, and that the final canonical correlation is not statistically significant when screening is taken into account.

Given the dangerousness of selecting predictors based on their relation with the predictand, we propose selecting predictors based on the order of their explained variances (which presumably is independent of the relation with the predictand). While this approach effectively eliminates selection bias, it potentially can lead to poor prediction models. To reduce the latter possibility, we choose the number of ordered PCs based on cross-validation experiments. We assume that the residuals of the regression model are serially independent, in which case leave-one-out cross validation is appropriate (Hastie et al. 1995). In leave-one-out cross validation, one sample of the dataset is withheld and the remaining samples are used to fit the model (32). Then, the resulting model is used to predict the withheld sample. The difference between prediction and withheld sample is

View Expanded

where and μ⁽ⁿ⁾ denote the least squares estimates of and μ derived from the data leaving out the nth sample. Note the mean term μ is needed because the predictors and predictands do not have zero mean when a sample is withheld. This procedure is repeated for n = 1, 2, … , N so that all samples will have been withheld exactly once. The cross-validated mean square error (CVMSE) is then defined as

View Expanded

The cross-validated mean square error determined by the above procedure is a function of the number of PCs. We evaluate CVMSE for all PC truncations.

In contrast to most studies, we do not choose the number of PCs that minimizes the CVMSE, because the CVMSE is often a “flat” function of the number of PCs, at least near the minimum, so choosing the minimum CVMSE ignores simpler models that have similar errors. Instead, we follow Hastie et al. (1995) and derive an approximate confidence interval for the error, and then we choose the simplest model (i.e., the model with the fewest PCs) that is within the confidence interval of the model with minimum CVMSE. Hastie et al. (1995) recommend a one-standard deviation confidence interval, which we estimate as

View Expanded

where denotes an unbiased estimate of the variance of z.

To facilitate comparison with the canonical correlations, we “convert” the cross-validated error into a “skill score” by

View Expanded

If the mean square error is computed from the full dataset (no cross validation), then 1 − (MSE/var[x]) reduces to the squared multiple correlation, which in turn is the variance explained by the predictors. By analogy, the cross-validated skill score (CVSS) is a measure of the amount of variance the model explains in a cross-validated sense. Negative values of CVSS imply that the model has larger cross-validated error than a prediction based on the mean. The confidence interval for CVSS is the appropriate linear transformation of the confidence interval of CVMSE.

c. Explained variance

It is often useful to quantify the amount of variance explained by the regression pattern. This measure can be derived from relation (9), which can be manipulated into the identity

View Expanded

The first term on the right quantifies the variability due to the regression pattern and the second term quantifies the residual variability after the regression pattern has been subtracted out. By taking the trace of both sides, we may define the fraction of variance explained by the regression pattern as

View Expanded

We remind the reader that is a row vector and Σ_x is a scalar in the case of a single predictor, so FEV is a scalar.

4. Application to climate time series

b. Diagnosing trends in annual mean SST

The pattern of trend coefficients for annual mean SST during the period 1948–2009 is shown in Fig. 1a. This regression pattern is calculated from defined in (4), where is the annual mean SST gridded data. The trend pattern reveals warming everywhere except the North Atlantic, North Pacific, and western boundary of the mid-Atlantic.

(a) The point-by-point regression coefficients for the trend (°K decade⁻¹), (b) the trend pattern as represented by the first three PCs of annual mean SST, (c) leading EOF (no trend calculation is involved), and (d) canonical projection pattern based on three PCs of annual mean SST for the period 1948–2009.

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

(a) The point-by-point regression coefficients for the trend (°K decade⁻¹), (b) the trend pattern as represented by the first three PCs of annual mean SST, (c) leading EOF (no trend calculation is involved), and (d) canonical projection pattern based on three PCs of annual mean SST for the period 1948–2009.

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

(a) The point-by-point regression coefficients for the trend (°K decade⁻¹), (b) the trend pattern as represented by the first three PCs of annual mean SST, (c) leading EOF (no trend calculation is involved), and (d) canonical projection pattern based on three PCs of annual mean SST for the period 1948–2009.

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

The squared canonical correlation ρ² (or equivalently, the squared multiple correlation) between the trend and the leading PCs, as a function of the number of PCs used to represent the data, is shown in Fig. 2a. For reference, ρ² for the first PC is 68%, which is statistically significant. Nevertheless, the first three PCs can be combined linearly to increase the squared canonical correlation to 92%. Comparison with the 5% significance curve for ρ², calculated from (18) and shown as the dashed line in Fig. 2a, indicates that the canonical correlation is significant for all PC truncations. However, the majority of this significance arises from the first three PCs, and very little gain in correlation occurs using more than three PCs.

(a) The squared canonical correlation (dots) and associated 5% significance level (dashed) between a linear trend and the annual mean SST during 1948–2009, as a function of the number of PCs used to represent the data. (b) The cross-validated skill score (38) for predicting the trend based on the PCs of annual mean SST, as a function of the number of PCs. The error bar shows the standard error of the maximum skill score, and the vertical dashed line indicates the selected number of PCs. (c) The correlation between the trend and individual PCs (bars), and the corresponding 5% significance levels (red lines).

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

(a) The squared canonical correlation (dots) and associated 5% significance level (dashed) between a linear trend and the annual mean SST during 1948–2009, as a function of the number of PCs used to represent the data. (b) The cross-validated skill score (38) for predicting the trend based on the PCs of annual mean SST, as a function of the number of PCs. The error bar shows the standard error of the maximum skill score, and the vertical dashed line indicates the selected number of PCs. (c) The correlation between the trend and individual PCs (bars), and the corresponding 5% significance levels (red lines).

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

(a) The squared canonical correlation (dots) and associated 5% significance level (dashed) between a linear trend and the annual mean SST during 1948–2009, as a function of the number of PCs used to represent the data. (b) The cross-validated skill score (38) for predicting the trend based on the PCs of annual mean SST, as a function of the number of PCs. The error bar shows the standard error of the maximum skill score, and the vertical dashed line indicates the selected number of PCs. (c) The correlation between the trend and individual PCs (bars), and the corresponding 5% significance levels (red lines).

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

We perform cross validation as described in section 3b. To illustrate the impact of leaving one year out, an extreme example based on 50 PCs is shown in Fig. 3. In this example, all years except 1959 were used to construct the projection pattern (27). The figure shows that the projection pattern gives a reasonable prediction of the trend in all years except 1959, the year that was left out of the analysis. The large prediction error in the withheld year is a classic symptom of overfitting. Cross validation involves repeating this procedure for each year in turn and collecting the statistics of the resulting prediction errors.

Prediction of a linear trend (dots) using as predictors the leading 50 PCs of annual mean SST. The prediction model was estimated by the least squares method using all years during 1948–2009 except 1959. The trend function being fitted is shown as the solid line. For reference, the year 1959 is indicated by a thin vertical dashed line.

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Prediction of a linear trend (dots) using as predictors the leading 50 PCs of annual mean SST. The prediction model was estimated by the least squares method using all years during 1948–2009 except 1959. The trend function being fitted is shown as the solid line. For reference, the year 1959 is indicated by a thin vertical dashed line.

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

Prediction of a linear trend (dots) using as predictors the leading 50 PCs of annual mean SST. The prediction model was estimated by the least squares method using all years during 1948–2009 except 1959. The trend function being fitted is shown as the solid line. For reference, the year 1959 is indicated by a thin vertical dashed line.

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

The cross-validated skill score (38) of predicting the trend based on the leading PCs of annual mean SST is shown in Fig. 2b. The figure shows that the maximum skill score occurs at six PCs. However, the figure shows that the skill scores using 3–8 PCs also lie within the estimated confidence interval of the maximum. By the principle of parsimony, we choose the simplest model (i.e., the model with the fewest number of PCs) that lies within the confidence interval of the maximum skill score, namely, the model with three PCs.

The correlation between the trend and individual PCs is shown in Fig. 2c; also shown as horizontal lines are the associated 5% significance levels. The figure shows that only the first three PCs have statistically significant trends. It is interesting to note that the ρ² shown in Fig. 2a is proportional to the sum of squared canonical correlations of individual PCs up to and including the PC indicated on the abscissa (as discussed in section 3a).

The trend pattern derived from the first three PCs is shown in Fig. 1b; this pattern is derived from (25). The similarity between the EOF-filtered trend pattern (Fig. 1b) and the regression pattern (Fig. 1a) indicates that there is little loss of trend information by using only the first three global EOFs to represent the data. For comparison, we also show the leading EOF in Fig. 1c. As anticipated from the fact that the leading PC contains a significant trend, the canonical trend pattern and the leading EOF are similar. The main difference is that the leading EOF has stronger amplitudes in the equatorial Pacific, suggesting a stronger contribution due to ENSO. The canonical component explains 24% of the SST spatial variance compared to 27% explained by the leading EOF. Thus, although the canonical component explains slightly less variance than the leading EOF, it explains significantly more long-term trend (e.g., ρ² are 92% versus 68%).

The canonical projection pattern is shown in Fig. 1d. This pattern is projected onto the SST to produce the canonical time series. The pattern has positive amplitudes concentrated in the Indian Ocean and south subtropical oceans, and negative amplitudes in the northern oceans. The projection pattern differs most significantly from the regression pattern by having negative amplitudes in the central equatorial Pacific, as well as generally weaker amplitudes in the western tropical Pacific. The similarity between the projection and regression patterns suggests that a significant fraction of the trend can be recovered by projecting the regression pattern onto the SSTs. However, this approach would not be optimal—differences between the two patterns imply that more trend can be captured by taking differences in SST along the equatorial Pacific, which presumably filters out interannual ENSO variability.

The time series for the leading EOF and the canonical component are shown in Figs. 4a and 4b, respectively, as computed from (22) and (28), respectively. The time series for the canonical pattern exhibits a stronger trend component than the leading PC, as anticipated from the fact that the ρ² for the leading EOF and canonical component are 68% and 92%, respectively. The correlation between the leading PC and the Niño-3.4 index is about 0.68, while the correlation between the canonical component and the Niño-3.4 index is 0.25. Loosely speaking, the canonical projection vector “filters out” ENSO noise to enhance the trend. Note that this filtering is based on spatial structure, not on temporal smoothing, which allows the amplitude of the canonical pattern to be monitored year by year.

Time series for (a) the leading PC and (b) canonical trend pattern and for annual average SST during the period 1948–2009.

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Time series for (a) the leading PC and (b) canonical trend pattern and for annual average SST during the period 1948–2009.

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

Time series for (a) the leading PC and (b) canonical trend pattern and for annual average SST during the period 1948–2009.

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

Most of the above results are insensitive to the number of PCs. The main exceptions are that the projection pattern shown in Fig. 1d tends to exhibit more small-scale noise as more PCs are included in the CCA, and the canonical time series tends to get smoother and approach an exact linear trend as the number of PCs increases (not shown). Nevertheless, the physical pattern shown in Fig. 1b and the ρ² shown in Fig. 2a are virtually indistinguishable for PC truncations greater than three.

c. Diagnosing trends in DJF 300-hPa zonal wind

The trend pattern for December–February 300-hPa zonal wind is shown in Fig. 5a. The figure reveals a “tripole” pattern with mostly positive amplitudes along 30°–45° latitude and negative amplitudes north and south. When superimposed on the background mean flow, this trend pattern indicates that the downstream end of the jet core is shifting eastward and that winds outside the jet core are decelerating by about 2 m s⁻¹ decade⁻¹.

(a) The point-by-point regression coefficients (shading) for the trend of DJF 300-hPa zonal wind (in meters per second per decade), (b) the trend pattern (shading) as represented by the first 21 PCs, (c) the leading EOF (shading), and (d) corresponding canonical projection pattern for the trend. The contours in (a)–(c) show the corresponding mean wind.

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

(a) The point-by-point regression coefficients (shading) for the trend of DJF 300-hPa zonal wind (in meters per second per decade), (b) the trend pattern (shading) as represented by the first 21 PCs, (c) the leading EOF (shading), and (d) corresponding canonical projection pattern for the trend. The contours in (a)–(c) show the corresponding mean wind.

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

(a) The point-by-point regression coefficients (shading) for the trend of DJF 300-hPa zonal wind (in meters per second per decade), (b) the trend pattern (shading) as represented by the first 21 PCs, (c) the leading EOF (shading), and (d) corresponding canonical projection pattern for the trend. The contours in (a)–(c) show the corresponding mean wind.

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

The squared canonical correlations between a linear trend and the leading PCs are shown in Fig. 6a. We see a dramatic difference compared to the canonical correlations for SST (Fig. 2a): the ρ² is negligible for the first few PCs, but then it increases and becomes statistically significant only after about 9 PCs. The cross-validated skill score of predicting the trend based on the PCs is shown in Fig. 6b. The figure shows that the maximum skill occurs at 28 PCs. Nevertheless, the skill for 21–35 PCs lie within the confidence interval of the maximum. Thus, the simplest model that is within the confidence interval of the best model is the 21-PC model. The 21-PC model also has statistically significant canonical correlation (as seen in Fig. 6a). The correlation between the trend and individual PCs, shown in Fig. 6c, reveals that the first two PCs do not have statistically significant trends. Indeed, only PCs 3, 9, 10, and 13 have statistically significant trends. This example illustrates the difficulty of choosing a single truncation based on the statistical significance of the trend in individual PCs.

(a) The squared canonical correlation (solid) and associated 5% significance level (dashed) between a linear trend and DJF 300-hPa zonal wind during 1948–2009, as a function of the number of PCs used to represent the data. (b) The CVSS (38) for predicting the trend based on the PCs of DJF 300-hPa zonal wind, as a function of the number of PCs. The error bar shows the standard error for the maximum skill score, and the vertical dashed line indicates the selected number of PCs. (c) The correlation between the trend and individual PCs (bars), and the corresponding 5% significance levels (red lines).

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

(a) The squared canonical correlation (solid) and associated 5% significance level (dashed) between a linear trend and DJF 300-hPa zonal wind during 1948–2009, as a function of the number of PCs used to represent the data. (b) The CVSS (38) for predicting the trend based on the PCs of DJF 300-hPa zonal wind, as a function of the number of PCs. The error bar shows the standard error for the maximum skill score, and the vertical dashed line indicates the selected number of PCs. (c) The correlation between the trend and individual PCs (bars), and the corresponding 5% significance levels (red lines).

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

(a) The squared canonical correlation (solid) and associated 5% significance level (dashed) between a linear trend and DJF 300-hPa zonal wind during 1948–2009, as a function of the number of PCs used to represent the data. (b) The CVSS (38) for predicting the trend based on the PCs of DJF 300-hPa zonal wind, as a function of the number of PCs. The error bar shows the standard error for the maximum skill score, and the vertical dashed line indicates the selected number of PCs. (c) The correlation between the trend and individual PCs (bars), and the corresponding 5% significance levels (red lines).

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

The regression pattern derived from the leading 21 PCs is shown in Fig. 5b. The similarity between this EOF-filtered pattern and the point-by-point regression pattern (Fig. 5a) indicates that there is little loss of trend information by using only 21 global PCs. For comparison, we also show the leading EOF in Fig. 5c. While the leading EOF also shows a tripole pattern, it is shifted southward relative to the trend pattern and generally has stronger amplitudes and more distinctive centers of action. The leading EOF explains 21% of the variance, whereas the canonical pattern explains only 4% of the variance. Thus, the canonical pattern represents a relatively small fraction of the seasonal variability in winds.

The projection pattern for the trend is shown in Fig. 5d. This pattern differs considerably from the canonical pattern and the leading EOF. In general, the projection pattern tends to have its largest amplitudes in regions where the background zonal wind is weakest. To clarify the reason for this, we show in Fig. 7 the standard deviation of the wind. A comparison between Figs. 5d and 7 shows that the largest amplitudes in the projection pattern tend to be in regions where the wind variance is weakest. While regions of weak variance may have weak trends, they also have less noise and therefore potentially larger signal-to-noise ratio. By weighting low-variance regions more strongly than regions with high variance, the projection pattern is able to filter out noise to enhance the signal-to-noise ratio.

Standard deviation (shading) and mean (contours) of the DJF 300-hPa zonal wind (m s⁻¹) during 1948–2009.

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Standard deviation (shading) and mean (contours) of the DJF 300-hPa zonal wind (m s⁻¹) during 1948–2009.

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

Standard deviation (shading) and mean (contours) of the DJF 300-hPa zonal wind (m s⁻¹) during 1948–2009.

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

The time series for the leading EOF and the canonical component are shown in Figs. 8a and 8b, respectively. The canonical time series exhibits a much stronger trend component than the leading PC, as expected. The squared correlation between the trend and the canonical component is 83% compared to 0.2% for the leading PC—a stark contrast in trends. This example provides a dramatic illustration of the fact that the leading principal components can fail to capture the trend. The leading principal component is highly correlated with the Niño-3.4 index (the correlation coefficient is 0.64), suggesting a relation with ENSO. In contrast, the correlation coefficient between the zonal wind canonical component and Niño-3.4 index is 0.05, suggesting that the trend component has little to do with ENSO.

Time series for the (a) leading PC and (b) canonical time series for the trend for DJF 300-hPa zonal wind during the period 1948–2009.

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Time series for the (a) leading PC and (b) canonical time series for the trend for DJF 300-hPa zonal wind during the period 1948–2009.

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

Time series for the (a) leading PC and (b) canonical time series for the trend for DJF 300-hPa zonal wind during the period 1948–2009.

Citation: Journal of Climate 24, 19; 10.1175/2011JCLI4105.1

• Download Figure
• Download figure as PowerPoint slide

5. Conclusions

This paper argues that multivariate regression provides a natural framework for testing the field significance of a regression pattern. Specifically, a regression model for predicting y given a single x leads naturally to the regression pattern. A test for the hypothesis that all regression coefficients vanish can then be derived from the likelihood ratio test. The statistic derived from the likelihood ratio test equals a ratio of determinants of relevant covariance matrices. This ratio takes into account correlations between predictors and predictands and takes into account the multiplicity of tests. It turns out that the statistic is invariant to reversing x and y, implying that testing the field significance of a regression pattern for predicting y given x, is equivalent to testing the model for predicting x given y. In this way, the field significance test is reduced to a standard univariate hypothesis test.

In general, the statistic derived from the likelihood ratio test depends only on the canonical correlations between predictors and predictands. This connection between multivariate regression and canonical correlation analysis follows naturally from the fact that vanishing of the regression coefficients implies that the predictors and predictands are independent. More generally, if x and y are independent, then the regression coefficients for predicting y given x vanish, the regression coefficients for predicting x given y vanish, and the canonical correlations vanish. Thus, each hypothesis leads to precisely the same decision rule because they are fundamentally the same—they each imply that x and y are independent. This reasoning also explains why the test is the same if y and x are reversed, since the concept of independence does not depend on which variables are identified as predictors or predictands.

The fact that field significance can be addressed naturally through multivariate regression raises the question of why it was overlooked in the past. Part of the answer is perhaps because most climate scientists are unaware of the likelihood ratio test in the context of multivariate regression, owing to its advanced nature statistically and mathematically. Also, the test cannot be applied when the number of variables exceeds the number of samples, and perhaps no one thought to apply EOF filtering to reduce the number of variables. Finally, the decision rule from the likelihood ratio test depends on Wilk’ s U distribution, whose critical values are not conveniently tabulated. That this distribution reduces to the F distribution for the case of one predictor does not seem to be mentioned often, even in the statistics literature.

A variety of other field significance tests have been proposed in the literature (see introduction). These alternative approaches lead to different decision rules, raising questions of interpretation. For instance, what is the meaning of a test that disagrees with CCA? An advantage of the proposed framework is that the decision rule for field significance is always consistent with that derived from multivariate regression and canonical correlation analysis.

When the number of regression coefficients exceeds the sample size, the field significance test cannot be evaluated because certain covariance matrices become singular. This singularity is more than a practical limitation, it signifies a fundamental problem. Field significance involves testing that all regression coefficients vanish simultaneously. If the number of coefficients exceeds the sample size, then field significance is tantamount to testing more hypotheses than data, which is ill conceived. To pose a proper field significance test, some constraints need to be imposed. We followed the standard approach of reducing the number of variables by using only a small number of leading principal components of the data. Furthermore, since testing field significance is equivalent to testing the significance of a model with reversed predictors and predictands, the problem of choosing the number of PCs is equivalent to a model selection problem, for which many strategies exist. We specifically avoided selecting PCs that were individually correlated with the predictand, as this strategy is a form of screening that leads to biased skill estimates. We proposed selecting PCs in the order of their explained variances, which avoids the screening problem but may lead to poor prediction models. To reduce the latter possibility, the number of ordered PCs was chosen based on cross-validation experiments in which the model predicts the prespecified time series given the PCs. In contrast to most studies, we do not select the model that minimizes the cross-validated error. Instead, we estimate a confidence interval for the minimum cross-validated error, and then select the most parsimonious model (i.e., the model with the fewest number of principal components) whose error is within the confidence interval of the minimum error. This additional step avoids picking complex models whose error is very close to those of much simpler models. While we believe that the above procedure merits attention, we also emphasize that the model selection issue is in most need of further research.

Because the significance test in multivariate regression is applicable to arbitrary predictors and predictands, the orthogonality of the PCs is in no way essential or critical to the field significance test—any set of variables, correlated or uncorrelated, can be used, and the statistical hypothesis test automatically accounts for correlations in the data.

The proposed field significance test was applied to diagnose trends in annual mean sea surface temperature and in December–February 300-hPa zonal wind in the Northern Hemisphere. The squared canonical correlation between the SST and trend is 92% compared with a squared correlation of 68% for the leading principal component alone. The canonical trend pattern and the leading EOF of SST are similar, except that the EOF generally has larger amplitudes, especially in the equatorial and North Pacific. Overall, much of the trend information in SST is contained in the leading EOF. The situation differs dramatically for upper-level zonal wind. Specifically, the leading EOF of zonal wind has no statistically significant trend, whereas the squared canonical correlation with the trend is 83%.

The canonical projection pattern is a spatial filter that maximizes the signal-to-noise ratio of the prespecified time series, and hence is analogous to climate change “fingerprints” that maximize the signal-to-noise ratio of the climate change signal (Hasselmann 1979; Hasselmann 1997; Allen and Tett 1999). To the extent that 50-yr trends are dominated by the response to anthropogenic and natural forcing (see DelSole et al. 2011), the canonical projection vector can serve as a “fingerprint” for real-time monitoring of the response to climate forcing.

The field significance test proposed in this paper makes several assumptions that may fail in practice. For instance, the test assumes that the errors are independent in time. In principle, autocorrelated errors can be taken into account by postulating an appropriate model for the autocorrelation, such as a multivariate autoregressive model, and deriving the corresponding likelihood ratio, although this entails increasing the number of parameters to be estimated. The test also assumes that the data are distributed as a multivariate Gaussian. Muirhead and Waternaux (1980) show that the likelihood ratio test is sensitive to departures from normality—even for large sample sizes, especially to long-tailed distributions. In cases in which violation of the underlying assumptions compromises the test, alternative techniques, such as based on the bootstrap or Monte Carlo methods, may become preferable.

The proposed test applies to any prespecified time series, not just trends. In particular, the proposed framework can be used to quantify the relation between a particular climate index and other variables, for example, the relation between ENSO and global precipitation. The framework (aside from EOF filtering) can be applied to variables with different units and different natural variances, since the component is invariant to nonsingular linear transformations of the data (see Anderson 1984, p. 490). Also, the framework can be generalized to multiple prespecified time series, such as orthogonal Fourier harmonics. The treatment of missing data within the proposed field significance test needs further investigation.