1. Introduction

If G is nonlinear and cannot be usefully linearized, then we are faced with a component extraction problem that is highly complex for many reasons. First, the definition of a well-adapted nonlinear component extraction model J is difficult without some a priori information about the nonlinear mixture model G. Using generic statistical models for the nonlinear regressions of J often introduces too many degrees of freedom, which ruins the inference process. Second, the determination of the extracted components is much more complex since the basis functions g_i[σ(j)], vary with location j. Third, the uniqueness of the result and its interpretation in physical-process terms is difficult to demonstrate. Some nonlinear techniques have been developed (Monahan 2000) but their main application is to represent complex data in a more compact way (i.e., compression). Here, we are not concerned with this use of such techniques, but with the extraction of “meaningful” (i.e., causal) components to understand how a dynamical system works. Currently, all the “classical” (and most frequently used) extraction methods are linear; the use of nonlinear models in (2) for component extraction is only just beginning. The nonlinear case is beyond the scope of this paper.

One approach that may be useful when G is complex is to linearize (1) using 𝗚[σ⁰(j)], the Jacobian matrix of the nonlinear operator G near a particular state, σ⁰(j):

View Expanded

where Δ is the difference operator, and the temporal basis functions g₁[σ⁰(j)], … , g_Q[σ⁰(j)], which are the columns of the matrix 𝗚[σ⁰(j)], are unknown time series describing a fixed dynamical behavior at each pixel, j. Each g_i[σ⁰(j)] is the temporal response to a perturbation Δσ_i(j) of ith component at pixel j when the state is given by σ⁰(j). For example, in the case where the physical component i is an oscillating wave propagating in space, the time series g_i[σ⁰(j)] have the same shape as the source but with a time delay dependent on the distance between their location at pixel j and the source of the wave.

When G is known a priori to be linear or has been linearized to 𝗚, the equivalent to (1)–(4) when the time series x(j) at particular spatial coordinate j ∈ {1, … , M} is decomposed in time, is:

View Expanded

where the temporal basis functions, g₁, … , g_Q, which are the columns of matrix 𝗚, are unknown time series describing a fixed dynamical behavior. In contrast with the nonlinear case, the basis functions g_i are independent of the geographical location j. Each g_i could be a signal with a different physical cause operative in a particular geographical region represented by a different component map {σ_i(j); j = 1, … , M}. The goal of the analysis is to infer the unknown contributing components σ(j) from the observed data x(j). In the linear case, we write (2) as

hJxσ(6)

where J is an estimate of the unknown matrix 𝗚⁻¹ (the superscript ⁻¹ represents the pseudoinverse if 𝗚 is not square) and h is an estimate of the unknown vector σ. The ability of statistical analysis techniques to retrieve good estimates, h, of the true components, σ, is highly dependent on the quality of the statistical dataset used (i.e., a sufficiently large number of independent examples is needed to sample all the variations involved) and on the technical assumptions that are made about 𝗝 and h.

There are many techniques that have been developed to estimate 𝗝 and h. The one most frequently used by the climate research community today is the Principal Components Analysis (PCA) or empirical orthogonal function (EOF) method (Lorenz 1951; von Storch and Frankignoul 1998). Sometimes, modifications of this method are used that either apply some other criterion besides maximizing the variance explained by each component or relax the requirement for orthogonal basis functions. These methods use this additional information to perform an orthogonal or oblique rotation of a previous PCA solution; we refer here to this large set of methods as rotational techniques (RT; Horel 1981; Richman 1986). We formulate a general linear case and the classical analysis techniques in section 2.

The PCA solution has some well-known difficulties. 1) The spatial orthogonality of the eigenvectors is not well-adapted to all problems. As a consequence, many EOF solutions display an artificial alternating sign structure. Horel (1981) quotes many cases in the literature of this artificial feature, for example for the sea level pressure in restricted regions such as Australia or the Arctic. 2) The principal components depend on the domain used for the analyses (see, e.g., different results obtained for the EOF analysis of the sea level pressure in the North American region; Buell 1975). 3) Weighting of geophysical data in the EOF analysis and in rotated analysis seems to have an important impact on the results obtained (Chung and Nugam 1999). Furthermore, (Karl et al. 1982) comment on possible distortions when using irregularly spaced data. 4) EOF components can be hard to interpret physically because of all these problems. All these concerns have led to the development of the rotational techniques (Horel 1981; Richman 1986).

The Independent Component Analysis (ICA) is described in section 3. This method, which can be interpreted as a rotation technique, is based on information theory and has been recently developed in the context of signal processing studies and of the development of neural coding models (Jutten and Herault 1991; Atick 1992; Bell and Sejnowski 1995). This technique has now been studied for some time by the statistical analysis research community and many recent applications of the ICA paradigm can be found in the ICA 1999 proceedings (Cardoso et al. 1999) or Hyvärinen and Oja (2000), but this method has not been used for analysis of climatological observations (Aires et al. 2000). The two major distinctions between the ICA approach and the classical techniques are the following.

• The method extracts statistically independent components, even if these components have non-Gaussian probability distribution functions, by making use of higher-order statistics, whereas the PCA and RT approaches use only second-order statistics.

• A linear mixture model is not assumed; any extraction model could be used with the ICA paradigm (Burel 1992), which allows for the introduction of pertinent a priori information about the mixture model, if it is available.

We will show that the “linear” (this term will be explained in the following) and PCA-initialized form of ICA, described here, performs a rotation of the PCA solution, eliminating the mixing problem: PCA has the tendency to mix several components, even when the signal is just the linear sum. We argue that, because of certain general features of the ICA approach, it is a particularly promising technique for rotations of the PCA solution. Furthermore, a previous study that applies a similar ICA to the analysis of variations of tropical sea surface temperature time series (Aires et al. 2000) illustrates its potential to separate a geophysical time series into more meaningful components.

To illustrate most clearly how ICA avoids the component mixing problem, we construct a synthetic dataset, where the true answer to the decomposition problem is known, and apply PCA-initialized Independent Component Analysis to extract components (section 4). We have deliberately devised a dataset that exagerates characteristics found in the real atmospheric observations to, on the one hand, make it even easier for PCA to separate the components and to, on the other hand, challenge whether ICA can separate distinct modes of variability when they are similar in their space and time distributions. In particular, the synthetic dataset is formed by a linear sum of components, some of which are better-separated in space and time than in real atmosphere, and some of which overlap in space, are correlated in time, or represent teleconnections as in the real case. Thus, this dataset is created so that it has structures that a linear component extraction techniques should find. If a method fails on this simple case, it is unlikely to succed when applied to the more ambiguous climate case. We show that, even in the simple linear case, the PCA technique mixes the components incorrectly. We also show that the ICA method performs a rotation to correctly separate the components, but, as for all statistical component extraction methods, this is only a necessary condition not a sufficient guarantee of success for climate analysis.

The goals of this paper are then to illustrate some of the problems of the most commonly used classical analysis technique, even in the simple linear case, and to introduce a new component extraction technique that overcomes these problems, at least in its linear form (section 5). We compare linear ICA to PCA to measure the effect of the rotation transformation by the ICA algorithm. We do not use a priori information for the component extraction experiments since we are interested in exploratory techniques, not confirmatory ones; that is, we want a technique to find the correct but unknown components, not confirm results from another analysis.

2. The linear case and classical component extraction techniques

One particular decorrelation solution is the well-known PCA or, in the geophysical community, EOF¹ analysis, first used in atmospheric sciences by Lorenz (1951). In this technique, an additional constraint is added to resolve the indeterminacy of the decorrelation solutions: successive extracted components have to explain the maximum remaining variance. This solution is given by taking Θ = I_Q×Q in (7). Depending on which space the PCA is applied to (space, time, frequency, multivariate data, etc), the PCA has also been called Singular Spectrum Analysis (Broomhead and King 1986; Vautard et al. 1992), Multichannel Singular Spectrum Analysis (Vautard et al. 1996), Extended Empirical Orthogonal Functions (Korres et al. 2000), Multivariate Empirical Orthogonal Function (Xue et al. 2000), etc.

Three well-known problems arise when using the PCA technique. 1) Even if the mixing of the components is linear as in Eq. (5), the maximum-explained-variance assumption can lead to a different mixing in the extracted components (Kim and Wu 1999) as we will be shown here (see Fig. 1a for an schematic illustration of this problem in a two-dimensional case). 2) This mixing problem is also particularly serious when the PCA is applied to data that have more than one component with about the same variance. In this case, the problem is not solvable since any orthogonal rotation of the principal components (i.e., in the space of the “degenerate” eigenvectors) will be a PCA solution (Fig. 1b). 3) Since PCA imposes orthogonality on the extracted basis functions, mixing problems also arise when the actual physical basis functions are not orthogonal (Fig. 1c). Another problem for the application of the PCA to geophysical data arises from an irregularly spaced grid of pixels that can lead to distorted basis functions (Karl et al. 1982).

The PCA assumptions (linearity, orthogonality, maximum variance explained by each successive component) used to resolve the solution indeterminacy are not known, a priori, to be valid for a particular dataset. They are not likely to be valid for climate variations. If these assumptions are not valid, variations that are not physically connected could be artificially mixed together into one extracted component (i.e., the mixing problem). This is the reason why PCA is often used in restricted geographical domains instead of global domains or applied to prefiltered data to try to isolate a single dominant mode of variation, which PCA can correctly identify. Thus, although PCA is useful for compressing information by describing the most variance with the fewest terms in an expansion (as a dimension-reduction/compression technique), it can lead to misinterpretation of physical relationships when used as a component extraction technique.

Rotational techniques were introduced (Horel 1981; Richman 1981), in part, to obtain a more physically interpretable solution and to avoid some of the problems of PCA. In these approaches, an additional constraint of localization, based on the so-called simple structure principle, is used to solve the indeterminacy of the decorrelation solutions. The rotations are said to be orthogonal (the rotation matrix is an orthogonal matrix) or oblique (this constraint is relaxed). There exist many proposed localization criteria: quartimax, varimax, transvarimax, quartimin, oblimax, etc. [see the review paper of Richman (1986) on this subject]. Two general distinct classes of RT solution can be distinguished: confirmatory RT where a priori information about the components is available and we want to verify the hypothesis, and exploratory RT where almost no a priori information about the problem is available. We are interested in the exploratory case where no a priori information is available. Since no general principle for choosing a particular localization criterion from this large set of proposed solutions is available, use of a particular RT method in exploratory mode may be equivalent to introducing a priori information about the localization that may not be any better suited to the particular problem than PCA.

The most commonly used criterion for orthogonal rotation is

View Expanded

where the constant γ gives a family of rotations with γ = 1.0 giving varimax rotations, and γ = 0.0 giving quartimax rotations. The implementation of these techniques is not trivial, but automatic routines are available (see, e.g., the routine G03BAF in the NAG FORTRAN Library Routine Document).

Despite the proposed alternatives to the variance maximization assumption and the orthogonality constraint used in various RT methods, they still all share two fundamental properties with PCA: they assume that the meaningful components are linearly mixed (classical techniques are intimately linked to the linear assumption and cannot be generalized to nonlinear models) and that only second-order statistics need be evaluated.

3. The Independent Component Analysis technique

In this section, we describe the main concepts underlying the ICA technique. For more details, the interested reader is referred to Bell et al. (1995) and Aires et al. (2000). The ICA technique aims to extract statistically independent components, a stronger constraint than the decorrelation requirement of the PCA.

It is also important to distinguish the non-Gaussian character of the components, σ, from the non-Gaussian character of the data x in Eq. (5). If the data have a non-Gaussian distribution, then at least one component is also non-Gaussian, since for the simplest linear mixture of Gaussian components (not to be confused with a “mixture of Gaussians,” which usually means that the random variable has one of a number of possible distributions), the distribution would be Gaussian, but a nonlinear combination of Gaussian distributions could also be non-Gaussian. Some previous studies examine the non-Gaussian behavior of geophysical data (Burgers and Stephenson 1999; Aires et al. 2000).

A variable is characterized by all its statistical cumulants: the first cumulant is the mean, the second is the variance, the third is the skewness, the fourth cumulant is the kurtosis, etc. (Press et al. 1992). For Gaussian variables, cumulants higher than 2 are zero. When data have a zero mean, the “skewness” skew(X) = 〈X³〉/σ³ and the “kurtosis” kurt(X) = 〈X⁴〉/σ⁴ − 3. These cumulants are often used to test the departure from Gaussian behavior. The skewness measures the symmetry of the probability distribution function: when the skewness is positive, larger events are more probable then smaller events, and the reverse is true when the skewness is negative. The kurtosis is a measure of the sharpness of the distribution: a negative kurtosis indicates that the distribution has a broader central peak and larger tails than a Gaussian distribution (sub-Gaussian), a positive kurtosis indicates that the distribution has a sharper central peak (super-Gaussian distribution). The non-Gaussian character of a variable is intimately linked to nonlinear dynamics (Palmer 1999). For example, a nonlinear dynamical system with two attractors can result in bimodal distributions. Thus, without a priori information on the Gaussianity of components in an analysis of geophysical time series, the use of ICA is recommended since its requirement of statistical independence is more general than the decorrelation assumption.

The parameters, W_i, are estimated by applying a gradient descent algorithm to a cost function that specifies the statistical independence of the {h_i; i = 1, … , Q}. Different equivalent cost functions can be used; we use here the infomax approach to ICA (Nadal and Parga 1994) for which simple algorithms have been derived (Bell and Sejnowski 1995). Information theory is used to specify the statistical independence cost function: the fundamental quality is information redundancy. Given Q variables, h₁, h₂, … , h_Q, the information redundancy, R(h₁, h₂, … , h_Q), is defined as the Kullback divergence (Dacunha-Castelle and Duflo 1982) between the joint distribution, P_h(h₁, h₂, … , h_Q), and the factorized distribution, P₁(h₁) · P₂(h₂), … , P_Q(h_Q):

View Expanded

This information redundancy measures the difference between the joint and the factorized distribution: when the redundancy R(h) = 0, P_h(h) = Π^Q_i=1 P_i(h_i), which means, by the definition in Eq. (9), that the components of vector h are statistically independent. An important remark is that, since no geometric constraint on the basis functions is specified in the redundancy quality criterion of (12), the basis functions extracted by ICA, in contrast to PCA, can be nonorthogonal.

A statistical regression model for the extraction model in Eq. (11) has to be specified. For the nonlinear mixture case the regression model needs to be nonlinear in order to simulate G⁻¹. The Multi-Layer Perceptron (MLP), an artificial neutral network model, could be chosen for such a case. The nonlinear mixture case will be the subject of future work.

The parameters, W_i, defining the matrix J and the optimal transfer functions f have to be determined by minimizing the redundancy criterion in (12). Practically, it has been demonstrated in various applications (Bell and Sejnowski 1995) that full optimization of the transfer functions is not necessary for performing ICA. Although promising results have been obtained, this analysis strategy can be improved by introducing some partial adaptation of the transfer functions to the particular problem. We use here the classical sigmoid function f(x) = 1/(1 + e^−β*x) that has proven generally useful.

With the information redundancy reduction criterion, (12) and a no-hidden-layer architecture, a straightforward algorithmic implementation of the ICA has been found (Bell and Sejnowski 1995) to estimate the matrix 𝗝:

View Expanded

where J_ik(n) is an element of the matrix 𝗝 at step n of the gradient descent, ρ is the learning rate parameter of the gradient descent, and

View Expanded

This algorithm is described in a more practical way in the appendix.² Note that, although the theory behind this analysis method may seem complex, the actual computational procedure that results for the linear case is relatively simple.

ICA can be applied to the raw data, x(j), but it has been shown (Nadal et al. 2000; Aires et al. 2000) that a PCA preprocessing of observations makes the gradient descent step stabler and faster. The N-dimensional data x(j) are projected onto their first N′ (where N′ < N) principal components using the matrix 𝗝₀ of PCA: observation noise is reduced and fewer free parameters need to be estimated because the dimensions of the matrix 𝗝 are considerably reduced, from Q × N to Q × N′. The dimension of the compression N′ is chosen to be equal to Q, the number of extracted components, thus, 𝗝 is a Q × Q matrix. The PCA-compression preprocessing step does not alter the components extracted by ICA if the neglected components (including noise) are statistically weak sources (see Nadal et al. 2000 for a definition of the term “weak”). In this configuration, the ICA can be considered equivalent to performing an oblique rotation of the PCA solution by generalizing decorrelation to statistical independence, but no additional criterion needs to be selected from among a large number of possibilities like the localization constraints in classical RT. The rotation matrix of the PCA solution 𝗝₀ is the ICA solution Q × Q matrix 𝗝. If the real physical components are Gaussian-distributed, then decorrelation is sufficient and the ICA algorithm would not change the PCA solution since zero gradient is already obtained. In the case where the components are non-Gaussian, the ICA rotates the initial PCA solution. So ICA improves the PCA solution only in the non-Gaussian case. We note that there are many examples in the literature showing non-Gaussian distributions of climate parameters, for example, cloud optical thickness (Rossow and Schiffer 1991), precipitation water path (Lin and Rossow 1996), and SST (Aires et al. 2000).

4. Application to a linear sum of components

5. Concluding remarks

For extraction of physically meaningful modes from observations, where the characteristics of the system's dynamics are not (well-) known, identifying statistically independent variation modes seems to be a sensible alternative for the rotation of a first PCA (i.e., EOF) solution to avoid the PCA mixing problem. Our simple example shows that in the most general, though still linear, case (the most favorable condition for PCA), PCA will mix modes of comparable magnitude, generating spurious regional overlaps or teleconnections where none exists or distorting existing overlaps or teleconnections. In the case of correlated modes, PCA produces mixing by combining the correlated parts of the modes and separating the less correlated parts, spuriously dividing some modes. The PCA is useful in a lot of applications, particularly when the user is interested in only one strong component in the observations that explains the maximum of variance. Note, however, that such use in recent studies requires substantial prefiltering of the data to isolate such a mode; this is equivalent to application of very strong a priori information about the mode in question. But to extract components when there are several, PCA results will to be misleading, even for a simple linear mixture.

We have shown the potential of the ICA technique for separating a complex signal in a more meaningful way. The mixing problem inherent in the PCA technique and the artifacts produced by the orthogonality and maximum-variance constraints of PCA are avoided when rotated by ICA. Moreover, the use of higher-order statistics by ICA to determine statistical independence assumes only the generalization of the decorrelation used in all classical approaches. In some cases, as we have shown, use of higher-order statistics is key to separating similar modes. Nevertheless, even statistical independence does not guarantee that the modes produced by different physical processes will be separated. Like other statistical methods, without a priori information about the actual physical modes in the observations, there is no guarantee that the components extracted have a physical meaning. The user of a particular technique should keep in mind the assumptions used in the technique, the qualities and deficiencies of the method, and be able to put these in the context of each application. The advantage of ICA is its straightforward criterion, that is, the statistical independence of its extracted components.

Some practical disadvantages of ICA exist. As with rotation techniques, it needs an a priori definition of the number of components to extract. But like oblique rotations (Richman 1981), ICA still should be able to extract meaningful components even if more components are extracted than there are actually in the observations (i.e., overfactoring). Practically, the use of higher-order statistics requires many more samples than using second-order statistics, placing stronger demands on the data needed. In addition, the higher-order moments are more sensitive to outliers; however, our use of PCA as a first step in the PCA helps reduce this problem by removing most of the noise and any low frequency-of-occurrence outliers. Computationally, ICA requires more resources than PCA, but the convergence of the ICA algorithm is very fast.

ICA, by finding statistically independent modes, may provide a better starting point to explore the unknown dynamics of a system. In the case of climate variations, where the components of the system are probably coupled (see, e.g., Salby and Callaghan 2000 or Krishnamurthy and Goswami 2000), considering the modes to be as statistically distinct as possible, even with a linear-ICA, would provide “prototypical” components that might serve as a guide to further investigation. As with the classical PCA technique or classical RT, this first (linear) ICA algorithm is not able to deal correctly with propagating components or components mixed nonlinearly. However, the ICA paradigm (statistical independence) may be a sufficiently powerful concept to be generalized using more advanced statistical models (e.g., more complicated neural networks) to treat nonlinear problems. This requires development of nonlinear solution algorithms and their testing for cases where the combination of modes is nonlinear, when components are physically linked, and for cases with propagating modes.