In their comment to my critique (Christiansen 2005a) of using nonlinear principal component analysis (NLPCA) for regime detection, Monahan and Fyfe (2007) claim that my central argument is that previous results of NLPCA (Monahan 2000; Monahan et al. 2000, 2001, 2003; Teng et al. 2004) are sensitive to sampling errors and not robust. This is a misunderstanding. Christiansen (2005a) showed that the NLPCA produces multiple regimes because of the intrinsic nonlinear structure of the algorithm. Because of this geometric effect, the NLPCA will find multiple regimes even in sufficiently isotropic Gaussian-distributed data.

In particular I showed that the NLPCA has theoretical problems:

• Gaussian inputs easily produce multimodality in the nonlinear principal component because of the nonlinearity of the neural network.

• The parameterization is ambiguous (Malthouse 1998) and bimodality in the nonlinear principal component can always be transformed away.

• For two neurons in the second and fourth layer of the neural network the NLPCA mode is either a simple C-shaped curve or a Z-shaped curve.

While the first or second of these analytical observations alone should keep us from using NLPCA for regime detection I showed the following numerically:

• The NLPCA detects bimodality even for Gaussian-distributed data when the isotropy of the data is sufficiently large.

Monahan and Fyfe (2007) hypothesize that a test of the reproducibility will allow one to distinguish between false positives, such as the regimes found in Gaussian-distributed data, and “real” regimes such as those found in the atmospheric data. Monahan and Fyfe (2007) now describe this test in much more details than in previous papers. However, I did study the reproducibility by analyzing different subsets each containing 80% of the data. I found the following:

• There is no difference between the reproducibility of the atmospheric data and the reproducibility of unimodal surrogate data with the same number of samples, the same serial correlations, and the same widths as the atmospheric data.

Monahan and Fyfe (2007) also claim that my results are not robust due to overfitting. This is not correct. I did study the effect of overfitting by adding a penalty term, and again I found no difference between the reproducibility of the atmospheric data and the unimodal surrogate data.

Therefore, false positives will not be rejected by tests of sampling sensitivity, and I find it hard to believe that the example in Monahan and Fyfe (2007) is representative. Note also that the theoretical problems listed above are unconstrained by sample size and overfitting. But, even if NLPCA was able to detect non-Gaussianity the ambiguity in the nonlinear principal component (Malthouse 1998) would make any inference about the existence of regimes impossible. Taken together the five points above show that the NLPCA will not be suited for regime detection even if the basic algorithm is embedded in a complicated and complicating validation scheme.

Monahan and Fyfe (2007) further argue that although the nonlinear principal component is ambiguous, the NLPCA approximation is unique. They imply that even if multimodality can be transformed away in the nonlinear principal component the distribution of the NLPCA approximation is physical. However, the clustering of the NLPCA approximation is a spurious effect—also present when Gaussian distributions are analyzed—of the fact that the method insists on producing a Z-shaped NLPCA mode with a sparsely populated middle branch. Figure 1 shows an example with data drawn from a bivariate Gaussian distribution. The NLPCA effectively resembles a least squares projection on two parallel lines with about half of the data projected onto each line. Because the distribution of the data has a peak at the center the NLPCA approximations (filled circles) will cluster around the projections of (0, 0) on the NLPCA mode (thick curve). In Fig. 1 these projections are near (−1, 0) and (1, 0).

The problems with NLPCA do not rule out the existence of circulation regimes. For example, studies of the wave amplitude index have shown a robust and statistical significant bimodality (Hansen and Sutera 1995; Christiansen 2005b). But NLPCA does teach us an important lesson. We are increasingly introducing more and more complicated and nonlinear methods to extract information about the atmospheric structure. It is of utmost importance to know the behavior and statistical properties of these methods before we apply them to atmospheric data. If not, we risk unconsciously misinterpreting the analysis to find the results we expect—the observer–expectancy effect. Finally, I find that several clustering methods show similar problems as the NLPCA; they often report regimes in data that are drawn from unimodal distributions (Christiansen 2007).