1. Introduction

The possibility of the existence of multiple atmospheric regimes in the large-scale flow has attracted much attention in the last decades. In a nonlinear system with multiple regimes an external forcing could result in a change of the frequencies of the regimes while leaving the spatial structure of the regimes almost unchanged as pointed out by Palmer (1993, 1999). In this conceptual picture, changes of the time average are a consequence of changes in the frequencies of the regimes. The spatial response patterns to possible anthropogenic forcings might then be almost identical to the patterns of natural variability. In this case discrimination between different anthropogenic changes and natural variability, for example, with the optimal fingerprint method (Hasselman 1993), would be almost impossible as no one-to-one correspondence would exist between forcing and response. In this paper we provide observational evidence for this nonlinear dynamical perspective on climate prediction.

Several methods have been used in the search for multiple regimes in a variety of atmospheric fields. These methods include the search for multimodality in probability density distributions (Kimoto and Ghil 1993; Hansen and Sutera 1986; Corti et al. 1999) and cluster analysis (Mo and Ghil 1988; Chen and Wallace 1993). See Ghil and Robertson (2002) for a recent review. It is fair to say that the existence of multiple regimes in the large-scale climate has not yet been firmly established (Stephenson et al. 2002). A major problem is the short length of the global-scale climate records and the consequent weak statistical significance.

Early observational evidence for multiple regimes in the tropospheric flow was provided by Hansen and Sutera (1986), who tested ideas about the zonal index cycle put forward by Charney and Devore (1979). Hansen and Sutera (1986) found bimodality in the probability density function of a wave amplitude index measuring the daily strength of the waves with wavenumbers 2–4 in the Northern Hemisphere midlatitudes. Subsequently, the robustness of this result with respect to changes in the details of the calculations has been debated (Nitsche et al. 1994; Hansen and Sutera 1995).

Recently, Corti et al. (1999) and Monahan et al. (2001) reported the existence of multiple regimes in the tropospheric circulation in studies of the probability density in the reduced space spanned by the two leading principal components (PCs) of the monthly mean 500-hPa geopotential height. However, there is a considerable sampling variability in the probability density estimates and Monte Carlo tests (Hsu and Zwiers 2001; Christiansen 2002), which show that the existence of multiple regimes is not statistically significant.

Multiple regimes in the stratosphere were reported by Pierce and Fairlie (1993) based on 10 yr of daily data. They identified three regions in the probability density distribution in the space spanned by the zonal wind and the amplitude of geopotential height wavenumber 1 at 10 hPa. However, the regimes are not well separated and regime borders must be subjectively defined, in particular when a longer period is studied (Pawson and Kubitz 1996).

In this paper we present evidence for two circulation regimes in the stratospheric inter-annual variability and for an abrupt regime shift in the late 1970s. The regimes are identified as two clearly separated peaks in the probability distribution of the leading PC of the geopotential height.

If one wants to study the global circulation patterns, the stratosphere offers the important advantage over the troposphere of having far fewer degrees of freedom (Perlwitz and Graf 2001). In the stratosphere only the largest scales are present in contrast to the situation in the troposphere where smaller scales are abundant. A related advantage is that the leading PCs in the stratosphere typically explain more of the variance than the leading PCs in the troposphere. With the winter means used in this work, the two leading PCs explain approximately 50% and 20% at 20 hPa but only approximately 20% and 15% at 500 hPa. We will therefore restrict the study to the leading PC avoiding the complications of calculating and analyzing multidimensional probability densities.

Special care should always be taken when drawing conclusions from short time series where sampling errors are expected to be large. A related problem is the possibility of serial correlations in the data, which reduce the number of independent points and therefore increase the sampling errors. To deal with these concerns we have performed extensive Monte Carlo simulations to test if the existence of the two regimes and the regime shift could be due to random sampling errors.

2. Data and methodology

In this study we have used the National Centers for Environmental Prediction–National Center for Atmospheric Research (NCEP–NCAR) reanalysis (Kalnay et al. 1996) geopotential heights, zonal winds, and temperatures from 1948 to 2001. The data have been subsambled from the original grid of 144 longitudes and 73 latitudes to a coarser grid of 72 longitudes and 36 latitudes.

Additionally we have used the monthly mean geopotential height analysis from the Free University of Berlin (FUB; Pawson et al. 1993) from 1957 to 1997 on a grid with 10° resolution.

To study the interannual variability we have used temporal averages over the Northern Hemisphere winters defined as the cold half-year from October to March. We thus have 53 winters from the NCEP–NCAR reanalysis and 40 winters from the FUB analysis. The winters are identified by the year that includes January. Long-term trends have not been removed.

Empirical orthogonal functions (EOFs) are calculated from data north of 20°N after each data point has been weighted with the square root of the area it represents. The scaling of the PCs and the EOFs has been chosen so that the spatial variance of the EOFs is one.

Probability density functions are calculated by the kernel density estimate procedure with a Gaussian kernel using the algorithm based on the fast Fourier transform (Silverman 1986). The kernel density algorithm contains a smoothing parameter, h, which is determined objectively by a least squares cross-validation procedure. Bimodal structure of the probability density estimate (PDE) will be taken as evidence for the existence of two different regimes. However, due to sampling errors a small sample from a unimodal distribution can often show multiple maxima in the PDE. To estimate this effect a Monte Carlo approach is applied where a large number of samples are drawn from the unimodal distribution that best describes the data (Silverman 1986).

3. Two regimes and a regime shift

Figures 1 and 2 show the leading PC and the associated EOF of the winter means of the 20-hPa geopotential height from the NCEP–NCAR reanalysis. The leading PC varies between approximately −70 and 100 m. Visual inspection suggests that there are data clusters around −40 and 30 m with an affinity for the lower value in the first half of the period and for the higher value in the second half of the period. We note that both clusters are visited in both periods. This plot is our basic evidence for the existence of two regimes and for a regime shift. Based on this plot we choose, somewhat arbitrarily, the year 1978 as the year of the shift.

The leading EOF explains 46% of the variability and is well separated (North et al. 1982) from the secondary EOFs, which explain 20%, 15%, and 9% of the variability. The leading EOF (Fig. 2) is mainly annular with a strong center over the pole and a ring of opposite polarity over midlatitudes. Positive values of the leading PC will therefore correspond to a strong vortex situation while negative values will correspond to a weak vortex situation. The leading PC therefore to a large extent amounts to a zonal index of the stratospheric circulation. The annular structure of the leading EOF is found for many different timescales and the pattern is characteristic for the stratospheric part of a coupled mode of stratospheric and tropospheric variability (Perlwitz and Graf 1995; Kodera et al. 1996). In particular, this pattern is found as the stratospheric part of the Arctic Oscillation (Thompson and Wallace 1998). The regime shift is therefore a change from a climate with a larger probability for a weak vortex (and a negative Arctic Oscillation index) to a climate with a larger probability for a strong vortex (and a positive Arctic Oscillation index).

Including all the years we find a linear trend of 1.5 m yr⁻¹ in the leading PC. This deepening of the vortex has previously been reported by several investigators (Perlwitz and Graf 1995; Graf et al. 1995; Kodera and Koide 1997; Randel and Wu 1999; Thompson et al. 2000). However, as indicated by Fig. 1, a linear trend is not a good description of the data, which, as mentioned above and elaborated on below, show an abrupt shift in the second half of the 1970s.

The PDE for the leading PC is shown in Fig. 3a for the full period (thick curve) and for the two subperiods before (1949–78) and after (1979–2001) the regime shift. The PC has been normalized to unit variance. The PDE for the full period is clearly bimodal with well-separated local maxima at −0.85 and 0.55. The PDE for the period 1949–78 is unimodal with maximum at −0.85 and a shoulder at 0.5. The PDE for the period 1979–2001 is bimodal with the largest maximum at 0.5 and a smaller maximum at −1.4. The PDEs are calculated with a smoothing parameter h = 0.4, the center of a rather broad minimum in the score function, which measures the integrated error between the PDE and the true probability density (Silverman 1986).

A similar bimodality and a similar regime shift are found for the first PC of the geopotential height for other levels in the stratosphere from 20 to 100 hPa but not in the troposphere. This is demonstrated in Figs. 3b and 3c, which show the PDE of the leading PC of the geopotential height at 70 and 500 hPa. At 70 hPa the leading PC explains 49% of the variance and is well separated from the secondary EOFs. This is not the case at 500 hPa, where the leading PC explains only 21% and the second PC explains 17% of the variance.

To consider the vertical depth of the two regimes, we stratify the 53 winters into weak and strong regime years according to the values of the leading PC at 20 hPa. Weak regime and strong regime winters are chosen be winters with the normalized PC less than −0.5 (21 yr) and larger than 0.5 (18 yr), respectively. The averages of the zonal mean zonal wind and the zonal mean temperature over the two sets of winters have been calculated and the differences are shown in Figs. 4a and 4b. As expected the weak regime and strong regime winters are dominated by weak and strong stratospheric winds, respectively. The largest difference of 10 m s⁻¹ is found at 10 hPa, 65°N. The difference decreases poleward and equatorward of 65°N and the difference also decreases with decreasing height. At 60°N differences statistically significant at the 99% level can be found all the way through the troposphere to the surface, but below 200 hPa the difference is everywhere smaller than 3 m s⁻¹. The statistical significance is calculated by a Monte Carlo procedure, where 2000 sets each consisting of two subsets of 18 and 21 yr are chosen randomly, their zonal mean zonal wind differences calculated, and the resulting distribution compared to the observed zonal mean zonal wind difference. This method is valid because the leading PC can be considered free of serial correlations as we will show in the next section. The pattern of Fig. 4a is almost identical to the pattern of the linear trend in the zonal mean zonal wind for the period 1968–97 presented in Thompson et al. (2000) and to the pattern obtained by regressing the Arctic Oscillation index on the zonal mean zonal wind (Thompson and Wallace 2000). The strong regime winters are much colder in the middle and lower polar stratosphere than the weak regime winters (Fig. 4b). In the lower stratosphere the temperature is almost totally determined by the leading PC of the geopotential height at 20 hPa with correlations reaching values stronger than −0.95. The negative temperature difference is statistically significant down to 400 hPa, while a positive but not significant temperature difference is found near the surface.

The average difference in the geopotential height at 500 hPa between strong regime and weak regime winters is shown in Fig. 4c. The pattern has an overall annular structure with a negative center over the pole and a positive elongated center over the North Atlantic. Positive centers are also found over the eastern part of the Eurasian continent and the western part of the Pacific. At the centers the difference is statistically significant at the 99% level. Patterns almost identical to Fig. 4c appear in several studies of the coupling between the stratosphere and troposphere. Baldwin et al. (1994) and Perlwitz and Graf (1995) found the pattern as the leading coupled mode of 50- and 500-hPa geopotential heights using singular value decomposition of seasonal means and canonical correlation analysis of monthly means, respectively. Kodera et al. (1996) and Kodera et al. (1999) obtained the pattern by calculating the correlation between a stratospheric index and the 500-hPa geopotential height. Recently, this connection between the stratospheric vortex and the tropospheric variability appears from studies of the Arctic Oscillation, and Fig. 4c resembles the pattern obtained by regressing the Arctic Oscillation index on the geopotential height at 500 hPa (Thompson and Wallace 1998). The pattern of Fig. 4c also appears in calculations of linear trends in the 500-hPa geopotential height (Graf et al. 1995; Kodera and Koide 1997). Furthermore, the pattern appears when the 500-hPa geopotential heights are regressed upon the hemispheric mean surface air temperature (Wallace et al. 1996). The patterns mentioned above and that of Fig. 4c only differ significantly over the Pacific Ocean.

The average difference in the sea level pressure and in the 1000–500-hPa height thickness between weak regime and strong regime winters are shown in Figs. 4d and 4e. The sea level pressure difference looks almost like the Arctic Oscillation pattern except that the positive center over the Pacific is located too far north. Compared to the 500-hPa difference it has weaker amplitudes over the continents. The pattern of the 1000–500-hPa thickness difference indicates that strong vortex years tend to be warmer in the lower troposphere than weak vortex years over most of the Northern Hemisphere. The largest signals are found over the continents and the pattern resembles the cold ocean–warm land (COWL) pattern (Wallace et al. 1996), which accounts for about half of variability in the hemispheric mean surface air temperature. The COWL pattern also contributes with more than 50% to the upward trend in the hemispheric mean temperature.

One could worry that the observed regime shift is unphysical and caused by inhomogeneities in the data, for example, related to the introduction of satellite observations in 1979. As a test we repeated the analysis with an independent dataset from the Free University of Berlin covering the years from 1957 to 1997. This dataset is based on subjectively analyzed radiosonde observations. As seen from Figs. 5 and 6 the FUB data agree with the NCEP–NCAR reanalysis on the existence of the two regimes and the regime transition in the last half of the 1970s.

4. Statistical tests

Although the existence of the two regimes seems clear from the time series (Fig. 1) and the PDE (Fig. 3), we need to test if the bimodality and the regime shift are not chance occurrences. Our results are based on a moderate number of 53 winters and they might be seriously compromised by sampling variability. If the winters are serially correlated the number of independent points is even lower. We will apply Monte Carlo tests to first establish the statistical independence of the 53 winters and then to test for the bimodality and for the shift. The Monte Carlo test will be performed by drawing many sets of numbers from a specified distribution, calculating a measure of the feature under consideration for each set, and then compare with the measure calculated from the original data. The stratospheric reanalysis is less reliable before the International Geophysical Year 1957/58 and we therefore repeat the statistical analysis including only the 44 years 1958–2001. For comparison the probability density estimate based on this reduced period is shown in Fig. 3d.

We first show that the 53 winters can be considered as independent. To this end we consider 10 000 sets of 53 numbers randomly drawn from the bimodal distribution shown in Fig. 3a. By construction these sets consist of independent numbers. We continue by calculating a measure of the decorrelation time for each of the 10 000 sets and by using these to estimate the distribution of the chosen measure. As the numbers are independently drawn the measure of the decorrelation time is only different from zero because of the finite samples. We then compare this distribution with the measure of the decorrelation time calculated for our original time series to see if it can be considered different from the randomly drawn sets. A simple measure of the decorrelation time is the sum Σ^N/2_i=1 C²_i over the autocorrelation spectrum C(i) with lags from 1 to N/2 (instead of N to avoid boundary effects). The calculated estimate of its distribution is shown in Fig. 7. A large part of the distribution, 30%, lies to the right of the value found from our original time series. Considering only the subperiods before and after 1978 gives 57% and 47%, respectively, and considering the period 1958–2001 gives 47%. Other measures of the decorrelation time give similar results. In the following we will therefore assume that the 53 samples of the leading PC are independent as this hypothesis is consistent with the data. Strictly speaking the regime shift in 1978 introduce a low-frequency dependence in the data, which is too weak to be detected with the method applied above and which is not important for the statistical tests below.

We now turn to the question of the statistical significance of the bimodality of the PDE in Fig. 3a. We do this by fitting the data to a unimodal distribution and then asking how often a set of 53 numbers randomly drawn from this unimodal distribution will produce a severe bimodal distribution when the PDE is calculated in the same way as the PDE in Fig. 3. The unimodal distribution, P^1.0_u(x), is found by increasing the smoothing parameter h until the distribution of the original data just becomes unimodal, which happens for h = 0.7. This test for bimodality was described in Silverman (1986) and used in Hansen and Sutera (1986). Unfortunately, the resulting distribution is too populated in the tails and it is not a good estimate of the probability distribution. It is a well-known problem of the kernel density estimation procedure with fixed h to overestimate the probability in sparsely populated regions. To overcome this problem we also consider the scaled distributions P^a_u(x) = aP^1.0_u(x/a), with a = 0.9, 1.0 ⋯ 1.4. Three of the unimodal distributions are shown in Fig. 8. When drawing 53 numbers from one of these distributions and calculating the PDE with h = 0.4 as in Fig. 3a, the PDEs will often be bimodal. To calculate a measure, B, of the bimodality we proceed as follows. First we locate the local minima x^min_i. For each local minima we calculate the depth D_i = P^a_u(x^max_i) − P^a_u(x^min_i) of the minimum by finding the smallest [x^max_i, P^a_u(x^max_i)] of its two neighboring local maxima. We then set D = max(D_i), that is, the largest of depth of all local minima. If the PDE is unimodal the depth is defined as 0. As a measure of the width of the distributions we use the standard deviation W.

Drawing 10 000 sets of 53 numbers randomly from P^a_u gives the statistics shown in Table 1. With increasing a, corresponding to more narrow distributions, the probability for obtaining a bimodal PDE decreases. For a = 0.9, we find that 17% of the 10 000 sets have a unimodal PDE while the fraction is 82% for a = 1.4. For a = 0.9 we find that 18% of the randomly generated sets have a depth D larger than that for the original time series and therefore can be considered as more severe bimodal. For a = 1.4 this fraction has decreased to 1.4%. Increasing a also, per construction, decreases the average width of the randomly drawn sets. For a = 0.9 almost all, 99.9%, of the random sets have a standard deviation larger than that of the original time series, while this number is only 3.7% for a = 1.4. We therefore reject the distributions P^0.9_u and P^1.0_u as estimates of the PDE of the original data, while the widths of P^1.1_u, P^1.2_u, P^1.3_u, and P^1.4_u are consistent with the original data. We now note that less than 7.5% of the randomly generated sets from these distributions have a bimodality more severe than the original data. Considering only the 44 yr since 1958 does not change the results significantly as can be seen from Table 1 where the results from the reduced period are shown in parentheses. We can therefore conclude that it is unlikely that the original data are drawn from a unimodal distribution.

We note that the test performed above is rather strict as it tests against a very flat unimodal PDE. If the test was done against the best Gaussian fit to the original data only 1.3% of the randomly generated sets would have a larger value of D than the original data.

Finally we try to establish statistical evidence for the regime shift in 1978. Our normalized leading PC contains 30 winters before the shift with a mean of −0.30 and 23 winters after the shift with a mean of 0.39. We want to test if this is a rare situation if all winters are from the bimodal PDE in Fig. 3a. We now draw 10 000 sets of 30 numbers randomly from the bimodal distribution in Fig. 3a and calculate the mean of all these sets. Only 8% have a mean smaller than −0.30. Likewise we draw 10 000 sets of 23 numbers from the bimodal distribution. Only 3% of these sets have a mean larger than 0.39. Considering only the 44 yr since 1958 the normalized leading PC contains 21 winters before the shift with a mean of −0.28 and 23 winters after the shift with a mean of 0.27. Drawing 10 000 sets of 21 and 23 numbers from the bimodal distribution in Fig. 3d we find that 12% have a mean smaller than −0.29 and 10% have a mean larger than 0.27. We therefore conclude that the proposed regime shift in 1978 is consistent with the data.

5. Conclusions and discussion

We have presented evidence for the existence of two different circulation regimes in the Northern Hemisphere stratospheric interannual variability. The regimes correspond to weak vortex and strong vortex situations. The two regimes are found by studying the PDEs of the leading PC of the geopotential heights and can be identified at levels in the stratosphere. The regimes have imprints in the troposphere, but we have not found bimodality in the PDEs below 100 hPa.

An abrupt regime shift has been identified in the last half of the 1970s. Both regimes are visited before and after the shift, but the frequency of the regimes changes drastically around 1978 in favor of the strong vortex regime. The regime shift describes the data better than the linear trend in the PC of 1.5 m yr⁻¹.

Our time series is short and the analysis is therefore vulnerable to effects of sampling errors. We have estimated the effect of sampling errors by Monte Carlo simulations and conclude that our results are statistically significant.

Most of our work was based on the NCEP–NCAR reanalysis spanning the period 1948–2001. One should worry that our observed regime shift is only a result of the inclusion of satellite observations in the reanalysis process after 1979. We have therefore repeated our calculations with the FUB analysis, which is based on radiosondes and covers the years 1957–97. The two datasets compare very well both to the existence of the two regimes and the regime shift.

In this study we have considered averages over the winter half-year. We get similar results if averages are taken over the three winter months December–February. If monthly means are used we find that the PDEs are unimodal as Gillet et al. (2002) reported for daily values. That the presence of higher frequencies can obscure the existence of bimodality was indicated by Palmer (1993) based on studies of the Lorenz model. Most other studies of multiple regimes have used monthly means or even higher temporal resolution such as Perlwitz and Graf (2001), who reported bimodality in the distribution of anomaly correlations of the zonal mean wind at 50 hPa, and Monahan et al. (2003), who found several regimes in a nonlinear EOF analysis of the 20-hPa geopotential height.

The regime shift has a significant imprint in the troposphere although no bimodality is found in PDEs below 100 hPa. The difference of the zonal mean zonal wind between the strong vortex regime and the weak vortex regime shows the deep vertical structure of the Arctic Oscillation. That the Arctic Oscillation pattern can be found as the difference between strong and weak vortex regimes was demonstrated in Castanheira and Graf (2003). One of the peaks in the PDE of the monthly 500-hPa geopotential heights found in Corti et al. (1999) (their regime D, described as the Arctic Oscillation regime) closely resembles, except for the sign, the pattern of the difference in the 500-hPa geopotential height between the strong vortex regime and the weak vortex regime. The peak was present when data for the period 1949–71 were used but nearly absent for the period 1971–94 consistent with the regime shift presented in this paper. However, according to Hsu and Zwiers (2001) the tropospheric regime is not statistically significant. Studies of the 1000–500-hPa thickness and the zonal mean temperature show that the lower-tropospheric air temperature is warmer in the strong vortex regime than in the weak vortex regime, while the upper troposphere and the lower stratosphere are colder. In the 1000–500-hPa thickness the largest signals are found over the continents just as in the cold ocean–warm land pattern. These results support the speculations of Corti et al. (1999) on the vertical structure of the cold ocean–warm land pattern and its possible role in explaining the different observed temperature trends in the free atmosphere and at the surface. We note that a jump in the global mean free troposphere temperature in 1976 has been argued to explain the discrepancy between the surface global mean temperature and atmospheric global mean temperature found in the last decades (Lindzen and Giannitsis 2002).

A statistically significant abrupt regime transition in an atmospheric field has to our knowledge not been presented before although several atmospheric changes in the second half of the 1970s have been reported. A shift toward lower values of the Southern Oscillation index in 1976/77 has been observed (Trenberth 1990; Trenberth and Hurrell 1994) as well as evidence for sudden changes in the North Pacific Ocean in the second half of the 1970s (Hare and Mantua 2000). Recently, a change in the link between the North Atlantic Oscillation (NAO) and indices of the Northern Hemisphere climate variability occurring around 1980 has been noticed and the change associated with a eastward shift in the NAO pattern (Lu and Greatbatch 2002). In studies involving the stratosphere, it has been found that correlations between the stratospheric vortex and both the 500-hPa geopotential height (Kodera et al. 1999) and the quasi-biennial oscillation (Dunkerton and Baldwin 1991) are different before and after 1978.

What is the cause of the regime shift? As discussed by Palmer (1993) in a system dominated by a few regimes the spatial patterns of forcing and response will not necessarily be simply related. Weak forcings will mainly be expressed in changes of the frequencies of the regimes and the frequencies will be sensitive to forcings that may have little spatial correlation with the regimes themselves (Corti et al. 1999). Thus, both forcings located in the stratosphere such as perturbations of the stratospheric ozone, forcings embracing the total atmosphere such as greenhouse gas emissions, and forcings deposited in the troposphere–surface system such as changes in the solar irradiance or changes related to the ocean circulation could be involved. There is also the possibility that the regime shift is simply a consequence of internal low-frequency variability of a chaotic atmosphere.

We also emphasize that the regime shift is coincident with a change in the sun–climate relationship. It has been reported that the North Atlantic Oscillation correlates with solar variations in the 1980s and 1990s while the correlation is weak or absent in earlier decades (Thejll et al. 2003). However, significant correlations between solar variations and the geopotential height at 20 hPa are found in both periods. Perlwitz et al. (2000) and Castanheira and Graf (2003) argue that only in strong vortex situations will the troposphere be influenced by the stratosphere. The regime shift in the late 1970s may therefore explain the change in the sun–climate relationship if the direct solar influence is mainly in the stratosphere.