1. Theoretical background

One of the most challenging problems in atmospheric dynamics is to understand the nature and limits of climate variability. It appears that the climate is intrinsically variable at all time and space scales. However, except from a general understanding that climate variability is nonlinear in origin, a clear formulation of its nature remains elusive. Here, we provide unique insights about climate variability by employing data relating to the general atmospheric circulation and time series analysis using “random walk” methods (Hurst et al. 1965;Tsonis and Elsner 1995; Viswanathan et al. 1996). Given a time series x(t) we can define a random walk on a plane by stepping in a random direction with a step of size equal to the corresponding value of x. As such the net displacement, y(t), after t time steps is defined by the running sum y(t) = Σ^t_i=1 x(i). For any walk a suitable statistical quantity that characterizes the walk is the root-mean-square fluctuation F(t) about the average displacement, F²(t) = [Δy(t)]² − [Δy(t)]², where y(t) = y(t₀ + t) − y(t₀) and the bars indicate an average over all positions t₀ in the walk.

A scaling (fractal) process y(t) satisfies the relationship y(t) = ^dσ⁻¹y(λt) where = ^d indicates equality in distribution and σ, λ > 0. This relationship indicates that the statistical properties at timescale t are related to the statistical properties at timescale λt. Consequently, any moment of order k, μ^′_k, satisfies the relation μ^′_k(t) = σ⁻¹μ^′_k(λt). It is easy to show that the power law μ^′_k(t) = AZ^H with H = log σ/log λ is a solution to the last equation (Triantafyllou et al. 1994). Considering the definition of F(t), we expect that if y(t) is scaling, then F(t) ∝ t^H. If y(t) is not scaling, then the relation between F(t) and t is unknown and not a power law. A value of H = 0.5 results from uncorrelated time series and corresponds to a purely random walk (Brownian motion). In this case, x(t) is a white noise. Note that Markov processes exhibiting local correlations extending up to some characteristic scale also give H = 0.5 for large t. It is well known (Feder 1988) that the correlation function C(t) of future increments, y(t), with past increments, y(−t), is given by C(t) = 2(2^2H−1 − 1). For H = 0.5, we have that C(t) = 0 as expected, but for H ≠ 0.5 we have that C(t) ≠ 0 independent of t. This indicates infinitely long correlations and leads to persistence or a scale invariance associated with positive long-range correlations for H > 0.5 (i.e., an increasing trend in the past implies an increasing trend in the future) and to antipersistence or a scale invariance associated with negative long-range correlations for H < 0.5 (i.e., an increasing trend in the past implies a decreasing trend in the future). Note, however, that positive long-range correlations do not imply simple persistence (which is defined as the continuance of a specific pattern and which provides no information about the intrinsic variability of the system in question). Scale invariance is a law that incorporates variability and transitions at all scales and is often a result of nonlinear dynamics. Note also that even though scaling associated with positive long-range correlations imply persistence, the reverse may not be true. Random walks with H ≠ 0.5 are called fractional Brownian motions (fBms). In theory the exponent H is related to the spectra of the y(t) function via a relation of the form S(f) ∝ f^−(2H+1), where f is the frequency. As such, higher values of H do not imply longer persistence but rather that the power of the smaller (shorter) scales is not as large. As H increases, the large-scale features of y(t) remain the same but the variability at shorter scales decreases. We may think of the y(t) functions corresponding to higher H values as “smoothed” versions of the y(t) functions corresponding to lower H values. In addition, the presence of long-range correlation dictates that the spectra of the original time series x(t) obeys a relationship of the form P(f) ∝ f^−2H+1. As such H > 0.5 is associated with redness in the spectra of x(t) and as H → 1 the redness increases (less power in smaller scales). Note, however, that positive long-range correlations do not just imply red spectra. As we mentioned above, scaling is a law that describes the exact relationship between the scales involved, not just the simple fact that larger scales have more power than smaller scales. Furthermore, while redness is one of the characteristics of positive long-range correlations the opposite is not true. Red spectra do not necessarily imply positive long-range correlations. Note, that even though in theory both P(f) and F(t) contain information about the exponent H, F(t) is superior in estimating H. Because the definition of F(t) involves averages over all positions t_o of the walk, F(t) is a smoother function than P(f). Also, P(f) fluctuates significantly and as a result scaling regions are often masked. Because of that the estimation of slopes in logP(f) versus logf plots is not straightforward (see Viswanathan et al. 1996).

2. Results

Because of the above properties of scaling random walks, it is important to investigate the extent to which long-range correlations exist in the atmosphere and if they exist, their implications with respect to our climate system. In order to address these questions we considered data relating to the general atmospheric circulation. At first, we considered a location for which uninterrupted long upper-air data exist, obtained from the National Centers for Environmental Prediction (NCEP, formerly the National Meteorological Center) compact disc dataset (Mass et al. 1987). The point has coordinates I = 22, J = 8 (29.7°N, 86.3°W) and represents a location in the southeastern United States (along the Florida gulf coast) adjacent to two rawinsonde stations (Appalachicola and Valparaiso/Eglin). Daily 500-hPa values are available with no interruptions for this point from 1964 to 1988 (total of 9132 values). From this record we first calculated daily 500-hPa anomalies by subtracting each value from the climatological mean for that day (defined as the average of the 25 available daily values). Then we produced weekly averages of these anomalies. Figure 1a shows the anomaly record x(t) for this grid point. Figure 1b shows the random walk in 2D that corresponds to this record. This walk is generated by stepping from some initial position in a random direction with a step size equal to the absolute value of the anomaly. Figure 1c shows the net displacement as a function of time. Figure 1d is a log–log plot of F(t) versus t for 1 ⩽ t ⩽ 512 weeks. As t approaches the sample size the estimation of F(t) involves fewer and fewer points. Thus, extending this type of analysis to longer timescales is not recommended. As the linear least squares fit indicates, a strong case can be made that the logF(logt) function is linear with a slope of 0.625. The null hypotheses H₀: H = 0 and H₀: H = 0.5 against the alternative H_a: H = 0.625 are rejected at a confidence level of 99.99%. Repeating the analysis by shuffling the x(t) record and producing a Markov process exhibiting similar lag-one autocorrelation as the original x(t) results in slopes equal to 0.5 as expected from random records. Most of the seasonality in the 500-hPa data is removed by constructing the anomaly record. As a test of the effect of any residual seasonality on our results, we differenced the anomaly record (which eliminates the remaining first-order seasonality) and repeated the analysis. Our results were unchanged. Note that differencing the record will not remove all the effect of higher-order moments (such as variance). However, since y(t) is the running sum of x(t), the magnitude of the fluctuations about a mean of zero is not as important as underlying low-frequency trends that result in a tendency for more positive rather than more negative fluctuations and vice versa.

While the above result proves that x(t) is not white noise nor a Markov process it may not necessarily constitute a proof of positive long-range correlations and scaling with H = 0.625. Since power law exponents are given by the slope of a corresponding log–log plot, the above-described procedure has become the common practice in studies exploring fractal dimensions or other scaling exponents, like H. Generally, a significant slope in a log–log plot has been considered adequate to claim fractality, nonlinearity, etc. However, the resolution and length of the time series introduce artifacts at small and large scales, respectively (Tsonis and Elsner 1995; Tsonis and Elsner 1990). As such the scaling region (if it exists) is somewhere in between small and large scales but exactly where is not always clear. Unfortunately, in log–log plots many functions appear linear and the exponents estimated from these slopes may be false and not necessarily represent actual scaling. For example, Fig. 2a shows a hypothetical logF(t) versus logt plot. A linear least squares fit in the range 0 ⩽ logt ⩽ 2.7 results in a slope of 0.7. Standard statistical tests show that the slope is significantly different from a slope of 0.0 and a slope of 0.5 at a confidence level of 99.99%. Does this indicate scaling associated with long-range correlations? As is shown in Fig. 2b a good argument can be made that the logF(t) versus logt function is nonlinear and therefore F(t) ≠ t^H. Thus, a rigorous determination that a process is scaling requires one to show that it is consistent with the family of processes that we know a priori exhibit similar scaling properties. As such, a proper test for scaling should be a goodness-of-fit type test.

Such a test for scaling over a given range of scales has been developed by Tsonis and Elsner (1995). For a scaling process and given an infinite sample size it follows that H = Δ logF(t)/Δ logt is the same at all timescales. However, due to limitations in the data, large fluctuations in the value of H may be observed locally (i.e., at different timescales) even in real scaling processes. In such cases, confidence intervals on the estimated H must be derived as a function of the timescale. This is achieved by using surrogate data generated by inverting power spectra of the form f^−(2H+1). Even though other approaches to generate fBms exist, this approach is considered the purest interpretation of fractional Brownian motion. The formula used to generate surrogate y(t) functions for t = 1, N is given by y(t) = Σ^N/2_k=1 [Ck^−a(2π/N)^1−a]^1/2 cos(2πik/N + ϕ_k), where C is a constant, N is the sample size, ϕ_k are N/2 random phases distributed in [0, 2π], and a = 2H + 1 (Osborne and Provenzale 1989; Tsonis 1992). A large number of computer-generated fractional Brownian motions with the desired exponent, resolution, and length are obtained. For each simulation and for a given logt we estimate a “local” H. Thus, from all simulations we can obtain the frequency distribution of H as a function of logt. Once we have this information, bounds indicating the 99th and first percentiles of these distributions can be produced, which can then be used to test the null hypothesis H_o: scaling with an exponent H against the alternative H_a: no scaling with an exponent H. If the Δ logF(t)/Δ logt function from the data under testing falls within these bounds, then the hypothesis cannot be rejected. Since no points are allowed to exist outside the bounds the test is very stringent. Figure 3 shows the results from 1000 simulated fBms with H = 0.625. The shaded area indicates the first and 99th percentiles of the frequency distribution of H as a function of logt. The black dots are the Δ logF(t)/Δ logt values from Fig. 1d. All points fall within the shaded area, indicating that the scaling reported in Fig. 1d with H = 0.625 is statistically significant at a confidence level of 98% over the whole range of scales 1 < t < 512 weeks. Note that the width of the bounds is a function of the sample size and it increases for large scales due to poorer statistics at those scales (the inflection at small scales is due to the fact that at those scales we approach the resolution of the data; see Tsonis and Elsner 1995). As such, at large scales the data points may be consistent with families of fractional Brownian motion (fBms) that exhibit an exponent H that is significantly different from 0.625. However, since fractional Brownian motion requires that the scaling extends over the whole range of scales (i.e., all points are inside the bounds), we find by producing the bounds for varying H’s (i.e., by testing the null hypothesis H_o: scaling with an exponent H against the alternative H_a: no scaling with an exponent H, for several H’s in the neighborhood of 0.625), that only families with an H between approximately ±0.015 of the estimated 0.625 will contain all points. This procedure indicates that the estimated value of H = 0.625 is robust. The results in Fig. 3 are strongly suggestive that the anomaly record in Fig. 1a exhibits scale invariance in time and positive long-range correlations such that if now a certain type of anomaly (negative or positive) exists it will most likely continue to exist in the future for any t < 512 weeks. Otherwise stated, this result would indicate that the dynamical properties of the scaling process at small scales are related to those at large scales via a relationship that involves a magnification factor λ^H, where λ is the ratio of the large timescale to the small timescale.

Having established the above we then tested the generality of this scaling law. In order to address this question we repeated the analysis for grid points providing full spatial coverage in the Northern Hemisphere (from about 20°N) as obtained from the NCEP compact disc dataset. For each grid point we estimated H and tested for significance. Figure 4 shows the spatial distribution of the estimated value of H. Almost everywhere the value of H exceeds 0.5 (indicating long-range correlations), with a hemispheric mean value of 0.65. Only a small area centered over Finland and the northern reaches of the former Soviet Union appears to exhibit values close to or lower than 0.5 (the lowest value is 0.48). The fact that virtually everywhere the value of H is greater than 0.5 is a direct consequence of natural processes exhibiting some degree of redness in their spectra (i.e., larger scales possess more energy than smaller scales). We observe a very coherent pattern that is characterized by a general tendency for H to decrease with increasing latitude. This result is consistent with the increasingly baroclinic nature of the dynamics as one progresses from the subtropics through the midlatitudes (more baroclinicity, more power to small scales, less“redness” in the spectra, smaller exponent H). Variations from this general tendency over the North Pacific and the North Atlantic Oceans are associated with the storm tracks where the influence of very short timescale cyclones and anticyclones is enhanced, resulting in local decreases in H. The consistency of these results with large-scale dynamics indicate that Fig. 4 would not arise by chance. Indeed, following the procedure outlined above for the point in the southeastern United Sates, we find that for 80% of the points the estimated value of H is statistically significant. Note that this result should not be interpreted as indicating a high probability of false negatives. The 20% of the points not classified as scaling may include points that are correctly classified as not scaling. Given the severity of the test and the overall coherent and consistent structure of Fig. 4 these results provide very strong evidence of the significance and universality of long-range correlations in the extratropical circulation.

This is an important result that raises questions about the implications and the physics responsible for the origin of this law. In order to address these issues we have examined the spatial distribution of the anomaly pattern over many years. Beginning with the year 1959 and a window of length 5 yr, we produced the corresponding 5-yr mean anomaly Northern Hemispheric map for the period 1959–63. Subsequently, we slid the window by 1 yr and repeated the analysis up to the last available 5-yr period (1989–93). The random walk analysis is sensitive to missing data and as such it requires uninterrupted data. However, producing 5-yr mean anomaly patterns is not as sensitive, since minor gaps do not alter the large-scale picture. As such we were able to extend the analysis beyond the period 1964–88 to years where short gaps do exist. By comparing these maps we were able to deduce that in the available 34-yr period the spatial 5-yr moving average anomaly distribution is characterized by a small wavenumber pattern that evolves in space and time and exhibits an apparent decadal-scale cycle. A major characteristic of this pattern is that its evolution is very slow. As such, a given distribution tends on the average to persist for many years before a transition takes place. Figure 5 demonstrates the above. From Figs. 5a and 5b we see that a seeming wavenumber-2 pattern persists during the 10-yr period of 1959–68, whereas Figs. 5c and 5d show that a dominant wavenumber-1 pattern persists during the 10-yr period of 1979–88. The transition from wavenumber 2 to wavenumber 1 can be traced to the period 1974–78, a transition that has previously been identified and studied as a substantial decade-long change in the North Pacific (Miller et al. 1994; Graham 1994; Trenberth and Hurrell 1994). It appears that a termination of this pattern occurred in the period 1989–93 and a different pattern has now been established (Miller et al. 1994). Note that due to the evolution of the anomaly patterns (that may include slow propagation, shifting, etc.), small-scale (local) differences between Figs. 5a and 5b or between Figs. 5c and 5d may be observed. However, the global pattern remains quite robust. We performed two spatial correlation analyses [one for the whole area and one for the Pacific–North American (PNA) sector] between Figs. 5a and 5b and between Figs. 5c and 5d. We find correlations of the order of 0.5 in all cases. This indicates two things: 1) both cases are equally persistent, and 2) since anomaly correlations of 0.5–0.6 between synoptic maps provide useful information (Hollingsworth et al. 1980), the leverage provided by similar correlations at timescales of many years is obvious. Thus, these figures provide useful information. It is this robustness that results in the emergence of long-range correlations with the local differences representing the intrinsic variability in the y(t) function.

Due to substantial gaps in the datasets we are not able to extend this analysis further into the past. However, given the fact that previous to this pattern another and different persistent pattern was in place, it would appear that decade-long patterns may be established as a result of the intrinsic variability of the complete climate system at those scales and that their persistence may be a result of scale invariance associated with long-range correlations. Our results, however, go further than explaining simple persistence as they indicate that the underlying dynamics and transitions in the atmospheric circulation are associated with a fractal law that dictates that no characteristic timescale exists and that all scales from a week to a decade are connected. A consequence of this law is that the memory of the large scales (low-frequency processes) is not independent of the memory of the small scales (high-frequency processes). Further, the decrease in H with latitude and the association of low values of H with the Pacific and Atlantic storm tracks shown in Fig. 4 suggest the fundamental role of these high-frequency midlatitude weather systems. This would indicate that the memory of the extratropical climate system does not reside only in the oceans (i.e., long timescales) with the atmosphere simply responding passively.

3. Concluding discussion: A possible scenario for the emergence of scaling

Recent climate research together with earlier conceptual ideas (Frankignoul 1985) offer a possible scenario of how this scaling may arise. It is well known that high-frequency atmospheric disturbances are crucial agents in achieving the long-term balance of energy, momentum, and water vapor in the atmosphere. It now appears that these systems may also play a fundamental aggregate role in modulating the low-frequency (seasonal to decadal) atmospheric flow by communicating the effects of anomalous surface properties (boundary forcing) to the slowly varying components of the climate system. This communication is directly tied to the location of the storm track (Trenberth and Hurrell 1994). The position of the storm track largely determines the seasonal distribution of temperature and precipitation, which leads to energetic exchanges of heat and momentum with the underlying surface. It has long been recognized that extratropical sea surface temperatures (SSTs) modulate midlatitude atmospheric variability (Latif and Barnett 1996; Namias 1969; Wallace and Jiang 1987; Lau and Nath 1990). Evidence exists to suggest that the atmospheric flow leads the oceanic changes by one to several months (Davis 1976; Lanzante 1984; Wallace et al. 1990); these results indicate that the driving of the ocean by the atmospheric circulation initiates the changes but that once induced, the strong persistence of oceanic SST features feeds back onto the low-frequency atmospheric dynamics. The mechanism for this feedback involves oceanic gyre modes (decadal timescale) generated by large-scale atmosphere–ocean interactions in midlatitudes (Latif and Barnett 1996; Latif and Barnett 1994; Latif et al. 1996). Given an anomalous subtropical ocean gyre, adjustments in the oceanic poleward transport of heat will result, leading to midlatitude SST anomalies. These anomalies force an atmospheric response in the form of adjustments in the atmospheric general circulation (e.g., PNA) and associated storm tracks. The aggregate effect of the latter is to modulate both surface heating (reinforcing the existing anomaly) and the wind stress curl (opposing the sense of the existing oceanic gyre), ultimately readjusting the poleward heat transport and the associated sign of the SST anomalies.

Our results have both practical and theoretical implications. From the practical point of view, since anomaly regimes may persist for timescales of at least up to a decade, persistence forecasts are reliable for long timescales once a regime has been established. This 10-yr timescale is consistent with recent results (Huang et al. 1996) on the use of persistent climate normals in forecasting. They suggest that such normals in order to be effective should be based on averages over periods less than 30 yr (if longer datasets were available, our method could define the exact averaging period from a break in the scaling). From the theoretical point of view, our results offer important insights into how we view extratropical air–sea interactions. It is customary to consider the high-frequency atmospheric processes as noise that randomly forces the coupled system. Since noise has no memory our results suggest that such a view is inappropriate. Rather, the complete interaction across all scales will be required in order to study the establishment and transition of general circulation patterns. An understanding of these scale interactions will in turn extend predictive ability beyond simple persistence and enhance our ability to estimate the response of the extratropical climate system.