1. Introduction

There is great interest in how global warming might be affecting the statistics of extreme weather (Meehl et al. 2000; IPCC 2014). Indeed the occurrence of any extreme event not previously observed “within living memory” or “since records began” (in both cases, about 100 years) immediately becomes a candidate for attribution to global warming. Both observational and modeling approaches are taken to the problem. A combined approach is often deemed necessary, given the limitations imposed on observational approaches by the shortness and inaccuracies of historical records, and on modeling approaches by known and unknown model errors.

It is not hard to see how detecting changes in extreme value probabilities using limited records can be problematic. Consider an event with an N-yr return period, whose probability of occurring in any single year is 1/N if the annual occurrences are independent, and of not occurring is 1 − 1/N. The probability of at least one occurrence in M years is then 1 − (1 − 1/N)^M, and of exactly one occurrence is [M/(N − 1)] (1 − 1/N)^M. These simple formulas highlight the difficulty in pinning down the probability (or equivalently the return period N) of rare events from short records. For example, a single occurrence of an event in an M ~100-yr record does not imply that N ~100 yr. Actually N could be as long as 2000 years, or as short as 22 years, because (using the binomial distribution) there is a 5% chance for a 2000-yr event as well as a 22-yr event to occur just once in 100 years. Similarly, the complete absence of an event in a 100-yr record does not imply that N > 100 yr, as is often assumed. It could be as short as 44 years, because there is a 10% chance for a 44-yr event to occur not even once in 100 years. Thus, although it might be tempting to conclude from the absence of an event in a 100-yr interval and its single occurrence in the subsequent 100-yr interval that its return period N has decreased from “infinity” to 100 years and the event has become “more likely,” the large uncertainties in N would render such a conclusion suspect. Indeed there is an appreciable probability [M/(N − 1)] (1 − 1/N)^2M for such a sequence of events to occur with no change in N at all, which is greater than 10% for 80 < N < 800 yr. Similar statements can be made regarding events that occur a few times in one century, and a few more (or less) times in the next.

Given the rarity of extreme events, it is not surprising that detecting significant changes in their probabilities through direct counts or raw (or cosmetically smoothed) histograms is prone to sampling errors. There are three ways around this difficulty. Approach 1 is to fit a probability distribution, usually of a universal form based on generalized extreme value (GEV) theory, to the sampled extreme values. Although more robust than using raw histograms, this approach is also prone to sampling errors, and the links between the parameters of the GEV distribution and underlying physical processes are often unclear. Approach 2 is to fit a probability distribution to all and not just the extreme values, whose more physically transparent parameters related to the mean and variance can also be more robustly estimated from limited records, and then to deduce changes in tail probabilities from the changes in those parameters. This approach is appealing from both sampling and physical perspectives, but can be compromised by using an inappropriate probability distribution, such as a Gaussian in cases where the distribution is clearly not a Gaussian. Approach 3 is to use a numerical climate model to estimate the tail probabilities from raw histograms derived from large ensembles of model integrations. This approach can be made as statistically reliable as desired depending upon computing resources, but may be compromised by known and unknown model errors. All three approaches, individually or in combination, are currently being followed to detect changes in extreme weather and climate risks and to attribute them to natural and/or anthropogenic influences.

Our first goal here is to highlight the need for such detection and attribution efforts to account for the distinctively skewed and heavy-tailed aspects of the probability distributions of daily weather anomalies. As discussed below, departures of several standard deviations from the mean, although rare, are far more common in such a distinctively non-Gaussian world than they are in a Gaussian world. In such a world a mean climate shift is also generally associated with changes in the width and shape of the probability distribution. Consequently, even the sign of the changes in tail probabilities cannot be inferred unambiguously from the mean shift alone. These realities further complicate the problems of detecting significant changes in tail probabilities from historical records of limited length and accuracy. In effect they make it harder to reject a null hypothesis of no change in extreme risks. They also raise the bar for the detection and attribution of changes in extreme risks using climate models, by requiring the models to accurately represent the salient non-Gaussian aspects of the observed distributions.

Our second goal is to suggest a simple way to account for the observed non-Gaussianity by using, as a null hypothesis, a general class of non-Gaussian distributions called stochastically generated skewed (SGS) distributions introduced to the meteorological literature by Sardeshmukh and Sura (2009, hereafter SS09). Sardeshmukh and Penland (2015) further discuss the broad physical foundations of such distributions in the climate system. Here we provide extensive justification for the particular relevance of SGS distributions in extreme value analysis. In addition to capturing the distinctive non-Gaussianity of the observed distributions, SGS distributions have two other notable properties: 1) they include Gaussian distributions as special cases, and 2) they remain SGS distributions following a mean climate shift. We describe how to estimate the parameters of an SGS distribution using the first four statistical moments of the data (mean, variance, skewness, and kurtosis) and, importantly, how to generate a statistically equivalent Markov process with the same parameters, whose physical interpretation is more transparent. The Markov model can be used to test for statistical significance through Monte Carlo simulations, and also to investigate changes in the temporal persistence statistics of extreme anomalies. Our approach may be viewed as an attempt to improve the full distribution–based approach 2 to the analysis of extreme anomalies. It also has implications for the extreme value distribution–based approach 1, and the climate model distribution–based approach 3, as will become clear below.

Figure 1 illustrates how non-Gaussianity can strongly impact tail probabilities, and how ignoring it can compromise the detection, quantification, and attribution of their changes even when they are associated with only a mean shift. The figure compares the probabilities of exceeding 2 and 4 standard deviations (sigma) before and after a positive mean “climate shift” of 1 sigma, for Gaussian and SGS distributions with the same mean and sigma, and a skewness of 1 and excess kurtosis of 5 for the SGS distribution. Before the shift, the probabilities of exceeding 2 sigma are 2.3% and 3.4% for the Gaussian and SGS distributions, respectively. After the shift, they increase to 15.8% and 13.3%, by different factors of 7 and 4. For exceedances of 4 sigma, the changes are more dramatic: from 0.003% (0.34%) before to 0.13% (1.0%) after the shift, and by very different factors (43 and 3 for Gaussian and SGS distributions, respectively). The detection and attribution of such changes would be strongly affected by what one assumes to be the true distribution. If the distribution is actually SGS but is assumed to be Gaussian (i.e., if the non-Gaussianity is ignored), then the occurrence of a 4-sigma event would be interpreted as highly significant (since its preshift risk of 0.003% is very small) and highly consistent with the 43 times higher risk of such events after the mean shift. One would thus, with high confidence, both reject a null hypothesis of no change in the risk of such events and accept the alternative hypothesis of their much higher risk in the shifted climate. If, on the other hand, the distribution is correctly recognized to be SGS, then the occurrence of the 4-sigma event would be interpreted as relatively less significant and less clearly attributable to the climate shift, given its already substantial risk of 0.34% before the shift (which is indeed higher than the 0.13% risk in the Gaussian case even after the shift) and its relatively small threefold increase of risk after the shift. Note that no such disparity of interpretation would occur if the distribution is actually Gaussian but is assumed to be SGS, since a Gaussian is a special case of an SGS distribution with zero skewness and excess kurtosis, as already mentioned.

View Full Size

An illustration of how changes in the probabilities of extreme values associated with a mean climate shift depend strongly on the shape of the PDF. Results are shown for (a) a Gaussian PDF and (b) a non-Gaussian SGS PDF (see text) with skewness S = 1 and excess kurtosis K = 5. In both panels the PDFs for the current climate (black curves) have zero mean and unit standard deviation, and are shifted by one standard deviation to the right (red curves) to illustrate a future climate scenario. This identical mean shift in the two cases implies very different changes in the probability P(x > 2) of exceeding 2 standard deviations, from 2.3% to 15.8% in (a) and from 3.4% to 13.3% in (b). The changes in the probability P(x > 4) of exceeding 4 standard deviations are even more striking, increasing by a large factor of 43 from 0.003% to 0.135% in (a) and by a relatively modest factor of 3 from 0.34% to 1.01% in (b).

Citation: Journal of Climate 28, 23; 10.1175/JCLI-D-15-0020.1

• Download Figure
• Download figure as PowerPoint slide

Granting that there may be substantial disparities in the magnitude of the estimated changes in extreme risks, one might still argue that their sign would be correctly deduced in all cases in the above example from the sign of the mean shift, using the popular rule of thumb that a positive mean shift increases the risk of positive extremes and decreases the risk of negative extremes. This rule, unfortunately, is generally valid only if the mean shift is not accompanied by a change in the width or shape of the distribution. It is easily shown for Gaussian distributions, for example, that if a mean shift is accompanied by any decrease of standard deviation, then the risks of sufficiently large positive and negative extremes both decrease, regardless of the sign and magnitude of the mean shift. More generally, it can be shown (see appendix A) that for shape-preserving changes of any probability density function (PDF) p(x) with a location parameter μ₀ (usually but not always identified with the mean of x) and a scale parameter σ₀ (usually but not always identified with the standard deviation of x), the sign of the change in probability of exceeding high (x > X_T) and low (x < 2μ₀ − X_T) thresholds is given by the sign of an extremal probability shift index (PSI), defined as

View Expanded

where μ and σ are the location and scale parameters after the change, and the + (−) sign refers to the change in probability of high (low) extreme values. It is obvious from this equation that if σ ≠ σ₀, the second term on the right-hand side dominates for sufficiently large X_T and yields the same sign for ψ(+) and ψ(−). This is essentially a restatement and extension of Katz and Brown’s (1992) result that “variability is more important than averages” in influencing changes in tail probabilities; that is, the second term can be more important than the first. Our extension is to combine these terms into a single predictive measure ψ in (1) for the sign of the change in tail probabilities. Applying it to the Gaussian case in Fig. 1, for example, shows that the 43-fold increased risk of 4-sigma events associated with a mean shift of 1 sigma could be completely nullified by a 25% reduction of sigma. We stress that ψ is applicable to a wider class of distributions than Gaussian distributions. We also note that the signs of changes in the tail probabilities of x are identical to those of any other variable y that is a monotonically increasing function of x. So even if the changes of p(y) are themselves not shape-preserving, the sign of changes in the tails of p(y) may be inferred through an appropriate transformation of variable and using (1) for the transformed variable, making ψ applicable to an even wider class of distributions. If y is precipitation, for instance, then given the monotonic relationship between precipitation and midtropospheric vertical velocity, one could use ψ to deduce changes in the sign of extreme precipitation probabilities associated with changes in vertical velocity, if the vertical velocity has a Gaussian or some other shape-preserving distribution.

Despite this wider applicability of ψ, one can imagine situations in which knowing even the mean and standard deviation would be insufficient to deduce the sign (let alone the magnitude) of changes in tail probabilities, if the distribution is not reducible to a shape-preserving form through a transformation of variable. Situations involving changes of strongly skewed and kurtotic SGS distributions belong to this category. In such situations the full distribution–based approach 2 and the climate model distribution–based approach 3 to determining tail probabilities are further compromised by ignoring or misrepresenting the SGS character of the observed distributions, eroding their advantages over the extreme value distribution–based approach 1.

The paper is organized as follows. Section 2 highlights the distinctive non-Gaussianity of the observed distributions of daily large-scale weather anomalies. Section 3 shows how this distinctive non-Gaussianity is captured by SGS distributions, and describes procedures for fitting SGS distributions to data and generating a statistically equivalent Markov process with the same parameters. Section 4 discusses the strong links between the distributions of midtropospheric vertical velocity (which are well approximated as SGS distributions) and precipitation anomalies. Section 5 shows how some striking observed asymmetries in the temporal persistence of positive and negative circulation anomalies are consistent with the SGS nature of the anomaly distributions. Section 6 shows how fitting SGS distributions to all available values in limited climate records yields extreme value distributions of block maxima with smaller sampling uncertainties than GEV distributions fitted to only the block maxima. This section also shows how the tails of SGS distributions can be directly linked to the corresponding asymptotic GEV distributions of block maxima and generalized Pareto (GP) distributions of threshold exceedances based on extreme value theory. Section 7 describes how a mean climate shift alters both the width and shape of an SGS distribution, and illustrates their combined impact on tail probabilities to contrast with the impacts of the mean shift alone shown in Fig. 1. Finally, section 8 presents an illustrative application of our SGS distribution–based approach to assess changes in the observed distributions of daily indices of large-scale wintertime atmospheric circulation variability over the North Atlantic and North Pacific Oceans, the North Atlantic Oscillation index and the North Pacific index, over the period 1872–2011. Section 9 follows with a discussion and concluding remarks.

2. The distinctive non-Gaussianity of observed daily weather anomalies

Figure 2 shows hemispheric maps of the skewness S and excess kurtosis K of the wintertime daily anomalies (departures from long-term averages) of upper tropospheric relative vorticity, lower tropospheric temperature, and midtropospheric vertical velocity, defined for an anomalous quantity x as

View Expanded

where , is the variance, and angle brackets refer to long-term averages. The maps are based on the Twentieth Century Reanalysis dataset (20CRv2; Compo et al. 2011) for the 1872 to 2011 period. Similar maps derived using other, shorter, datasets have been presented for these and other atmospheric variables by, for example, SS09 and Perron and Sura (2013). Note that S and K are zero for Gaussian distributions. The maps in Fig. 2 show interesting spatial structure near the midlatitude jets and over the polar region, which will be investigated elsewhere. Our focus here is on the fact that the distributions are not only non-Gaussian, but non-Gaussian in a distinctive way. In particular, for all three variables, regions of large positive or negative S generally coincide with regions of large positive K. The left panels in Fig. 3 depict this relationship more clearly, through joint histograms of the S and K values from Fig. 2. An approximately parabolic K–S relationship, K > (3/2)S², is evident. This relationship for daily atmospheric variability was first presented and investigated by SS09. Although not our focus here, a similar relationship for daily sea surface temperature (SST) variability was also noted and investigated by Sura and Sardeshmukh (2008).

View Full Size

(left) Skewness S and (right) excess kurtosis K of daily anomalies in winter (December–February) of (top) relative vorticity at the upper tropospheric 250-hPa level, (middle) temperature at the lower tropospheric 850-hPa level, and (bottom) vertical velocity at the midtropospheric 500-hPa level, estimated using the 20CRv2 dataset for the 1872–2011 period. Positive values are indicated by red and negative by blue shading. For a Gaussian variable, these maps would be blank.

Citation: Journal of Climate 28, 23; 10.1175/JCLI-D-15-0020.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

The distinctive non-Gaussianity of the probability distributions of daily anomalies in winter. (left) Joint histograms of the S and K values from Fig. 2. Note the approximately parabolic relationship between K and S. In each panel the lower thick black curve is a theoretical parabolic lower bound on K of the form K = 1.5S² − Δ, where Δ = 1.1, 1.1, and 0.5 in the top, middle, and bottom panels. A weaker universal lower bound on K, if K exists, is the parabola K = S² − 2, the lowest gray curve. The upper thick black curve in each panel is a theoretical upper bound on K if the fifth moment 〈x⁵〉 also exists. (right) Hemispherically averaged PDFs of transformed standardized variables y = (x/σ) × S/|S| estimated as histograms. The local factors S/|S| (equal to +1 or −1 at each grid point) ensure that p(y) at each grid point is positively skewed even if p(x) is not. The hemispherically averaged values of p(y > 0) and p(y < 0) are shown as blue and red curves, respectively, with the p(y < 0) curve flipped over to the positive side. Note that the asymmetry between p(y > 0) and p(y < 0) is substantial even for small magnitudes of y, and changes sign at a crossover point somewhere between |y| = 1.41 and 1.73, beyond which the tail of p(y > 0) is heavier than that of p(y < 0).

Citation: Journal of Climate 28, 23; 10.1175/JCLI-D-15-0020.1

• Download Figure
• Download figure as PowerPoint slide

The right panels of Fig. 3 highlight another distinctive feature of the observed non-Gaussianity. They show PDFs of the transformed standardized variables y = (x/σ)S/|S| from Fig. 2, estimated as histograms at each grid point and averaged over the hemisphere. The local factors S/|S| (equal to +1 or −1 at each grid point) ensure that p(y) is positively skewed at each grid point regardless of whether p(x) is positively or negatively skewed. The values of p(y > 0) and p(y < 0) are shown as blue and red curves, respectively, with the p(y < 0) curve flipped over to the positive side for easier comparison with the p(y > 0) curve. The two curves would coincide if the PDFs were symmetric. Note that the asymmetry is substantial even for small magnitudes of y (i.e., the PDFs do not have a Gaussian core) and changes sign at a “crossover” point between |y| = 1.41 and 1.73, beyond which the tail of p(y > 0) becomes heavier than that of p(y < 0). It is interesting that this heaviness is already evident at moderate anomaly magnitudes of 2 sigma that are often used as thresholds for defining extreme anomalies and assessing their statistical significance. Such heavy tails clearly need to be accounted for in extreme value analysis.

3. SGS distributions and corresponding Markov processes

The distinctive K–S relationship, asymmetry, and heavy tails of the observed distributions highlighted in Fig. 3 are well captured by SGS distributions. An SGS distribution has a PDF of the form

View Expanded

with parameters E, g, and b, and the normalization constant ,

View Expanded

which ensures that p(x) integrates to unity. Here ν = 1/E², q = 2gν/b, and Γ(z) is the standard gamma function of complex argument z. The PDF has a unique maximum at x_max = −Eg/(1 + E²).

The moments of the distribution are

View Expanded

View Expanded

View Expanded

A necessary condition for to exist is that E² < 2/(n − 1). Using (4), the skewness S and excess kurtosis K defined in (2) may be expressed as

View Expanded

View Expanded

where the K–S inequality K > (3/2)S² in (5b) results from the fact that the first term within square brackets on the right-hand side is greater than 1 and the second term is greater than 0 (SS09). The left panels of Fig. 3 are generally consistent with this inequality.

It is clear from (5a) that both E and g must be nonzero for S to be nonzero. Note that E is nondimensional, and g and b have the same dimension as x (and σ). Also, without loss of generality, E and b are positive, whereas g can be positive or negative. Thus the sign of g determines the sign of S.

As already mentioned, SGS distributions include Gaussian distributions as special cases. Although not obvious from (3), these correspond to cases of E = 0. Then both S and K are zero from (5), as for a Gaussian distribution. The convergence of the entire PDF to a Gaussian PDF as E → 0 becomes more obvious upon expressing the PDF of the standardized anomalies using (3) as

View Expanded

View Expanded

where and as E → 0. Note that from (4). For small E, the first term on the right-hand side of (6a) dominates and the PDF approximates a standard Gaussian. The second term, whose magnitude is proportional to S, represents a leading-order departure from Gaussianity, and provides a simple explanation for the distinctive asymmetry crossover of the observed distributions in Fig. 3. It shows that for a positively skewed distribution, is larger (smaller) than for larger (smaller) than 3, and vice versa for a negatively skewed distribution. The predicted asymmetry crossover at is close to the observed values in Fig. 3. This first-order approximation for small E is valid for . Including the second-order terms in (6) gives an even closer match of the PDF and the predicted crossover at with respect to the observations.

SS09 discussed other distinctive non-Gaussian features of SGS distributions, such as a cubic relationship between the fifth moment and S, and tails that decay approximately as a power of x. This is easy to see from (3). For , p(x) decays approximately as

View Expanded

where is a constant, and the + (−) sign applies to positive (negative) tails. SS09 provided examples of such approximate power-law tails in the observed PDFs of upper-tropospheric vorticity and midtropospheric geopotential height anomalies. Note that the inference of such tails from (3) does not require even the variance to exist. Penland and Sardeshmukh (2012) indeed used this fact to suggest an alternative interpretation of power-law distributions encountered in many natural contexts as possibly arising from SGS processes, instead of Lévy processes as is often assumed.

Both g and E are important in determining the skewness and heavy tails of SGS distributions. This is illustrated in Fig. 4, which contrasts the tails of two SGS distributions with the same mean, variance, and E, but one (the same as in Fig. 1) with g ≠ 0 and the other with g = 0. The positive tail is much heavier, and the negative tail much lighter, in the g ≠ 0 case. Note, however, that even the negative tail is much heavier than that of a Gaussian distribution with the same mean and variance, also shown in the figure, beyond about 4σ. Interestingly, a pure power-law decay of the PDF does not obtain until well beyond 10σ, which makes it difficult to establish from even very long time series, as indicated by the gray curves obtained from a long integration of a Markov process (described below) with the same PDF.

View Full Size

Log–log plot of the positive (blue) and negative (red) segments of the asymmetric SGS PDF in Fig. 1 with S = 1 and K = 5, contrasted with a symmetric SGS PDF with S = 0 and the same power-law tail exponent, and also with a standard Gaussian PDF (thin black curve). The PDFs have zero mean and unit variance in all cases. The wiggly gray curves show the estimated PDF in the asymmetric SGS case from a long simulation of the Markov process model [(10)] with the same parameters as the SGS distribution.

Citation: Journal of Climate 28, 23; 10.1175/JCLI-D-15-0020.1

• Download Figure
• Download figure as PowerPoint slide

There are several ways to estimate the parameters of an SGS distribution from a sample series of x. A simple “method of moments” is to specify the sample σ², S, and K in (4b) and (5) and solve for E, g, and b as

View Expanded

View Expanded

View Expanded

A necessary and sufficient condition for both E² and b² to be positive is that , where . If E² estimated from (8a) falls below this lower bound, the value of K specified in (8a) is increased until it exceeds the bound, and steps (8b) and (8c) are repeated.

One can fit an SGS distribution to any sample data series using this simple robust method. It may be a poor fit if the sample distribution has more than one peak, or if the sample E² estimated from (8a) strongly violates the constraints on E² for an SGS process. The degree to which the fitted SGS distribution differs significantly from the sample distribution may be assessed through Monte Carlo integrations of the SGS process model described in (10) below. It is possible that other methods of SGS parameter estimation and their significance, such as maximum likelihood estimation (MLE) and Markov chain Monte Carlo (MCMC) methods might be superior to our method, especially given the explicit forms of (3) and (10). Although we have not considered it here, this issue is certainly worthy of further exploration.

The constraint on E², if exists up to some n ≥ 4, implies stricter bounds on S and K than is obvious from (5). Specifically,

View Expanded

View Expanded

The fact that these stricter bounds are also generally satisfied in the left panels of Fig. 3 for n = 5 provides additional evidence for the relevance of SGS distributions, and for the existence of the fifth moments of those variables. The upper bounds for S are generally satisfied even for n = 6, but those for K are not, especially for vertical velocity at many gridpoints. This suggests that the sixth moment of vertical velocity generally does not exist, and that its PDF has relatively heavy tails.

The physical basis for SGS distributions, as discussed by SS09, is their association with a stochastically perturbed damped linear Markov process x(t) whose change dx in a small time interval dt is given by

View Expanded

where η₁ and η₂ are temporally uncorrelated random Gaussian variables with zero mean and unit variance, and λ > 0 is a damping time constant. [Note that (10) is identical to Eq. (5) of SS09 with a different notation, with E, g, and b in that equation scaled by here, and A in that equation equal to −[1 + (1/2)E²]λ here]. The stationary PDF of this process is identical to p(x) in (3), with identical parameters E, g, and b. Its lag-autocorrelation function is ρ(τ) = . Note that λ does not affect the stationary probability density p(x), since it can be used to standardize time in (10) as t′ = λt. It does, however, crucially affect the temporal evolution and hence the persistence statistics of x. We specify its value as λ = 1/τ_c, where τ_c is the temporal correlation scale of x estimated using ρ(τ).

The terms including in (10) represent Stratonovich stochastic forcing: these consist of bη₁ and gη₂ as “additive” and Exη₂ as “multiplicative” noise forcing terms. The combined (Ex + g)η₂ term was called correlated additive and multiplicative noise (CAM noise) by SS09. The skewness of p(x) arises from the asymmetry of this term with respect to the sign of x. It immediately makes clear why E and g must both be nonzero in (5a) for S to be nonzero.

This statistically equivalent Markov process model (10) is useful for quantifying the sampling uncertainties of SGS distributions, with a proper accounting of the “serial dependence” (i.e., the temporal correlation) of x. Long integrations of (10) also provide information on the persistence statistics of extreme values of x that is difficult to obtain in any other way.

Figure 5 illustrates the dramatically different behavior of extremes in the Gaussian and SGS (S = 1, K = 5) cases of Fig. 4 from a time series perspective. It compares the daily time series of x(t) in the two cases in long integrations of the Markov model (10) using a Heun method. The length of the integrations is 10⁸ correlation time units (ctu) of standardized time t′ = λt = t/τ_c, with a specified τ_c of 1 ctu and time step of 1/24 ctu. For concreteness, if τ_c is 1 day, one may think of these as 10⁸-day integrations, with the first 10⁵ days displayed in Fig. 5 as 1000-winter series of 100-day winters. Consistent with the heavier tails in Fig. 4, excursions beyond ±3 standard deviations are more frequent in the SGS case. Note that the two series have the same mean and standard deviation, with little sampling variability in the 1000-winter series of their values estimated from each winter. The figure also shows the 100-decade series of the largest and fifth-largest daily values in each 10-winter decade. These are dramatically different in the SGS and Gaussian cases, with excursions beyond +4 standard deviations far more common in the SGS case. Note also the substantial long-term trends in the series of decadal maxima in the SGS case, lasting a century or even longer, that are associated purely with sampling fluctuations. Consistent with the discussion of Fig. 1, such long-term trends of “record-breaking” values in an SGS world would be mistakenly attributed to “climate change” by adopting an inappropriate null hypothesis that they are drawn from a Gaussian distribution.

View Full Size

The dramatically different behavior of extremes in the SGS (S = 1, K = 5) and Gaussian cases of Fig. 1 from a time series perspective, obtained from long integrations of the Markov model [(10)]. The length of the integrations is 10⁸ days, performed using a specified correlation time scale τ_c of 1 day and time step of 1 h. Results are shown for the first 10⁵ days, displayed as 1000-winter series of 100-day winters. Consistent with the heavier tail in Fig. 1, excursions beyond ±3 standard deviations are more frequent in the SGS case. The two series have the same mean and standard deviation, with little sampling variability in the 1000-winter series of their values estimated from each winter. Also shown are 100-decade series of the largest (blue) and fifth-largest (orange) daily values in each 10-winter decade. These are dramatically different in the SGS and Gaussian cases, with excursions beyond +4 standard deviations far more common in the SGS case. Note the substantial trends in the series of the decadal maxima in the SGS case, lasting a century or even longer, that are associated purely with sampling fluctuations.

Citation: Journal of Climate 28, 23; 10.1175/JCLI-D-15-0020.1

• Download Figure
• Download figure as PowerPoint slide

4. The link between vertical velocity and precipitation distributions

The fact that the observed distributions of midtropospheric vertical velocity are approximately SGS distributions has important implications for the distributions of precipitation and precipitation extremes. In the simplest approximation, the daily precipitation rate R(t) at a location may be related to the standardized daily vertical velocity w(t) = W(t)/σ_w at that location as

View Expanded

where β is a proportionality constant that depends on the mean local humidity and thermal structure and H(w) is the Heaviside function that is 0 for negative w and 1 for positive w. Note that here is the full (not anomalous) vertical velocity W(t) standardized by its standard deviation σ_w. One may think of positive w as the standardized precipitation rate . This simple model allows one to relate precipitation probabilities to w probabilities. In essence, it states that the PDF of precipitation is the (appropriately normalized) PDF of positive w, and the probability of no precipitation is the probability of negative w.

Figure 6 illustrates how the probabilities of w in scenarios of = −1, 0, and +1 are impacted by whether the PDF of w′ is Gaussian or SGS (with S = 1, K = 5 as in Fig. 1). The impact on the probability of negative w (and therefore also on that of positive w) is relatively modest. However, the impact on the probability of extreme positive w (defined as w > 3) is dramatic, with the SGS/Gaussian probability ratio increasing from 1.5 in the mean ascent scenario to 115 in the mean descent scenario.

View Full Size

An illustration of how the probabilities of standardized midtropospheric vertical velocities w in scenarios of = −1, 0, and +1 are impacted by whether the PDF of the anomalous w′ is Gaussian (red curves and values) or SGS (with S = 1, K = 5 as in Fig. 1). The impact on the probability of negative w (and therefore also on that of positive w) is relatively modest, as evident from probability of “no rain” indicated in the upper left portions of each panel. However, the impact on the probability of extreme positive w (defined as w > 3), and therefore on the probability of “heavy rain,” is dramatic, with the SGS/Gaussian probability ratio increasing from about 1.5 in the mean ascent scenario (far right panel) to about 115 in the mean descent scenario (far left panel).

Citation: Journal of Climate 28, 23; 10.1175/JCLI-D-15-0020.1

• Download Figure
• Download figure as PowerPoint slide

The correspondence of the PDFs of daily precipitation and positive w through (11) also has implications for the PDFs of seasonal-mean precipitation. Focusing only on the PDFs of positive w in Fig. 6, it is clear that they are positively skewed in both the SGS and Gaussian cases, with the skewness relatively larger in the SGS case, increasing from the mean ascent to the mean descent scenarios. One would expect this skewness to be reduced for seasonal averages in accord with the central limit theorem, but to remain strongly positive in regions of strong mean descent. This is indeed observationally evident in regions of mean descent in Fig. 7. The right panels of Fig. 7 demonstrate the robustness of this relationship in five different observational and reanalysis datasets. One would thus expect the SGS versus Gaussian character of the daily w PDFs to have a strong impact on the extremes of even seasonal-mean precipitation in arid regions of mean descent.

View Full Size

(left) Skewness S of seasonal-mean precipitation in northern (top) winter and (bottom) summer estimated using the GPCP precipitation dataset (Adler et al. 2003). The stippled and white areas indicate areas of mean midtropospheric ascent ( ≥ 0) and descent ( < 0) respectively obtained using the long-term seasonal mean 500-mb vertical velocity fields in the NCEP–NCAR reanalysis dataset (Kalnay et al. 1996, Kistler et al. 2001). Note the tendency for S to be positive in areas of mean descent. (right) A demonstration of the robustness of this relationship using five different observational and reanalysis datasets pairs of precipitation skewness and long-term seasonal mean 500-mb vertical velocity fields: GPCP (Adler et al. 2003)/NCEP, CMAP (Xie and Arkin 1997)/NCEP, NCEP/NCEP, NCEP2/NCEP2 (Kanamitsu et al. 2002), and ERA-40/ERA-40 (Uppala et al. 2005). The skewness S is notably more positive in areas of mean descent than ascent, although it is weakly positive even in areas of mean ascent as anticipated from Fig. 6.

Citation: Journal of Climate 28, 23; 10.1175/JCLI-D-15-0020.1

• Download Figure
• Download figure as PowerPoint slide

5. Implications for the persistence of extreme anomalies

An important consideration in extreme value analysis is not only the magnitude but also the duration of extreme anomaly events. There is substantial observational evidence for the asymmetry of both with respect to the sign of circulation anomalies (e.g., Dole and Gordon 1983; Masato et al. 2009; Woollings et al. 2010). Figure 8a shows an early example from Dole and Gordon (1983) of the asymmetric persistence of positive versus negative 500-hPa geopotential height anomalies in the North Pacific sector. It shows the counts P_M(T) of T-day persistent anomaly events of amplitude M, defined as events in which the anomaly extends beyond and then persists beyond a threshold M for at least T days. Results are shown for M = ±0, ±0.5, and ±1 standard deviations. Large positive anomalies evidently last longer than large negative anomalies, and also longer than large anomalies of a Gaussian red noise process with the same variance and lag correlation. Such asymmetries are usually attributed to a greater role of nonlinearity in the dynamics of large-amplitude anomalies. Note, however, that the figure shows an asymmetry in the persistence of all positive and negative anomalies, not just large-amplitude anomalies, and the positive anomalies overall are actually less persistent than negative anomalies. There is also a striking exponential decay of P_M(T) with T in all cases. No simple explanations of such features currently exist.

View Full Size

Persistent anomaly statistics of positive and negative anomalies beyond specific amplitude thresholds contrasted with similar statistics for a univariate Gaussian red noise process (light gray shaded bands). (a) Counts P_M(T) of T-day persistent 500-hPa geopotential height anomaly events of amplitude M, defined as events in which the anomaly extends beyond and then persists beyond a threshold M for at least T days [adapted from Dole and Gordon (1983)]. Results are shown for positive (solid) and negative (dashed) thresholds of M = ±0, ±0.5, and ±1 standard deviations. (b) as in (a), but for P_M(T) of x(t) from 10⁸-day simulations of Markov processes with 1-day correlation time scales described in the text associated with SGS (S = 1, K = 5) and Gaussian distributions. Results are shown for positive (dark gray hatching) and negative (red hatching) thresholds of M = ±0, ±1, and ±2 standard deviations, with 95% confidence intervals estimated from the sampling uncertainty among 1000 100-winter blocks of 100-day winters. Note the striking overall similarity between (a) and (b).

Citation: Journal of Climate 28, 23; 10.1175/JCLI-D-15-0020.1

• Download Figure
• Download figure as PowerPoint slide

The general character of such persistence probability curves is remarkably well captured, without invoking nonlinearity, by the CAM noise–driven linear Markov model (10) associated with SGS distributions. Figure 8b shows the P_M(T) of x(t) from the 10⁸-day model integrations presented earlier for the SGS (S = 1, K = 5) and Gaussian cases, in an identical format to that of Fig. 8a. Note again that time here is actually measured in correlation time units t/τ_c, and is referred to as “days” using τ_c = 1 day purely for illustrative purposes. Results are shown for thresholds of M = ±0, ±1, and ±2 standard deviations, with 95% confidence intervals estimated from the sampling uncertainty among 1000 100-winter blocks (“centuries”) of 100-day “winters.” The similarities with Fig. 8a are striking.

The exponentially decaying curves for both the SGS and Gaussian cases in Fig. 8b are associated with the same expected exponential decay from an initial condition x(t). What distinguishes them are the unconditional probabilities of x(t), as well as the conditional probabilities (not just expected values) of x(t + τ) given x(t). Both of these are altered in the SGS case by the asymmetric CAM noise forcing (Ex + g)η₂ in (10). Given the heavier positive tails in this positively skewed SGS case, it is then not surprising that large positive anomalies last longer than large negative anomalies. It is also not surprising that, overall, positive anomalies do not last as long as negative anomalies, because the unconditional probability of positive x [i.e., p(x) in (3) integrated over all positive x] is smaller than that of negative x (see Fig. 6b). Note that the signs of all such asymmetries are reversed for negatively skewed SGS distributions.

The ability of the linear Markov process (10) to represent such temporal persistence statistics—correctly discriminating among anomaly persistence beyond positive and negative, as well as large and small, amplitude thresholds—makes it useful for investigating the statistical significance of a wide variety of changes in weather statistics associated with climate change. It also enables any changes that are established as significant in a variable to be attributed, as a first step, to changes in the deterministic dynamics (encapsulated in λ = 1/τ_c) versus the stochastic forcing (E, g, and b) of the variable.

6. Extreme value analysis using SGS versus GEV or generalized Pareto distributions

In this section we show that fitting SGS distributions to all available values in limited records yields extreme value distributions of block maxima with smaller sampling uncertainties than generalized extreme value distributions fitted to only the block maxima. We also show how the tails of SGS distributions are linked to the corresponding GEV distributions of block maxima and generalized Pareto (GP) distributions of exceedances over specified thresholds.

The widespread use of GEV distributions in extreme value analysis is motivated by their quasi-universal applicability. If F(x) is the cumulative probability distribution function (CDF) of x, and is the maximum value of x in a sample of N independent values (a “block”) of x, then the CDF of is , which for large N approximates a GEV distribution

View Expanded

where , , and are shape, scale, and location parameters. It is remarkable that F(x) need not be known explicitly to estimate the distribution of in this limit. The question is how large does N need to be for the limit to apply, and given that each block yields only one sample value of , how accurately can the distribution be estimated using the relatively small number J of such blocks in available climate records.

Figure 9 illustrates the large sampling uncertainties of GEV fits when only 25 blocks (“winters”) of data are used. The thick black curves show the “true” PDFs of 100-day block maxima of the Markov process (10) for the Gaussian and SGS cases discussed earlier, determined using 10⁶ such blocks in 10⁸-day simulations. The shaded bands indicate 95% sampling error bars on the PDFs if they are estimated using just 25 blocks. Not surprisingly, the errors are large if only the raw histograms of the 25-block maxima are used. Fitting GEV distributions to the 25-block maxima reduces the errors (although not substantially in the Gaussian parent case), but using F(x) fitted to all 2500 daily values of x reduces them even more. [Note that to avoid ambiguity in specifying the precise number N of independent x values in blocks of length N_d = 100 days, the PDFs in the latter case were not estimated directly as , but from 2000 additional 10⁶-day Markov model runs with SGS parameters estimated from 2000 randomly selected 25-block segments of the full 10⁸-day runs, and processed similarly as in estimating the “true” PDFs.]

View Full Size

A demonstration that for limited-length records, the PDF of winter (100-day) maxima of daily values (extreme value PDFs) can be more accurately estimated by fitting SGS distributions to all daily values than by fitting GEV distributions to just the maximum values from each winter. The thick black curve in each panel shows the “true” extreme value PDF of the winter maxima in 10⁸-day integrations of the Markov process (10) associated with the (a) Gaussian and (b) SGS parent distributions of Fig. 1. Shading indicates 95% confidence intervals for the extreme value PDFs estimated using 25-winter (2500-day) segments of the integrations, using raw histograms of the 25 winter maxima (light gray shading), fitting a GEV distribution to the 25 winter maxima (cross-hatching), and using the SGS distribution fit to the 2500 daily winter values (dark gray shading).

Citation: Journal of Climate 28, 23; 10.1175/JCLI-D-15-0020.1

• Download Figure
• Download figure as PowerPoint slide

The results in Fig. 9 are consistent with the intuition that if the parent distribution F(x) is known, the PDF of can be more accurately estimated using F(x) fitted to all J × N_d values of x in the record than by fitting a GEV distribution to just the J block maxima , in essence because the former approach uses more information. The GEV approach entails a trade-off in splitting the J × N_d record into J blocks of length N_d. Ideally, both J and N_d should be large, but in practice a compromise is necessary. A smaller J increases sampling error, whereas a smaller N_d makes the GEV approximation itself less valid, especially if the parent distribution is Gaussian. As shown in appendix B (Fig. B1), the GEV fit to the “true” PDF of N_d-day block maxima from the 10⁸-day Markov model runs described earlier is much less accurate in the Gaussian parent case than in the SGS parent case for N_d less than 200 days.

The extreme value distribution associated with an SGS distribution approaches a GEV form for large enough block sizes N that in each block occurs in the heavy tail of p(x) in (7). It is then easy to derive an expression for and the corresponding GEV distribution (see appendix C), whose parameters are related to those of the parent SGS distribution as

View Expanded

View Expanded

View Expanded

Note that the shape parameter ξ does not depend upon N. The increase of and with N is associated with the fact that a larger is more likely to occur in a longer block.

Insofar as SGS distributions are relevant, the correspondence between GEV and SGS distribution parameters made explicit by (13) enables the GEV parameters to be interpreted physically through the Markov model (10), and provides insight into their variation with block size N. The role of E, the amplitude of the multiplicative noise in (10), is key, since E completely determines ξ and the dependence of and on N. Most importantly, the upper bound 2/(n − 1) of E², if exists, implies an upper bound 1/n of ξ through (13a). The fact that ξ in most climatic contexts is estimated to be less than 1/4 (e.g., Kharin and Zwiers 2005) is thus consistent with both the existence of the fourth moment (kurtosis) and the relevance of SGS anomaly dynamics. The key role of E also explains the slow convergence of the extreme value distribution to the GEV form for a Gaussian parent associated with E → 0, since for small E the block size N needs to be relatively large to ensure that occurs in the heavy tail of p(x) in (7). (The results in Fig. B1 are consistent with this.)

Extreme value analysis based on generalized Pareto (GP) distributions circumvents some of the sampling issues of GEV distributions, while still retaining some of their quasi-universal applicability. The idea is to shift the focus from the probabilities of just block maxima to the probabilities of exceedances over specified thresholds X_T. According to theory (e.g., Pickands 1975; Smith 1989; Davison and Smith 1990), for “high enough” thresholds the conditional PDF of x, given that x exceeds X_T, approaches the GP form

View Expanded

where ξ, , and X_T are shape, scale, and location parameters. This quasi-universal form of tail PDFs is posited on a self-similarity requirement of “threshold stability,” which in essence is a requirement that “the tail of a tail should look like the tail.” Note that unlike the GEV distribution, which is a model of the sampling distribution of a function of tail values (their maxima in large blocks), the GP distribution is a model of the distribution of the tail values themselves. In general, the tails need not be of GP form, of course. Even in contexts in which they are of GP form, specifying the threshold X_T beyond which (14) is valid can be an important concern, requiring additional assumptions about the distribution of nonextreme values that allow for the possibility that a single distribution may not fit all values (e.g., Tancredi et al. 2006).

Fortunately, it turns out that the tails of SGS distributions are precisely of the GP form. Specifically, for high enough X_T that p(x) for x > X_T can be approximated by (7), the conditional PDF of x given that x exceeds X_T is identical to (14) (see appendix C), with a location parameter X_T and shape and scale parameters

View Expanded

The shape parameter ξ is identical to that of the GEV distribution in (13a), but the scale and location parameters, which now depend on X_T instead of block size, are different.

As in the case of the GEV distribution, the correspondence between the GP and SGS distribution parameters made explicit by (15) enables the GP parameters to be interpreted physically through the Markov model (10). The role of E is again key, since E completely determines ξ. In the limit of E → 0 associated with a Gaussian parent, both ξ → 0 and → 0, underscoring that fact that the tail of a pure Gaussian (E = 0) is not of the form (7) even for X_T → ∞.

This analysis highlights the contrast between the extreme value distribution–based approach 1 and the full distribution–based approach 2 to estimating tail probabilities discussed in section 1. The GEV and GP approaches are clearly valuable when the parent distribution F(x) is unknown, but are unnecessary when F(x) is known, especially if F(x) is associated with a process model such as (10) that can be used to quantify sampling uncertainties and provide physical insight. The issue, of course, is to what extent F(x) is known. We have argued for the relevance of SGS distributions as the simplest physically based prototypes for non-Gaussian distributions that include Gaussian distributions as special cases. Even if F(x) is not strictly SGS but is only approximately so, our analysis suggests a way to improve the estimation of GEV and GP parameters by constraining, for example, the shape parameter ξ to be consistent with (13a), with E estimated from (8a) using the full record and not just extreme values. This utility of the SGS approach would clearly be compromised if F(x) deviates significantly and substantially from an SGS distribution, for instance if p(x) has more than one maximum. Note however that even in such instances the GP distribution may not necessarily provide a more accurate model of tail probabilities.

7. Form invariance with respect to climate changes

The SGS distribution is an anomaly distribution associated with a one-dimensional linear Markov model (10). This model is a highly idealized model of the dynamics of individual system components of the multicomponent nonlinear climate system. A rationale for all the multicomponent linear and nonlinear interactions to be approximately represented by parameters E, g, b, and λ in (10) was given in SS09, and the validity of (10) as a process model for the stationary statistics and some temporal persistence statistics has been provided here. No additional assumptions need be made to retain (10) as an equally valid model of anomaly statistics in an altered climate. To that extent, the anomaly distribution of a variable in the new climate should remain an SGS distribution, with possibly different parameters. This rationale is the same as that for expecting anomaly distributions to remain Gaussian in an altered climate if there is a physical basis for them to be Gaussian in the current climate.

For small climate shifts, the form invariance of SGS distributions can be rigorously derived by interpreting the mean shift as the mean anomalous response to a mean anomalous external forcing f_ext in (10). Incorporating this constant forcing as in (10) modifies it to

View Expanded

If we now write , with , the equation for the anomalies x′ around the new mean becomes

View Expanded

where . This anomaly equation in the new climate is identical to (10) in the original climate, with the same E and b, but with g replaced by . The anomaly distribution in the new climate is then also an SGS distribution, with a PDF (3) in which g is replaced by .

The change in g implies changes in the variance, skewness, and kurtosis of the SGS distribution through (4b) and (5), and hence changes in the width and shape of the distribution. It also implies changes in the types of persistence statistics shown in Fig. 8b, even without a change in λ.

Figure 10 shows how accounting for the changes in σ², S, and K modifies the tail probabilities of the shifted SGS and Gaussian distributions in Fig. 1. (For a cleaner comparison we specify the same change of σ² in the Gaussian case.) The probabilities of exceeding 2 sigma, which were previously stated as increasing to 15.8% (13.3%) in the Gaussian (SGS) cases due to the mean shift alone, are modified to 21.2% (16.7%) after also accounting for the changes in width and shape, and the probabilities of exceeding 4 sigma are modified even more strongly, from 0.13% (1.0%) to 0.92% (2.65%). Correctly accounting for the changes in width and shape in addition to the mean shift is clearly important in quantifying changes in tail probabilities.

View Full Size

As in Fig. 1, but now also with changes in the width and shape of the SGS distribution associated with the mean shift taken into account. For a cleaner comparison the shifted Gaussian PDF is also shown with the same width as the SGS distribution. See text for a fuller explanation.

Citation: Journal of Climate 28, 23; 10.1175/JCLI-D-15-0020.1

• Download Figure
• Download figure as PowerPoint slide

We stress that for even small climate shifts, the change in the SGS distribution is not shape preserving. However, the change in the associated GEV distribution is shape preserving, because the constancy of E implies the constancy of the shape parameter ξ in (13a), as has indeed been found (e.g., Kharin and Zwiers 2005), and sometimes even assumed (as in Zwiers et al. 2011). This allows the sign of the change in the GEV tail (i.e., in the “extreme extremes”) to be deduced from the sign of the extremal probability shift index (PSI) in (1), by specifying the changes in the estimated location and scale parameters.

For larger climate shifts, it is possible for not just g but the other parameters in (10) to change as well. We have not yet found a simple way to characterize the combined effect of the changes in E, g, and b on the tail probabilities, both of extremes and extreme extremes. It appears that in this more general scenario there would be no simpler alternative to deducing the changes in tail probabilities from SGS distributions fitted to data before and after the mean shift.

8. An application to assessment of twentieth-century circulation changes

Finally, we present an illustrative application of our SGS distribution–based approach to assess the changes in two sea level pressure (SLP) based indices of large-scale wintertime atmospheric circulation variability in the Northern Hemisphere, the North Atlantic Oscillation (NAO) index and the North Pacific (NP) index, over the period 1872–2011. To this end we use the 56-member ensemble 20CRv2 dataset (Compo et al. 2011), in which the time series of each ensemble member represents an equally likely estimate of global atmospheric evolution over the 140-yr period within observational error bounds.

To provide context, Fig. 11 shows the ensemble-averaged 7-yr running mean anomalies of the NAO and NP indices. Corresponding results obtained using the NCEP–NCAR reanalysis and ERA-40 datasets for shorter periods are also shown for comparison. There are large multidecadal variations in both indices, including an apparent upward trend in the NAO index and a downward trend in the NP index during much of the second half the twentieth century. However, the trends over the full 140-yr period are much weaker or even nonexistent. This has important implications for understanding the atmospheric circulation response to global warming, and casts doubt on inferences about this response drawn in studies that focus only on the second half, or other subsets, of the full record.

View Full Size

7-yr running means of anomalous sea level pressure (SLP)-based North Pacific (NP) and North Atlantic Oscillation (NAO) indices for the 1872–2011 period. Results are shown for ensemble means of the 56-member ensemble 20CRv2 (Compo et al. 2011; black curves). Corresponding results obtained using the NCEP–NCAR reanalysis (NNR; orange, 1948–2010) and ERA-40 (cyan, 1958–2001) datasets are also shown for comparison with the 20CRv2 results. The NAO index is defined as the daily area-averaged anomaly in a 5° cell centered on 38°N, 26°W minus that centered on 64°N, 22°W, using all available data in each dataset. The NP index (Trenberth and Hurrell 1994) is the area-average anomaly in the region 65°–30°N, 160°E–140°W. Anomalies are defined as departures from the 1981–2000 December through February mean.

Citation: Journal of Climate 28, 23; 10.1175/JCLI-D-15-0020.1

• Download Figure
• Download figure as PowerPoint slide

Despite the absence of a long-term mean trend in these indices, one might still think it possible for the probabilities of their extreme values to have changed in association with possible changes in the width and shape of their PDFs. We consider such a possibility, even though the argument given in the previous section precludes it for small (in these cases, nearly zero) mean shifts. Since our interest is in daily weather extremes and the large-scale circulation contexts in which they occur, we focus on changes in the PDFs of daily NAO and NP indices. These have autocorrelation time scales τ_c of approximately 5.2 and 7 days, respectively.

For each one of the 56 members of the 20CRv2 ensemble, we estimated the mean, variance, skewness, and kurtosis of the daily index anomalies separately from the first and second halves (1872–1941 and 1942–2011) of the 1872–2011 record, and used (8) to fit SGS distributions in each half. As Fig. 12 shows, the ensemble spread of the 56 fitted PDFs, associated with observational uncertainty, is much smaller than that of the raw histograms. The ensemble-mean fitted PDFs do show some changes from the first half to the second half of the record. This still leaves open the question, however, as to how statistically significant those changes are given that we used only 70 years of data to estimate the distributions in each half. We addressed this question through 10⁶ winter simulations of the Markov process model (10), with parameters specified from the ensemble-mean fitted SGS distributions for the second half. We identified the sampling uncertainty of the fitted observational SGS distributions with the sampling variability of SGS distributions fitted to 70-yr segments of the Markov model simulations. This sampling uncertainty is indicated as the gray shaded bands in Fig. 12.

View Full Size

PDFs of standardized daily NP and NAO indices in northern winter. The standardization is based on the full variance of daily data from 1872 to 2011. The standardized daily NAO index is defined as the standardized daily anomaly near Ponte Delgada, Azores (38°N, 26°W), minus that near Reykjavik, Iceland (64°N, 22°W). Anomalies are with respect to the 1871–2011 mean plus 3 harmonics of the annual cycle. The PDFs are estimated for two 70-yr periods, 1872 to 1941 (red) and 1942 to 2011 (blue), both as raw histograms (rectangles) and as fitted SGS PDFs [curves; see (3)]. Results for December through February are shown for each one of the 56 members of the observational 20CRv2 (Compo et al. 2011). There are thus 56 red and 56 blue curves in each panel. Upper and lower segments of the red and blue rectangles show the range of raw counts for each anomaly size bin. The gray swaths indicate 95% confidence intervals on the PDFs associated with using limited 70-yr records, estimated from long Markov model simulations [see (10)]. The internal spread among the red and blue curves is a measure of observational uncertainty, whereas the gray swath is a measure of sampling uncertainty. The shifts of the mean are indicated by the red and blue circles near the center of each panel.

Citation: Journal of Climate 28, 23; 10.1175/JCLI-D-15-0020.1

• Download Figure
• Download figure as PowerPoint slide

Figure 12 suggests that there was not only no change in the mean, but also no statistically significant change in the entire PDFs of the NAO and NP indices from the first to the second half of the period 1872–2011. This is consistent with the theoretical expectation from the previous section that if there is no mean shift, there should be no change in the entire PDF if the PDF is an SGS distribution.

9. Discussion and concluding remarks

The difficulties of detecting significant long-term changes in the atmospheric circulation using historical records of limited length and accuracy are well appreciated in the climate research community. They provide a major impetus for ongoing efforts to improve atmospheric reanalyses and extend them as far back in time as feasible. Indeed the very different trends in Fig. 11 over the last 70 years versus the last 140 years in the first such long-term 20CRv2 reanalysis underscore the danger of drawing conclusions about long-term changes from relatively short records. Given the strong link between daily weather statistics (“storm tracks”) and the ambient flow (Whitaker and Sardeshmukh 1998; Compo and Sardeshmukh 2004), there are evident pitfalls in establishing significant long-term changes in weather statistics as well. Changes in extreme weather statistics are even harder to pin down using short records. Since large-scale weather events typically last a day or longer, and there are only about 9000 days of data for each 3-month season in a 100-yr record, a discussion of changes in extreme weather events that occur only a few times in a century is compromised by large uncertainties in their small probabilities on the order of 0.01%. Discussions of such small (and sometimes even smaller) probabilities indeed force one to confront the Bayesian versus frequentist meanings of probability itself based on “degree of belief” versus counts of outcomes. The issue, at its core, is the meaningfulness of assigning quantitative probabilities to events that have never occurred or have occurred only a few times in a finite record. A claim that the probability of an extreme weather event has “doubled” from 0.01% to 0.02% is of a very different character in this regard, given the likelihood of unrealized outcomes in a 100-yr record, than a claim that it has doubled from, say, 10% to 20% or even from 1% to 2%.

The basic note of caution sounded in this paper regarding quantifiable tail probabilities is that they are strongly affected by the non-Gaussianity of the observed distributions, and that their changes cannot generally be deduced from the mean shifts alone. The skewed and heavy tailed character of the distributions warrants accurate representation not only of the mean changes but also of changes of variance, skewness, and kurtosis for deducing the changes in tail probabilities. This has large implications for the detection, attribution, and projection of changes in extreme weather statistics using limited and inaccurate observations and imperfect climate models.

From a physical perspective, understanding the probabilities of extreme “unusual” values of an atmospheric variable requires an understanding of how such values are generated, whether through amplified mechanisms for generating the “usual” values or through concatenation of “a series of unfortunate events.” In either case the probabilities of the unusual and usual values are physically related. The tails of the probability distribution are thus related to the properties of the entire distribution, including its mean, width (variance), and shape (skewness and kurtosis). To get tail probabilities right, one has to get all of these properties right.

We have presented evidence that the distinctively skewed and heavy tailed character of the observed distributions of daily weather anomalies is basically due to their evolution being perturbed by stochastic noise whose amplitude increases linearly with anomaly amplitude, but asymmetrically for positive and negative anomalies. This statistically accurate description is associated with a precise model of the anomaly probability distribution as a stochastically generated skewed (SGS) distribution generated by correlated additive and multiplicative (CAM) noise forcing. The relevance of SGS distributions is not inconsistent with the notion that additional feedbacks play an important role in the evolution and persistence of some extreme anomaly events. Rather, it suggests that such additional feedbacks are not systematic, and can effectively be treated as a CAM noise forcing in modeling anomaly statistics.

We have described how to fit SGS distributions to data using the first four moments (mean, variance, skewness, and kurtosis), and how to generate a statistically equivalent CAM noise–driven Markov process with the same parameters, which is useful for estimating sampling uncertainties and persistence statistics. A virtue of this approach is that it makes use of all values in a time series to estimate tail probabilities, whose sampling uncertainties are therefore smaller than those associated with GEV and GP approaches that use just the extreme values. We stress that our approach is applicable not only to SGS variables but also to other variables that are monotonically related to SGS variables, such as precipitation via its strong link to midtropospheric vertical velocity. Overall, this approach provides a sharper tool for investigating the statistical significance of observed changes in extremes over the twentieth century and of projected changes over the twenty-first century.

Current statements concerning changing extreme weather risks associated with global warming mainly involve quantities related to surface temperature (IPCC 2014) and essentially amount to simple shifts of existing PDFs. Such statements downplay the fact that there is more to regional climate change than surface warming, and that assessing the changing risks of extreme storminess, droughts, floods, and heat waves requires accurate model representations of multidecadal and longer-term changes in the large-scale modes of natural atmospheric circulation variability and the complex nonlinear climate–weather interactions associated with them. The detection of changes in such modes from the limited observational record is much less clear cut than for surface temperature. Figure 12 demonstrated this for indices of large-scale modes of wintertime atmospheric circulation variability in the North Atlantic sector (the NAO index) and the North Pacific sector (the NP index). We found no significant changes either in the mean or in the entire PDFs of these indices over the last 140 years. This is of course not to imply that there have been no changes in other modes of circulation variability in other parts of the globe, just that any such changes need to be carefully evaluated on a case-by-case basis after due consideration of the non-Gaussian aspects of the probability distributions.

To detect, attribute, and make credible projections of changing extreme weather risks in a changing climate using climate models, the models need to represent not only the shifting means but also changes in the width and shape of the distributions of daily weather anomalies. Model misrepresentations of these changes can cast doubt on such efforts. Indeed current models have difficulty in representing even the mean changes, such as their inability to adequately capture the post-1998 “hiatus” in global warming (Fyfe et al. 2013). The mean errors are relatively large on regional scales, in substantial part because of model difficulties in capturing the pattern of tropical ocean temperature changes and their global impacts (Shin and Sardeshmukh 2011) that are associated with misrepresentations of tropical feedback processes (Shin et al. 2010). To estimate changes in tail probabilities using such imperfect models, it is important to account for biases in the probability distributions. However, it is clearly inappropriate to do so through a simple a posteriori “bias correction,” given the links between the mean, variance, and shapes of the predicted probability distributions. There is limited understanding of such links at present, mainly due to the limited understanding of climate–weather interactions on regional scales.

Given the importance of non-Gaussianity highlighted in this paper, it would be interesting to use our SGS–Markov process approach to analyze the tail probabilities of other weather-related variables such as near-surface temperature and precipitation in observational datasets and climate model simulations and projections. A multivariate Markov process model, of which (10) is a univariate approximation as discussed by SS09, could also be used as a more accurate “probability model” for this purpose. These are topics of current research.