1. Introduction
Empirical orthogonal function (EOF) analysis, principal component analysis, or eigenvector analysis is a widely used method in climate sciences. For the complex climate system, the EOF method can decompose a space–time field into spatial patterns and associated time series (von Storch and Zwiers 1999) and reduce the dimensionality of system to a few prominent modes (Hannachi et al. 2007). Therefore, this method has been extensively used to identify and illustrate the dominant modes or teleconnection patterns in the climate system such as El Niño–Southern Oscillation (ENSO; Takahashi et al. 2011; Timmermann et al. 2018), the North Atlantic Oscillation (NAO; Hurrell and Deser 2010), the Pacific–North American pattern (Barnston and Livezey 1987), and Pacific decadal oscillation (Newman et al. 2016), and detect climate signals (North and Wu 2001). The EOF method has also been extended to different research fields such as physics (Lloyd et al. 2014), bioinformatics (Karamizadeh et al. 2013), and economics (Zhang and Wang 2022).
However, the sampling size is variously assigned when researchers use EOF analyses to derive the modes of variability as reported in the literature (e.g., Stoner et al. 2009; Lee and Black 2013; Lee et al. 2014; Wang et al. 2017; Coburn and Pryor 2021; Li et al. 2021), ranging generally from about 30 to 100. Some studies investigated the sampling error of EOF analysis (e.g., Quadrelli et al. 2005; North et al. 1982), focusing on the separation of a particular mode from other modes. Assigning relatively small sampling size stems from various reasons, including particularly the limitation of available data and the snapshot analyses that emphasize the stepwise statuses in changing systems (such as global warming). Therefore, a question arises naturally: How large a sampling size is adequate to guarantee the reliability of results obtained by EOF analyses? In other words or more generally, how does the uncertainty of EOF results decrease with increasing sampling size? In particular, the EOF method is widely used in the evaluation and projection of dominant modes of variability in climate models. The uncertainty of EOF would affect remarkably the credibility of the EOF results on model evaluation and future projection. Thus, it would be a vital issue to judge whether the leading modes or patterns obtained by the EOF method are reliable or not in climate models. However, to our knowledge, this question has not been clarified yet, despite some studies on how many ensemble members can guarantee the credibility of climatological means obtained as the averages over a certain period (equivalent to sampling size) (e.g., Deser et al. 2012; Milinski et al. 2020; Peings et al. 2021) and intensity of variability in terms of standard deviation (e.g., Maher et al. 2018; McKenna and Maycock 2021).
In this study, we use the model-simulated results of preindustrial control (piControl) experiments in phase 6 of the Coupled Model Intercomparison Project (CMIP6), which can provide adequate sampling sizes, and quantify how the uncertainty of EOF patterns changes with the sampling size in climate models. We choose the two well-known phenomena: ENSO, which is characterized by the strong variability in sea surface temperatures (SSTs) in the equatorial central and eastern Pacific, and the NAO, which is a teleconnection pattern in the North Atlantic, to illustrate the uncertainty of EOF patterns.
2. Datasets and methods
a. Datasets
In this study, we focus on the NAO defined by sea level pressure (SLP) and ENSO defined by SST in winter [December–February (DJF)]. The dataset used in this work to identify NAO in observations is the Twentieth Century Reanalysis, version 3 (20CRv3), monthly SLP. The 20CRv3 is constrained by observations of surface pressure and provides data from 1836 to 2015 (Compo et al. 2011; Giese et al. 2016; Slivinski et al. 2019). The data have a spatial resolution of 1° × 1°. The SST data in observations used in this study are the NOAA Extended Reconstructed Sea Surface Temperature dataset, version 5 (ERSSTv5) (Huang et al. 2017), which provides spatially complete SST data on a 2° × 2° grid at a monthly time step from 1854 to present.
We also use 50 piControl experiments in CMIP6 (for detailed information, refer to https://wcrp-cmip.github.io/CMIP6_CVs/docs/CMIP6_source_id_experiment_citation.html). The variables selected in piControl experiments include monthly mean SLP and SST. The piControl experiments provide long integrations without external forcings and exclude the spinup period (Eyring et al. 2016). We select models with its integration longer than 450 years, particularly HadGEM3-GC31-LL and IPSL-CM6A-LR, which have 2000-yr integration. Most models have the same length of output for SLP and SST except for two models: The EC-Earth3-Veg provides 1000-yr output for SLP but 500-yr output for SST, and the FIO-ESM-2-0 provides 575-yr output for SLP but 500-yr output for SST. Before analyzing, the monthly mean SLP data in 20CRv3 and models are regridded to 2.5° × 2.5° grid and the SST data in models are regridded to 2° × 2°.
b. The NAO and ENSO pattern
For SLP and SST, linear trends and decadal variations are subtracted from original DJF mean time series on each grid point in observations and models first, which reduces the influence of global warming in observations and eliminates model drifts in piControl experiments and thus highlight interannual variability. We then use the EOF analysis to derive the prominent patterns. The NAO pattern is defined by the SLP anomalies regressed onto the leading principal component (PC1) of EOF analysis on SLP over the domain 20°–80°N, 90°W–40°E, with area weighted (Hurrell and Deser 2010). Similarly, the ENSO pattern is defined, but with SST anomalies and the EOF domain over 25°S–25°N, 140°E–80°W (Timmermann et al. 2018).
c. Sampling method
We use random sampling with replacement, which can diminish variance loss in comparison with sampling without replacement, to obtain the probability distributions of CCs corresponding to different size of samples. For instance, sampling size of 30 means collecting 30 samples from the pool of the entire time period for observations or models. Then, we perform EOF analysis to obtain an NAO or ENSO pattern and calculate the CC of the NAO or ENSO pattern with the reference field. Repeating this process 3000 times, we get the probability distribution of CCs corresponding to 30 samples. Similarly, we can obtain probability distributions of CCs corresponding to 40, 50, …, 100, 150, 200, 250, 300, 350, and 400 samples. The maximum size of samples is fixed to 150 in observations due to limited length. For models, the 5%–95% confidence interval (CI) of CC is generally very small when the sampling size is larger than 100, for both the NAO and ENSO in section 4. Therefore, how the CI of CC changes with the increase of sampling size is actually more determined by the CIs of the sampling sizes less than 100 and thus the maximum sampling size is 400 in this study for models.
d. Fitting metrics
3. The uncertainty of EOF patterns
Figure 1a shows the probability distribution of CCs of NAO pattern with the sampling size of 30 in observations obtained by 3000 random samplings. That is, we collect 30 years from the entire period (1836–2015) randomly and repeat the process 3000 times. It should be mentioned that the first EOF mode in the reference field for each model resembles well the observed pattern, for either the NAO or ENSO (not shown). The median of CCs is 0.97, and the 5%–95% CI of CCs is from 0.85 to 0.99. Approximately 84% and 26% of the samples have a good match (CC > 0.92) and an excellent match (CC > 0.98), respectively. The rank of 95% can yield an NAO pattern that closely resembles the reference field (Fig. 1b), that is, the NAO pattern obtained by using the observational data of the entire period. However, the rank of 5% shows an NAO pattern with southern lobe shifting eastward and weakened intensity in comparison with the reference field (Fig. 1c). This suggests that there is at least a 5% probability of getting an inappropriate NAO pattern if using 30 years’ data. In addition, the probability distribution of CCs shows a negatively skewed distribution, indicated by the strongly negative skewness (−6.0). There are some extraordinary CCs near zero, which is related to the phenomenon of EOF swapping (Lee et al. 2019). Note that the NAO is defined as the first EOF mode, but some extraordinary samplings may reverse with the second mode, resulting in very low CCs. The uncertainty of EOF pattern also exists in ENSO, although much smaller when compared with the NAO (Fig. 1d). There are approximately 99.9% and 74% of the samples showing a good match (CC > 0.92) and an excellent match (CC > 0.98), respectively, and these ratios are much higher than those for the NAO. The rank of 95% is very akin to the reference pattern (Fig. 1e), but the rank of 5% yields an ENSO pattern with weakened intensity (Fig. 1f).
Thirty-year DJF mean NAO and ENSO patterns derived from random sampling in observations: (a) The 5%–95% CI (the black error bar) and the median of CC (the black circle). The blue crosses represent outliers smaller than the 5th percentile of CC. Also shown are the NAO patterns (hPa; shading) corresponding to the (b) 95th percentile and (c) 5th percentile of CC. The black contours are the ±0.5, ±1, ±2, ±3, and ±4 hPa values and represent the reference NAO pattern derived from 1836 to 2015 DJF mean SLP in 20CRv3. (d)–(f) As in (a)–(c), but for ENSO patterns in ERSSTv5. The reference ENSO pattern is calculated from 1854 to 2021 DJF mean SST. The black contours are the ±0.1°, ±0.2°, ±0.4°, …, ±1.0°C values.
Citation: Journal of Climate 37, 7; 10.1175/JCLI-D-23-0165.1
The CI of CC becomes narrower and the median of CC increases with the sampling size increasing in observations. Similar results can be found for each model. Here, for brevity, we only show the results for observations and the models with the available data of both SLP and SST longer than or equal to 1000 years (Fig. 2), since these model results can adequately illustrate model diversities. When the sampling size is 30, the percentages of CC smaller than 0.92 range from 8.3% to 28.5% for the NAO in the 9 models. For ENSO, only INM-CM5-0 shows a remarkable percentage (14.3%), the CCs of other models are almost larger than 0.92 in 3000 samples. In addition, it can be found that the CIs vary considerably between the models. For instance, the CI for NAO in ACCESS-ESM1-5 (0.25) is about 2.8 times that in IPSL-CM6A-LR (0.09) when the sampling size is 30, and the CI in the former model (0.05) is about 2.5 times that in the latter model (0.02) when the sampling size is 100. For ENSO, the CI in INM-CM5-0 (0.08) is about 8 times that in CESM2 (0.01) when the sampling size is 30, and the CI in the former model (0.02) is about 6 times that in the latter model (0.003) when the sampling size is 100.
CCs change with size of samples increasing in observations and models longer than or equal to 1000 years for (top),(top middle) the NAO and (bottom middle), (bottom) ENSO. In each panel, the abscissa represents the size of samples (yr) from 30 to 100 and the ordinate represents the values of CCs. The blue error bars and the blue circles respectively indicate the 5%–95% CIs and medians of CCs.
Citation: Journal of Climate 37, 7; 10.1175/JCLI-D-23-0165.1
Actually, the CI and median of CC, which are two important indicators of reliability of EOF results, show a highly negative correlation in models: The CIs become narrower while the medians become larger with the sampling size increasing (Fig. 3). The relationship between CI and median is roughly linear, although some models, such as ACCESS-ESM1-5, show a relatively higher CI when the sampling size is small. In comparison with the median, the varying CI shows a wider range, which is consistent with the negatively skewed distribution of CC. Considering the close relationship between CI and median and the larger range of CI changes, we only focus on the CI variations in the following.
The relationship between the medians (abscissa) and 5%–95% CIs (ordinate) of CCs for (top),(top middle) the NAO and (bottom middle),(bottom) ENSO. The correlation coefficients are shown in the upper-right corner of each panel.
Citation: Journal of Climate 37, 7; 10.1175/JCLI-D-23-0165.1
4. Quantifying the variations of CIs with sampling sizes
Figure 4 shows the variations of 5%–95% CI with the sampling sizes in two models (MIRCO-ES2L and INM-CM4-8; the reason for selecting them is to be shown later) and observations. In this figure we show the results for only two models in an attempt to reveal the details of fitting effect. The CI decreases rapidly with the sampling size increasing when the sampling size is small. When the sampling size becomes large, such as larger than 100, the rate of the CI decrease becomes much slower. After many attempts, we found that the concise function y = a/(x − b) fits the relationship well regardless of NAO or ENSO. Here, y is the 5%–95% CI of CC and x is the sampling size. Parameters a and b can be derived from the least squares method, and then the fitting effect can be estimated by the MSE and r2, which are smaller than 3 × 10−5 and larger than 0.99, respectively, for 50 models and observations, regardless of NAO or ENSO.
Changes of the 5%–95% CI of CC with the size of samples increasing in (a),(b) two models with the largest MSE and (c) for observations for NAO (blue) and ENSO (red). The abscissa represents the size of samples (years) from 30 to 400, and the ordinate represents the 5%–95% CI of CC. Open circles are derived from random sampling, and the curves are fitting results based on the least squares method. The numbers in the upper-right corner are the minimum sampling size to obtain reliable NAO and ENSO patterns. Given the limited length of data, the maximum sampling size is 150 in the observations [(c)].
Citation: Journal of Climate 37, 7; 10.1175/JCLI-D-23-0165.1
The models MIRCO-ES2L and INM-CM4-8 have the largest MSE for the NAO and ENSO, respectively, and thus they are deliberately shown in Figs. 4a and 4b, parallel to the observations in Fig. 4c. It can be found that the fitting effect is good even for these two models (the blue line in Fig. 4a and the red line in Fig. 4b). The concise function is also valid for NAO and ENSO in observations and other 48 models (Figs. 4c and 5).
As in Fig. 4, but for the other 48 models with respect to NAO (blue) and ENSO (red).
Citation: Journal of Climate 37, 7; 10.1175/JCLI-D-23-0165.1
The curve of CI is determined by the parameters a and b. The smaller the parameters a and b, the narrower the CI for a fixed size of samples. The parameter b modulates the sampling sizes directly, and the parameter a scales the modulated sampling sizes, according to the fitting function. Moreover, the parameter a is closely related to the difference between the variance fractions of EOF1 (VF1) and EOF2 (VF2) in individual models: a decreases with VF1-VF2 increasing in a power function fitting, based on the least squares method (Figs. 6a,b). Here, the VF1-VF2 is obtained from all integrations for individual models. The VF1-VF2 ranges from 0.17 to 0.39 for the NAO, i.e., the EOF1 explains 17%–39% more total variance than the EOF2, depending on individual models, and correspondingly, a decreases from 7.19 to 1.25. Similarly, the VF1-VF2 ranges from 0.17 to 0.75 for ENSO, and a decreases from 2.24 to 0.15. The MSE is 0.08 and 0.008 for the NAO and ENSO, respectively, and r2 is 0.93 and 0.94, respectively, suggesting the goodness of fit. Even if we remove two extreme values in the left tail, the r2 of the power function is as high as 0.84 for ENSO.
Scatter diagram of the parameters (top) a and (bottom) b vs the difference between variance fractions of EOF1 and EOF2 in 50 models for the (a),(c) NAO and (b),(d) ENSO. The VF1 and VF2 are derived from the EOF analysis on all integration in each model. The dashed curves are fitting function based on the least squares method, and the expressions are given in the upper-right corner for NAO [in (a)] and ENSO [in (b)].
Citation: Journal of Climate 37, 7; 10.1175/JCLI-D-23-0165.1
The parameter b, by contrast, is vaguely related to the VF1-VF2 (Figs. 6c,d). For the NAO, the parameter b shows a negative correlation with the VF1-VF2 (Fig. 6c), suggesting that relatively larger sampling sizes are required to obtain adequately narrow CIs for the models with lower VF1-VF2. For instance, for the model SAM0-UNICON, which has the largest parameter b (16.17), when the sampling size is 30 and the CC corresponds to the 5th percentile, the first and second EOF modes (Figs. 7a,b) are quite different from the reference modes (Figs. 7c,d). By contrast, for ENSO, the model INM-CM5-0, which has the largest parameter b (9.51), can capture the first EOF mode that is similar to the reference ENSO pattern (Fig. 8). However, the first mode shows eastward-shrunken and weaker anomalies in the tropics and some extra trivial anomalies in the subtropics in Fig. 8a, when compared with the reference mode shown in Fig. 8b.
The (a) EOF1 and (b) EOF2 patterns corresponding to the 5th percentile of 30 years’ CC of NAO in SAM0-UNICON which has the largest parameter b for the NAO. The CCs when compared with the reference EOF1 [above (a)] and EOF2 [above (b)] patterns and the variance fractions of EOF modes are given. The reference (c) EOF1 and (d) EOF2 patterns, derived from the entire period of SAM0-UNICON, are also shown.
Citation: Journal of Climate 37, 7; 10.1175/JCLI-D-23-0165.1
As in Fig. 7, but for the ENSO pattern in INM-CM5-0, with the largest parameter b for ENSO.
Citation: Journal of Climate 37, 7; 10.1175/JCLI-D-23-0165.1
5. The minimum sampling size for obtaining reliable EOF patterns
In this study, we use 5%–95% CI to discuss the uncertainty in EOF patterns. The varying of 5%–95% CI is determined mostly by the 5th percentile of CC, because of much stronger variability in the 5th percentile of CC than the 95th percentile (Fig. 2). For the NAO, the 95th percentile of CC ranges from 0.98 to 0.99 among all the 50 models, very close to 1.0, but the 5th percentile covers a much larger range from 0.65 to 0.94, when the sampling size is 30. Similarly, For ENSO, the 95th percentile of CC ranges from 0.97 to 0.99 among all the 50 models, very close to 1.0, but the 5th percentile covers a much larger range from 0.87 to 0.98, when the sampling size is 30. Therefore, the CI equal to 0.1 can be considered as the 5th percentile of CC close to 0.9, which means the possibility of obtaining the acceptable patterns is approximately 95%, in comparison with the reference pattern. For example, the ENSO pattern of the 5th percentile with the sampling size of 30 in observations shows a CC of 0.96 and is similar to the reference pattern (Fig. 1f).
One of the applications of the fitting function is the estimate of the sampling size that assures reliable EOF patterns. An adequately narrow CI indicates that the CCs do not depend significantly on the sampling. As mentioned above, a CI equal to 0.1 is considered to be a reliable EOF pattern, and thus the minimum sampling size x can be estimated by solving the equation a/(x − b) = 0.1. The minimum sampling size shows a great intermodel spread, as denoted by the numbers of Fig. 5. For the NAO, the minimum sampling size varies from 19 (NorCPM1) to 81 (CMCC-ESM2). By contrast, fewer samples are needed to obtain a reliable pattern for ENSO, with the minimum sampling size being about 30 for only two models (INM-CM4-8, INM-CM5-0).
Furthermore, the relative importance of parameters a and b can also be estimated. ∂x/∂a = 10, ∂x/∂b = 1, deduced by the fitting function. The parameter a varies from 1.25 to 7.19 for the NAO and from 0.15 to 2.24 for ENSO among 50 models, generally equivalent to the change in minimum sampling size being 62 and 24 for the NAO and ENSO, respectively. By contrast, the parameter b varies from 4.22 to 16.17 for the NAO and from 1.48 to 9.51 for ENSO. Hence, it can be concluded that the parameter a plays a dominant role in determining the CIs and minimum sampling sizes in individual models.
6. Conclusions and discussion
We investigate how the uncertainty of EOF modes, which is represented by the 5%–95% CI of CC, changes with sampling size in observations and 50 CMIP6 piControl experiments, taking the NAO and ENSO as examples. The relationship between the CI and sampling size can be quantified by the concise function y = a/(x − b) both in observations and models. While the parameter b modulates the sampling size in the fitting function, the parameter a scales the sampling size and thus plays a more important role. The parameters a and b show a large intermodel spread for both the NAO and ENSO. The parameter a is highly related to the difference between the variance fractions explained by the first and the second EOF modes, approximately in the form of a power function.
The minimum sampling size that guarantees the reliability of EOF modes is closely related to the CI, and thus can be estimated by the fitting function. The minimum sampling size varies from 19 to 81 for the NAO and from 6 to 30 for ENSO in models. Larger sampling sizes are required for the EOF1 modes when the EOF1 and EOF2 explain similar variance fractions. Similarly, if one performs EOF analyses over extensively large domains such as hemisphere (e.g., Barnston and Livezey 1987; Ambaum et al. 2001) or globe (Schmidt et al. 2008), the first EOF modes would explain much smaller variances, which usually decreases the differences in explained variances between the first and second modes, and thus more samplings would be necessary. Aside from this, EOF analyses over extensively large domains may result in patterns with multiple action centers, and some centers might be weakly correlated (Huth and Beranová 2021).
One example for the similar variance fractions explained by the EOF1 and EOF2 is the Madden–Julian oscillation (MJO). The eigenvalues for the first two modes are very close, and the combination of them are used as EOF representations of the MJO (Hendon et al. 1999; Kessler 2001). It should be mentioned that the present method cannot be extended to this kind of EOF representations. How to determine the uncertainty of EOF representations when the variance fractions explained by the EOF1 and EOF2 are equivalent? This question remains an open one and should be investigated in the future. In addition, the results in this study are based on EOF analysis, and they may not be extended to the rotated EOF analysis, which is a postprocess for the EOF modes and can simplify the EOF eigenvectors with close eigenvalues (Richman 1986; Jolliffe 1989).
In this study, we select 5%–95% CI as a threshold. This threshold would be appropriate for the predominant modes, such as the NAO and ENSO examined in this study. For the dominant modes that can be identified as leading modes but are with close adjacent variance fractions such as smaller than 10%, the 5%–95% threshold may be too strict. For these modes, somewhat lower thresholds such as 10%–90% CI would be practical. On the other hand, our study provides probability distributions of EOF patterns based on a more comprehensive coefficient to assure the similarity of structure and magnitude between the sampling EOF pattern and the reference EOF pattern. This differs from the criterion of North et al. (1982) that is based on one-order deviation and simplified to the significant separation of adjacent eigenvalues, for the sampling EOF patterns. This difference between the present study and the North test may yield the different judgement for sampling EOF. For instance, in some cases, the sampling EOF1 patterns are different from the reference pattern but with sampling variance fractions separating significantly, and there are also some cases that show sampling EOF1s matching the reference pattern well but with close sampling VF1s and VF2s (not shown).
In the present study, the samplings are randomly selected, and thus theoretically there is no autocorrelation. However, in real analyses of continuous data, there might be nonnegligible autocorrelations, particularly when one examines slowly varying modes of climate variability, such as the Pacific decadal oscillation or Atlantic multidecadal oscillation. Therefore, the effective sampling size may be appreciably smaller than the actual sampling size, and this issue should be of concern when estimating the minimum sampling size. In addition, in reality there would be nonnegligible effects of climate change, and thus these effects might be removed before analyses. Another caveat is that the modes of climate variability might be more complicated than the linear modes obtained through EOF analyses. For instance, ENSO comprises asymmetric phases of El Niño and La Niña (An and Jin 2004) and distinct spatial patterns of the eastern Pacific type and central Pacific type (Ren and Jin 2011).
Note that our study mainly focuses on the reliability, rather than the validity, of EOF modes. Using enough samples to obtain reliable EOF modes cannot assure the validity of EOF modes. In fact, EOF analysis will merge all sources of variabilities and might obtain artificial patterns. Therefore, one should treat the EOF modes carefully and assure that they are physically meaningful rather than artificial patterns (Buell 1979). Furthermore, Huth and Beranová (2021) have pointed out that at least the robust spatial autocorrelations of EOF patterns and the insensitivity to spatial and temporal subsampling should be considered to ensure the validity of EOF patterns.
The present approach may have an implication for estimating the sampling sizes required for the snapshot EOF analysis. The snapshot method has been widely used to reveal dominant modes in time-dependent dynamical systems, such as the climate system forced with external forcings including anthropogenic emissions (e.g., Chu et al. 2014; Lee et al. 2014; Li et al. 2020; Sun et al. 2022). The physical basis for this method is that the climate system can be considered stationary over short time spans, but the short time spans seriously limit the sampling sizes. Currently, a way to produce adequate sampling sizes for the snapshot EOF method is performing EOF analysis on many ensemble members with different initial conditions, and collecting a fixed short time span from entire integrations of individual ensemble members (Haszpra et al. 2020a,b), which makes the snapshot EOF method somewhat expensive. Therefore, prior knowledge of minimum sampling sizes for EOF analysis in models would be greatly helpful for obtaining reliable snapshot EOF results using the minimum possible resources.
Acknowledgments.
The authors highly appreciate the anonymous reviewers for their constructive comments, which helped to improve the paper. The authors also thank Lijing Cheng for helpful discussion. This research was supported by the National Natural Science Foundation of China (Grant 42130504).
Data availability statement.
The data used in this study—ERSSTv5 (https://psl.noaa.gov/data/gridded/data.noaa.ersst.v5.html), 20CRv3 (https://psl.noaa.gov/data/gridded/data.20thC_ReanV3.html), and CMIP6 datasets (https://esgf-node.llnl.gov/search/cmip6/)—are available online for download. The function (curve_fit in scipy.optimize) to derive the optimal estimate of parameters of fitting function can be found online (https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html).
REFERENCES
Ambaum, M. H. P., B. J. Hoskins, and D. B. Stephenson, 2001: Arctic Oscillation or North Atlantic Oscillation? J. Climate, 14, 3495–3507, https://doi.org/10.1175/1520-0442(2001)014<3495:AOONAO>2.0.CO;2.
An, S.-I., and F.-F. Jin, 2004: Nonlinearity and asymmetry of ENSO. J. Climate, 17, 2399–2412, https://doi.org/10.1175/1520-0442(2004)017<2399:NAAOE>2.0.CO;2.
Barnston, A. G., and R. E. Livezey, 1987: Classification, seasonality and persistence of low-frequency atmospheric circulation patterns. Mon. Wea. Rev., 115, 1083–1126, https://doi.org/10.1175/1520-0493(1987)115<1083:CSAPOL>2.0.CO;2.
Buell, C. E., 1979: On the physical interpretation of empirical orthogonal functions. Preprints, Sixth Conf. on Probability and Statistics in Atmospheric Sciences of the American Meteorological Society, Banff, Alberta, Canada, Amer. Meteor. Soc., 112–117.
Chu, J.-E., K.-J. Ha, J.-Y. Lee, B. Wang, B.-H. Kim, and C. E. Chung, 2014: Future change of the Indian Ocean basin-wide and dipole modes in the CMIP5. Climate Dyn., 43, 535–551, https://doi.org/10.1007/s00382-013-2002-7.
Coburn, J., and S. C. Pryor, 2021: Differential credibility of climate modes in CMIP6. J. Climate, 34, 8145–8164, https://doi.org/10.1175/JCLI-D-21-0359.1.
Compo, G. P., and Coauthors, 2011: The Twentieth Century Reanalysis Project. Quart. J. Roy. Meteor. Soc., 137 (654), 1–28, https://doi.org/10.1002/qj.776.
Deser, C., A. Phillips, V. Bourdette, and H. Teng, 2012: Uncertainty in climate change projections: The role of internal variability. Climate Dyn., 38, 527–546, https://doi.org/10.1007/s00382-010-0977-x.
Eyring, V., S. Bony, G. A. Meehl, C. A. Senior, B. Stevens, R. J. Stouffer, and K. E. Taylor, 2016: Overview of the Coupled Model Intercomparison Project phase 6 (CMIP6) experimental design and organization. Geosci. Model Dev., 9, 1937–1958, https://doi.org/10.5194/gmd-9-1937-2016.
Giese, B. S., H. F. Seidel, G. P. Compo, and P. D. Sardeshmukh, 2016: An ensemble of ocean reanalyses for 1815–2013 with sparse observational input. J. Geophys. Res. Oceans, 121, 6891–6910, https://doi.org/10.1002/2016JC012079.
Guirguis, K. J., and R. Avissar, 2008: An analysis of precipitation variability, persistence, and observational data uncertainty in the western United States. J. Hydrometeor., 9, 843–865, https://doi.org/10.1175/2008JHM972.1.
Hannachi, A., I. T. Jolliffe, and D. B. Stephenson, 2007: Empirical orthogonal functions and related techniques in atmospheric science: A review. Int. J. Climatol., 27, 1119–1152, https://doi.org/10.1002/joc.1499.
Haszpra, T., D. Topál, and M. Herein, 2020a: On the time evolution of the Arctic Oscillation and related wintertime phenomena under different forcing scenarios in an ensemble approach. J. Climate, 33, 3107–3124, https://doi.org/10.1175/JCLI-D-19-0004.1.
Haszpra, T., M. Herein, and T. Bódai, 2020b: Investigating ENSO and its teleconnections under climate change in an ensemble view—A new perspective. Earth Syst. Dyn., 11, 267–280, https://doi.org/10.5194/esd-11-267-2020.
Hendon, H. H., C. Zhang, and J. D. Glick, 1999: Interannual variation of the Madden–Julian oscillation during austral summer. J. Climate, 12, 2538–2550, https://doi.org/10.1175/1520-0442(1999)012<2538:IVOTMJ>2.0.CO;2.
Huang, B., and Coauthors, 2017: Extended Reconstructed Sea Surface Temperature, version 5 (ERSSTv5): Upgrades, validations, and intercomparisons. J. Climate, 30, 8179–8205, https://doi.org/10.1175/JCLI-D-16-0836.1.
Hurrell, J. W., and C. Deser, 2010: North Atlantic climate variability: The role of the North Atlantic Oscillation. J. Mar. Syst., 79, 231–244, https://doi.org/10.1016/j.jmarsys.2009.11.002.
Huth, R., and R. Beranová, 2021: How to recognize a true mode of atmospheric circulation variability. Earth Space Sci., 8, e2020EA001275, https://doi.org/10.1029/2020EA001275.
Ibebuchi, C. C., and M. B. Richman, 2023: Circulation typing with fuzzy rotated T-mode principal component analysis: Methodological considerations. Theor. Appl. Climatol., 153, 495–523, https://doi.org/10.1007/s00704-023-04474-5.
Jolliffe, I. T., 1989: Rotation of ill-defined principal components. J. Roy. Stat. Soc., 38C, 139–147.
Karamizadeh, S., S. M. Abdullah, A. A. Manaf, M. Zamani, and A. Hooman, 2013: An overview of principal component analysis. J. Signal Inf. Process., 4, 173, https://doi.org/10.4236/jsip.2013.43B031.
Kessler, W. S., 2001: EOF representations of the Madden–Julian oscillation and its connection with ENSO. J. Climate, 14, 3055–3061, https://doi.org/10.1175/1520-0442(2001)014<3055:EROTMJ>2.0.CO;2.
Lee, J., K. R. Sperber, P. J. Gleckler, C. J. W. Bonfils, and K. E. Taylor, 2019: Quantifying the agreement between observed and simulated extratropical modes of interannual variability. Climate Dyn., 52, 4057–4089, https://doi.org/10.1007/s00382-018-4355-4.
Lee, J.-Y., B. Wang, K.-H. Seo, J.-S. Kug, Y.-S. Choi, Y. Kosaka, and K.-J. Ha, 2014: Future change of Northern Hemisphere summer tropical–extratropical teleconnection in CMIP5 models. J. Climate, 27, 3643–3664, https://doi.org/10.1175/JCLI-D-13-00261.1.
Lee, Y.-Y., and R. X. Black, 2013: Boreal winter low-frequency variability in CMIP5 models. J. Geophys. Res. Atmos., 118, 6891–6904, https://doi.org/10.1002/jgrd.50493.
Li, S., W. Cai, and L. Wu, 2020: Attenuated interannual variability of austral winter Antarctic sea ice over recent decades. Geophys. Res. Lett., 47, e2020GL090590, https://doi.org/10.1029/2020GL090590.
Li, S., W. Cai, and L. Wu, 2021: Weakened Antarctic dipole under global warming in CMIP6 models. Geophys. Res. Lett., 48, e2021GL094863, https://doi.org/10.1029/2021GL094863.
Lloyd, S., M. Mohseni, and P. Rebentrost, 2014: Quantum principal component analysis. Nat. Phys., 10, 631–633, https://doi.org/10.1038/nphys3029.
Lorenzo-Seva, U., and J. M. F. ten Berge, 2006: Tucker’s congruence coefficient as a meaningful index of factor similarity. Methodology, 2, 57–64, https://doi.org/10.1027/1614-2241.2.2.57.
Maher, N., D. Matei, S. Milinski, and J. Marotzke, 2018: ENSO change in climate projections: Forced response or internal variability? Geophys. Res. Lett., 45, 11 390–11 398, https://doi.org/10.1029/2018GL079764.
McKenna, C. M., and A. C. Maycock, 2021: Sources of uncertainty in multimodel large ensemble projections of the winter North Atlantic Oscillation. Geophys. Res. Lett., 48, e2021GL093258, https://doi.org/10.1029/2021GL093258.
Milinski, S., N. Maher, and D. Olonscheck, 2020: How large does a large ensemble need to be? Earth Syst. Dyn., 11, 885–901, https://doi.org/10.5194/esd-11-885-2020.
Newman, M., and Coauthors, 2016: The Pacific decadal oscillation, revisited. J. Climate, 29, 4399–4427, https://doi.org/10.1175/JCLI-D-15-0508.1.
North, G. R., and Q. Wu, 2001: Detecting climate signals using space–time EOFs. J. Climate, 14, 1839–1863, https://doi.org/10.1175/1520-0442(2001)014<1839:DCSUST>2.0.CO;2.
North, G. R., T. L. Bell, R. F. Cahalan, and F. J. Moeng, 1982: Sampling errors in the estimation of empirical orthogonal functions. Mon. Wea. Rev., 110, 699–706, https://doi.org/10.1175/1520-0493(1982)110<0699:SEITEO>2.0.CO;2.
Peings, Y., Z. M. Labe, and G. Magnusdottir, 2021: Are 100 ensemble members enough to capture the remote atmospheric response to +2°C Arctic sea ice loss? J. Climate, 34, 3751–3769, https://doi.org/10.1175/JCLI-D-20-0613.1.
Quadrelli, R., C. S. Bretherton, and J. M. Wallace, 2005: On sampling errors in empirical orthogonal functions. J. Climate, 18, 3704–3710, https://doi.org/10.1175/JCLI3500.1.
Ren, H.-L., and F.-F. Jin, 2011: Niño indices for two types of ENSO. Geophys. Res. Lett., 38, L04704, https://doi.org/10.1029/2010GL046031.
Richman, M. B., 1986: Rotation of principal components. Int. J. Climatol., 6, 293–335, https://doi.org/10.1002/joc.3370060305.
Schmidt, R., S. Petrovic, A. Güntner, F. Barthelmes, J. Wünsch, and J. Kusche, 2008: Periodic components of water storage changes from GRACE and global hydrology models. J. Geophys. Res., 113, B08419, https://doi.org/10.1029/2007JB005363.
Slivinski, L. C., and Coauthors, 2019: Towards a more reliable historical reanalysis: Improvements for version 3 of the Twentieth Century Reanalysis System. Quart. J. Roy. Meteor. Soc., 145, 2876–2908, https://doi.org/10.1002/qj.3598.
Stoner, A. M. K., K. Hayhoe, and D. J. Wuebbles, 2009: Assessing general circulation model simulations of atmospheric teleconnection patterns. J. Climate, 22, 4348–4372, https://doi.org/10.1175/2009JCLI2577.1.
Sun, W., B. Wang, J. Liu, and Y. Dai, 2022: Recent changes of Pacific decadal variability shaped by greenhouse forcing and internal variability. J. Geophys. Res. Atmos., 127, e2021JD035812, https://doi.org/10.1029/2021JD035812.
Takahashi, K., A. Montecinos, K. Goubanova, and B. Dewitte, 2011: ENSO regimes: Reinterpreting the canonical and Modoki El Niño. Geophys. Res. Lett., 38, L10704, https://doi.org/10.1029/2011GL047364.
Timmermann, A., and Coauthors, 2018: El Niño–Southern Oscillation complexity. Nature, 559, 535–545, https://doi.org/10.1038/s41586-018-0252-6.
von Storch, H., and F. W. Zwiers, 1999: Statistical Analysis in Climate Research. Cambridge University Press, 484 pp.
Wang, X., J. Li, C. Sun, and T. Liu, 2017: NAO and its relationship with the Northern Hemisphere mean surface temperature in CMIP5 simulations. J. Geophys. Res. Atmos., 122, 4202–4227, https://doi.org/10.1002/2016JD025979.
Zhang, Y., and Y. Wang, 2022: Forecasting crude oil futures market returns: A principal component analysis combination approach. Int. J. Forecasting, 39, 659–673, https://doi.org/10.1016/j.ijforecast.2022.01.010.
