Using Conditional Nonlinear Optimal Perturbation to Generate Initial Perturbations in ENSO Ensemble Forecasts

Qian Zhou aKey Laboratory of Marine Hazards Forecasting, National Marine Environmental Forecasting Center, Ministry of Natural Resources, Beijing, China

Search for other papers by Qian Zhou in
Current site
Google Scholar
PubMed
Close
,
Lei Chen bShanghai Central Meteorological Observatory, Shanghai, China

Search for other papers by Lei Chen in
Current site
Google Scholar
PubMed
Close
,
Wansuo Duan cLASG, Institute of Atmospheric Physics, Chinese Academy of Sciences, Beijing, China

Search for other papers by Wansuo Duan in
Current site
Google Scholar
PubMed
Close
,
Xu Wang dMinistry of Education Key Laboratory for Earth System Modeling, Department for Earth System Science, Tsinghua University, Beijing, China

Search for other papers by Xu Wang in
Current site
Google Scholar
PubMed
Close
,
Ziqing Zu aKey Laboratory of Marine Hazards Forecasting, National Marine Environmental Forecasting Center, Ministry of Natural Resources, Beijing, China

Search for other papers by Ziqing Zu in
Current site
Google Scholar
PubMed
Close
,
Xiang Li aKey Laboratory of Marine Hazards Forecasting, National Marine Environmental Forecasting Center, Ministry of Natural Resources, Beijing, China

Search for other papers by Xiang Li in
Current site
Google Scholar
PubMed
Close
,
Shouwen Zhang eNational Marine Environmental Forecasting Center, Ministry of Natural Resources, Beijing, China

Search for other papers by Shouwen Zhang in
Current site
Google Scholar
PubMed
Close
, and
Yunfei Zhang aKey Laboratory of Marine Hazards Forecasting, National Marine Environmental Forecasting Center, Ministry of Natural Resources, Beijing, China

Search for other papers by Yunfei Zhang in
Current site
Google Scholar
PubMed
Close
Open access

We are aware of a technical issue preventing figures and tables from showing in some newly published articles in the full-text HTML view.
While we are resolving the problem, please use the online PDF version of these articles to view figures and tables.

Abstract

Using the latest operational version of the ENSO forecast system from the National Marine Environmental Forecasting Center (NMEFC) of China, ensemble forecasting experiments are performed for El Niño–Southern Oscillation (ENSO) events that occurred from 1997 to 2017 by generating initial perturbations of the conditional nonlinear optimal perturbation (CNOP) and climatically relevant singular vector (CSV) structures. It is shown that when the initial perturbation of the leading CSV structure in the ensemble forecast of the CSVs scheme is replaced by those of the CNOP structure, the resulted ensemble ENSO forecasts of the CNOP+CSVs scheme tend to possess a larger spread than the forecasts obtained with the CSVs scheme alone, leading to a better match between the root-mean-square error and the ensemble spread, a more reasonable Talagrand diagram, and an improved Brier skill score (BSS). All these results indicate that the ensemble forecasts generated by the CNOP+CSVs scheme can improve both the accuracy of ENSO forecasting and the reliability of the ensemble forecasting system. Therefore, ENSO ensemble forecasting should consider the effect of nonlinearity on the ensemble initial perturbations to achieve a much higher skill. It is expected that fully nonlinear ensemble initial perturbations can be sufficiently yielded to produce ensemble forecasts for ENSO, finally improving the ENSO forecast skill to the greatest possible extent. The CNOP will be a useful method to yield fully nonlinear optimal initial perturbations for ensemble forecasting.

© 2021 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Wansuo Duan, duanws@lasg.iap.ac.cn

Abstract

Using the latest operational version of the ENSO forecast system from the National Marine Environmental Forecasting Center (NMEFC) of China, ensemble forecasting experiments are performed for El Niño–Southern Oscillation (ENSO) events that occurred from 1997 to 2017 by generating initial perturbations of the conditional nonlinear optimal perturbation (CNOP) and climatically relevant singular vector (CSV) structures. It is shown that when the initial perturbation of the leading CSV structure in the ensemble forecast of the CSVs scheme is replaced by those of the CNOP structure, the resulted ensemble ENSO forecasts of the CNOP+CSVs scheme tend to possess a larger spread than the forecasts obtained with the CSVs scheme alone, leading to a better match between the root-mean-square error and the ensemble spread, a more reasonable Talagrand diagram, and an improved Brier skill score (BSS). All these results indicate that the ensemble forecasts generated by the CNOP+CSVs scheme can improve both the accuracy of ENSO forecasting and the reliability of the ensemble forecasting system. Therefore, ENSO ensemble forecasting should consider the effect of nonlinearity on the ensemble initial perturbations to achieve a much higher skill. It is expected that fully nonlinear ensemble initial perturbations can be sufficiently yielded to produce ensemble forecasts for ENSO, finally improving the ENSO forecast skill to the greatest possible extent. The CNOP will be a useful method to yield fully nonlinear optimal initial perturbations for ensemble forecasting.

© 2021 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Wansuo Duan, duanws@lasg.iap.ac.cn

1. Introduction

As one of the strongest interannual variabilities in the tropical Pacific Ocean, El Niño–Southern Oscillation (ENSO) can bring about extreme weather and climate events all over the world (Wang et al. 2000; Diaz et al. 2001; Alexander et al. 2002; Henderson et al. 2018). It is therefore necessary to predict ENSO and to improve the skill of these predictions (Cane et al. 1986; Latif et al. 1998; Chen and Cane 2008; Zhu et al. 2013; Zhang et al. 2020). ENSO prediction results are often contaminated by prediction errors caused by uncertainties in the initial conditions and model parameters (Tang et al. 2018), atmospheric noise (Kleeman and Moore 1997; Moore and Kleeman 1998) and other high-frequency variabilities, such as Madden–Julian oscillations and westerly wind burst events (Vecchi and Harrison 2003; Gebbie et al. 2007).

Ensemble forecasting is a useful forecasting method that not only provides a deterministic prediction result by taking the ensemble mean but also obtains an estimation of the uncertainties present in this prediction result by evaluating the ensemble spread (Leith 1974). However, due to the insufficient reliability of ensemble forecast systems, the uncertainties present in the prediction results are often aggressively underestimated (Buizza et al. 2005; Palmer 2019). The reliability of an ensemble forecast system depends on the methods used to yield the ensemble members. For an ensemble forecast dealing with initial uncertainties, the ensemble members are usually generated by superimposing initial perturbations on a control forecast. Thus, determining how to yield ensemble initial perturbations becomes a key question when obtaining a reliable ensemble forecast system.

Some methods, such as random initial perturbations (Leith 1974), lagged average forecasting (Hoffman and Kalnay 1983), singular vector (SV) methods (Lorenz 1965), and breeding vector (BV) methods (Toth and Kalnay 1993), have been proposed to yield ensemble initial perturbations, and these methods have achieved successes in operational forecasting to a certain degree. In particular, the SV method has been applied in European Centre for Medium-Range Weather Forecasts (ECMWF) for numerical weather forecasting and has obtained great success (Molteni et al. 1996; Buizza et al. 2008). SVs stand out because they have clear dynamical meaning and can capture the fast-growing initial perturbations of control forecasts. The SV method has also been widely applied to studies of climatic ENSO forecasting and predictability. Some studies have calculated SVs by using intermediate or hybrid (simple atmosphere and OGCM) coupled ENSO models (Xue et al. 1997; Fan et al. 2000) to explore ENSO predictability from the perspective of error growth (Kug et al. 2010); in these studies, the SVs were calculated by deriving the adjoint of the relevant models. However, for very complex models, such as coupled GCMs, it is very difficult to derive the adjoint. Fortunately, Kleeman et al. (2003) proposed an approach to calculating SVs in the presence of weather noise; in this approach, SVs are called climatically relevant singular vectors (CSVs). CSVs have been applied to ENSO ensemble forecasts to improve the skill of ENSO predictions by using a fully coupled GCM (Tang et al. 2006).

ENSO phenomena have been verified to include nonlinearity; correspondingly, ENSO forecasting systems should also be constructed by fully nonlinear models. Beyond doubt, SVs, as an approach to yielding ensemble initial perturbations, certainly have limitations in ENSO forecasting due to their linear approximations. To overcome the linearity of SVs, (Mu et al. 2003) focused on the leading SV (LSV) with the fastest growth rate in the linearized model and proposed the conditional nonlinear optimal perturbation (CNOP) approach; this approach is a natural extension of the LSV in the nonlinear regime and represents the largest-growing initial perturbation in the nonlinear model. The CNOP approach has been used to explore the largest-growing initial errors and the optimal precursors of weather or climate events; the CNOP method has also been used to determine the sensitive areas for targeted observations associated with the forecasting of weather and climate events by using an intermediate ENSO model (Duan et al. 2004; Mu et al. 2007a; Yu et al. 2012; Duan et al. 2013; Tao et al. 2017; Duan et al. 2018). For ensemble forecasting, the CNOP method has also been used to yield ensemble initial perturbations by replacing the LSV with the CNOP using a quasigeostrophic model (Mu and Jiang 2008; Jiang and Mu 2009). The results showed that this new ensemble forecast method improved the skill of forecasts compared with the ensemble forecast method of SVs. Inspired by the good performance of the CNOP in studies of ensemble forecasts using a simple model, it is natural to ask how to calculate the CNOP initial perturbations in an operational ENSO forecast system. Can this CNOP initial perturbation improve the skill of ensembles to forecast ENSO when the LSV in the SVs ensemble forecast method is replaced by the CNOP? These questions will be addressed in the present study.

In the following sections, the ENSO forecast system and ensemble forecasting method adopted here are introduced in section 2. Section 3 gives a brief introduction of the CSV and CNOP of the CESM. All the ensemble forecast results are given in section 4. The summary and discussion follow in section 5.

2. The ENSO forecast system and ensemble forecasting method

In this article, we adopt an ENSO operational forecast system from the National Marine Environmental Forecasting Center (NMEFC) of China to construct an ENSO ensemble forecast system through perturbations of the CNOP and SVs. Next, the ENSO forecast system and the perturbation methods are introduced.

a. The ENSO forecast system

The ENSO forecast system adopted here is an operational forecast system of the NMEFC. Its core model was established based on the Community Earth System Model (CESM) developed by NCAR/UCAR. This Earth system model consists of atmospheric, oceanic, land and ice components, with a flux coupler linking these components. Both biogeochemical process and global carbon cycle are included in this model. The horizontal resolution of the atmospheric component, the Community Atmosphere Model (CAM), is 0.9° latitude × 1.25° longitude with 26 vertical levels. The oceanic component of CESM, the Parallel Ocean Program version 2 (POP2), has a resolution of 1° that is reduced to 1/3° near the equator. The CESM model produces acceptable ENSO simulations and simulations of the decadal variability in ENSO (Deser et al. 2012; Zheng et al. 2018).

The NMEFC ENSO forecast system assimilates sea temperature in depths ranging from 15 to 400 m from GODAS data (Derber and Rosati 1989) by using the nudging method. The hindcasting results generated by the forecast system for the ENSO events that occurred during 1981–2011 suggest that the system has an acceptable skill when forecasting ENSO events. Currently, this ENSO forecast system is operationally running in the NMEFC (Zhang et al. 2018). In the present study, we will use this system to conduct ensemble forecast experiments to predict ENSO.

b. The methods used to yield ensemble initial perturbations

1) The CSVs

Although the traditional SV method is often used to generate initial perturbations in ensemble forecasting, the necessity of calculating adjoints limits the wide application of this method in operational climate event forecasting. As mentioned in the introduction, Kleeman et al. (2003) proposed an efficient technique to calculate singular vectors. They used the EOF modes derived from a model’s historical integration to obtain CSVs in the presence of weather noise. Specifically, a dynamical system Ψ(t) can be written as follows:
Ψ(t)=F[Ψ(t)],
where Ψ is a state vector, F is a nonlinear operator, and t > t′ is the time. For a “small” perturbation δΨ(t), which does not grow considerably during the time interval, the system expressed in Eq. (1) can be linearized as follows:
δΨ(t)=R(t,t)δΨ(t),
where R is the tangent linear operator. The singular vectors for the time interval (t′, t) are the eigenvectors of the operator R*(t,t)R(t,t), where R*(t,t) is the transposed matrix of R(t, t′).
Let Emn be the EOF modes derived from the model’s historical integration, where m is the phase-space dimension and n denotes the number of EOF modes we obtained. Since the total variances of the leading 5 EOFs in the tropical Pacific in this model can reach to 90.37%, n = 5 is enough to construct a reduced set of small perturbations. In this situation, the Eq. (2) can be rewritten as follows:
δΨmn=RmmEmn+residual,
where the residual term is often fairly small in amplitude and predominantly consists of small-scale noise. Accordingly, the eigenvectors of the operator Rmm*Rmm represent the CSVs we need.

For ENSO, the CSVs should be calculated based on a neutral year state. From the years of interest 1997–2017, 2013 was a normal year, and its corresponding state was chosen to calculate the CSVs. The CSVs are calculated by following four steps.

  1. An ensemble of 30 predictions with lead times of 8 months are constructed by perturbing a set of 30 white-noise pattern initial errors with a typical amplitude of 0.01°C in a normal year. The mean is denoted as Ψ¯0(t).

  2. Each of the five leading normalized correlation-EOF modes EOFmi (i = 1, 2, …, 5) are added to the initial state of the normal year, and the initial errors resulting from the white-noise pattern mentioned in the last step are superimposed to produce new predictions with 30 random initial perturbations. The corresponding mean is denoted by Ψ¯i(t).

  3. With Ψ¯i(t) (i = 1, 2, …, 5) and Ψ¯0(t), δΨm5 can be calculated following δΨm5=[Ψ¯1(t)Ψ¯0(t),,Ψ¯5(t)Ψ¯0(t)]T.

  4. Rmm is then calculated according to Eq. (3), and the singular vectors are obtained.

2) The CNOP

As mentioned in the introduction, the CNOP has been used to generate initial perturbations for ensemble forecasts by replacing the LSV in ensemble forecasts while keeping other CSVs as the ensemble members; this method has achieved success in improving forecast skill because the CNOP involves the effect of nonlinearity and overcomes the limitations of LSV. The application of the CNOP first requires computations. Traditionally, the gradient of the objective function associated with the CNOP is needed in its optimization. Unfortunately, most coupled CGCMs do not possess corresponding adjoint models; furthermore, coding the adjoint model of a complex model is a massive and time-consuming task. To overcome these limitations associated with traditional optimization algorithms, some intelligent optimization algorithms, such as particle swarm optimization and generic algorithms, have been applied to calculate the CNOP (Sun and Mu 2013; Mu et al. 2015; Zhang et al. 2017). Particularly, based on the ensemble projection approach proposed by Wang and Tan (2010), Chen et al. (2015), Mu et al. (2019), and Xu et al. (2021) combined this method with the SPG2 solver (Birgin et al. 2000) and developed a singular vector decomposition (SVD)-based ensemble projection algorithm. This new method can be successfully applied to the Zebiak–Cane model and CESM model for the calculation of the CNOP associated with ENSO predictions. In the present study, we adopt such an approach to calculate the CNOP of the CESM.

For the dynamical system Ψ(t) mentioned above, let x0 be an initial perturbation superimposed on the initial value X0 of a reference state. For the selected norms δ and , the CNOP (denoted by x0δ) is the solution to the following optimization problem:
J(x0δ)=maxx0δδF(X0+x0,t,t)F(X0,t,t)2,
where ||x0||δδ is the constraint of the initial perturbation amplitude and δ is a prescribed positive number that defines the amplitude of the initial perturbation x0. The norm measures the evolution of the initial perturbations. Optimization algorithms are often used to solve minimization problems. We then rewrite J1(x0) = − J(x0) and solve the minimization problem of J1(x0) to obtain the CNOP.

The projection of a continuous system into a discrete numerical model can be expressed as x(t)=i=1NσiuiviT, where N is the degree of freedom of a numerical model, σi is a singular value arranged from highest to lowest, ui is the spatial mode corresponding to σi, and vi is the time series of ui. A forced and dissipative dynamical system will tend toward a low-dimensional attractor after a long evolution (Osborne and Pastorello 1993; Foias 1997). The spatial modes ui can be chosen so that |σi| monotonously decreases sufficiently quickly as i increases. If k (kN) former main spatial modes are used as the bases to construct the approximate state space of the whole system, the original N-dimensional system can be truncated to a k-dimensional approximate system.

SVD statistically provides a standard method that reduces the dimension of the system effectively by determining the number of spatial modes. If k spatial modes are chosen and combined linearly to approximate the state vector of the discrete system, then x0=i=1kaiui, and the optimization problem associated with computing the CNOP is then transformed into the following equation:
J1(aδ)=minx0δδF(X0+i=1kaiui,t,t)F(X0,t,t)2,
where ai is the weighted coefficient of the chosen mode ui. Equation (5) searches for the optimal combination of weighted coefficients of the chosen bases. The validity of the SVD-based EP algorithm when computing the CNOP has been verified in a ZC model that has an adjoint model. Chen et al. (2015) combined the SVD-based EP method with the traditional SPG2 algorithm to calculate the CNOP and explore the optimal precursor conditions of ENSO events. The results show that the CNOP obtained by the SVD-based EP algorithm can effectively approximate the CNOP calculated by the adjoint method, and the SVD-based EP algorithm can also retain the general spatial characteristics of the latter. Wen et al. (2015) combined this method with four intelligent algorithms to compute the CNOP in the ZC model. Their results show that not only does the spatial pattern of the CNOP generated by the reduced-dimension method look similar to that generated by the adjoint method, but the variation trend of the cost functions in different calendar months obtained with the new method is almost the same as that obtained with the latter method. These studies demonstrated that, independent of the optimization algorithm chosen, the SVD-based EP algorithm can be used to approximately compute the CNOP. In this study, the CNOP of the operational ENSO forecast system is calculated using this SVD-based EP algorithm.

3. The CSV and CNOP of the CESM

It has been shown that the initial errors associated with conditions in the tropical Pacific Ocean have a significant influence on ENSO predictability (Mu et al. 2007a; Duan and Hu 2016). In the present study, we mainly focus on the influence of errors in the initial sea temperature in the tropical Pacific Ocean on the ENSO forecast skill in the ensemble forecast. The experiments are conducted using the ENSO forecast system of the NMEFC, as was introduced in the last section. The control forecasts are yielded by the ENSO forecast system; hindcasts are obtained for SST anomalies during the years from 1997 to 2017. Due to expensive computing costs, the ensemble forecasts conducted here are only initialized on 1 January in each year. Both the CSVs and CNOP are also obtained on January 1st in each year following the procedures presented in section 2b.

In Fig. 1, the spatial structures of both the leading CSV (for convenience, we also refer to it as the LSV) and the CNOP-type initial perturbations in the tropical Pacific Ocean are presented. The LSV-type perturbation reveals positive SST anomalies in the eastern Pacific Ocean and negative SST anomalies in the western region, while in the subsurface, three sea temperature anomaly centers presenting a positive–negative–positive pattern appear along the tropical Pacific Ocean from west to east. However, for the CNOP-type initial perturbations, positive SST anomalies dominate nearly the whole tropical Pacific Ocean; along the vertical section, the positive anomalies are mainly distributed in the central and eastern Pacific Ocean with larger amplitudes than those provided by the LSV, and negative anomalies dominantly appear in the subsurface in the western tropical Pacific Ocean. The differences between the CNOP-type and LSV-type initial perturbations in spatial structures are understandable since the CNOP-type initial perturbation has the largest error growth in the nonlinear scenario while the LSV initial perturbation is the fastest growing perturbation under the linear assumption. It is clear that the CNOP and LSV have different patterns. We cannot help but wonder whether these differences may result in different performances of the CNOP and LSV in the ensemble forecasting of ENSO. In the following section, we will use CNOP and CSVs to conduct ensemble forecasting of ENSO and compare their ENSO prediction skills.

Fig. 1.
Fig. 1.

The spatial structures of the (a) LSV and (b) CNOP initial errors in the tropical Pacific Ocean for ENSO events. (top) SSTs and (bottom) the equatorial (5°N–5°S) subsurface temperatures (°C) averaged by the meridian 5°N–5°S.

Citation: Weather and Forecasting 36, 6; 10.1175/WAF-D-21-0063.1

First, the control forecast is obtained without any initial perturbation added at the initial field. Then, the ensemble forecast members associated with the CSVs-type initial perturbations are generated by adding and subtracting the leading five CSVs to and from the initial analysis field of the control forecast to, together with the control forecast, yield a total of 11 ensemble members. For convenience, we call such ensembles CSVs-scheme ensembles. For the CNOP-type initial perturbations, we simply use the CNOP to replace the leading CSV in the CSVs-scheme ensemble, referring to such an ensemble as CNOP+CSVs-scheme ensemble. For comparison, all the initial perturbations in these two schemes are scaled to have the same amplitude as the climatological variance in the sea temperatures in the tropical Pacific Ocean.

For both CSVs and CNOP+CSVs schemes, the SST anomalies associated with ENSO events during 1997–2017 are predicted for 8 months starting from January in each year. The ENSO prediction skills are comparably evaluated to explore the validity of applying the CNOP in ENSO ensemble forecasts yielded by the ENSO forecast system of the NMEFC.

4. Results

a. The deterministic skill and reliability of the ensemble forecast

To examine the role of the CNOP in improving the skill of the ensemble when forecasting ENSO events, ensemble forecasting experiments are carried out with both the CSVs scheme and the CNOP+CSVs scheme, and ENSO predictions of SST anomalies during 1997–2017 are obtained; the anomaly correlation, root-mean-square error (RMSE), and ensemble spread are used to measure the deterministic skill of the ensemble forecasting.

Figure 2 shows the ENSO forecast skills, with black lines denoting the skill of the control forecast and the red and blue lines representing the skills of the CSVs scheme and CNOP+CSVs scheme, respectively. For the control forecast, the anomaly correlation coefficients are relatively higher at short lead times and then abruptly drop from above 0.7 to approximately 0.5 at a lead time of 5 months. This dramatic drop in skill may be caused by the spring prediction barrier (SPB) when ENSO predictions are made spanning the spring season (Webster and Yang 1992; Mu et al. 2007b; Lai et al. 2018). When the ensemble forecasting strategy of either the CSVs or CNOP+CSVs scheme is adopted, the ensemble mean forecasts of the SST anomalies in the tropical Pacific Ocean obtain much larger correlation coefficients than those of the control forecast at a lead time of 5 months, suggesting that an ensemble forecasting strategy is an effective way to improve the ENSO forecasting skill in the NMEFC ENSO forecast system. In addition, we notice that the anomaly correlation skill of the CNOP+CSVs scheme is slightly higher than that of the CSVs scheme at lead times from 3 to 6 months, while at other lead times, the correlation skill of the CSVs scheme is slightly higher than that of the CNOP+CSVs scheme.

Fig. 2.
Fig. 2.

Evolution of the Niño-3 index correlation between the CNOP+CSVs scheme (red), CSVs scheme (blue), control run (black), and observations as a function of the lead time.

Citation: Weather and Forecasting 36, 6; 10.1175/WAF-D-21-0063.1

Regarding the RMSEs, the ensemble forecasts with lead times longer than 2 months show much higher skill than the control run; that is, the RMSEs in the ensemble forecasts are smaller than those in the control forecast. However, comparing the ensemble forecasts with the CNOP+CSVs and CSVs schemes, the former has slightly smaller RMSEs than the latter for lead times from 4 to 7 months. From these comparisons, it seems that the CNOP trivially improves the skill of the CSVs-ensemble forecast in ENSO predictions. In fact, whether an ensemble forecast system is acceptable depends on two aspects: the deterministic forecasting skill and accountability. Accountability is the skill associated with estimating uncertainties in the ensemble mean forecast. Therefore, if the ensemble forecasts obtained with the CNOP+CSVs scheme show little improvement in the deterministic forecasting skill as measured by anomaly correlations and RMSEs but a large increase in the forecasting skill of prediction uncertainties measured by the ensemble spread, these forecasts can also be thought of as having much larger improvement in the ensemble forecast skill and are much more accountable than the other forecasts. Therefore, we will investigate the accountability of the ensemble forecasts obtained with the CNOP+CSVs scheme in the following context.

As shown in Fig. 2, the CSVs-scheme ensemble forecasts have a significantly small spread at all lead times, which is actually a common shortcoming of almost all ensemble forecasting methods (Tompkins et al. 2017), while the CNOP+CSVs-scheme ensemble forecasts have a much larger spread. Then, the following question is raised: Is it optimal to apply the CNOP+CSVs-scheme ensemble forecasts to estimate the forecast uncertainties of the ensemble mean forecast?

As discussed in Zhu (2005), the spread is used to measure the distance from the ensemble mean of the forecasts to each ensemble member while the RMSE measures the distance from the ensemble mean of the forecasts to the true value represented by the observation. Therefore, the ensemble members are not likely to include the true values and have a significant improved estimate against the control forecast unless the spread is not less than the RMSE. Buizza et al. (2005) demonstrated that a perfect ensemble forecasting system should have the same ensemble spread magnitude as that of the RMSE; and a large difference between them is an indication of statistical inconsistency. Therefore, the ratio of the RMSE to the ensemble spreads are here investigated, where the ratio λ is expressed as λ = RMSE/spread. According to Buizza et al. (2005), when the ratio λ has a value of approximately 1, it indicates an optimal match between the ensemble spread and the error of the ensemble mean, suggesting that the ensemble spread can represent the prediction errors in ensemble mean forecasts measured by the RMSE. The ratio λ values of the ensemble forecasts obtained with both schemes are presented in Fig. 3. It is shown that the ratio λ of the CSVs scheme has an extremely large value, 33.18, at a lead time of 1 month because of its very small ensemble spread; the value then decreases to a flat curve for lead times from 3 to 8 months. On average, the ratio λ of the CSVs scheme can reach 6.26 over the forecasting time period, even if the ratio of the flat curve from lead times of 3–8 months can still reach 1.98. However, the ratio λ of the CNOP+CSVs scheme is always flat for lead times from 1 to 8 months, and its average is only 1.54, which is significantly smaller than that of the CSVs scheme. To show the spatial structure of the forecast result in the whole tropical Pacific Ocean, other than only the Niño-3 index, the spatial pattern of ratio λ is plotted in Fig. 4. For the ensemble forecasts with the CSVs scheme, the ratios are much larger than 1.0, and most large values locate in the central Pacific Ocean; while for the ensemble forecasts carried out with the CNOP+CSVs scheme, all the ratios are less than 2.0, which is obviously much smaller than that of the CSVs scheme. Therefore, it is clear that the application of the CNOP+CSVs scheme in the ensemble forecast system can significantly improve the reliability of ensemble forecasts when compared with the CSVs scheme.

Fig. 3.
Fig. 3.

Evolution of the ratio λ (which is defined as λ = RMSE/spread) values of the CNOP+CSVs scheme (red) and CSVs scheme (blue) as a function of the lead time.

Citation: Weather and Forecasting 36, 6; 10.1175/WAF-D-21-0063.1

Fig. 4.
Fig. 4.

The spatial pattern of the averaged ratio λ (which is defined as λ = RMSE/spread) values of (a) the CSVs scheme and (b) the CNOP+CSVs scheme.

Citation: Weather and Forecasting 36, 6; 10.1175/WAF-D-21-0063.1

The Talagrand diagram is also used to evaluate the reliability of the ensemble forecasts. In the case of ensemble forecasting with N ensemble forecast members, we first rank these N members in increasing order and define N + 1 intervals. The verifying observations should fall with equal frequency into each of the intervals for a perfectly consistent ensemble forecast system. Therefore, the more reliable an ensemble forecast is, the flatter the Talagrand diagram will be. In the region of concern (i.e., the region between 20°N and 20°S in the tropical Pacific Ocean) in the present study, there are 16 086 grid points associated with the adopted CESM model, and the effective sample size is M = grid point number × N = 16 086 × 11 = 176 946, where N = 11 and corresponds to the number of ensemble ENSO forecasting members in the present study. As mentioned before, N ensemble members have N + 1 intervals; for the kth interval [k ∈ (1, N + 1)], the numbers of observations that fall into the kth interval can be accumulated and marked as Sk, and the probability can be calculated according to βi = Sk/M. Normally, the verifying observations fall into the extreme intervals (i.e., the first and last intervals) much more frequently than they fall into the middle intervals (Talagrand et al. 1997).

The plot of βi, i.e., the Talagrand diagram, is shown in Fig. 5. For the ensemble forecasts obtained with the CSVs scheme, the diagram shows a U-shape, while for those obtained with the CNOP+CSVs scheme, the diagram is more like an L-shape, with much fewer forecasts falling into the first interval. This shows that the percentages of the observations falling into the middle intervals increase and the percentages of those falling into the first and last intervals decrease, indicating that the reliability of the ensemble forecasts obtained with CNOP+CSVs scheme is improved against the reliability of the forecasts obtained with the CSVs scheme when these schemes are applied in the ENSO ensemble forecasts generated by the NMEFC.

Fig. 5.
Fig. 5.

Evolution of the rank histograms of the CNOP+CSVs scheme (red) and CSVs scheme (blue).

Citation: Weather and Forecasting 36, 6; 10.1175/WAF-D-21-0063.1

A more accurate quantitative diagnosis is given by the sum-of-squares differences between Sk and its expectation, i.e., M/(N + 1). The deviation from flatness in the histogram can be measured as Δ=k=1N+1{Sk[M/(N+1)]}2. For a perfectly reliable system, Δ has an expectation Δ0, where Δ0 = M[N/(N + 1)]. Then, the ratio δ = Δ/Δ0 can be used to measure the flatness of the histogram (Talagrand et al. 1997; Candille and Talagrand 2005). This measurement is also used in Jiang and Mu (2009). According to the definition of ratio δ, for a perfect ensemble forecast system, δ will be equal to 1. However, the value of δ is always larger than 1 in realistic ensemble forecasting. Therefore, the closer δ is to 1, the more reliable an ensemble system is. Figure 6 presents the temporal variation in the ratio δ of the rank histogram. Although both the CSVs and CNOP+CSVs schemes have ratio δ values obviously larger than 1, they tend to significantly decrease with a lengthening lead time; in particular, the ratio of the CNOP+CSVs scheme is smaller than that of the CSVs scheme. This indicates that the CNOP+CSVs scheme provides more reliable ensemble forecasts than the CSVs scheme.

Fig. 6.
Fig. 6.

The diagnostic score δ values obtained from the rank histograms of the CNOP+CSVs scheme (red) and CSVs scheme (blue) as a function of the lead time.

Citation: Weather and Forecasting 36, 6; 10.1175/WAF-D-21-0063.1

b. The probability skill in ensemble forecasts

The Brier skill (BS) is often used to verify the accuracy of a probability forecast. The BS is always calculated as the mean squared difference between the forecast probability Pi and the corresponding observed binary variable Oi, the latter of which is defined as 1 when an event occurs and 0 otherwise. The BS is calculated using the formula BS=(1/M)i=1M(PiOi)2, where M is the number of total verification samples. A small BS value indicates a better ensemble forecast. The BS values are calculated for three categorical events in the present study, i.e., cold, neutral, and warm events. The three categorical events possess equal climatological frequencies of 1/3 and are applied to all the forecasting results and the observations. This is done to exclude the potential influences of the different climatological state of the observations and the forecast data. This analysis method is also used by Zhang et al. (2019) to evaluate the SST predictability using the multimodel hindcast data.

The BSs of the ensemble forecasts generated by the CSVs and CNOP+CSVs schemes and those obtained with the control forecast are calculated. Among these calculations, the BSs of the ensemble forecasts are much smaller than those of the control run. To reveal the relative accuracy of the probabilistic forecasts compared to the accuracy of the control run, the Brier skill score (BSS) is further calculated by BSS = 1 − (BS/BSctrl). For a perfect ensemble forecast, the BSS value is equal to 1, while for realistic ensemble forecasts, the larger the BSS value is, the more reliable the ensemble forecast system is. Figure 7 presents the evolution of the BSSs of both the CSVs and CNOP+CSVs schemes. Generally, the BSSs of both kinds of ensemble schemes increase with the lead time. For neutral years, the performances of both kinds of ensemble methods are similar, with the BSS of the CSVs scheme being slightly larger than that of the CNOP+CSVs scheme. For cold events and warm events, the BSS of the CNOP+CSVs scheme is much larger than that of the CSVs scheme, implying that the CNOP+CSVs scheme can significantly improve the prediction skill in the ENSO ensemble forecast as measured by the BSS.

Fig. 7.
Fig. 7.

Evolution of the BSSs of the CNOP+CSVs scheme (red) and CSVs scheme (blue) as a function of the lead time: (a) cold events, (b) warm events, and (c) neutral state.

Citation: Weather and Forecasting 36, 6; 10.1175/WAF-D-21-0063.1

In summary, applying an ensemble forecast method, either the CSVs or CNOP+CSVs scheme, can significantly improve the ENSO predictions obtained using the NMEFC ENSO forecast system. In particular, the CNOP+CSVs scheme, i.e., the scheme involving the replacement of the LSV with the CNOP, can further improve the accuracy of ENSO forecasts, especially by increasing the reliability of the associated ensemble forecast system.

5. Summary and discussion

In this paper, we calculate the CSVs and CNOP of the sea temperatures in the tropical Pacific Ocean to forecast ENSO events in the operationally running ENSO forecast system of the NMEFC. Two groups of ensemble initial perturbations are constructed: one is the CSVs scheme using the leading five CSVs; the other is the CNOP+CSVs scheme with the CNOP replacing the LSV of the CSVs scheme. Ensemble forecast experiments of ENSO are conducted using these two groups of ensemble initial perturbations initialized in January of each year from 1997 to 2017. The results show that the NMEFC ENSO forecast system benefits greatly when ensemble forecast strategies are introduced. In particular, when comparing the CSVs with the CNOP+CSVs scheme, the ensemble forecast of CNOP+CSVs scheme is shown to be able to obviously improve the skill when predicting ENSO events; specifically, the ensemble spread and associated reliability of the ensemble forecasts are significantly improved, although the accuracy of the ensemble forecasts is increased to a lesser extent. The difference between the two groups of ensemble initial perturbations only lies in the difference between the LSV and CNOP schemes. The LSV represents the fastest linearly growing perturbation, while the CNOP acts as the nonlinear optimal perturbation. ENSO possesses irregular variabilities and amplitudes and is generally thought of as being controlled by nonlinear dynamical systems; furthermore, its prediction error growth is also demonstrated as having an almost common dynamical mechanism that often causes ENSO events to be over or underpredicted (Duan and Wei 2013; Duan and Hu 2016). The CNOP, due to its nonlinearity, can capture the nonlinear dynamical behavior of the initial error growth of ENSO forecasts, causing the ensemble forecasts of CNOP+CSVs scheme to be able to partially describe the nonlinear effects of error growth on ENSO forecasts; this scheme thus possesses higher skill for ENSO forecasting than the ensemble forecasts obtained with the CSVs scheme.

As demonstrated by Huo and Duan (2019), the magnitude of the initial perturbation, the lead time of the ensemble forecast may also influence the predictability of the ensemble forecasts. Therefore, more studies could be carried out to further clarify those questions. Besides, the CSVs and CNOP may also be sensitive to the phase of the ENSO cycle. Nevertheless, due to the associated expensive computational costs, we only calculate herein the CNOP and CSVs for ENSO forecasts starting from January. In addition, since ENSO dynamics are often governed by nonlinear dynamical systems, all ensemble initial perturbations used in ENSO forecasting should consist of nonlinear ENSO effects. However, for the ensemble forecasting strategy of CNOP+CSVs scheme, although this scheme possesses a much higher forecasting skill due to the introduction of the CNOP, the rest of the ensemble initial perturbations are still linearly optimized. That is, the ensemble initial perturbations generated by the CNOP+CSVs scheme do not fully include nonlinear effects. Regarding this effect, Duan and Huo (2016) proposed an orthogonal CNOPs strategy to generate ensemble initial perturbations. This strategy was applied in the fifth-generation mesoscale model (MM5) in an attempt to obtain better forecasts of tropical cyclone tracks (Huo and Duan 2019). The results showed that the ensemble forecasts obtained using orthogonal CNOPs have a higher forecasting skill for tropical cyclone tracks than that of the CNOP+SVs scheme. Therefore, it is important to fully consider the influence of nonlinearity on ensemble initial perturbations in ensemble forecasts. The biggest challenge we face now is the overwhelmingly large computational cost. So a more efficient algorithm needs to be developed in order to compute the orthogonal CNOPs for the application of the CNOP in complex climate models. Actually, the work conducted under our investigation and the primary results are encouraging. It is expected that orthogonal CNOPs can be effectively applied in ENSO ensemble forecasts to achieve much higher forecast skills in the future.

Acknowledgments

This study was supported by the National Key Research and Development Program of China (Grant 2020YFA0608800), together with the National Natural Science Foundation of China (Grants 41930971, 41805069, 41525017, 41606031, and 41975069).

Data availability statement

The datasets generated and/or analyzed during this study are available from the corresponding author.

REFERENCES

  • Alexander, M. A., I. Blade, M. Newman, J. R. Lanzante, N. C. Lau, and J. D. Scott, 2002: The atmospheric bridge: The influence of ENSO teleconnections on air–sea interaction over the global oceans. J. Climate, 15, 22052231, https://doi.org/10.1175/1520-0442(2002)015<2205:TABTIO>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Birgin, E., J. M. Martínez, and M. Raydan, 2000: Nonmonotone spectral projected gradient methods on convex sets. SIAM J. Optim., 10, 11961211, https://doi.org/10.1137/S1052623497330963.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Buizza, R., P. L. Houtekamer, G. Pellerin, Z. Toth, Y. Zhu, and M. Wei, 2005: A comparison of the ECMWF, MSC, and NCEP global ensemble prediction systems. Mon. Wea. Rev., 133, 10761097, https://doi.org/10.1175/MWR2905.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Buizza, R., M. Leutbecher, and L. Isaksen, 2008: Potential use of an ensemble of analyses in the ECMWF Ensemble Prediction System. Quart. J. Roy. Meteor. Soc., 134, 20512066, https://doi.org/10.1002/qj.346.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Candille, G., and O. Talagrand, 2005: Evaluation of probabilistic prediction systems for a scalar variable. Quart. J. Roy. Meteor. Soc., 131, 21312150, https://doi.org/10.1256/qj.04.71.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cane, M. A., S. E. Zebiak, and S. C. Dolan, 1986: Experimental forecasts of El Niño. Nature, 321, 827832, https://doi.org/10.1038/321827a0.

  • Chen, D., and M. A. Cane, 2008: El Niño prediction and predictability. J. Comput. Phys., 227, 36253640, https://doi.org/10.1016/j.jcp.2007.05.014.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Chen, L., W. Duan, and H. Xu, 2015: A SVD-based ensemble projection algorithm for calculating the conditional nonlinear optimal perturbation. Sci. China Earth Sci., 58, 385394, https://doi.org/10.1007/s11430-014-4991-4.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Derber, J., and A. Rosati, 1989: A global oceanic data assimilation system. J. Phys. Oceanogr., 19, 13331347, https://doi.org/10.1175/1520-0485(1989)019<1333:AGODAS>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Deser, C., and Coauthors, 2012: ENSO and Pacific decadal variability in the Community Climate System Model version 4. J. Climate, 25, 26222651, https://doi.org/10.1175/JCLI-D-11-00301.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Diaz, H. F., M. P. Hoerling, and J. K. Eischeid, 2001: ENSO variability, teleconnections and climate change. Int. J. Climatol., 21, 18451862, https://doi.org/10.1002/joc.631.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Duan, W., and C. Wei, 2013: The ‘spring predictability barrier’ for ENSO predictions and its possible mechanism: Results from a fully coupled model. Int. J. Climatol., 33, 12801292, https://doi.org/10.1002/joc.3513.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Duan, W., and J. Hu, 2016: The initial errors that induce a significant “spring predictability barrier” for El Niño events and their implications for target observation: Results from an earth system model. Climate Dyn., 46, 35993615, https://doi.org/10.1007/s00382-015-2789-5.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Duan, W., and Z. Huo, 2016: An approach to generating mutually independent initial perturbations for ensemble forecasts: Orthogonal conditional nonlinear optimal perturbations. J. Atmos. Sci., 73, 9971014, https://doi.org/10.1175/JAS-D-15-0138.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Duan, W., M. Mu, and B. Wang, 2004: Conditional nonlinear optimal perturbations as the optimal precursors for El Niño–Southern Oscillation events. J. Geophys. Res., 109, D23105, https://doi.org/10.1029/2004JD004756.

    • Search Google Scholar
    • Export Citation
  • Duan, W., Y. Yu, H. Xu, and P. Zhao, 2013: Behaviors of nonlinearities modulating the El Niño events induced by optimal precursory disturbances. Climate Dyn., 40, 13991413, https://doi.org/10.1007/s00382-012-1557-z.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Duan, W., X. Li, and B. Tian, 2018: Towards optimal observational array for dealing with challenges of El Niño–Southern Oscillation predictions due to diversities of El Niño. Climate Dyn., 51, 33513368, https://doi.org/10.1007/s00382-018-4082-x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fan, Y., M. R. Allen, D. L. T. Anderson, and M. A. Balmaseda, 2000: How predictability depends on the nature of uncertainty in initial conditions in a coupled model of ENSO. J. Climate, 13, 32983313, https://doi.org/10.1175/1520-0442(2000)013<3298:HPDOTN>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Foias, C., 1997: What do the Navier–Stokes equations tell us about turbulence? Contemp. Math., 208, 151180, https://doi.org/10.190/conm/208/02739.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gebbie, G., I. Eisenman, A. Wittenberg, and E. Tziperman, 2007: Modulation of westerly wind bursts by sea surface temperature: A semistochastic feedback for ENSO. J. Atmos. Sci., 64, 32813295, https://doi.org/10.1175/JAS4029.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Henderson, D. S., C. D. Kummerow, and W. Berg, 2018: ENSO influence on TRMM tropical oceanic precipitation characteristics and rain rates. J. Climate, 31, 39793998, https://doi.org/10.1175/JCLI-D-17-0276.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hoffman, R. N., and E. Kalnay, 1983: Lagged average forecasting, an alternative to Monte Carlo forecasting. Tellus, 35A, 100118, https://doi.org/10.1111/j.1600-0870.1983.tb00189.x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Huo, Z., and W. Duan, 2019: The application of the orthogonal conditional nonlinear optimal perturbations method to typhoon track ensemble forecasts. Sci. China Earth Sci., 62, 376388, https://doi.org/10.1007/s11430-018-9248-9.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jiang, Z., and M. Mu, 2009: A comparison study of the methods of conditional nonlinear optimal perturbations and singular vectors in ensemble prediction. Adv. Atmos. Sci., 26, 465470, https://doi.org/10.1007/s00376-009-0465-6.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kleeman, R., and A. M. Moore, 1997: A theory for the limitation of ENSO predictability due to stochastic atmospheric transients. J. Atmos. Sci., 54, 753767, https://doi.org/10.1175/1520-0469(1997)054<0753:ATFTLO>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kleeman, R., Y. Tang, and A. M. Moore, 2003: The calculation of climatically relevant singular vectors in the presence of weather noise as applied to the ENSO problem. J. Atmos. Sci., 60, 28562868, https://doi.org/10.1175/1520-0469(2003)060<2856:TCOCRS>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kug, J.-S., Y.-G. Ham, M. Kimoto, F.-F. Jin, and I.-S. Kang, 2010: New approach for optimal perturbation method in ensemble climate prediction with empirical singular vector. Climate Dyn., 35, 331340, https://doi.org/10.1007/s00382-009-0664-y.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lai, A. W.-C., M. Herzog, and H.-F. Graf, 2018: ENSO forecasts near the spring predictability barrier and possible reasons for the recently reduced predictability. J. Climate, 31, 815838, https://doi.org/10.1175/JCLI-D-17-0180.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Latif, M., and Coauthors, 1998: A review of the predictability and prediction of ENSO. J. Geophys. Res., 103, 14 37514 393, https://doi.org/10.1029/97JC03413.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Leith, C. E., 1974: Theoretical skill of Monte Carlo forecasts. Mon. Wea. Rev., 102, 409418, https://doi.org/10.1175/1520-0493(1974)102<0409:TSOMCF>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lorenz, E. N., 1965: A study of the predictability of a 28-variable atmospheric model. Tellus, 17, 321333, https://doi.org/10.3402/tellusa.v17i3.9076.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Molteni, F., R. Buizza, T. N. Palmer, and T. Petroliagis, 1996: The ECMWF ensemble prediction system: Methodology and validation. Quart. J. Roy. Meteor. Soc., 122, 73119, https://doi.org/10.1002/qj.49712252905.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Moore, A. M., and R. Kleeman, 1998: Skill assessment for ENSO using ensemble prediction. Quart. J. Roy. Meteor. Soc., 124, 557584, https://doi.org/10.1002/qj.49712454609.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mu, B., S. Wen, S. Yuan, and H. Li, 2015: PPSO: PCA based particle swarm optimization for solving conditional nonlinear optimal perturbation. Comput. Geosci., 83, 6571, https://doi.org/10.1016/j.cageo.2015.06.016.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mu, B., J. Ren, S. Yuan, R.-H. Zhang, L. Chen, and C. Gao, 2019: The optimal precursors for ENSO events depicted using the gradient-definition-based method in an intermediate coupled model. Adv. Atmos. Sci., 36, 13811392, https://doi.org/10.1007/s00376-019-9040-y.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mu, M., and Z. Jiang, 2008: A new approach to the generation of initial perturbations for ensemble prediction: Conditional nonlinear optimal perturbation. Chin. Sci. Bull., 53, 20622068, https://doi.org/10.1007/s11434-008-0272-y.

    • Search Google Scholar
    • Export Citation
  • Mu, M., W. S. Duan, and B. Wang, 2003: Conditional nonlinear optimal perturbation and its applications. Nonlinear Processes Geophys., 10, 493501, https://doi.org/10.5194/npg-10-493-2003.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mu, M., W. Duan, and B. Wang, 2007a: Season-dependent dynamics of nonlinear optimal error growth and El Niño–Southern Oscillation predictability in a theoretical model. J. Geophys. Res., 112, D10113, https://doi.org/10.1029/2005JD006981.

    • Search Google Scholar
    • Export Citation
  • Mu, M., H. Xu, and W. Duan, 2007b: A kind of initial errors related to “spring predictability barrier” for El Niño events in Zebiak–Cane model. Geophys. Res. Lett., 34, L03709, https://doi.org/10.1029/2006GL027412.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Osborne, A. R., and A. Pastorello, 1993: Simultaneous occurrence of low-dimensional chaos and colored random noise in nonlinear physical systems. Phys. Lett. A, 181, 159171, https://doi.org/10.1016/0375-9601(93)90914-L.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Palmer, T., 2019: The ECMWF ensemble prediction system: Looking back (more than) 25 years and projecting forward 25 years. Quart. J. Roy. Meteor. Soc., 145, 1224, https://doi.org/10.1002/qj.3383.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sun, G., and M. Mu, 2013: Understanding variations and seasonal characteristics of net primary production under two types of climate change scenarios in China using the LPJ model. Climatic Change, 120, 755769, https://doi.org/10.1007/s10584-013-0833-1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Talagrand, O., R. Vautard, and B. Strauss, 1997: Evaluation of probabilistic prediction systems. Workshop Proc. Workshop on Predictability, ECMWF, Reading, United Kingdom, 1–26.

  • Tang, Y., R. Kleeman, and S. Miller, 2006: ENSO predictability of a fully coupled GCM model using singular vector analysis. J. Climate, 19, 33613377, https://doi.org/10.1175/JCLI3771.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Tang, Y., and Coauthors, 2018: Progress in ENSO prediction and predictability study. Natl. Sci. Rev., 5, 826839, https://doi.org/10.1093/nsr/nwy105.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Tao, L.-J., R.-H. Zhang, and C. Gao, 2017: Initial error-induced optimal perturbations in ENSO predictions, as derived from an intermediate coupled model. Adv. Atmos. Sci., 34, 791803, https://doi.org/10.1007/s00376-017-6266-4.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Tompkins, A. M., and Coauthors, 2017: The climate-system historical forecast project: Providing open access to seasonal forecast ensembles from centers around the globe. Bull. Amer. Meteor. Soc., 98, 22932301, https://doi.org/10.1175/BAMS-D-16-0209.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Toth, Z., and E. Kalnay, 1993: Ensemble forecasting at NMC: The generation of perturbations. Bull. Amer. Meteor. Soc., 74, 23172330, https://doi.org/10.1175/1520-0477(1993)074<2317:EFANTG>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Vecchi, G. A., and D. E. Harrison, 2003: On the termination of the 2002–03 El Niño event. Geophys. Res. Lett., 30, 1964, https://doi.org/10.1029/2003GL017564.

    • Search Google Scholar
    • Export Citation
  • Wang, B., and X. Tan, 2010: Conditional nonlinear optimal perturbations: Adjoint-free calculation method and preliminary test. Mon. Wea. Rev., 138, 10431049, https://doi.org/10.1175/2009MWR3022.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wang, B., R. Wu, and X. Fu, 2000: Pacific–East Asian teleconnection: How does ENSO affect East Asian climate? J. Climate, 13, 15171536, https://doi.org/10.1175/1520-0442(2000)013<1517:PEATHD>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Webster, P. J., and S. Yang, 1992: Monsoon and ENSO: Selectively interactive systems. Quart. J. Roy. Meteor. Soc., 118, 877926, https://doi.org/10.1002/qj.49711850705.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wen, S., S. Yuan, B. Mu, and H. Li, 2015: Robust PCA-based genetic algorithm for solving CNOP. Intelligent Computing Theories and Methodologies, D. S. Huang, V. Bevilacqua, and P. Premaratne, Eds., Springer, 597–606, https://doi.org/10.1007/978-3-319-22180-9_59.

    • Crossref
    • Export Citation
  • Xu, H., L. Chen, and W. Duan, 2021: Optimally growing initial errors of El Niño events in the CESM. Climate Dyn., 56, 37973815, https://doi.org/10.1007/s00382-021-05668-1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Xue, Y., M. A. Cane, and S. E. Zebiak, 1997: Predictability of a coupled model of ENSO using singular vector analysis. 1. Optimal growth in seasonal background and ENSO cycles. Mon. Wea. Rev., 125, 20432056, https://doi.org/10.1175/1520-0493(1997)125<2043:POACMO>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Yu, Y., M. Mu, W. Duan, and T. Gong, 2012: Contribution of the location and spatial pattern of initial error to uncertainties in El Niño predictions. J. Geophys. Res., 117, C06018, https://doi.org/10.1029/2011JC007758.

    • Search Google Scholar
    • Export Citation
  • Zhang, L.-L., S.-J. Yuan, B. Mu, and F.-F. Zhou, 2017: CNOP-based sensitive areas identification for tropical cyclone adaptive observations with PCAGA method. Asia-Pac. J. Atmos. Sci., 53, 6373, https://doi.org/10.1007/s13143-017-0005-8.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zhang, R. H., and Coauthors, 2020: A review of progress in coupled ocean-atmosphere model developments for ENSO studies in China. J. Oceanol. Limnol., 38, 930961, https://doi.org/10.1007/s00343-020-0157-8.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zhang, S., C. Song, H. Wang, H. Jiang, and L. Du, 2018: Evaluation of the hindcasting main SSTA modes of the global key regions based on the CESM forecasting system. Haiyang Xuebao, 40, 1830, https://doi.org/10.3969/j.issn.0253-4193.2018.09.002.

    • Search Google Scholar
    • Export Citation
  • Zhang, S., H. Jiang, and H. Wang, 2019: Assessment of the sea surface temperature predictability based on multimodel hindcasts. Wea. Forecasting, 34, 19651977, https://doi.org/10.1175/WAF-D-19-0040.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zheng, X.-T., C. Hui, and S.-W. Yeh, 2018: Response of ENSO amplitude to global warming in CESM large ensemble: Uncertainty due to internal variability. Climate Dyn., 50, 40194035, https://doi.org/10.1007/s00382-017-3859-7.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zhu, J., B. Huang, M. Balmaseda, J. Kinter, P. Peng, Z.-Z. Hu, and L. Marx, 2013: Improved reliability of ENSO hindcasts with multi-ocean analyses ensemble initialization. Climate Dyn., 41, 27852795, https://doi.org/10.1007/s00382-013-1965-8.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zhu, Y., 2005: Ensemble forecast: A new approach to uncertainty and predictability. Adv. Atmos. Sci., 22, 781788, https://doi.org/10.1007/BF02918678.

    • Crossref
    • Search Google Scholar
    • Export Citation
Save
  • Alexander, M. A., I. Blade, M. Newman, J. R. Lanzante, N. C. Lau, and J. D. Scott, 2002: The atmospheric bridge: The influence of ENSO teleconnections on air–sea interaction over the global oceans. J. Climate, 15, 22052231, https://doi.org/10.1175/1520-0442(2002)015<2205:TABTIO>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Birgin, E., J. M. Martínez, and M. Raydan, 2000: Nonmonotone spectral projected gradient methods on convex sets. SIAM J. Optim., 10, 11961211, https://doi.org/10.1137/S1052623497330963.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Buizza, R., P. L. Houtekamer, G. Pellerin, Z. Toth, Y. Zhu, and M. Wei, 2005: A comparison of the ECMWF, MSC, and NCEP global ensemble prediction systems. Mon. Wea. Rev., 133, 10761097, https://doi.org/10.1175/MWR2905.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Buizza, R., M. Leutbecher, and L. Isaksen, 2008: Potential use of an ensemble of analyses in the ECMWF Ensemble Prediction System. Quart. J. Roy. Meteor. Soc., 134, 20512066, https://doi.org/10.1002/qj.346.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Candille, G., and O. Talagrand, 2005: Evaluation of probabilistic prediction systems for a scalar variable. Quart. J. Roy. Meteor. Soc., 131, 21312150, https://doi.org/10.1256/qj.04.71.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Cane, M. A., S. E. Zebiak, and S. C. Dolan, 1986: Experimental forecasts of El Niño. Nature, 321, 827832, https://doi.org/10.1038/321827a0.

  • Chen, D., and M. A. Cane, 2008: El Niño prediction and predictability. J. Comput. Phys., 227, 36253640, https://doi.org/10.1016/j.jcp.2007.05.014.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Chen, L., W. Duan, and H. Xu, 2015: A SVD-based ensemble projection algorithm for calculating the conditional nonlinear optimal perturbation. Sci. China Earth Sci., 58, 385394, https://doi.org/10.1007/s11430-014-4991-4.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Derber, J., and A. Rosati, 1989: A global oceanic data assimilation system. J. Phys. Oceanogr., 19, 13331347, https://doi.org/10.1175/1520-0485(1989)019<1333:AGODAS>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Deser, C., and Coauthors, 2012: ENSO and Pacific decadal variability in the Community Climate System Model version 4. J. Climate, 25, 26222651, https://doi.org/10.1175/JCLI-D-11-00301.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Diaz, H. F., M. P. Hoerling, and J. K. Eischeid, 2001: ENSO variability, teleconnections and climate change. Int. J. Climatol., 21, 18451862, https://doi.org/10.1002/joc.631.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Duan, W., and C. Wei, 2013: The ‘spring predictability barrier’ for ENSO predictions and its possible mechanism: Results from a fully coupled model. Int. J. Climatol., 33, 12801292, https://doi.org/10.1002/joc.3513.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Duan, W., and J. Hu, 2016: The initial errors that induce a significant “spring predictability barrier” for El Niño events and their implications for target observation: Results from an earth system model. Climate Dyn., 46, 35993615, https://doi.org/10.1007/s00382-015-2789-5.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Duan, W., and Z. Huo, 2016: An approach to generating mutually independent initial perturbations for ensemble forecasts: Orthogonal conditional nonlinear optimal perturbations. J. Atmos. Sci., 73, 9971014, https://doi.org/10.1175/JAS-D-15-0138.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Duan, W., M. Mu, and B. Wang, 2004: Conditional nonlinear optimal perturbations as the optimal precursors for El Niño–Southern Oscillation events. J. Geophys. Res., 109, D23105, https://doi.org/10.1029/2004JD004756.

    • Search Google Scholar
    • Export Citation
  • Duan, W., Y. Yu, H. Xu, and P. Zhao, 2013: Behaviors of nonlinearities modulating the El Niño events induced by optimal precursory disturbances. Climate Dyn., 40, 13991413, https://doi.org/10.1007/s00382-012-1557-z.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Duan, W., X. Li, and B. Tian, 2018: Towards optimal observational array for dealing with challenges of El Niño–Southern Oscillation predictions due to diversities of El Niño. Climate Dyn., 51, 33513368, https://doi.org/10.1007/s00382-018-4082-x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fan, Y., M. R. Allen, D. L. T. Anderson, and M. A. Balmaseda, 2000: How predictability depends on the nature of uncertainty in initial conditions in a coupled model of ENSO. J. Climate, 13, 32983313, https://doi.org/10.1175/1520-0442(2000)013<3298:HPDOTN>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Foias, C., 1997: What do the Navier–Stokes equations tell us about turbulence? Contemp. Math., 208, 151180, https://doi.org/10.190/conm/208/02739.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gebbie, G., I. Eisenman, A. Wittenberg, and E. Tziperman, 2007: Modulation of westerly wind bursts by sea surface temperature: A semistochastic feedback for ENSO. J. Atmos. Sci., 64, 32813295, https://doi.org/10.1175/JAS4029.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Henderson, D. S., C. D. Kummerow, and W. Berg, 2018: ENSO influence on TRMM tropical oceanic precipitation characteristics and rain rates. J. Climate, 31, 39793998, https://doi.org/10.1175/JCLI-D-17-0276.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hoffman, R. N., and E. Kalnay, 1983: Lagged average forecasting, an alternative to Monte Carlo forecasting. Tellus, 35A, 100118, https://doi.org/10.1111/j.1600-0870.1983.tb00189.x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Huo, Z., and W. Duan, 2019: The application of the orthogonal conditional nonlinear optimal perturbations method to typhoon track ensemble forecasts. Sci. China Earth Sci., 62, 376388, https://doi.org/10.1007/s11430-018-9248-9.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jiang, Z., and M. Mu, 2009: A comparison study of the methods of conditional nonlinear optimal perturbations and singular vectors in ensemble prediction. Adv. Atmos. Sci., 26, 465470, https://doi.org/10.1007/s00376-009-0465-6.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kleeman, R., and A. M. Moore, 1997: A theory for the limitation of ENSO predictability due to stochastic atmospheric transients. J. Atmos. Sci., 54, 753767, https://doi.org/10.1175/1520-0469(1997)054<0753:ATFTLO>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kleeman, R., Y. Tang, and A. M. Moore, 2003: The calculation of climatically relevant singular vectors in the presence of weather noise as applied to the ENSO problem. J. Atmos. Sci., 60, 28562868, https://doi.org/10.1175/1520-0469(2003)060<2856:TCOCRS>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kug, J.-S., Y.-G. Ham, M. Kimoto, F.-F. Jin, and I.-S. Kang, 2010: New approach for optimal perturbation method in ensemble climate prediction with empirical singular vector. Climate Dyn., 35, 331340, https://doi.org/10.1007/s00382-009-0664-y.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lai, A. W.-C., M. Herzog, and H.-F. Graf, 2018: ENSO forecasts near the spring predictability barrier and possible reasons for the recently reduced predictability. J. Climate, 31, 815838, https://doi.org/10.1175/JCLI-D-17-0180.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Latif, M., and Coauthors, 1998: A review of the predictability and prediction of ENSO. J. Geophys. Res., 103, 14 37514 393, https://doi.org/10.1029/97JC03413.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Leith, C. E., 1974: Theoretical skill of Monte Carlo forecasts. Mon. Wea. Rev., 102, 409418, https://doi.org/10.1175/1520-0493(1974)102<0409:TSOMCF>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lorenz, E. N., 1965: A study of the predictability of a 28-variable atmospheric model. Tellus, 17, 321333, https://doi.org/10.3402/tellusa.v17i3.9076.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Molteni, F., R. Buizza, T. N. Palmer, and T. Petroliagis, 1996: The ECMWF ensemble prediction system: Methodology and validation. Quart. J. Roy. Meteor. Soc., 122, 73119, https://doi.org/10.1002/qj.49712252905.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Moore, A. M., and R. Kleeman, 1998: Skill assessment for ENSO using ensemble prediction. Quart. J. Roy. Meteor. Soc., 124, 557584, https://doi.org/10.1002/qj.49712454609.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mu, B., S. Wen, S. Yuan, and H. Li, 2015: PPSO: PCA based particle swarm optimization for solving conditional nonlinear optimal perturbation. Comput. Geosci., 83, 6571, https://doi.org/10.1016/j.cageo.2015.06.016.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mu, B., J. Ren, S. Yuan, R.-H. Zhang, L. Chen, and C. Gao, 2019: The optimal precursors for ENSO events depicted using the gradient-definition-based method in an intermediate coupled model. Adv. Atmos. Sci., 36, 13811392, https://doi.org/10.1007/s00376-019-9040-y.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mu, M., and Z. Jiang, 2008: A new approach to the generation of initial perturbations for ensemble prediction: Conditional nonlinear optimal perturbation. Chin. Sci. Bull., 53, 20622068, https://doi.org/10.1007/s11434-008-0272-y.

    • Search Google Scholar
    • Export Citation
  • Mu, M., W. S. Duan, and B. Wang, 2003: Conditional nonlinear optimal perturbation and its applications. Nonlinear Processes Geophys., 10, 493501, https://doi.org/10.5194/npg-10-493-2003.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mu, M., W. Duan, and B. Wang, 2007a: Season-dependent dynamics of nonlinear optimal error growth and El Niño–Southern Oscillation predictability in a theoretical model. J. Geophys. Res., 112, D10113, https://doi.org/10.1029/2005JD006981.

    • Search Google Scholar
    • Export Citation
  • Mu, M., H. Xu, and W. Duan, 2007b: A kind of initial errors related to “spring predictability barrier” for El Niño events in Zebiak–Cane model. Geophys. Res. Lett., 34, L03709, https://doi.org/10.1029/2006GL027412.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Osborne, A. R., and A. Pastorello, 1993: Simultaneous occurrence of low-dimensional chaos and colored random noise in nonlinear physical systems. Phys. Lett. A, 181, 159171, https://doi.org/10.1016/0375-9601(93)90914-L.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Palmer, T., 2019: The ECMWF ensemble prediction system: Looking back (more than) 25 years and projecting forward 25 years. Quart. J. Roy. Meteor. Soc., 145, 1224, https://doi.org/10.1002/qj.3383.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sun, G., and M. Mu, 2013: Understanding variations and seasonal characteristics of net primary production under two types of climate change scenarios in China using the LPJ model. Climatic Change, 120, 755769, https://doi.org/10.1007/s10584-013-0833-1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Talagrand, O., R. Vautard, and B. Strauss, 1997: Evaluation of probabilistic prediction systems. Workshop Proc. Workshop on Predictability, ECMWF, Reading, United Kingdom, 1–26.

  • Tang, Y., R. Kleeman, and S. Miller, 2006: ENSO predictability of a fully coupled GCM model using singular vector analysis. J. Climate, 19, 33613377, https://doi.org/10.1175/JCLI3771.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Tang, Y., and Coauthors, 2018: Progress in ENSO prediction and predictability study. Natl. Sci. Rev., 5, 826839, https://doi.org/10.1093/nsr/nwy105.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Tao, L.-J., R.-H. Zhang, and C. Gao, 2017: Initial error-induced optimal perturbations in ENSO predictions, as derived from an intermediate coupled model. Adv. Atmos. Sci., 34, 791803, https://doi.org/10.1007/s00376-017-6266-4.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Tompkins, A. M., and Coauthors, 2017: The climate-system historical forecast project: Providing open access to seasonal forecast ensembles from centers around the globe. Bull. Amer. Meteor. Soc., 98, 22932301, https://doi.org/10.1175/BAMS-D-16-0209.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Toth, Z., and E. Kalnay, 1993: Ensemble forecasting at NMC: The generation of perturbations. Bull. Amer. Meteor. Soc., 74, 23172330, https://doi.org/10.1175/1520-0477(1993)074<2317:EFANTG>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Vecchi, G. A., and D. E. Harrison, 2003: On the termination of the 2002–03 El Niño event. Geophys. Res. Lett., 30, 1964, https://doi.org/10.1029/2003GL017564.

    • Search Google Scholar
    • Export Citation
  • Wang, B., and X. Tan, 2010: Conditional nonlinear optimal perturbations: Adjoint-free calculation method and preliminary test. Mon. Wea. Rev., 138, 10431049, https://doi.org/10.1175/2009MWR3022.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wang, B., R. Wu, and X. Fu, 2000: Pacific–East Asian teleconnection: How does ENSO affect East Asian climate? J. Climate, 13, 15171536, https://doi.org/10.1175/1520-0442(2000)013<1517:PEATHD>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Webster, P. J., and S. Yang, 1992: Monsoon and ENSO: Selectively interactive systems. Quart. J. Roy. Meteor. Soc., 118, 877926, https://doi.org/10.1002/qj.49711850705.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wen, S., S. Yuan, B. Mu, and H. Li, 2015: Robust PCA-based genetic algorithm for solving CNOP. Intelligent Computing Theories and Methodologies, D. S. Huang, V. Bevilacqua, and P. Premaratne, Eds., Springer, 597–606, https://doi.org/10.1007/978-3-319-22180-9_59.

    • Crossref
    • Export Citation
  • Xu, H., L. Chen, and W. Duan, 2021: Optimally growing initial errors of El Niño events in the CESM. Climate Dyn., 56, 37973815, https://doi.org/10.1007/s00382-021-05668-1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Xue, Y., M. A. Cane, and S. E. Zebiak, 1997: Predictability of a coupled model of ENSO using singular vector analysis. 1. Optimal growth in seasonal background and ENSO cycles. Mon. Wea. Rev., 125, 20432056, https://doi.org/10.1175/1520-0493(1997)125<2043:POACMO>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Yu, Y., M. Mu, W. Duan, and T. Gong, 2012: Contribution of the location and spatial pattern of initial error to uncertainties in El Niño predictions. J. Geophys. Res., 117, C06018, https://doi.org/10.1029/2011JC007758.

    • Search Google Scholar
    • Export Citation
  • Zhang, L.-L., S.-J. Yuan, B. Mu, and F.-F. Zhou, 2017: CNOP-based sensitive areas identification for tropical cyclone adaptive observations with PCAGA method. Asia-Pac. J. Atmos. Sci., 53, 6373, https://doi.org/10.1007/s13143-017-0005-8.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zhang, R. H., and Coauthors, 2020: A review of progress in coupled ocean-atmosphere model developments for ENSO studies in China. J. Oceanol. Limnol., 38, 930961, https://doi.org/10.1007/s00343-020-0157-8.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zhang, S., C. Song, H. Wang, H. Jiang, and L. Du, 2018: Evaluation of the hindcasting main SSTA modes of the global key regions based on the CESM forecasting system. Haiyang Xuebao, 40, 1830, https://doi.org/10.3969/j.issn.0253-4193.2018.09.002.

    • Search Google Scholar
    • Export Citation
  • Zhang, S., H. Jiang, and H. Wang, 2019: Assessment of the sea surface temperature predictability based on multimodel hindcasts. Wea. Forecasting, 34, 19651977, https://doi.org/10.1175/WAF-D-19-0040.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Zheng, X.-T., C. Hui, and S.-W. Yeh, 2018: Response of ENSO amplitude to global warming in CESM large ensemble: Uncertainty due to internal variability. Climate Dyn., 50, 40194035,