1. Introduction
In an ensemble Kalman filter (EnKF; e.g., Houtekamer and Zhang 2016), when the analysis update of an ensemble member is computed using error statistics estimated from an ensemble that includes the background of the member being updated, the spread of the resulting analysis ensemble usually systematically underestimates the uncertainty of the ensemble mean analysis as noted first by Houtekamer and Mitchell (1998). This so-called inbreeding problem is the result of the deviation from the Monte Carlo methodology, which requires that an existing assimilation system be tested with an independent test ensemble (Houtekamer and Zhang 2016). Various inflation methods have been proposed to compensate for this loss of reliability of the analysis ensemble (e.g., Houtekamer et al. 2009; Whitaker and Hamill 2012). These mitigation techniques are effective enough to have permitted the adoption of inbreeding-affected EnKFs in operational or quasi-operational systems at NCEP (Zhou et al. 2017), DWD (Schraff et al. 2016), JMA (Miyoshi et al. 2010), KMA (Shin et al. 2016), and elsewhere.
Since the operational implementation of an EnKF in the global ensemble prediction system (GEPS) at Environment and Climate Change Canada (ECCC) in 2005 (Houtekamer et al. 2005), the concept of cross validation has been used to counter the inbreeding problem. The full ensemble is divided into k subensembles of equal size (Houtekamer and Mitchell 1998). Then, for each subensemble, the analysis update is computed using a Kalman gain matrix estimated using the other, independent, members. Experiments in a perfect-model framework show that this technique makes it possible to maintain an ensemble spread that closely matches the uncertainty in the ensemble mean throughout the data assimilation (DA) cycle (Houtekamer et al. 2009) without using any inflation approach. To date, no other national weather center is using a cross-validation approach in an operational context, in part because the type of EnKF historically used at ECCC (a stochastic EnKF, i.e., a class of filters that assimilates perturbed observations, see Houtekamer and Zhang 2016) permits more easily the implementation of such an approach. However, Buehner (2020) recently showed how cross validation can be efficiently implemented in the local ensemble transform Kalman filter (LETKF) that does not rely on perturbed observations (i.e., a deterministic type of EnKF).
A drawback of the cross-validation approach is that it can lead, in some circumstances, to the divergence of one or more ensemble members from both observations and the remainder of the ensemble. This can culminate in a catastrophic filter divergence (or failure) as the atmospheric state in the diverging members becomes increasingly unrealistic (see appendix A of Houtekamer and Zhang 2016). Such a problem occurs when one of the subensembles becomes sufficiently different from the others through nonlinear processes in the forecasting step (e.g., the triggering of deep convection in a single member of the whole ensemble), leading to an uneven spread distribution among the subensembles. When this happens, the subensembles with smaller spread are updated using a larger gain than the subensemble with larger spread, thus amplifying the differences between the spread of the subensembles (Mitchell and Houtekamer 2009). So far, such instabilities have been reported only in the context of highly nonlinear low-dimensional models, using either a scalar model like the logistic map (Strogatz 2015) as in Mitchell and Houtekamer (2009) and Buehner (2020), or the Lorenz and Emanuel (1998) 1D model as in Bowler et al. (2013) and Buehner (2020). Catastrophic filter divergence related to the use of cross validation in low-dimensional models also seems to manifest itself more easily with the deterministic class of EnKF (e.g., Buehner 2020).
However, the first known incidence of catastrophic filter divergence due to the use of cross validation in an NWP context was observed during the real-time testing of the latest upgrade to the GEPS at ECCC (2021) (whose configuration is described in section 2b). The top two rows of Fig. 1 show the zonal wind in the upper stratosphere (∼3 hPa) for the background state (i.e., 6-h forecast) of two ensemble members (234 and 254) during the real-time execution of this new version of the GEPS in a DA cycle in parallel to the operational one. Member 254 shows a broad and intense easterly jet over the tropics (Fig. 1a), exceeding 350 kt (180 m s−1; 1 kt ≈ 0.51 m s−1) locally, which appears unrealistically too strong according to some decade-long estimates at this level (Smith et al. 2017). On the other hand, member 234 exhibits a well-defined narrow westerly jet over the same area (Fig. 1b), resulting in differences between these members of as much as 500 kt. The background states from these two members are both far away from the estimated zonal winds provided by the analysis from the operational global deterministic prediction system (GDPS; Buehner et al. 2015),1 which shows no well-defined jet near the equator (Fig. 1c). During the first hours of the following forecast, the temperature reached 95°C in the upper stratosphere in ensemble member 254, exceeding the limit of the Planck function fit in the radiation scheme of the model, causing the model to abort.
Zonal wind (kt) at ∼3 hPa valid at 1200 UTC 12 Oct 2021. (a) Background state from ensemble member 254 of a new version of the GEPS; (b) as in (a), but for ensemble member 234; and (c) the analysis from the operational GDPS. The black lines correspond to the parallels at 10°N and 10°S.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-22-0108.1
This paper first presents objective evaluations of filter divergence occurrences in some of the subensembles of the GEPS ranging from the extreme case presented in Fig. 1 to more minor events that were found a posteriori in both the configuration referred above and the former operational version. Note that the type of filter divergence addressed here is one that leads to excessive ensemble spread and should not be confused with the classical type of divergence where the ensemble spread becomes much too small (see appendix A in Houtekamer and Zhang 2016 for a discussion). To significantly reduce the risk of such filter divergence, a small—but key—modification to the cross-validation approach is then presented. Instead of always assigning each ensemble member to the same subensemble, the members forming each subensemble are randomly chosen at every analysis step (i.e., every 6 h in the context of the GEPS). The success of this new approach to prevent filter divergence permitted the operational implementation of the revised version of the GEPS on 1 December 2021.
Key details on the GEPS are presented in section 2. Section 3 presents the different metrics used to diagnose the occurrence of filter divergences while section 4 shows the results from their application to versions of the GEPS that use the traditional cross-validation formulation. In section 5, the effectiveness of the proposed randomized subensemble approach in preventing filter divergence in the upgraded version of the GEPS and its impact on the ensemble forecast performance are presented. Finally, the outcomes of this study are summarized and further discussed in section 6.
2. The Canadian Global Ensemble Prediction System
a. Overview of the former operational configuration (GEPS-Old-Static)
The GEPS is formed of two components: 1) an EnKF-based DA cycle that generates 256 analyses and background states every 6 h and 2) a forecasting component where 20 out of the 256 analyses are integrated up to 16 days every 12 h at 0000 and 1200 UTC and up to 1 month every Thursday at 0000 UTC. Both components use the Global Environmental Multiscale (GEM) model defined on two overlapping limited-area domains (in Yin–Yang grid formation; Qaddouri and Lee 2011) that provide a global coverage with a fairly uniform grid spacing of approximately 39 km. The GEM model is a two-time-level implicit, semi-Lagrangian, gridpoint model. The vertical grid is defined on the terrain-following log-hydrostatic pressure vertical coordinate described by Girard et al. (2014). In this version of the GEPS, the model configuration in the DA cycle uses 81 vertical levels whereas only 46 levels are used in the forecast component. In both cases, the model lid is located at 0.1 hPa.
The stochastic EnKF algorithm assimilates perturbed observations sequentially (i.e., by batches) to make solving the analysis equations computationally feasible. The cross-validation approach consists of subdividing the 256 ensemble members into 8 subensembles of 32 members where each member is always assigned to the same subensemble: a “static” membership strategy. The Kalman gain matrix for the assimilation of the 32 members of one subensemble is therefore computed using the background-error covariances estimated from the 224 members of the other 7 subensembles. To mitigate the deleterious impact of the sampling noise in the background-error covariances due to the limited size of the ensemble, the impact of the observations is localized spatially with a Schur product [Houtekamer and Zhang 2016, section 3e(1)]. The horizontal localization length varies in the vertical from 2100 km near the surface to 3000 km near the model lid (see Table 1 in Houtekamer et al. 2018) whereas the vertical localization limits the impact of an observation to a vertical distance of three units of lnp.
Model uncertainty is represented using three distinct techniques. A multiphysics approach is adopted, which consists of applying different parameterizations and parameter values to different ensemble members (Houtekamer et al. 1996; Houtekamer 2011). In addition, the forecast component uses a stochastic kinetic energy backscatter scheme (Shutts 2005; Charron et al. 2010) and stochastically perturbed parameterization tendencies scheme (Buizza et al. 1999; Palmer et al. 2009; Charron et al. 2010).
Finally, to account for neglected error sources, an additive inflation procedure is used [Houtekamer and Zhang 2016; section 4a(1)] which adds random perturbations, generated from the climatological background-error covariances used in the EnVar scheme of the GDPS, to the temperature, specific humidity, horizontal winds and surface pressure. The version of the GEPS described above was operational at ECCC between 3 July 2019 and 1 December 2021. Further details on this configuration referred hereafter to as GEPS-Old-Static can be found in ECCC (2019) and references therein.
b. The first version of the new configuration (GEPS-New-Static)
Significant changes to both atmospheric DA and modeling aspects were proposed and tested in 2021 (hereafter referred to as GEPS-New-Static). First, the stochastic EnKF algorithm was replaced by a deterministic LETKF scheme using the gain form approach (Bishop et al. 2017) in order to enable cross validation as described by Buehner (2020). The observations are therefore no longer perturbed in this new scheme. A more advanced atmospheric physics parameterization package (McTaggart-Cowan et al. 2019) and vertical discretization on 84 levels (with a model lid still at 0.1 hPa) were adopted for both the DA and forecast components of the GEPS. The multiphysics-based model uncertainty representation was removed. Model uncertainty across the updated system is now represented using an updated stochastic kinetic energy backscatter scheme and stochastically perturbed parameterizations (McTaggart-Cowan et al. 2022a,b). The hybrid-gain approach was modified so that the γ parameter in (1) is member-specific, taking on a fixed, randomly assigned value between 0 (centering around the EnVar analysis) and 1 (no recentering). This change introduces a new source of ensemble spread for variables and locations where the LETKF ensemble mean analysis and the EnVar analysis differ. Due to the addition of uncertainty sources and the increase in the forecast accuracy resulting from the updates, the amplitude of the random additive perturbations to inflate the spread of the ensemble of analyses was reduced by 20% in the DA and 30% in the forecast components of the GEPS. Other minor changes were also made, a complete list of which can be found in ECCC (2021).
Note that for each GEPS-New-Static experiment reported in this paper, the first analysis step used an ensemble of background states originating from archived outputs of the operational GEPS-Old-Static configuration and the resulting ensemble of analyses was then cycled using this new configuration.
3. The divergence diagnostics
Given that the DA schemes employed in the GEPS (i.e., the EnKF, LETKF, and EnVar) are based on the assumption that the background errors, as well as the observation errors, follow a Gaussian distribution, we also evaluate the probability that the ensemble of background states are consistent with a Gaussian distribution by computing, first, the sample skewness and kurtosis. These two quantities can be combined in an omnibus test (K2; D’Agostino and Pearson 1973) that measures deviations from normality. Assuming that the null hypothesis of normality is true, K2 can be transformed into a probability by computing the integral of a cumulative chi-squared distribution function (e.g., Moore et al. 2013) with two degrees of freedom. The whole data processing described above can be achieved, for example, with the SciPy library function “normaltest” (Jones et al. 2001). This diagnostic offers another view on the impact of the divergence on the ensemble of background states but does not provide any information on the cause of the divergence.
4. The manifestation of divergence
a. The catastrophic divergence of 2021
1) The pathological background states of 1200 UTC 12 October 2021
Given the dramatic differences encountered in the upper stratospheric wind distribution of GEPS-New-Static near the equator as previously shown in Fig. 1, the presence and the scale of the divergence is examined here using zonal-mean zonal winds between 10°N and 10°S. The display of these values for each of the ensemble members (Fig. 2a), with the frontiers between each subensemble overlaid, clearly demonstrates the pathological background states in the 8th subensemble of GEPS-New-Static at 1200 UTC 12 October 2021, though some larger than average in-subensemble variabilities are also visible in subensembles 2 and 7. Note that Fig. 1a (member 254) corresponds to the strongest easterly (largest negative value) reaching 200 kt while Fig. 1b (member 234) corresponds to the strongest westerly (largest positive value) of about 90 kt found in the ensemble. Applying (4) on the zonal averages shown in Fig. 2a demonstrate that all the 32 members in subensemble 8 are correctly labeled outliers with this approach (Fig. 2b), with absolute values of modified Z score exceeding 40 for the three members exhibiting an easterly regime. Ten other members in subensembles 2 and 7 also somewhat exceed the adopted threshold, consistent with the variability of the zonal wind averages seen in these subensembles.
The (a) zonal-mean zonal winds at 3 hPa over the tropics (10°N–10°S) and the (b) corresponding absolute value of the modified Z score for GEPS-New-Static background states valid at 1200 UTC 12 Oct 2021. In (a), red corresponds to a westerly flow regime while blue corresponds to an easterly regime. In (b), values below 3.5 are colored in green while values above 3.5 are colored in purple. The dashed lines represent the frontiers between each of the 8 subensembles of 32 members and the numbers on the right (in gray) indicate each subensemble identification number.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-22-0108.1
Applying the normality test described in section 3 to the distribution of zonal averages of zonal wind shown in Fig. 2a returns a null probability that the tropical upper stratospheric circulation in the background states of GEPS-New-Static valid at 1200 UTC 12 October 2021 was drawn from a Gaussian distribution.
2) The evolution over time
The emergence and the evolution of the divergence in the real-time execution of GEPS-New-Static is examined here by looking at the diagnostics based on zonal means as shown above since the start of the execution of GEPS-New-Static in parallel to the then-operational version, GEPS-Old-Static, on 0000 UTC 2 April 2021, more than 6 months before the catastrophic filter divergence apex.
Displaying the absolute value of the modified Z score for the zonal-mean zonal winds over the tropics with a heat map approach (Fig. 3a) reveals that the first larger-than-normal values appeared in subensemble 7 (members 193–224) in mid-May, just a month after the start of the cycle. The first signal of zonal wind divergence in subensemble 8 appears a month later, in mid-June, while the divergence in subensemble 7 decayed temporarily and as minor divergences show up briefly in subensemble 2 (members 33–64) and 4 (members 97–128). The unique aspect about subensemble 8 is that the divergence persists and amplifies over time, particularly in three specific ensemble members, one of which (member 254) experienced the 1200 UTC 12 October 2021 model failure. The pathological behavior of this limited subset of members is better illustrated by Fig. 4a. The three members exhibiting absolute values of modified Z score greater than 40 near the end of the period are associated with an Easterly wind regime over the tropics as shown in Fig. 2a. In comparison, the absolute values of the modified Z score for GEPS-Old-Static over the same period do not indicate any divergent behavior across the ensemble (Figs. 3b and 4b).
The absolute value of the modified Z score for the zonal average of the zonal winds at 3 hPa over the tropics (10°N–10°S) for GEPS background states valid from 0000 UTC 2 Apr 2021 to 0000 UTC 15 Oct 2021, every 24 h. (a) GEPS-New-Static and (b) GEPS-Old-Static. The values in (a) stop at 0000 UTC 12 Oct 2021 due to the catastrophic filter divergence that occurred in the following 24 h in that experiment. The dashed lines represent the frontiers between each of the 8 subensembles of 32 members and the numbers on the right (in gray) indicate each subensemble identification number.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-22-0108.1
Time series of various statistics for the zonal-mean zonal winds at 3 hPa over the tropics (10°N–10°S) for GEPS background states valid from 0000 UTC 2 Apr 2021 to 0000 UTC 15 Oct 2021, every 24 h. (a),(b) The absolute value of the modified Z score for each of the ensemble members where each member is represented by a line of a different color. (c),(d) Numbers of outliers, i.e., number of ensemble members with an absolute value of the modified Z score greater than 3.5. (e),(f) The estimated probabilities that the ensembles members of the GEPS are coherent with a Gaussian distribution (note that the code used to perform this test returned a null value when the computed probability was smaller than 10%–6%). (left) GEPS-New-Static. (right) GEPS-Old-Static. The values in the left column stop at 0000 UTC 12 Oct 2021 due to the catastrophic filter divergence that occurred in the following 24 h in that experiment.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-22-0108.1
The time evolution of the number of outliers is shown in Fig. 4c, defined by the number of ensemble members with an absolute value of the modified Z score greater than 3.5 (section 3). Despite the relatively modest signal of the mid-May divergence event in subensemble 7 (Fig. 3a), 15 members could be identified as outliers (Fig. 4c). When subensemble 8 began to diverge in mid-June, the number of outliers rose rapidly above 20 and remained above that value for the remainder of the cycle. The fact that outlier counts peak above 40 demonstrates that more than one subensemble can be impacted by outliers at the same time (each subensemble contains only 32 members). In GEPS-Old-Static (Fig. 4d), only one to three ensemble members occasionally and barely exceed the somewhat subjective threshold adopted to be labeled as outliers, which is expected even for random samples drawn from a perfectly Gaussian distribution. Indeed, the level of false alarm from a Gaussian distribution when using a threshold of 3.5 for the absolute value of the modified Z score was empirically estimated to 15% when using 106 realizations of a 256-member randomly generated distribution. In comparison, this threshold is exceeded by at least one member 25% of the time in GEPS-Old-Static.
As for the probability that the ensemble zonal-mean zonal winds are consistent with a Gaussian distribution, the time evolution of this probability is coherent with the number of estimated outliers. Probabilities remained >1% throughout most of the first month of the GEPS-New-Static cycle (Fig. 4e), but dropped dramatically following the emergence of divergence in subensemble 7. The probability rose briefly to 0.1% in mid-June but reverted back to near-zero values3 when subensemble 8 began to diverge. In comparison, the probabilities in GEPS-Old-Static (Fig. 4f) were usually well above 1% apart from briefs drops, in particular in April, for which we do not have an explanation.
The results presented here have focused on the zonal winds around the equator at 3 hPa. Signatures of divergence were also detected in the temperature and the meridional wind fields of GEPS-New-Static at the same level but showed up gradually only after manifesting first in the zonal wind (not shown). The dominance of zonal circulation anomalies is believed to be related to the semiannual oscillation, a natural mode of the upper stratosphere that is characterized by transitions from strong easterly to strong westerly flow in the equatorial band (Hamilton and Mahlmann 1988). The low density of air parcels at this level makes such rapid accelerations possible through the actions of gravity waves, eddy-flux forcing and advection (Ern et al. 2021). Though the subensemble divergence clearly emerged in the tropics, it eventually contaminated extratropical regions when subensemble 8 diverged considerably from mid-July onward. Finally, all divergences found in GEPS-New-Static for winds and temperature were confined to the upper stratosphere, with maxima between 1 and 3 hPa.
Note that many significant changes were introduced in the new version of the GEPS and identifying those responsible for triggering these divergences was outside the scope of this paper. Nevertheless, one hypothesis will be discussed later in section 6.
b. Minor occurrences
Before testing a new version of the GEPS in real-time, experiments on retrospective periods are always conducted. In preparation for proposing the GEPS-New-Static configuration, two 2.5-month-long analysis and forecast cycles were employed, one over the boreal summer of 2019 (from 0000 UTC 13 June 2019 to 0000 UTC 1 September 2019) and the other over the boreal winter of 2020 (from 0000 UTC 12 December 2019 to 0000 UTC 1 March 2020). Applying the divergence diagnostics developed in section 3 to the ensemble of background states of the equatorial zonal-mean zonal wind at 3 hPa revealed no sign of divergence during the boreal winter cycle (not shown). However, during the boreal summer cycle, both GEPS-New-Static and GEPS-Old-Static exhibit a clear signature of divergence in subensemble 7 (Figs. 5 and 6).
As in Fig. 3, but from 0000 UTC 13 Jun 2019 to 0000 UTC 1 Sep 2019.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-22-0108.1
As in Fig. 4, but from 0000 UTC 13 Jun 2019 to 0000 UTC 01 Sep 2019.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-22-0108.1
As in 2021, the divergence in GEPS-New-Static in the boreal summer of 2019 starts a few weeks after the beginning of the cycle. However, the divergence is always confined to subensemble 7 (Fig. 5a) and the absolute values of modified Z scores never exceed 20 (Fig. 6a). Similarly, the number of ensemble members deemed an outlier never exceeds 28 (Fig. 6c), indicating that the members of subensemble 7 were never all considered to be outliers at the same time. As in 2021, the probability that the ensemble of zonal-mean zonal winds are consistent with a Gaussian distribution (Fig. 6e) decreased rapidly when outliers started to appear and fell to near-zero from mid-July onward.
The divergence event in GEPS-Old-Static is quite different from those documented in the GEPS-New-Static system. First, it starts only a few days after the beginning of the cycle and gradually vanishes after peaking in the second week of July (Fig. 5b). The absolute values of the modified Z scores are also very similar over time in all the members of the diverging subensemble (7; Fig. 6b). Consequently, all the 32 members of this subensemble are considered to be outliers during most of the event (Fig. 6d). The probability that the ensemble zonal-mean zonal winds are consistent with a Gaussian distribution (Fig. 6f) rebounds from near-zero to exceed 1% in August, demonstrating that a divergence event can be transient. These results show that GEPS-New-Static is not the only configuration prone to filter divergence and suggest that other (undetected) transient events have almost certainly occurred in the operational system (GEPS-Old-Static).
The relative uniformity of the large absolute values of the modified Z scores found in subensemble 7 of GEPS-Old-Static (Fig. 5b) can be explained by the fact that similar easterly flow departures developed in all subensemble members (Fig. 7a). However, variability within the subensemble remained similar to that observed across the remainder of the ensemble. In contrast, the spread in the diverging subensemble of GEPS-New-Static was notably larger than that of the other subensembles (Figs. 7b and 2a), which explains the large variability in the absolute values of the modified Z scores. The different divergence behavior is clearly illustrated by comparing the in-subensemble mean and spread for these two experiments (Fig. 8). However, the root-cause of these differences in divergence behavior between GEPS-Old-Static and GEPS-New-Static remains unknown. Their examination as well as the understanding of the mechanisms that trigger, drive and, sometimes, stop these filter divergence events are outside the scope of this paper. Nevertheless, the changes introduced in GEPS-New-Static that may favor the formation of outliers will be discussed in section 6.
As in Fig. 2a, but for (a) GEPS-Old-Static background states valid at 0000 UTC 6 Jul 2019 and (b) GEPS-New-Static background states valid at 0000 UTC 6 Aug 2019. These validity times correspond to the moments when the largest absolute values of the modified Z score were found in each of these experiments in the boreal summer of 2019 as shown in Fig. 5 and Figs. 6a and 6b.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-22-0108.1
The in-subensemble (a),(b) mean and (c),(d) spread (i.e., standard deviation) for (left) GEPS-Old-Static background states valid at 0000 UTC 6 Jul 2019 and (right) GEPS-New-Static background states valid at 0000 UTC 6 Aug 2019. These validity times correspond to one shown in Fig. 7 for the same two experiments.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-22-0108.1
Unlike the catastrophic filter divergence event observed in GEPS-New-Static in 2021, the minor divergence observed in the zonal-mean zonal winds around the equator in GEPS-New-Static and GEPS-Old-Static in the boreal summer of 2019 did not propagate outside this region. The impact on the zonal averages of the temperature was also rather smaller than in 2021 and no sign of divergence was noted in the zonal averages of the meridional wind (not shown).
Finally, we remark that all divergences reported here showed up first in subensemble 7. One hypothesis for the boreal summer of 2019 experiments is that some kind of latent signal was present in the initial ensemble of background states—that is shared between these cycles—and favored this divergence. As for the catastrophic filter divergence event of 2021, it is likely that simply chance favored again the appearance of the filter divergence in subensemble 7 as nothing in the system, as far as we can see, could introduce a bias toward one specific subensemble.
5. Randomized subensembles
a. Rationale
Until now, all implemented or tested cross-validation schemes use static subensemble membership: members are assigned to the same subensemble throughout the DA cycle (e.g., Houtekamer et al. 2005; Bowler et al. 2013); however, this approach is simply a pragmatic choice. To reduce the risk of filter divergence, we propose instead to randomly distribute the ensemble members over the 8 subensembles4 at each analysis step. With this new strategy, it should not be possible for a group of ensemble members to behave differently in a systematic way simply because groupings are ephemeral. However, in theory, a single member could still diverge if the errors that develop within it are poorly described by the error-covariances in the remaining ensemble members. The likelihood of such an event occurring are thought to be remote in GEPS-New configurations because of the exchangeability of members afforded by the removal of the multiphysics-based model uncertainty representation (McTaggart-Cowan et al. 2022b). In GEPS-Old-Static, the limited number of different model configurations adopting fixed parameters were carefully assigned to avoid duplication in each fixed subensemble. Because the use of randomly formed subensembles could occasionally introduce intergroup biases under such conditions (e.g., all ensemble members sharing the same model configuration could briefly end up in the same subensemble), the results discussed here focus uniquely on the introduction of the randomized subensemble generation approach in GEPS-New-Static (a configuration hereafter referred to GEPS-New-Random).
b. Impact on the occurrences of divergence
GEPS-New-Random experiments were performed over the reference retrospective periods of boreal summer 2019 and boreal winter 2021 (section 4b). Unlike GEPS-New-Static, no sign of divergence shows up in the zonal-mean zonal winds over the tropics during boreal summer 2019 (Fig. 9a and the left column in Fig. 10) nor in any other variables and regions (not shown). In Fig. 10c, one to two outliers are present about 30% of the time, close to the 25% rate seen in GEPS-Old-Static in 2021 (see Fig. 4d again) and only twice the expected rate from a perfectly Gaussian distribution. The probabilities that the ensemble of zonal-mean zonal winds are consistent with a Gaussian distribution remains well above 1% (Fig. 10e), except for a two-week period in August when a single member slightly exceeds the outlier threshold value (Figs. 10a,c). During the experiment over the boreal winter of 2021, as in GEPS-New-Static, no divergence was observed (not shown).
As in Fig. 3, but for GEPS-New-Random (a) from 0000 UTC 13 Jun 2019 to 0000 UTC 1 Sep 2019 and (b) from 0000 UTC 26 Aug 2021 to 0000 UTC 28 Oct 2021. The dashed lines that represented the frontiers between each of the 8 subensembles were removed because the subensembles are randomly formed in GEPS-New-Random. Note that in (b), the GEPS-New-Random cycle was initialized with ensemble of background stated from the real-time testing of GEPS-New-Static, when the ensemble members in subensemble 8 had significantly diverged.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-22-0108.1
As in Fig. 4, but for GEPS-New-Random (a) from 0000 UTC 13 Jun 2019 to 0000 UTC 1 Sep 2019 and (b) from 0000 UTC 26 Aug 2021 to 0000 UTC 28 Oct 2021. Note that in the right column, the GEPS-New-Random cycle was initialized with the data from the real-time testing of GEPS-New-Static, when the ensemble members in subensemble 8 had significantly diverged.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-22-0108.1
These results suggest that randomized subensembles can prevent the onset of filter divergence, though the sample size is rather limited. To further test the robustness of this new approach, we initialized a new GEPS-New-Random experiment by providing an ensemble of background states where a group of members had diverged. This experiment was carried out in the boreal summer of 2021 and used as inputs the ensemble of background states from the real-time testing of GEPS-New-Static valid at 0000 UTC 26 August 2021, when the members of subensemble 8 in that experiment had already significantly diverged (see again Fig. 3a and the left panels of Fig. 4). The randomized-subensembles approach removes most of the 30 initial outliers within a few days (Fig. 10d) and the ensemble becomes free of any persistent outliers after about one month as the probability of normality rises well above 1% (Fig. 10f). The absolute values of the modified Z scores from the ensemble members within the divergent subensemble do not decrease uniformly (Figs. 9b and 10b): some members converge faster than others. As a result, only a single member exceeds the outlier threshold value after mid-September (Fig. 10b) but, nevertheless, the zonal-mean zonal winds over the tropics from that member continue to gradually align with the rest of the ensemble. These results provide further evidence that randomized subensembles can prevent filter divergence, not only for a group of ensemble members but also for a single ensemble member.
c. Impact on the comparison to observations
The impact of the randomly formed subensembles approach on the fit to the observations of the GEPS was first evaluated by comparing the background ensemble mean to radiosonde observations of temperature and zonal wind from the surface up to 10 hPa. Assessment of error standard deviations and biases shows no measurable difference in skill between GEPS-New-Static and GEPS-New-Random for boreal summer (Fig. 11) or winter (not shown). Unfortunately, no direct wind observations are available around 3 hPa. It is therefore not possible to assess directly the impact of the removal of the divergence in the zonal winds around the equator during the boreal summer of 2019. However, temperature-sensitive satellite observations are available in the upper stratosphere. Observations from the Advanced Microwave Sounding Unit-A (AMSU-A; Mo 1996) between 20°N and 20°S for 11 channels with estimated peak response levels ranging from 850 to 2 hPa again reveal no discernable differences (Fig. 12).
Verification for the background ensemble mean from the GEPS-New-Random (red) and GEPS-New-Static (blue; mostly hidden by the red lines representing GEPS-New-Random results) experiments against radiosondes over (top) Northern Hemisphere extratropics (20°–90°N), (middle) tropics (20°N–20°S), and (bottom) Southern Hemisphere extratropics (20°–90°S). The standard deviation (solid lines) and bias (forecast minus observation; dashed lines) are shown for (left) zonal wind and (right) temperature. Scores are averaged over the 2.5-month-long cycle in boreal summer 2019.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-22-0108.1
Verification for the background ensemble mean from the GEPS-New-Random (red) and GEPS-New-Static (blue; mostly hidden by the red lines representing GEPS-New-Random results) experiments against AMSU-A brightness temperatures between 20°N and 20°S. The standard deviation (solid lines; normalized by GEPS-New-Static values) and biases (forecast minus observation; dashed lines) are shown for the 11 channels assimilated in the GEPS with channel numbers shown on the left and the corresponding estimated pressure level of maximum response shown on the right. Scores are averaged over the 2.5-month-long cycle in boreal summer 2019.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-22-0108.1
The 15-day forecasts were then executed at 0000 and 1200 UTC using 20 out of the 256 analyses for GEPS-New-Random and GEPS-New-Static. Their quality was assessed by comparing them to both radiosonde and surface observations using the probabilistic verification procedures described in Candille et al. (2007). Again, no significant changes were noted when using randomly formed subensembles in both boreal summer and winter periods (not shown). Given the neutral results, GEPS-New-Random was accepted for operational use at ECCC from 1 December 2021.
6. Summary and discussion
The first known occurrence of catastrophic ensemble filter divergence in an NWP context resulting from the use of the cross-validation approach was documented in this study. The event occurred during the real-time testing of the latest upgrade to the GEPS (GEPS-New-Static) at ECCC (2021). In May 2021, about a month after the start of the DA cycle, outliers developed in the background states of the zonal winds in the upper stratosphere around the equator. At the same time, the probability that the ensemble was drawn from a Gaussian distribution dramatically dropped. Later, the divergence migrated from one subensemble to another subensemble and gradually amplified in the latter while oscillating in the former. About five months later, the zonal wind differences in the upper stratosphere near the equator between the two most extreme ensemble members reached 500 kt. A few forecast hours after the ensemble forecast initialized of 1200 UTC 12 October 2021, uncontrolled upper-stratospheric warming led to unrealistic temperatures and model failure. The then-operational version of the GEPS (GEPS-Old-Static) did not show any sign of subensemble divergence throughout this period.
However, when applying the same diagnostics to the same two configurations of the GEPS executed on two retrospective periods, divergence was identified in both GEPS configurations during the boreal summer period of 2019. In GEPS-New-Static, the signature of the divergence was similar to the 2021 event but the magnitude and the number of outliers was much smaller than in 2021. Still, the probability that the ensemble was drawn for a Gaussian distribution became persistently near-zero. It is not clear how this divergence event would have evolved if this experiment had been continued beyond 2.5 months, the duration of pre-real-time tests for new versions of NWP systems at ECCC. On the other hand, the divergence in GEPS-Old-Static occurred only in the first half of the experimental period, demonstrating that a filter divergence event can sometimes be transient. These results show that GEPS-New-Static is not the only configuration prone to divergence events and suggest that other transient events have probably occurred in the past in the operational system.
Many significant changes were introduced in the new version of the GEPS and identifying those responsible for the changes in divergence behavior was outside the scope of this paper. Nevertheless, it is likely that moving away from perturbing the observations increased the likelihood and the magnitude of the divergences. This is supported by the idealized experiments with low-dimensional models performed by Buehner (2020) who found that the LETKF with cross validation and no perturbations to the observations was much more likely to have outliers than approaches with perturbed observations. In addition, Lawson and Hansen (2004) showed that deterministic filters are more susceptible to enhanced non-Gaussianity, and thus outliers, than stochastic filters. Still, we believe that deterministic filters have the advantage of not introducing additional noise through the random observation perturbations, which should be beneficial with relatively small ensemble sizes. We therefore decided to keep using the deterministic version of the LETKF with cross validation but with the modification described below.
Instead of always assigning each ensemble member to the same subensemble, we propose randomly assigning ensemble members to the 8 subensembles of the GEPS at each analysis step. This new approach has been tested in the context of the updated version of the GEPS (GEPS-New-Random). It prevented the development of filter divergence previously observed during the experiment over the boreal summer of 2019 and it gradually removed divergence introduced in the initial ensemble of background states in a sensitivity experiment initialized in August 2021. A comparison against observations for the experiment in boreal summer 2019 showed no impact on the short or long-range forecast quality both in the troposphere and in the stratosphere, though no direct wind observations are available at the height of the divergence manifestation. The success of this new approach to prevent filter divergence, combined with the absence of impacts on previously performed and accepted verifications, permitted the operational implementation of GEPS-New-Random at ECCC on 1 December 2021. Since then, a real-time monitoring system of filter divergence based on the diagnostics presented in this paper has been implemented. After more than 6 months of operation, the number of outliers identified follows what was previously observed with this configuration in the retrospective experiments of 2019 and 2020, implying that no signature of filter divergence is evident.
Despite the success of the randomized-subensembles approach, it is still unclear if this strategy is the best one to eliminate any risk of filter divergence. In theory, a single member could still diverge despite the encouraging results presented in this paper. However, if only a single member diverges, this is less likely to lead to a catastrophic divergence because the impact on the ensemble covariances will be much less that when an entire static subensemble diverges. We plan to use low-dimensional models to examine the robustness of this approach in more detail.
Finally, we believe that the upper stratospheric circulation near the equator is prone to experience filter divergence due to the absence of direct wind observations assimilated at this altitude and the absence of geostrophic temperature-wind correlations in the tropics. On the other hand, the driving mechanisms of the different filter divergence events reported in this paper remain largely unknown. Experiments to shed light on these mechanisms in a high-dimensional context and to better understand their possible manifestation near the equator are thus recommended.
This analysis is completely independent from the pathological GEPS, i.e., the ensemble-derived background-error covariances used in the operational GDPS come from the then operational version of the GEPS.
The sum of the weights intentionally exceeds 1 and is not an unusual choice (e.g. Clayton et al. 2013). In fact, Ménétrier and Auligné (2015) stated that there is no objective reason to require that the weights sum to 1.
Note that the code used to perform this test returned a null value when the computed probability was smaller than 10−6%.
The number of subensembles was kept the same to not increase the computational cost of the analysis step as each different gain matrix computation involves an eigenvalue decomposition.
Acknowledgments.
The authors thank their colleagues Seung-Jong Baek, Xingxiu Deng, and Nicolas Gasset for their contributions to performing the DA and forecast cycles presented here as well as forecast verifications against observations and companion experiments. The authors also address a special additional thanks to Nicolas Gasset who first spotted that something unusual was happening during the real-time testing of the GEPS during the fall of 2021. His timely report made it possible to find a solution to the problem reported here shortly before the model abort event that occurred in October 2021. The authors would also like to thank the anonymous reviewers for their careful reading of our manuscript.
Data availability statement.
Due to the large volume of space requires (tens of terabytes), GEPS analyses and forecasts on the native grid are archived for a period of 2 years only. A subset could be made available to anyone upon request. However, the outlier diagnostics as well as the comparison against observations are more easily accessible and will be retained for a period of 5 years.
REFERENCES
Bishop, C., J. S. Whitaker, and L. Lei, 2017: Gain form of the ensemble transform Kalman filter and its relevance to satellite data assimilation with model space ensemble covariance localization. Mon. Wea. Rev., 145, 4575–4592, https://doi.org/10.1175/MWR-D-17-0102.1.
Bowler, N. E., J. Flowerdew, and S. R. Pring, 2013: Tests of different flavours of EnKF on a simple model. Quart. J. Roy. Meteor. Soc., 139, 1505–1519, https://doi.org/10.1002/qj.2055.
Buehner, M., 2020: Local ensemble transform Kalman filter with cross validation. Mon. Wea. Rev., 148, 2265–2282, https://doi.org/10.1175/MWR-D-19-0402.1.
Buehner, M., J. Morneau, and C. Charette, 2013: Four-dimensional ensemble-variational data assimilation for global deterministic weather prediction. Nonlinear Processes Geophys., 20, 669–682, https://doi.org/10.5194/npg-20-669-2013.
Buehner, M., and Coauthors, 2015: Implementation of deterministic weather forecasting systems based on ensemble-variational data assimilation at Environment Canada. Part I: The global system. Mon. Wea. Rev., 143, 2532–2559, https://doi.org/10.1175/MWR-D-14-00354.1.
Buizza, R., M. Miller, and T. N. Palmer, 1999: Stochastic representation of model uncertainties in the ECMWF ensemble prediction system. Quart. J. Roy. Meteor. Soc., 125, 2887–2908, https://doi.org/10.1002/qj.49712556006.
Candille, G., C. Côté, P. L. Houtekamer, and G. Pellerin, 2007: Verification of an ensemble prediction system against observations. Mon. Wea. Rev., 135, 2688–2699, https://doi.org/10.1175/MWR3414.1.
Charron, M., G. Pellerin, L. Spacek, P. L. Houtekamer, N. Gagnon, H. L. Mitchell, and L. Michelin, 2010: Toward random sampling of model error in the Canadian Ensemble Prediction System. Mon. Wea. Rev., 138, 1877–1901, https://doi.org/10.1175/2009MWR3187.1.
Clayton, A. M., A. C. Lorenc, and D. M. Barker, 2013: Operational implementation of a hybrid ensemble/4D-Var global data assimilation system at the Met Office. Quart. J. Roy. Meteor. Soc., 139, 1445–1461, https://doi.org/10.1002/qj.2054.
D’Agostino, R., and E. S. Pearson, 1973: Tests for departure from normality. Empirical results for the distributions of b2 and √b1 Biometrika, 60, 613–622, https://doi.org/10.1093/biomet/60.3.613.
ECCC, 2019: Upgrade of the Global Ensemble Prediction System (GEPS) from version 5.0.0 to version 6.0.0. Tech. Note, ECCC, 72 pp., https://collaboration.cmc.ec.gc.ca/cmc/cmoi/product_guide/docs/tech_notes/technote_geps-600_e.pdf.
ECCC, 2021: Upgrade of the Global Ensemble Prediction System (GEPS) from version 6.1 to version 7.0. Tech. Note, ECCC, 110 pp., https://collaboration.cmc.ec.gc.ca/cmc/cmoi/product_guide/docs/tech_notes/technote_geps-700_e.pdf.
Ern, M., M. Diallo, P. Preusse, M. G. Mlynczak, M. J. Schwartz, Q. Wu, and M. Riese, 2021: The Semiannual Oscillation (SAO) in the tropical middle atmosphere and its gravity wave driving in reanalyses and satellite observations. Atmos. Chem. Phys., 21, 13 763–13 795, https://doi.org/10.5194/acp-21-13763-2021.
Girard, C., and Coauthors, 2014: Staggered vertical discretization of the Canadian Environmental Multiscale (GEM) model using a coordinate of the log-hydrostatic-pressure type. Mon. Wea. Rev., 142, 1183–1196, https://doi.org/10.1175/MWR-D-13-00255.1.
Hamill, T. M., and C. Snyder, 2000: A hybrid ensemble Kalman filter-3D variational analysis scheme. Mon. Wea. Rev., 128, 2905–2919, https://doi.org/10.1175/1520-0493(2000)128<2905:AHEKFV>2.0.CO;2.
Hamilton, K., and J. D. Mahlmann, 1988: General circulation model simulation of the semiannual oscillation of the tropical middle atmosphere. J. Atmos. Sci., 45, 3212–3235, https://doi.org/10.1175/1520-0469(1988)045<3212:GCMSOT>2.0.CO;2.
Houtekamer, P. L., 2011: The use of multiple parameterizations in ensembles. Proc. ECMWF Workshop on Representing Model Uncertainty and Error in Numerical Weather and Climate Prediction Models, Shinfield Park, Reading, United Kingdom, ECMWF, 163–173.
Houtekamer, P. L., and H. L. Mitchell, 1998: Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev., 126, 796–811, https://doi.org/10.1175/1520-0493(1998)126<0796:DAUAEK>2.0.CO;2.
Houtekamer, P. L., and F. Zhang, 2016: Review of the ensemble Kalman filter for atmospheric data assimilation. Mon. Wea. Rev., 144, 4489–4532, https://doi.org/10.1175/MWR-D-15-0440.1.
Houtekamer, P. L., L. Lefaivre, J. Derome, H. Ritchie, and H. L. Mitchell, 1996: A system simulation approach to ensemble prediction. Mon. Wea. Rev., 124, 1225–1242, https://doi.org/10.1175/1520-0493(1996)124<1225:ASSATE>2.0.CO;2.
Houtekamer, P. L., H. L. Mitchell, G. Pellerin, M. Buehner, M. Charron, L. Spacek, and B. Hansen, 2005: Atmospheric data assimilation with an ensemble Kalman filter: Results with real observations. Mon. Wea. Rev., 133, 604–620, https://doi.org/10.1175/MWR-2864.1.
Houtekamer, P. L., H. L. Mitchell, and X. Deng, 2009: Model error representation in an operational ensemble Kalman filter. Mon. Wea. Rev., 137, 2126–2143, https://doi.org/10.1175/2008MWR2737.1.
Houtekamer, P. L., M. Buehner, and M. De La Chevrotière, 2018: Using the hybrid gain algorithm to sample data assimilation uncertainty. Quart. J. Roy. Meteor. Soc., 145, 35–56, https://doi.org/10.1002/qj.3426.
Iglewicz, B., and D. C. Hoaglin, 1993: How to detect and handle outliers. The American Society for Quality Control (ASQC) Basic References in Quality Control: Statistical Techniques, Vol. 16, E. F. Mykytka, Ed., American Society for Quality Control, 77 pp.
Jones, E., and Coauthors, 2001: SciPy: Open source scientific tools for Python. SciPy, accessed 14 April 2021, http://www.scipy.org/.
Lawson, W. G., and J. A. Hansen, 2004: Implications of stochastic and deterministic filters as ensemble-based data assimilation methods in varying regimes of error growth. Mon. Wea. Rev., 132, 1966–1981, https://doi.org/10.1175/1520-0493(2004)132<1966:IOSADF>2.0.CO;2.
Lorenz, E. N., and K. A. Emanuel, 1998: Optimal sites for supplementary weather observations: Simulation with a small model. J. Atmos. Sci., 55, 399–414, https://doi.org/10.1175/1520-0469(1998)055<0399:OSFSWO>2.0.CO;2.
McTaggart‐Cowan, R., and Coauthors, 2019: Modernization of atmospheric physics parameterization in Canadian NWP. J. Adv. Model. Earth Syst., 11, 3593–3635, https://doi.org/10.1029/2019MS001781.
McTaggart-Cowan, R., and Coauthors, 2022a: Using stochastically perturbed parameterizations to represent model uncertainty. Part I: Implementation and parameter sensitivity. Mon. Wea. Rev., 150, 2829–2858, https://doi.org/10.1175/MWR-D-21-0315.1.
McTaggart-Cowan, R., L. Separovic, M. Charron, X. Deng, N. Gagnon, P. L. Houtekamer, and A. Patoine, 2022b: Using stochastically perturbed parameterizations to represent model uncertainty. Part II: Comparison with existing techniques in an operational ensemble. Mon. Wea. Rev., 150, 2859–2882, https://doi.org/10.1175/MWR-D-21-0316.1.
Ménétrier, B., and T. Auligné, 2015: Optimized localization and hybridization to filter ensemble-based covariances. Mon. Wea. Rev., 143, 3931–3947, https://doi.org/10.1175/MWR-D-15-0057.1.
Mitchell, H. L., and P. L. Houtekamer, 2009: Ensemble Kalman filter configurations and their performance with the logistic map. Mon. Wea. Rev., 137, 4325–4343, https://doi.org/10.1175/2009MWR2823.1.
Miyoshi, T., Y. Sato, and T. Kadowaki, 2010: Ensemble Kalman filter and 4D-Var inter-comparison with the Japanese operational global analysis and prediction system. Mon. Wea. Rev., 138, 2846–2866, https://doi.org/10.1175/2010MWR3209.1.
Mo, T., 1996: Prelaunch calibration of the Advanced Microwave Sounding Unit-A for NOAA-K. IEEE Trans. Microwave Theory Tech., 44, 1460–1469, https://doi.org/10.1109/22.536029.
Moore, D. S., W. I. Notz, and M. A. Flinger, 2013: The Basic Practice of Statistics. 6th ed. W. H. Freeman and Company, 745 pp.
Palmer, T. N., R. Buizza, F. Doblas-Reyes, T. Jung, M. Leutbecher, G. J. Shutts, M. Steinheimer, and A. Weisheimer, 2009: Stochastic parametrization and model uncertainty. ECMWF Tech. Memo. 598, European Centre for Medium-Range Weather Forecasts, Reading, United Kingdom, https://doi.org/10.21957/ps8gbwbdv.
Penny, S. G., 2014: The hybrid local ensemble transform Kalman filter. Mon. Wea. Rev., 142, 2139–2149, https://doi.org/10.1175/MWR-D-13-00131.1.
Qaddouri, A., and V. Lee, 2011: The Canadian Global Environmental Multiscale model on the Yin-Yang grid system. Quart. J. Roy. Meteor. Soc., 137, 1913–1926, https://doi.org/10.1002/qj.873.
Schraff, C., H. Reich, A. Rhodin, A. Schomburg, K. Stephan, A. Periáñez, and R. Potthast, 2016: Kilometre-scale ensemble data assimilation for the COSMO model (KENDA). Quart. J. Roy. Meteor. Soc., 142, 1453–1472, https://doi.org/10.1002/qj.2748.
Shin, S., J. S. Kang, and Y. Jo, 2016: The Local Ensemble Transform Kalman Filter (LETKF) with a global NWP model on the cubed sphere. Pure Appl. Geophys., 173, 2555–2570, https://doi.org/10.1007/s00024-016-1269-0.
Shutts, G., 2005: A kinetic energy backscatter algorithm for use in ensemble prediction systems. Quart. J. Roy. Meteor. Soc., 131, 3079–3102, https://doi.org/10.1256/qj.04.106.
Smith, A. K., R. R. Garcia, A. C. Moss, and N. J. Mitchell, 2017: The semiannual oscillation of the tropical zonal wind in the middle atmosphere derived from satellite geopotential height retrievals. J. Atmos. Sci., 74, 2413–2425, https://doi.org/10.1175/JAS-D-17-0067.1.
Strogatz, S., 2015: Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry, and Engineering. 2nd ed. CRC Press, 532 pp., https://doi.org/10.1201/9780429492563.
Whitaker, J. S., and T. M. Hamill, 2012: Evaluating methods to account for system errors in ensemble data assimilation. Mon. Wea. Rev., 140, 3078–3089, https://doi.org/10.1175/MWR-D-11-00276.1.
Zhou, X., Y. Zhu, D. Hou, Y. Luo, J. Peng, and R. Wobus, 2017: Performance of the new NCEP Global Ensemble Forecast System in a parallel experiment. Wea. Forecasting, 32, 1989–2004, https://doi.org/10.1175/WAF-D-17-0023.1.
