1. Introduction
Forecast “busts” or “dropouts” can be defined as intermittent but significant losses of model forecast performance. Such events have been the focus of recent studies in the modeling, predictability, and data assimilation communities (e.g., Langland and Maue 2012; Rodwell et al. 2013; Lillo and Parsons 2017; Magnusson 2017; Grams et al. 2018). The aim of these studies has been to associate dropout events with deficiencies in the forecast model or data assimilation systems, or to provide insight into situations that demonstrate inherent predictability limits.
A particular theme emerging from these previous studies is the limits of predictability tied to large-scale pattern variability. The link between sudden drops in value of T + 120 h Northern Hemisphere (NH) 500-hPa geopotential height (Φ500) anomaly correlation coefficient (ACC) and large-scale pattern variability was explored by Langland and Maue (2012) using three forecast systems: the European Centre for Medium-Range Weather Forecasts (ECMWF) Integrated Forecast System (IFS), the National Centers for Environmental Prediction (NCEP) Global Forecast System (GFS), and the U. S. Navy Operational Global Atmospheric Prediction System (NOGAPS). They linked the ACC with transitions of the Arctic Oscillation (AO) between the negative and positive phases. Subsequently, Rodwell et al. (2013) studied dropouts over Europe using T + 144 h forecasts from the Interim ECMWF Re-Analysis (ERA-Interim; Dee et al. 2011). They constructed composites of several hundred dropout events and linked the events to flow patterns characterized by the presence of a mesoscale convective system (MCS) in the initial conditions over North America and blocked flow downstream over Europe. Lillo and Parsons (2017) extended the work of Rodwell et al. (2013) by applying an empirical orthogonal function (EOF) analysis to the same data. Their analysis linked the dropouts over Europe with a chain of events wherein the MCS over North America triggers a Rossby wave train that then amplifies across the Atlantic and initiates large-scale pattern transition. This analysis also showed seasonal peaks of forecast dropout occurrence in September–October associated with tropical cyclone recurvature and secondary peaks in June–July, consistent with Rodwell et al. (2013). Parsons et al. (2019) further elaborated on the relationship between the North American MCSs, Rossby wave packets, and dropouts, highlighting the importance of improved representation of deep convection. Separately, Magnusson (2017) investigated several cases of extreme error in the ECMWF Integrated Forecasting System (IFS) using a combination of three methods (manual error tracking, ensemble sensitivity, and relaxation experiments). Similar to Rodwell et al. (2013) and Lillo and Parsons (2017), who connected dropout cases to blocking pattern transitions, Magnusson (2017) related his three cases to the Scandinavian blocking pattern. The connection to blocking transition also emerged in Grams et al. (2018), who focused on a single severe forecast dropout in the ECMWF IFS that occurred in March 2016 and associated it with a European blocking regime. Recently, Yamagami and Matsueda (2021) highlighted once more the common thread between dropouts and large-scale patterns, linking dropouts over the Arctic in forecasts from five leading NWP centers to forecasts initialized on the Greenland blocking and Arctic cyclone patterns.
The above studies are diagnostic in nature. Several additional studies have explored frameworks to improve dropout predictability. Rodwell et al. (2018) prescribe improvements to forecast reliability, to remedy (among other things) dropout events over Europe that are linked to occasions when there is insufficient ensemble spread associated with MCSs over North America. Uno et al. (2018) used multicenter global ensemble spread to predict large forecast error for surface solar radiation. A project initiated between the Joint Center for Satellite Data Assimilation (JCSDA) and NCEP developed a Global Forecast Dropout Prediction Tool (GFDPT) to monitor and analyze regional differences between the NCEP and ECMWF global models (Kumar et al. 2016).
The patchwork of studies noted above has made important inroads into the dropout problem. Nonetheless, one observes that these studies are most often in the context of an individual model, and particularly the ECMWF model. Furthermore, while many of the studies suggest a link between dropouts and large-scale flow patterns, the results show strong case dependence. With these points in mind, this work seeks to build on the existing dropout literature by presenting an inter-model comparison of dropouts, and by further investigating the statistical links between dropouts and large-scale flow indices. The manuscript proceeds as follows: section 2 describes the data and methodology. In section 3, we use a short-term climatology to characterize the seasonality and magnitude of recent dropouts in the Navy Global Environmental Model (NAVGEM). We also present an inter-model comparison of dropouts across NAVGEM and the ensemble control members from the Environment and Climate Change Canada (ECCC) Global Ensemble Prediction System (GEPS, Lin et al. 2016) and the NCEP Global Ensemble Forecast System (GEFS, Zhou et al. 2017), to assess whether the NAVGEM dropouts are due to inherent limits of predictability or to a deficiency of the NAVGEM model. In section 4, we use compositing procedures and other diagnostics to explore the statistics of large-scale pattern variability surrounding dropout events, with a focus on the temporal variance and correlation of standard flow indices. Finally, in section 5 we survey the performance of the NAVGEM ensemble in conjunction with several severe dropout events. Section 6 follows with concluding remarks.
2. Data and methodology
a. Data
NAVGEM is a primitive equation spectral atmospheric model with a hydrostatic, three-time level, semi-Lagrangian/semi-implicit dynamical core (Hogan et al. 2014). We accumulated operational NAVGEM deterministic fields and verification statistics for a period dating from 1 April 2014 to 1 January 2018 at times of 0000 and 1200 UTC. This short-term climatology covers several model operational upgrades, notably an increase in resolution from T359L50 (37 km, 50 vertical levels) to T425L60 (31 km, 60 vertical levels, operational on 15 June 2015).
For inter-model comparisons, we leverage the THORPEX Interactive Grand Global Ensemble (TIGGE) archive (Richardson 2005), hosted at ECMWF. Both comparisons are performed using the ensemble control members from the NCEP GEFS and ECCC GEPS. For the GEFS, the control member has been run at TL574 (33 km) resolution out to forecast day 8, since December 2015. While not an exact match to the NAVGEM control resolution, we are assuming that GEFS (33 km) and NAVGEM (31 km, 60 vertical levels) are of sufficiently comparable resolution to produce an informative comparison without being overly dominated by resolution differences. The slightly lower resolution GEPS control member has been run at 50-km resolution since November 2014. See Table 1 for a complete list of resolution upgrades over this period of study. For both the GEFS and GEPS, the control member is run without initial perturbations, parameter modification, or stochastic forcing. Finally, we note that while Table 1 focuses on resolution, there were significant data assimilation upgrades to all systems during this time period. For NAVGEM, hybrid 4DVar became operational in 2016, with ensemble perturbations created using the ensemble transform (ET, McLay et al. 2010). For the GEFS, the ensemble Kalman filter (EnKF) replaced the ET with rescaling technique in 2015. Also in 2015, new stochastic physics schemes were implemented in the EnKF to represent model error. GEPS version 4, which was operational in 2015, included EnKF based initial conditions and multiparameterization physics.
Resolution changes during the period of study: 1 Apr 2014–1 Jan 2018.
b. Metrics and verification
For verifying analyses, we use self-analysis or ECMWF analyses (9-km resolution) obtained on 1° × 1° grid from the TIGGE archive. The results shown are with respect to self-analysis, unless indicated otherwise in order to remove the impact of differences in the initial condition.
c. Definition of “dropout”
We define forecast dropouts in two ways: 1) a constant threshold definition implies that the ACC has fallen below a predefined constant (e.g., ACC falls below 0.8) and 2) a percentile threshold definition implies that the ACC has fallen below a given percentile (e.g., ACC falls below the lowest 10th percentile). When percentile based thresholds are used, they are defined for a particular model and year, and by design, result in each center having the same number of dropout cases.
3. Initial findings from a short-term climatology
We initially focus on a short-term climatology to assess seasonality and any influence or statistical signature of model upgrades. This short-term climatology also serves as the basis for an inter-model comparison.
a. Seasonality of dropouts for the NAVGEM model
Figure 1 shows the seasonality of those T + 120 h ACC that drop below a percentile based threshold (defined by year). For Fig. 1, we consider 12-hourly intervals between successive valid times and define “recovery time” to be the total number of consecutive hours (based on 12-hourly data) in which the ACC remains below a predefined threshold. Therefore, a single dropout event can be counted several times if the recovery time exceeds 12 h (e.g., twice if recovery time was 24 h). Figure 1 shows that the number of dropout cases has clear seasonality, with a primary peak in NH summer and a secondary peak in NH fall. This is in agreement with the seasonality of dropouts in the ECMWF model, as discussed by Lillo and Parsons (2017). Stratifying by severity of the dropout, we see a similar seasonal pattern for even the most severe cases. We note that while most years show primary peaks in June–July and secondary peaks in September–October, we see summer peaks pushed later in the season for 2016, with drops below the 10th percentile peaking in August (Fig. 1c). Finally, we note that 2016 was a more predictable year. For NAVGEM, the 10th percentile threshold was found to be 0.808 over the entire period, 0.785 for 2014 after 1 April, 0.796 for 2015, 0.842 for 2016, and 0.806 for 2017. For reference, the 10th percentile ACC thresholds for each model considered are shown in Table 2. When we use a threshold based definition of 0.8 (not shown), the peaks for 2016 move to September and December, in agreement with the yellow bars in Fig. 1.
(a) The number of T + 120 h Φ500 ACC scores (using self-analyses as verification) that dropped below the percentile-based threshold as a function of month, shown for NAVGEM in the year 2014. Color shades indicate percentile based threshold bins: below 10th percentile (blue), below 7.5th percentile (red), below 5th percentile (green), and below 2.5th percentile (yellow). (b) As in (a), but for the year 2015. (c) As in (a), but for the year 2016. (d) As in (a), but for the year 2017.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-21-0208.1
The 10th percentile of ACC scores calculated for NAVGEM, NCEP GEFS control, and ECCC GEPS control by time period.
b. Inter-model comparison
In this subsection, we focus on the comparisons between the NAVGEM deterministic forecast and the GEFS and GEPS control members obtained from the TIGGE database. For multimodel comparisons, we exclude all date–time groups where any model is missing data resulting in 590 T + 120 h forecasts (0000 and 1200 UTC) for 2015 and 513 T + 120 h forecasts for 2016. A large portion of data were missing for the GEPS control member on the TIGGE portal for 2017, we therefore exclude 2017 for comparisons using all models.
Figure 2 shows probability density functions (PDFs) of T + 120 h ACC for all three models (Fig. 2a), as well as PDFs of their paired differences (Fig. 2b). We find the probability distribution of ACC fairly similar between the models, with NAVGEM having a slightly lower mean, but the GEPS control member displaying the most variability. Qualitatively similar behavior holds when we exclude dates prior to the GEPS resolution upgrade (not shown). The PDF of paired differences (Fig. 2b) again shows qualitatively similar behavior between models with the mean (−0.017 for NAVGEM-GEFS, −0.010 for NAVGEM-GEPS) and standard deviation (0.038 for NAVGEM-GEFS, 0.041 for NAVGEM-GEPS) of differences being comparable. One can also compute the Pearson’s correlation coefficient between the models’ T + 120 h Φ500 ACC, after the data have been detrended and the seasonal mean has been removed. The correlation coefficient typically ranges between 0.5 and 0.6 for 2015 and 2016 (not shown).
(a) PDF of T + 120 h Φ500 ACC, and (b) paired ACC differences. Shown for NAVGEM (blue), ECCC GEPS control member (green), and NCEP GEFS control member (red). All dates are considered between 1 Apr 2014 and 1 Jan 2018 where data for all three models are available.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-21-0208.1
We now explore the persistence of forecast dropout events. For dropout events based on the 10th percentile (Fig. 3), we see that the models have fairly comparable recovery times. The longest recovery times are 96 h for 2015 (Fig. 3a) and 72 h for 2016 (Fig. 3b), both witnessed for the NAVGEM model.
(a) Histogram of the recovery time of dropout events based on the 10th percentile for NAVGEM in the year 2015. (b) As in (a), but for the year 2016. (c) As in (a), but for the NCEP GEFS control member. (d) As in (c), but for the year 2016. (e) As in (a), but for the ECCC GEPS control member. (f) As in (e), but for the year 2016.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-21-0208.1
Figure 4 shows the seasonality of those dropout events that took longer than 24 h to recover. The seasonal peaks in long recovery time events are consistent with the seasonal peaks in dropout events shown in Fig. 1. Further, the 2015 peaks in June and October (Fig. 4, left panels) and the 2016 peaks in August/September and December (Fig. 4, right panels) are consistent between models.
(a) Number of dropout events based on the 10th percentile in which the recovery times exceeded 24 h as a function of month, for the NAVGEM model in the year 2015. (b) As in (a), but for the year 2016. (c) As in (a), but for the NCEP GEFS control member. (d) As in (c), but for the year 2016. (e) As in (a), but for the ECCC GEPS control member. (f) As in (e), but for the year 2016.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-21-0208.1
We now turn our attention to whether different models experience dropouts at the same time. Figures 5a and 5b indicate how many models, of the three considered, experience a given percentile-based dropout at the same time. One finds that the proportion of dropout events that are experienced by more than one model is significant: For 2015, 36% of dropout events are experienced by more than one model, and for 2016 the percentage is 28%. Figures 5c and 5d show the same analysis by month. The peaks seen in Fig. 1b for 2015 (June and October) and Fig. 1c for 2016 (August and December) are also seen to be periods when multiple models experience dropout events. The behavior is consistent when we lower the threshold to the (model relative) 5th and 2.5th percentiles (not shown). We note that October 2015 displays a peak only for single model dropout events for the most severe cases.
(a) Number of (model-relative) percentile-based dropouts in the year 2015 that are experienced by one, two, and three models. (b) As in (a), but for the year 2016. (c) As in (a), but broken out as a function of month. (d) As in (b), but broken out as a function of month.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-21-0208.1
The inter-model comparisons performed in this section indicate that the ACC is correlated across the models, the shape of the ACC distribution is similar across the models, and the proportion of dropout events that are experienced by more than one model is significant. Since the models compared in this section have differences in the dynamical core and physical parameterizations, as well as assimilation techniques and assimilated observations, this behavior suggests that these dropout events are, at least in part, due to inherent limits of predictability (at the model resolution investigated) rather than relative deficiencies in a particular prediction system. We now turn our attention to the relationship between dropout events and large-scale patterns of variability.
4. Dropouts and large-scale pattern indices
Nonlinear systems tend to be attracted to particular regions of phase space and to make abrupt and difficult to anticipate transitions between these regions (e.g., Reinhold and Pierrehumbert 1982; Kimoto and Ghil 1993; Matsueda and Palmer 2018). From a meteorological perspective, these regions manifest as large-scale weather patterns and are characterized by commonly used indices e.g., the AO index. This section examines whether there is any correspondence between certain statistics of these large-scale flow indices and severe dropout events. It is reasonable to hypothesize that severe dropouts might be related to particular large-scale states, as predictability is known to be strongly dependent on the flow. To give just one example of such dependence, deterministic forecast performance tends to be lower in episodes of the positive phase of the AO (denoted +AO) than in episodes of the negative phase (−AO) (Minami and Takaya 2020).
The focus of this section is on three indices of the NH extratropical flow: the AO index, the Pacific–North American (PNA) index, and the North Atlantic Oscillation (NAO) index (e.g., Wallace and Gutzler 1981; Thompson and Wallace 1998). Each index is in the form of a daily time series for the period 1950–2020 obtained from the National Oceanographic and Atmospheric Administration (NOAA) Physical Sciences Division (PSD). Also, this section considers only the top-10 and top-50 most severe dropout events of this paper’s NAVGEM model dataset (Fig. 6). Note that to select the dropout events in Fig. 6, an exclusionary window is applied which ensures that no candidate dropout event is within ±14 days of any already selected dropout event. This is to ensure independence of the events. One can infer from Fig. 6 that the top-50 events encompass nearly all of the major dropout events. However, Fig. 7 makes this point explicitly. Here one sees that when the ACC scores for this paper’s NAVGEM model dataset are ranked from smallest to largest, the resulting ordering exhibits a radical left tail. The top-10 events capture the extremity of the radical tail, and the top-50 events encompass virtually the entirety of the radical tail.
Time series of NAVGEM ACC for the evaluation period. The top-10 (top-50) dropout events are indicated by the red (blue) filled circles.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-21-0208.1
The NAVGEM ACC scores for the evaluation period, ranked from smallest to largest. The top-10 (top-50) dropout events are indicated by the red (blue) filled circles.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-21-0208.1
To facilitate the compositing of the flow-index statistics across the diverse dates of the dropouts, the flow indices are first subjected to the same classical time series decomposition used in McLay et al. (2020). Each time series is assumed to consist of three components: A mean trend, seasonal fluctuations, and stochastic fluctuations. Then the seasonality and mean trend are removed to isolate the stochastic fluctuations (or “anomalies”) with respect to the seasonal cycle. The statistics of such index anomalies can be safely composited across dropout events in different months and seasons. Unless otherwise stated, all of the diagnostics that follow are based on these index anomalies.
Two basic statistics of the flow indices are examined in this section: the temporal variance of each individual index, and the temporal correlation of given pairs of indices. The temporal variance is included to test the hypothesis that the dropouts tend to occur during periods of heightened volatility of the indices. This idea is supported by findings that forecast predictability is often relatively low in the neighborhood of pattern transitions, which by definition are periods during which the flow indices are changing substantially, e.g., from one signed phase to the other (e.g., Lillo and Parsons 2017; McLay et al. 2020). The temporal correlation is included based on the hypothesis that certain points of the joint distribution of the indices might correspond to flows that are relatively less predictable and hence more conducive to dropouts. This idea is speculative but follows from the fact that predictability is strongly flow dependent.
a. Temporal variance of the flow indices for severe dropouts
The temporal variance of the flow indices in proximity to the top-10 and top-50 most severe dropout events is examined through a compositing procedure. First, the temporal variance of a flow index of interest is calculated for a time series segment of length N that is coincident with a given dropout event date.1 This time series segment could be centered on the dropout date or it could be immediately prior to the date. Then, the resulting 10 (or 50) variance values are averaged to produce a composite variance value. For this composite variance to be meaningful it needs to be put into context, i.e., one needs to determine whether the composite value is relatively large or small, or fairly typical, as compared to the value one would expect to get by chance. To do this, a sampling distribution of composite variance is generated using a bootstrap procedure. This procedure replicates the construction of the composite variance values, but for the situation where the underlying time series segments are randomly drawn from the full 1950–2020 index anomaly time series. The construction is repeated many times, yielding a sampling distribution of randomly generated composite variance values. The pseudocode for the procedure is as follows:
For trial = 1, …, a large number (e.g., 1 × 104)
For m = 1, …, number of severe dropout events (10 or 50)
Randomly sample a date in the period 1950–2020. This date must fall within the same month as the dropout event corresponding to index m.
Given the random date and the index anomaly time series of interest, extract a segment of length N from the series, where it is specified a priori that the segment is either centered on or ending on the random date.
Calculate the temporal variance of the index over the segment.
Calculate the composite temporal variance over the 10 (or 50) events.
Construct the sampling distribution from the results of all the trials.
Figures 8a and 8d show the composite temporal variance of the AO and NAO indices during selected segments of time in proximity to the top-10 dropouts, as well as the benchmark bootstrap sampling distribution of composite temporal variance. For the time window 14–7 days prior to the dropout dates (Figs. 8a,c), one finds that the composite values for the two indices are well below the median, near the 10th percentile of the sampling distribution. By contrast, for the window 6–0 days prior to the dropout date, the composite values for the two indices are well above the median, with the composite NAO value being at the 90th percentile of the sampling distribution (Figs. 8b,d). In other words, in the 14 days leading up to a severe dropout event the indices appear to swing from a state of depressed volatility to a state of elevated volatility. This swing is especially pronounced in the case of the NAO, but is notable as well in the case of the AO. The composite temporal variance of the PNA also exhibits a swing from below-median to above-median value during this time frame, but the change is more modest (not shown).
(a) Composite over the top-10 dropout events of the temporal variance of the AO index based on a window from 14 to 7 days prior to the dropout dates (red line) and benchmark bootstrap sampling distribution (shaded histogram). The 10th, 50th, and 90th percentiles of the sampling distribution are indicated by the vertical dashed lines. See text for details of the sampling distribution. (b) As in (a), but where the composite is based on a window from 6 to 0 days prior to the dropout dates. (c),(d) As in (a) and (b), respectively, but for the NAO index. (e),(f) As in (a) and (b), respectively, but where the composite is over the top-50 dropout events. (g),(h) As in (c) and (d), respectively, but where the composite is over the top-50 dropout events.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-21-0208.1
Considering the results for the top-50 dropouts (Figs. 8e,h), the composite temporal variance of the AO is seen to be elevated well above the median in both the windows 14–7 days prior and 6–0 days prior (Figs. 8e,f). Meanwhile, the composite variance of the NAO is found to move from a state of extremely low volatility (below the 10th percentile) in the window 14–7 days prior to a state of near-median volatility in the window 6–0 days prior (Figs. 8g,h). For the PNA, the composite volatility again shows a modest increase across the time windows (not shown). Thus, in the lead up to the most severe dropouts, all three indices either show an increase in volatility (which in some cases is dramatic) or show a consistently elevated level of volatility.
To further investigate this result, the temporal variance of each of the indices is computed in a running window centered on each of the 28 consecutive days leading up to a given dropout event. The running window is eight days in length, covering ±4 days of a given date. The result of this calculation is a 29-day-long time series of running-mean temporal variance for each index, beginning 28 days prior to a given dropout date (i.e., at the −28-day lag from the dropout date) and ending on the dropout date itself (i.e., at the 0-day lag).
Figure 9a shows for each of the three indices the composite across the top-10 dropout events of this 29-day-long time series of running-mean temporal variance. Note that for presentation purposes the composite series for each index has been normalized by its respective maximum value, such that the normalized maximum value is one. This figure shows in finer detail what was suggested by Fig. 8. Namely, one sees that roughly 10–20 days prior to the dropout date the indices show a distinct relative minimum of volatility. Then, the volatility escalates, peaking roughly 5–7 days prior to the dropout date. The escalation of volatility in the AO and NAO indices is quite rapid and dramatic. Significantly, the peak in volatility happens within roughly 2 days of the initialization date of the dropout forecast, suggesting that the volatility may be a contributing factor in producing the dropouts.
(a) Composite over the top-10 dropout events of the running-mean temporal variance of each index as a function of lag (in days) relative to the dropout valid dates. The curve for each index has been normalized by its respective maximum value. The blue vertical line indicates the initialization date of the forecast that experiences the dropout. See text for further details of the running-mean calculation. (b) As in (a), but where the composite is over the top-50 dropout events.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-21-0208.1
Analogous, albeit less dramatic, results are seen in Fig. 9b, which shows the corresponding composites across the top-50 dropouts. Prior to the dropout initialization date, both the AO and NAO move from a state of relatively low volatility to a state of relatively high volatility, with the volatility reaching a local peak near the initialization date. For the PNA, an escalation from relatively low to relatively high volatility is still seen, but the escalation occurs in the interval between the dropout forecast initialization and the forecast valid date. One notes that the AO volatility largely plateaus over the interval from −10- to −2-day lag, which is consistent with the indications of persistently elevated temporal variance in Figs. 8e,f.
The individual 29-day-long time series of running-mean temporal variance that underlie the composite time series in Fig. 9a for the top-10 dropouts were also examined, as a check to see if the behavior seen in Fig. 9a is due to sampling variability e.g., outliers in the data. This examination (not shown) indicates that the AO and NAO behavior seen in Fig. 9a is representative and is not due to outliers. For the PNA index, however, there is considerable underlying sampling variability and the situation is less conclusive.
b. Temporal correlation of the flow indices for severe dropouts
The temporal correlation of the flow indices in proximity to severe dropouts is examined through a compositing procedure similar to that for the temporal variance. First, the temporal correlation of a given pair of indices is calculated for a time series segment of length 28 days that either is centered upon or immediately precedes the given dropout date. Then the resulting 10 (or 50) correlation values are averaged to produce a composite correlation. For comparison, a sampling distribution for the composite correlation is generated using a procedure analogous to that described in section 4a for the temporal variance.
Figure 10 shows the results obtained when the 28-day interval is centered upon the given dropout date. Considering the correlations for the top-10 dropouts, one finds that the AO–NAO and AO–PNA index pairs tend to exhibit notably below-median composite correlation (Figs. 10a,b). In fact, the AO–PNA correlation value is less than the 5th percentile of the sampling distribution, placing it in the extreme left tail. For the AO–NAO index pair, the implication is that the correlation is a weaker positive value than typical, while for the AO–PNA index pair the implication is that the correlation is both much more negative and much larger in magnitude than typical. Meanwhile, the PNA–NAO composite correlation is also below-median, but is otherwise unremarkable (Fig. 10c).
(a) Composite over the top-10 dropout events of the temporal correlation of the AO–NAO index pair based on a 28-day window centered on the dropout dates (red line) and benchmark bootstrap sampling distribution (shaded histogram). The 10th, 50th, and 90th percentiles of the sampling distribution are indicated by the vertical dashed lines. See text for details of the sampling distribution. (b),(c) As in (a), but for the AO–PNA and PNA–NAO index pairs, respectively. (d)–(f) As in (a)–(c), but where the composite is over the top-50 dropout events.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-21-0208.1
Considering the composite correlations for the top-50 dropouts (Figs. 10d,f), the AO–PNA correlation is again notably below-median, at the 10th percentile (Fig. 10e). The AO–NAO correlation is again below-median as well, but not remarkably so (Fig. 10d). Interestingly, the PNA–NAO correlation is well above-median, near the 90th percentile, in contrast to the corresponding result for the top-10 dropouts (Fig. 10f). Thus, the result from Fig. 10 that stands out the most and that is consistent across both the top-10 and top-50 dropouts is the notably below-median value of the AO–PNA correlation.
c. Further assessing the relationship between the AO–PNA correlation and ACC scores
While Figs. 10b and 10e imply an association between severe dropouts and unusually strong negative AO–PNA correlation, this information is insufficient to infer that unusually strong negative AO–PNA correlation is a predictor of low ACC scores. For instance, bearing in mind that dropouts are rare events, it is possible that most instances of AO–PNA correlation less than the 10th percentile are actually associated with average or even above-average ACC scores. To gain insight into a possible predictive relationship between the AO–PNA correlation and ACC scores, here a binned scatterplot diagnostic is employed. As a preliminary to constructing the diagnostic, the time series of ACC scores and AO–PNA index correlation for the dropout period of 1 April 2014–31 December 2017 are subjected to the same classical time series decomposition that is applied earlier in this section to the AO and PNA indices. This decomposition yields anomalies of ACC scores and anomalies of AO–PNA correlation with respect to the low-frequency (seasonal) cycle. To give some sense of the resulting anomalies, Figs. 11a and 11b show the detrended time series of ACC scores and AO–PNA index correlation, respectively, against their underlying seasonal cycles. Note that the anomalies are obtained by subtracting the seasonal cycle from the detrended series. Given the anomalies, the binned scatterplot diagnostic is constructed by ordering the pairs of ACC anomalies and AO–PNA correlation anomalies from smallest AO–PNA correlation anomaly to largest AO–PNA correlation anomaly. The ordered list is divided into equally populated bins, and then the bin-average AO–PNA correlation anomaly is plotted against the bin-average ACC anomaly. This type of binned scatter diagnostic is a standard means of assessing the predictive relationship between ensemble spread and verifying error (e.g., McLay et al. 2008).
(a) Detrended time series of ACC (gray line) and corresponding estimated seasonal cycle (blue line) for the evaluation period. (b) Detrended time series of temporal correlation between the AO–PNA index pair (gray line) and corresponding estimated seasonal cycle (blue) line for the evaluation period.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-21-0208.1
Figure 12a shows the binned scatter diagnostic and accompanying simple linear regression for the case with 10 bins. A direct relationship is evident between the anomalous AO–PNA correlation and the anomalous ACC scores. This result is insensitive to the number of bins used, as the diagnostic was produced using bin counts up to 100 with analogous results. Thus, not only is anomalously negative AO–PNA correlation a predictor of anomalously negative ACC scores, the converse also holds: anomalously positive AO–PNA correlation is a predictor of anomalously positive ACC scores.
(a) Binned scatterplot of anomalous AO–PNA correlation vs anomalous ACC score for the case with 10 bins. Black line shows the simple linear regression on the 10 scatter points. (b) Binned scatterplot of anomalous ACC score vs anomalous AO–PNA correlation for the case with 10 bins; otherwise, as in (a).
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-21-0208.1
A companion to the binned scatter diagnostic of Fig. 12a can also be constructed, by ordering the pairs of ACC anomalies and AO–PNA correlation anomalies from smallest ACC anomaly to largest ACC anomaly and then plotting the bin-average ACC anomaly against the bin-average AO–PNA correlation anomaly. This allows one to verify that the predictive relationship between the anomalies of AO–PNA correlation and ACC is a two-way relationship. The companion diagnostic is shown in Fig. 12b, and a direct relationship similar to that in Fig. 12a is evident. Thus, the ACC anomalies are in fact a predictor of the AO–PNA correlation anomalies.
d. A local solution of the ACC
In one additional attempt at insight, a local solution of the ACC equation [Eq. (1)] is carried out, such that the expectations in Eq. (1) are solved over 10° × 10° latitude–longitude boxes centered on each grid point of a 1° × 1° grid. The aim of the local ACC solution is to see whether certain regions might contribute more than others to the low ACC scores of the dropout events. A gridded local ACC field is obtained for each of the top-10 (or top-50) dropout events, and then the 10 (or 50) different local ACC fields are composited to produce Figs. 13a and 13b. Note that the local ACC is calculated only for those grid points at latitudes ≤ 80°N.
(a) Composite over the top-10 dropout events of the local solution of the ACC. See text for details of the local ACC calculation. (b) As in (a), but where the composite is over the top-50 dropout events.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-21-0208.1
Considering the composite for the top-10 events (Fig. 13a), one finds a prominent minimum of local ACC over the southeastern United States and less prominent minima over several other regions. Meanwhile, the local ACC over the northeastern Atlantic Ocean and Europe is relatively high, indicating that anomaly errors in these regions are not major contributors to the top-10 dropout events. The composite for the top-50 events (Fig. 13b) is grossly similar but also exhibits certain notable differences. For example, the minimum over the southeastern United States is much less prominent, and the local ACC over the Pacific Ocean near the West Coast of the United States is relatively high. Also, a local minimum is present over the west coast of Europe.
Noting that the composites of Fig. 13 are based on the mean local ACC across the top-10 (or top-50) events, and that the mean can be sensitive to outliers, the corresponding median local ACC was also calculated. Although not shown here, the median local ACC yields similar findings to Fig. 13. Importantly, the median diagnostic indicates that the prominent minimum of local ACC over the southeastern United States in Fig. 13a is not attributable to any single event (i.e., it is not due to an outlier).
Perhaps the most informative aspect of Fig. 13 is that it shows that the local ACC minima tend to coincide with regions associated with prominent features in the EOFs for the PNA and AO.2 For instance, the main features in the EOF for the PNA are found over the Pacific Ocean south of the Aleutian Islands, northwestern North America, the southeastern United States, and the northern Atlantic Ocean directly south of Greenland. These are all regions emphasized in Fig. 13. Thus, Fig. 13 reinforces (or at least is consistent with) the earlier suggestions from Figs. 8, 10, and 12 of a relationship between the ACC dropouts and the PNA and AO patterns.
5. An ensemble forecasting perspective of selected dropouts
A general notion in the forecasting community is that a deterministic forecast is only complete if there is an attendant examination of forecast uncertainty or confidence. Thus, this section surveys three selected dropout cases from an ensemble forecasting perspective. We note that focusing on only three cases prevents any generalizing. However, we see value in exploring the following ensemble diagnostics in that they bring attention to certain aspects of the dropout problem that have received little, if any, scrutiny.
Many questions arise, e.g., does the ensemble-mean forecast also suffer a dropout? Does the ensemble spread behavior indicate larger than typical forecast uncertainty in proximity to the dropout date? To consider such questions, here ensemble forecasts are generated using the operational configuration of the NAVGEM global ensemble forecasting system. This configuration includes the local ET initialization methodology of McLay et al. (2010), a horizontal spectral resolution T359 (∼37 km) with 60 vertical levels, and an ensemble size of 20. The ensembles employ model error representation in the form of stochastic kinetic energy backscatter (SKEB) (Shutts 2005) and they include sea surface temperature (SST) variation analogous to that described in McLay et al. (2012).
Owing to computing limitations, the ensembles were constructed in association with three dropout cases from 2017 with the dropout dates 1200 UTC 29 April, 0000 UTC 11 June, and 1200 UTC 6 July. For the 11 June and 6 July cases, the ensemble forecasts are initialized twice per day at 0000 and 1200 UTC starting 13 days prior to the dropout date and ending 3 days following the dropout date. For the 29 April case, the ensemble forecasts are initialized twice per day starting 22 days prior to the dropout date and ending 3 days following. Thus, for all three cases the ensemble forecasts are initialized over an interval bracketing the dropout date, allowing one to see how the forecasts evolve leading up to the dropout.
To begin with, Fig. 14 shows time series of ACC for successive T + 120 h Φ500 forecasts valid on an interval encompassing each dropout date. The forecasts include the deterministic forecast (the forecast that exhibits a dropout), the ensemble-mean forecast, and the so-called best-member forecast (the individual perturbed member of the forecast ensemble that has the maximum ACC). Accompanying these forecasts is the T + 120 h Φ500 ensemble spread (right axis). The dropouts in the deterministic forecast are acutely apparent in each of the cases of Figs. 14a and 14c. Also apparent in each case is that both the ensemble-mean forecast and the best-member forecast display prominent local minima in close proximity to the dropout date (i.e., ±1 cycle). Thus, even the best performing deterministic forecasts from the ensemble exhibit a substantial decline in performance on or about the dropout date. Further, as with the deterministic forecast, both the ensemble mean and best member begin to display a decline in performance one or more cycles prior to the dropout date. In two of the three cases (Figs. 14a,c) the ensemble-mean and best-member forecasts have somewhat higher ACC on the dropout date compared to the deterministic forecast. Additionally, on a percentage basis the performance decrease of the ensemble-mean and best-member forecasts is not as large as for the deterministic forecast (not shown). Hence, while the forecasts from the ensemble display analogous dropout behavior to the deterministic forecast, the ensemble forecasts may still yield comparatively better results.
(a) Left axis: Time series of ACC for successive T + 120 h Φ500 forecasts valid on an interval centered on the dropout date of 1200 UTC 29 Apr 2017. The deterministic, ensemble-mean, and best-member forecasts are denoted by the blue filled circles, black solid line, and gray solid line, respectively. Right axis: Time series of ensemble spread (i.e., standard deviation) (red line) for the same T + 120 h Φ500 forecasts. (b) As in (a), but for an interval centered on the dropout date of 0000 UTC 11 Jun 2017. (c) As in (a), but for an interval centered on the dropout date of 1200 UTC 6 Jul 2017.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-21-0208.1
Considering next the ensemble spread evolution, one finds that in all three cases (Figs. 14a,c) the T + 120 h spread is in the midst of a modest upward trend at the time of the dropout date. There are no indications that the spread for the dropout date is substantively different from the recent history, e.g., there is no overt peak in spread on or immediately preceding the dropout date. Interestingly, however, in all three cases a local maximum in T + 120 h spread occurs for dates immediately following the dropout initialization. This circumstance where the peak in spread follows the dropout forecast cycle, as though the system is slow to respond to a rapidly emerging mode of error growth, may relate in part to the nature of the ET ensemble algorithm. Specifically, the ET is a cycling system, and in such systems it can take some finite number of cycles for new growing modes to fully emerge in the perturbations. This fact may also help to explain the result that the dropouts tend to occur immediately following rapid escalations of the temporal volatility of the AO/NAO/PNA indices: The pace of the flow changes may be challenging the ability of the ET system to adapt.
Figure 15 shows what is referred to as the sequence of lagged forecasts for the 29 April dropout case. This is the sequence of ACC for all those forecasts with lead times ≤ 384 h that are valid on the dropout date of 1200 UTC 29 April. One knows that as the valid date approaches and the lead time goes to zero that any given forecast score must eventually converge to its best possible value, e.g., for the ACC used here, that value is one. Note, though, that this convergence does not necessarily occur smoothly, e.g., the forecast could flip-flop, jump, or fluctuate from one initialization date to the next until finally the forecast system “locks-in” to the ultimate verifying state. Sequences of lagged forecasts like those in Fig. 15 are useful in revealing just how this convergence transpires in practice. In Fig. 15, both the ensemble-mean and best-member lagged score sequences are shown, as are composite ensemble-mean and composite best-member lagged score sequences. The composite sequences are intended to serve as benchmarks of expected sequence behavior and are obtained as follows: First, recognize that each of the 17 dates in the set (1200 UTC 24 April, 0000 UTC 25 April, …, 1200 UTC 2 May) has a corresponding lagged score sequence of length 384 h that terminates on that date. E.g., the score sequence that terminates on the date 1200 UTC 24 April consists of the scores for the lag-384-h forecast initialized at 1200 UTC 8 April, the lag-372-h forecast initialized at 0000 UTC 9 April, and so forth, up to the lag-0-h forecast (i.e., the verifying analysis) initialized at 1200 UTC 24 April. Next, exclude from consideration those five sequences that terminate on dates within ±1 day of the dropout date (i.e., the dates 1200 UTC 28 April–1200 UTC 30 April). The composite sequence is the average of the remaining 12 lagged sequences.
(a) The thick and thin red lines depict the sequences of lagged ACC corresponding to all those ensemble-mean and best-member forecasts, respectively, with lead times ≤ 384 h that are valid on the dropout date of 1200 UTC 29 Apr 2017. The filled blue circle indicates the ACC of the T + 120 h deterministic forecast that suffers the dropout. The thick and thin black lines depict the reference composite sequences of lagged ACC for the ensemble-mean and best-member forecasts, respectively. See text for further details of the composite construction. (b) Left axis: Red lines and filled blue circle are as defined in (a). Right axis: Dark blue line depicts the sequence of lagged ensemble forecast spread corresponding to all those ensemble forecasts with lead times ≤ 384 h that are valid on the dropout date of 1200 UTC 29 Apr 2017. Turquoise line depicts the reference composite sequence of lagged ensemble forecast spread.
Citation: Weather and Forecasting 37, 11; 10.1175/WAF-D-21-0208.1
Considering the composite lagged sequences in Fig. 15a, one unsurprisingly sees a basically monotonic increase in the ACC as the lead time decreases, with the ensemble-mean and best-member forecasts attaining a 0.6 ACC roughly around the T + 216 h lead time. Compared to these composite sequences, the ensemble-mean and best-member forecasts for the 29 April dropout case have lower than typical ACC at the longest leads. However, they rapidly gain ACC value over the interval from roughly T + 288 h to T + 168 h, such that by T + 168 h they have approximately the same value as would be expected from the composites. Interestingly, the ensemble-mean and best-member sequences then very discernibly plateau for the two days’ worth of forecast cycles immediately prior to the dropout forecast, i.e., for the lead times T + 168 h to T + 120 h. Promptly following the T + 120 h dropout forecast, these sequences then resume a rapid convergence to the composite lagged sequences. Thus, from the lagged score sequence perspective the forecast dropout is not a sharp and isolated decline in forecast skill. Rather, it appears to be the culmination of an interval in which the forecast system fails to achieve the typical cycle-to-cycle gains in forecast performance.
Figure 15b also presents the sequence of lagged ensemble forecast spread for the 29 April dropout case and a reference composite sequence of lagged forecast spread (derived in a manner analogous to that used to obtain the composite sequences of ensemble-mean and best-member ACC in Fig. 15a). One is looking to see if the sequence of spread provides any obvious indications that the forecast error associated with the dropout date might be larger than typical. To this end, note that the lagged sequence of spread must converge toward zero as lead time decreases. Thus, indications of larger than typical forecast error for the dropout date might manifest as either a local leveling off of the spread sequence or as a local peak in the sequence. In fact, in Fig. 15b one does find a brief plateauing of the spread sequence beginning at T + 192 h. This plateauing is sufficient to elevate the spread sequence for 29 April above the reference composite spread sequence. However, the elevation of spread is modest and short-lived, fading away 12 h prior to the T + 120 h dropout forecast. By T + 120 h the spread is in a monotonic, rapid decline toward zero. Hence, there is no overt indication from the sequence of spread that the T + 120 h forecast error might by atypically large.
Several statistical factors may help to explain the lack of a strong signal in the spread in Figs. 14 and 15. One is the fact that a large increase in spread is not necessary for there to be a proportionally large increase in the probability of realizing an outsize error. As a toy example, for a Gaussian-distributed random variable with mean 0.0 and spread 1.0, a 10% increase in spread increases the probability of realizing a value ≥ 2.0 by O(100%). Another is the fact that the temporal variability of the spread will always (over a sufficiently long time sequence) be less than that of the verifying error. A third factor is the possibility that the large error on the dropout date is simply a low-likelihood draw from the upper tail of an otherwise typical T + 120 h error distribution. Finally, the standard parametric confidence interval for an estimate of spread based on 20 samples is quite wide (not shown). Thus, sampling uncertainty associated with the 20-member ensemble may be overwhelming any relevant signal in the spread.
6. Conclusions
In the first part of this study, a short climatology is used to evaluate the seasonality and inter-model relationships associated with dropout events in the NAVGEM, GEPS, and GEFS forecast systems. Then, certain statistics of the large-scale flow in proximity to NAVGEM dropout events are explored. Finally, the dropout problem is surveyed from the standpoint of ensemble prediction using several cases of severe NAVGEM dropouts.
The inter-model comparisons indicate that the ACC is correlated between models, the shape of the distribution of ACC is similar between models, and the proportion of dropout events that are experienced by more than one model is significant. Since the models compared have differences in the dynamical core, physical parameterizations, as well as assimilation techniques and assimilated observations, this behavior points toward dropout events being, at least in part, due to inherent limits of predictability (for the model resolution investigated) rather than relative deficiencies in a particular prediction system.
Using time series and compositing techniques, one finds that the temporal volatility of the AO/NAO/PNA indices increases prior to the dropout event, and that the volatility tends to peak in the vicinity of the initialization date of the dropout forecast. Meanwhile, considering the temporal correlation of the indices, one finds that the period centered on the dropout event is characterized by much stronger than typical negative correlation between the AO and PNA anomaly time series. In other words, the most severe dropouts tend to occur when the AO and PNA exhibit unusually strong dependence but have opposite signed anomalies. Of note, the AO–PNA correlation anomalies are seen to exhibit a general predictive relationship with the ACC anomalies, such that not only do negative correlation anomalies tend to portend negative ACC anomalies, but positive correlation anomalies tend to portend positive ACC anomalies.
For the ensemble prediction view of the problem, both the ensemble-mean and best-member forecasts display analogous dropout behavior to the operational deterministic forecast, albeit with comparatively better performance scores. Also, the time series of T + 120 h spread does not provide obvious advance warning of an elevated potential for very large forecast errors on the dropout date. Rather, in all three cases examined here a local maximum in T + 120 h spread occurs for dates after the dropout initialization.
From the lagged forecasting perspective, the dropout appears to be the culmination of a period in which the forecast performance “plateaus” and fails to achieve the expected cycle-to-cycle gains. Considering the corresponding sequence of lagged ensemble spread, in the case examined there is no obvious indication from this sequence that the T + 120 h forecast error might be atypically large. A number of statistical factors are discussed that might explain the lack of a strong signal in the ensemble spread. There is also the possibility that the lack of a signal relates in part to the ET cycling algorithm, which may require a finite number of cycles to respond to rapidly emerging modes of error growth.
For future work, the foremost objective is to provide a dynamical understanding of the tendency for dropouts to occur during time intervals where the anomalies of the AO and PNA patterns exhibit much stronger than usual negative correlation. One can speculate that these intervals of elevated negative correlation might have lower intrinsic predictability, and hence be more disposed to large forecast errors, but this idea needs to be substantiated.
Note that here and elsewhere, “dropout date” refers to the valid date of the forecast that exhibits the dropout.
The EOFs (or loading diagrams) for the PNA and AO are available for viewing at https://www.cpc.ncep.noaa.gov/products/precip/CWlink/daily_ao_index/loading.html and https://www.cpc.ncep.noaa.gov/products/precip/CWlink/pna/pna_loading.html.
Acknowledgments.
This research is supported by the Chief of Naval Research through the NRL Base Program (Program Element 0601153N). The forecasts were obtained from the THORPEX Interactive Grand Global Ensemble (TIGGE) data portal at ECMWF.
Data availability statement.
NAVGEM model data used in this study can be obtained at ftp://usgodae.org/pub/outgoing/fnmoc/models/navgem_0.5/, and data for GEFS and ECCC GEPS were obtained from THORPEX Interactive Grand Global Ensemble (TIGGE) data portal at ECMWF.
REFERENCES
Dee, D. P., and Coauthors, 2011: The ERA-Interim reanalysis: Configuration and performance of the data assimilation system. Quart. J. Roy. Meteor. Soc., 137, 553–597, https://doi.org/10.1002/qj.828.
Grams, C. M., L. Magnusson, and E. Madonna, 2018: An atmospheric dynamics perspective on the amplification and propagation of forecast error in numerical weather prediction models: A case study. Quart. J. Roy. Meteor. Soc., 144, 2577–2591, https://doi.org/10.1002/qj.3353.
Hogan, T. F., and Coauthors, 2014: The Navy Global Environmental Model. Oceanography, 27, 116–125, https://doi.org/10.5670/oceanog.2014.73.
Kimoto, M., and M. Ghil, 1993: Multiple flow regimes in the Northern Hemisphere winter. Part I: Methodology and hemispheric regimes. J. Atmos. Sci., 50, 2625–2644, https://doi.org/10.1175/1520-0469(1993)050<2625:MFRITN>2.0.CO;2.
Krishnamurti, T. N., and Coauthors, 2003: Improved skill for the anomaly correlation of geopotential heights at 500 hPa. Mon. Wea. Rev., 131, 1082–1102, https://doi.org/10.1175/1520-0493(2003)131<1082:ISFTAC>2.0.CO;2.
Kumar, V. K., E. Maddy, J. C. Alpert, and S. A. Boukabara, 2016 : Global forecast dropout prediction tool in support of the NCEP Model Evaluation Group (MEG)—A collaborative project between JCSDA/NESDIS and NWS. 20th Conf. on Integrated Observing and Assimilation Systems for the Atmosphere, Oceans, and Land Surface (IOAS-AOLS), New Orleans, LA, Amer. Meteor. Soc., 6.3, https://ams.confex.com/ams/96Annual/webprogram/Paper288630.html.
Langland, R. H., and R. N. Maue, 2012: Recent northern hemisphere mid-latitude medium-range deterministic forecast skill. Tellus, 64 A, 17531, https://doi.org/10.3402/tellusa.v64i0.17531.
Lillo, S. P., and D. B. Parsons, 2017: Investigating the dynamics of error growth in ECMWF medium-range forecast busts. Quart. J. Roy. Meteor. Soc., 143, 1211–1226, https://doi.org/10.1002/qj.2938.
Lin, H., N. Gagnon, S. Beauregard, R. Muncaster, M. Markovic, B. Denis, and M. Charron, 2016: GEPS-based monthly prediction at the Canadian meteorological Centre. Mon. Wea. Rev., 144, 4867–4883, https://doi.org/10.1175/MWR-D-16-0138.1.
Magnusson, L., 2017: Diagnostic methods for understanding the origin of forecast errors. Quart. J. Roy. Meteor. Soc., 143, 2129–2142, https://doi.org/10.1002/qj.3072.
Matsueda, M., and T. N. Palmer, 2018: Estimates of flow-dependent predictability of wintertime Euro-Atlantic weather regimes in medium-range forecasts. Quart. J. Roy. Meteor. Soc., 144, 1012–1027, https://doi.org/10.1002/qj.3265.
McLay, J. G., C. H. Bishop, and C. A. Reynolds, 2008: Evaluation of the ensemble transform analysis perturbation scheme at NRL. Mon. Wea. Rev., 136, 1093–1108, https://doi.org/10.1175/2007MWR2010.1.
McLay, J. G., C. H. Bishop, and C. A. Reynolds, 2010: A local formulation of the ensemble transform (ET) analysis perturbation scheme. Wea. Forecasting, 25, 985–993, https://doi.org/10.1175/2010WAF2222359.1.
McLay, J. G., M. K. Flatau, C. A. Reynolds, J. Cummings, T. Hogan, and P. J. Flatau, 2012: Inclusion of sea-surface temperature variation in the U.S. Navy ensemble-transform global ensemble prediction system. J. Geophys. Res., 117, D19120, https://doi.org/10.1029/2011JD016937.
McLay, J. G., D. Hodyss, and C. A. Reynolds, 2020: Favoured cycles and other features of anomalous forecast uncertainty as revealed by time series of TIGGE data. Quart. J. Roy. Meteor. Soc., 146, 870–888, https://doi.org/10.1002/qj.3713.
Minami, A., and Y. Takaya, 2020: Enhanced Northern Hemisphere correlation skill of subseasonal predictions in the strong negative phase of the Arctic oscillation. J. Geophys. Res. Atmos., 125, e2019JD031268, https://doi.org/10.1029/2019JD031268.
Parsons, D. B., S. P. Lillo, C. P. Rattray, P. Bechtold, M. J. Rodwell, and C. M. Bruce, 2019: The role of continental mesoscale convective systems in forecast busts within global weather prediction systems. Atmosphere, 10, 681, https://doi.org/10.3390/atmos10110681.
Reinhold, B. B., and R. T. Pierrehumbert, 1982: Dynamics of weather regimes: Quasi-stationary waves and blocking. Mon. Wea. Rev., 110, 1105–1145, https://doi.org/10.1175/1520-0493(1982)110<1105:DOWRQS>2.0.CO;2.
Richardson, D. S., 2005: The THORPEX Interactive Grand Global Ensemble (TIGGE) . Geophysical Research Abstracts, Vol. 7, Abstract 02815, http://meetings.copernicus.org/www.cosis.net/abstracts/EGU05/02815/EGU05-J-02815.pdf.
Rodwell, M. J., and Coauthors, 2013: Characteristics of occasional poor medium-range weather forecasts for Europe. Bull. Amer. Meteor. Soc., 94, 1393–1405, https://doi.org/10.1175/BAMS-D-12-00099.1.
Rodwell, M. J., D. S. Richardson, D. B. Parsons, and H. Wernli, 2018: Flow-dependent reliability: A path to more skillful ensemble forecasts. Bull. Amer. Meteor. Soc., 99, 1015–1026, https://doi.org/10.1175/BAMS-D-17-0027.1.
Shutts, G., 2005: A kinetic energy backscatter algorithm for use in ensemble prediction systems. Quart. J. Roy. Meteor. Soc., 131, 3079–3102, https://doi.org/10.1256/qj.04.106.
Thompson, D. W. J., and J. M. Wallace, 1998: The arctic oscillation signature in the wintertime geopotential height and temperature fields. Geophys. Res. Lett., 25, 1297–1300, https://doi.org/10.1029/98GL00950.
Uno, F., H. Ohtake, M. Matsueda, and Y. Yamada, 2018: A diagnostic for advance detection of forecast busts of regional surface solar radiation using multi-center grand ensemble forecasts. Sol. Energy, 162, 196–204, https://doi.org/10.1016/j.solener.2017.12.060.
Wallace, J. M., and D. S. Gutzler, 1981: Teleconnections in the geopotential height field during the Northern Hemisphere winter. Mon. Wea. Rev., 109, 784–812, https://doi.org/10.1175/1520-0493(1981)109<0784:TITGHF>2.0.CO;2.
Wilks, D. S., 2006 : Statistical Methods in the Atmospheric Sciences. 2nd ed. International Geophysics Series, Vol. 100, Academic Press, 648 pp.
Yamagami, A., and M. Matsueda, 2021: Statistical characteristics of arctic forecast busts and their relationship to arctic weather patterns in summer. Atmos. Sci. Lett., 22, e1038, https://doi.org/10.1002/asl.1038.
Zhou, X., Y. Zhu, D. Hou, Y. Luo, J. Peng, and R. Wobus, 2017: Performance of the new NCEP global ensemble forecast system in a parallel experiment. Wea. Forecasting, 32, 1989–2004, https://doi.org/10.1175/WAF-D-17-0023.1.
