a. The ECMWF Ensemble Prediction System

Since 11 March 2008, ECMWF has been running a unified medium-range and monthly ensemble system, formed by merging the previous Variable Resolution Ensemble Prediction System (VAREPS) and monthly configurations (Buizza et al. 2007; Vitart et al. 2008). The merged system includes 51 members integrated twice a day up to 15 days, except once a week when they are extended to 32 days. More precisely, these are the characteristics of the ECMWF Ensemble Prediction System (EC-EPS):

• 1200 UTC: T_L399L62 (50-km resolution, 62 levels, days 0–10) and T_L255L62 (80-km resolution, days 10–15) with persisted SST anomaly;

• 0000 UTC: T_L399L62 (days 0–10) with persisted SST anomaly, and T_L255L62 (days 10–15) with coupled ocean model; on Thursdays the integration is extended to day 32.

Initial uncertainties are simulated using a combination of initial-time and evolved singular vectors (Buizza and Palmer 1995) computed at T42L62 resolution, with 48-h optimization time interval and a total-energy norm. Singular vectors are computed to maximize the final-time norm over different areas (Barkmeijer et al. 1999, 2001), combined and scaled to have initial amplitude comparable to an estimate of the analysis error. Model uncertainties due to physical parameterizations are simulated using a stochastic scheme (Buizza et al. 1999). The reader is referred to Palmer et al. (2007) for a review of the developments of the ECMWF ensemble system since its implementation in 1992 to date.

b. The Met Office Ensemble Prediction System

The 15-day Met Office Ensemble Prediction System (UK-EPS) is an extension of the Met Office Global Regional Ensemble Prediction System (MOGREPS; see Bowler et al. 2008). The UK-EPS is run twice a day, at 0000 and 1200 UTC, has a resolution of 90 km, 38 vertical levels, and includes 24 members (23 perturbed plus the unperturbed control forecast). Initial uncertainties are simulated using an ensemble transform Kalman filter (ETKF) approach (Wei et al. 2006; Bishop et al. 2001; Bowler et al. 2007). Model uncertainties are modeled using a stochastic backscatter scheme (Shutts 2005). The reader is referred to Park et al. (2008) and Johnson and Swinbank (2008) for two recent papers that compare the performance of the EC-EPS and the UK-EPS, and discuss the potential value of combining the two ensembles.

c. The dataset used in this study

The EC-EPS and the UK-EPS forecasts used in this study were extracted from the TIGGE archive hosted online at ECMWF (more information available online at http://tigge.ecmwf.int/) on a regular 1° × 1° latitude–longitude grid. To account for the poleward distortion of the grid, weights defined by the cosine of the latitude have been given to each grid point when area averages have been computed. Since the UK-EPS started being available in the TIGGE archive in January 2007, we chose to base our statistical computations on the period from February 2007 to August 2008. In effect this meant 18 months worth of forecasts, issued twice a day at 0000 and 1200 UTC, a sufficiently large sample to provide robust results. Furthermore, to assess the seasonal variability of the ensemble jumpiness, at least for the EC-EPS, ECMWF ensemble data from June 2003 to August 2008 (i.e., about 5 yr) have also been considered.

d. The methodology and indices used to measure forecast jumpiness

Five different measures of forecast jumpiness, one basic index, and four related concepts are introduced. First, the inconsistency index of two forecasts valid for the same verification time is defined. Then, considering the average of the inconsistency index for a given sampling period, the concepts of different forecast jumps—the “flip,” “flip-flop,” and “flip-flop-flip”—and their corresponding indices, are defined in their two flavors, the single and the parallel one. Finally, the inconsistency correlation between two forecasts’ inconsistency index time series is introduced to measure how similar the inconsistency of two different forecasts is.

Denote with f_j^C(d, t) the forecast started on date/time d and valid for date/time d + t, defined for all grid points of an area Σ, where the superscript C = EC identifies the ECMWF forecast, and C = UK identifies the Met Office forecast. The subscript j = 0 identifies the control forecast, while j = 1, N^C identifies the N^C perturbed ensemble forecasts. Denote with 〈f_m^C(d, t)〉 the ensemble-mean forecast, defined by averaging the control and all the perturbed members:

View Expanded

Consider two forecast fields f (..,..) and g(..,..). Denote with d_Σ[ f, g] the difference between the two forecasts over the area Σ:

View Expanded

where gp denotes the grid points in the area, and denote with std_Σ[ f] the standard deviation of the field values over the area Σ:

View Expanded

where A_Σ is the average of the field values over the area Σ:

View Expanded

1) Definition of the inconsistency index used to measure forecast jumpiness

Consider now two consecutive forecasts issued at date/time d and (d + δ) and valid for the same verification time, where δ has been set to 12 h in our case. The inconsistency index INC_Σ[ f_j^C(d, t), δ] between the two forecast fields issued δ hours apart (here j is either 0 or m corresponding to control or ensemble mean) over the area Σ is defined as the ratio between their distance and their average standard deviation:

View Expanded

The denominator has been introduced to take into account the fact that different forecasts have different characteristics (e.g., resolution and variability). For example, the control forecast has more detail (i.e., includes smaller scales) and thus is expected to have larger inconsistency than the smoother ensemble-mean forecast. This scaling factor normalizes the differences between consecutive forecasts to the variability of the forecasts themselves. The normalization is also important to enable comparison of inconsistency indices computed for different forecast steps, different weather parameters, or different forecasting systems.

Given a set of forecasts over N dates, the N-period root-mean-square average of the daily inconsistency index is defined as (hereafter referred to as the period-average inconsistency):

View Expanded

2) definition of the concept of a forecast flip

Consider two consecutive forecasts issued at date/time d and (d + δ), and valid for the same verification time (d + t). The forecast f_j^C(d, t) is defined to have made a single flip if the absolute value of the inconsistency index is larger than half of the period-average inconsistency over N forecast runs:

View Expanded

3) Definition of the concept of a forecast flip-flop

Consider three consecutive forecasts issued at date/time (d − δ), d and (d + δ), and valid for the same verification time. The forecast f_j^C(d, t) is defined to have made a single flip-flop if the forecast made two consecutive single flips but with an opposite sign. In other words, a forecast made a single flip-flop if the following three conditions are met:

View Expanded

6) Definition of the inconsistency correlation

Consider the time series of an inconsistency index over the area Σ, for forecasts f (d_i, t) and g(d_i, t) over N forecast runs, denoted here for simplicity by F_i and G_i where i = 1, 2, … , N refers to the forecast runs. The inconsistency correlation between the two inconsistency index time series, F_i and G_i over the area Σ, is defined by the product-moment correlation coefficient:

View Expanded

The inconsistency correlation is computed either for an individual EPS system with f and g representing the control and the ensemble-mean forecasts, or for two EPS systems with f and g representing the two controls or the two ensemble-mean forecasts.

a. Case 1: Jumpiness during zonal-flow conditions in February 2008

During this case of February 2008 (this event actually triggered our investigation), the ECMWF control and the EC-EPS ensemble-mean forecasts were very jumpy for a few days in a row over northern Europe, and the EC-EPS ensemble mean followed the control forecast very closely. This raised the question of whether this was an unusual event, and more generally of whether the ensemble-mean forecast was on average following the control forecast too closely.

At the time of this event, the northern part of Europe was under the influence of a strong zonal flow, with a ridge building up and moving eastward rather rapidly (Fig. 2). Figure 3 shows the 180-, 168-, and 156-h control and ensemble-mean forecasts valid for 1200 UTC 23 February. In terms of the large-scale, zonal flow the forecasts look consistent and are in quite good agreement with the observed flow. However, a closer look highlights large inconsistencies in the small scales (e.g., in the prediction of the ridge that was affecting the northern European countries). The high level of uncertainty in the region is also indicated by the unusually large ensemble spread (not shown), as measured by the ensemble standard deviation. The “jumpy” behavior of the consecutive forecasts is highlighted on Fig. 3 by the arrows indicating the dominant flow direction. The forecasts started on 16 (t + 180 h) and 18 (t + 156 h) February show a ridge stretching over the United Kingdom, while the intermediate forecasts started on 17 February (t + 168 h) indicate a significantly more zonal flow with only slightly anticyclonic curvature over the United Kingdom. Note that the control and the ensemble-mean forecasts show a very similar behavior, although the jumps in the ensemble mean have smaller amplitude.

Figure 4 shows the inconsistency index of the EC-EPS control and ensemble-mean forecasts as a function of forecast range for all forecasts verifying at 1200 UTC 23 February. The ensemble-mean forecast follows the control forecast rather closely between forecast steps t + 240 h and t + 144 h, while it jumps much less for longer forecast ranges. At a short forecast range both forecasts change less and follow each other very closely (this is not surprising in the short range, since by construction the ensemble-mean and the control forecasts coincide at initial time, and perturbations grow linearly during the first 24–48 h (see e.g., Gilmour et al. 2001).

The dotted and dash–dotted lines in Fig. 4 show the threshold values used in the definition of forecast jumps. The control and/or the ensemble mean makes a flip if the corresponding inconsistency indices (open circles for the control, triangles for the ensemble mean) are outside of the area bounded by the two dotted (control) or dash–dotted (ensemble mean) lines (i.e., if the modulus of the inconsistency index is larger than half of the period-average inconsistency). The values enclosed in the large circles in Fig. 4 (at t + 180 h and t + 168 h) correspond to the forecasts displayed in Fig. 3. The lower circle includes the inconsistency indices computed between the first (t + 180 h) and second (t + 168 h) forecast fields (delivering the inconsistency index for t + 180 h) in the upper (control) and lower (ensemble mean) rows of Fig. 3. Similarly, the upper circle includes the inconsistency indices computed between the second (t + 168 h) and third (t + 156 h) fields (inconsistency index for t + 168 h). The two circled inconsistency-index pairs actually correspond to a single flip-flop of the control and also of the ensemble mean, because they zigzag outside of the jump-threshold lines (first both geopotential height fields get lower, then on the next run both get higher again). Moreover, since the control and the ensemble-mean jump in the same direction, they also make a parallel flip-flop. A closer inspection actually indicates a parallel flip-flop-flip of the three consecutive 180-, 168-, and 156-h forecasts. Note also that from t + 228 h to t + 144 h the control and the ensemble mean always jump in phase.

b. Case 2: Jumpiness during a fast-moving cyclone in January 2008

The second case was associated with a fast-moving, very intense cyclone sweeping across northern Europe in January 2008 and causing disruption and damages to the surrounding countries (Fig. 5). Figure 6 shows the inconsistency indices of the EC-EPS control and ensemble-mean forecasts as a function of forecast range for this event. This case differs substantially from the previous one in the extent to which the ensemble mean follows the control. In the February case (Fig. 4) the control and the ensemble mean changed from run to run very closely for days, while in this case the relationship is much less apparent and the parallel zigzagging is rather limited.

c. Time series of the inconsistency index for the t + 168-h forecasts valid for January–February 2008

Before section 4’s discussion of the 18-month average results, it is interesting to analyze in some detail a time series of the inconsistency index for a shorter, 2-month period. Figure 7 shows the inconsistency indices for the t + 168-h forecasts from January to February 2008. The inconsistency index of the EC-EPS ensemble-mean forecasts is smaller than that of the control forecast for most cases, with synchronized zigzag events for both the ensemble mean and the control occurring only very rarely (see e.g., the consecutive runs around 4–7 January 2008, when both indices have roughly the same values). The correlation coefficient between the two curves is 0.60.

a. Period-average inconsistency statistics

Figure 8 shows that all average inconsistency indices increase with the forecast range in agreement with the practical experience that the forecasts are usually more consistent at short forecast ranges. Figure 8 also shows that for both ensemble systems, the ensemble-mean period-average inconsistency index is smaller than the control one, especially at long forecast ranges. Although it is difficult to relate the differences between the indices to precise system characteristics, factors that might have contributed to them are model resolution, model activity, and forecast skill. Figure 8 shows, for example, that the EC-EPS control is more inconsistent than the UK-EPS control, which could be because the EC-EPS has a higher resolution, and its members are more active. By contrast, Fig. 8 shows that the EC-EPS ensemble mean is less inconsistent than the UK-EPS ensemble mean: this could be due to the fact that the EC-EPS has a larger, better tuned ensemble spread (Park et al. 2008).

b. Control and ensemble-mean inconsistency correlation statistics

To measure the relationship between the inconsistency indices of the control and ensemble-mean forecasts, the product-moment correlation coefficient (Neter et al. 1988) between their corresponding inconsistency indices has been computed. In both the EC-EPS and UK-EPS, the initial perturbations are centered on the control forecast, therefore the correlation between control and ensemble-mean inconsistency will be high at short ranges, while initial perturbations evolve linearly, and then decrease as perturbation growth becomes more nonlinear (Fig. 9). The correlation is slightly stronger in the EC-EPS than in the UK-EPS during the first few days, while it is weaker in the medium range. This is also consistent with the spread characteristics of the two systems: in the short forecast range the UK-EPS has a larger spread since it uses larger initial perturbations. This makes each perturbation’s growth more nonlinear, and the ensemble-mean/control divergence stronger at shorter forecast ranges. The opposite is valid in the long range, where the UK-EPS is underdispersive (Park et al. 2008).

It is interesting to contrast these correlations with the correlation between the inconsistencies of the two control forecasts or the two ensemble-mean forecasts. Figure 9 shows that the inconsistency correlation between the two control forecasts (pluses) and the two ensemble-mean forecasts (crosses) are much lower than the correlations obtained for each individual system (squares and triangles). The large difference between the same-system and the different-system correlations throughout the forecast range could be seen as an indication that each single system is still not capable of properly simulating the effect of model or analysis uncertainty.

c. Forecast jump frequency statistics

Occasionally (e.g., the case discussed in section 3a), the ensemble mean follows the control very closely in a zigzagging manner for a number of runs. If such situations happened too often, the capability of the EPS to improve the prediction of potentially damaging weather scenarios could be reduced. To measure the frequency of occurrence of such events we counted the number of cases when the control and the ensemble-mean forecasts—either individually or in parallel—made flips, flip-flops, and flip-flop-flips.

Figure 10 shows that the frequency of individual flips is similar throughout the forecast range for the control and ensemble-mean forecasts for both systems. The parallel flip frequency between the control and the ensemble-mean forecasts of each system reflects the weakening relationship between the control and the ensemble mean with the forecast range (cf. Fig. 9): at short ranges almost every jump occurs simultaneously, but this decreases steadily through the medium range. The parallel flips between EC-EPS and UK-EPS also reflect the correlations discussed in the previous section: the two systems jump together substantially less often than the same-system control and ensemble mean.

The frequencies of flip-flops (Fig. 11) and flip-flop-flips (Fig. 12) are lower than for flips. For the ensemble mean, the flip-flops and flip-flop-flips indices are noticeably smaller than the control indices for both EPS systems, especially in the medium range. Thus, although the control and ensemble-mean forecasts have a similar numbers of flips, the ensemble mean has a definite smaller frequency of flip-flops and flip-flop-flips. This confirms the earlier results that the ensemble mean jumps less than the control forecast. Note also that this is more evident for the EC-EPS: this, as already mentioned, is probably linked to the fact that the EC-EPS has a better tuned spread than the UK-EPS.

d. Results sensitivity to parameters, areas, and sampling periods

To assess the robustness of the results presented above, the inconsistency indices have been assessed for the other variables, different areas, and sampling periods. The discussion is limited to the EC-EPS system.

Figure 13 shows the sensitivity of the period-average inconsistency to the parameter. Although the relationship between the control and the ensemble-mean curves is similar for all parameters, the absolute values vary substantially. Over the medium northwestern European area the most inconsistent parameter is the 1000-hPa geopotential height; and forecasts are more consistent for temperature than for geopotential height.

Regarding the sensitivity of the inconsistency to the area, first consider the areas nested in the large northwestern European area (Fig. 1), which by construction, are affected on average by similar weather climatology. Figure 14 shows that the period-average inconsistency is very sensitivity to the area size. This strong sensitivity to the area size is not surprising, since it reflects the fact that, in general, any forecast, or a forecast rms error, has more variability if computed over a smaller than a larger area. Consider now the indices for the middle of the northwesterly areas and the one for the southest area, which have the same size but are affected, on average, by different synoptic activity and weather regimes. Figure 14 shows that the indices for the two areas are very similar, suggesting that there is little sensitivity to the weather climate. The different flip indices and the inconsistency correlation between the control and the ensemble mean also show little sensitivity to the choice of the weather parameter and the area size (not shown).

The sensitivity of the inconsistency indices to the sampling period has also been investigated. Although there is some moderate variability on the seasonal scale, it becomes very small if averages are computed for longer periods (i.e., 12 months or more, not shown), confirming the limited sensitivity of the inconsistency to the weather regimes.