An Assessment of the Performance of the Operational Global Ensemble Forecast Systems in Predicting the Forecast Uncertainty

Carlee F. Loeser; Michael A. Herrera; Istvan Szunyogh

doi:10.1175/WAF-D-16-0126.1

Jump to Content Jump to Main Navigation

Weather and Forecasting

ProCite

RefWorks

Reference Manager

BibTeX

Zotero

EndNote

View raw image

Fig. 1.

Temporal evolution of the diagnostics , , , and in the forecasts for the (left) ECMWF and (right) NCEP ensembles in the NH extratropics.

View raw image

Fig. 2.

Temporal evolution of the diagnostics , , , and in the forecasts for the (top left) CMC, (top right) JMA, (bottom left) KMA, and (bottom right) UKMO ensembles in the NH extratropics. Notice that the maximum forecast time is different for the different ensembles.

View raw image

Fig. 3.

Spatial distribution of the bias [(J kg⁻¹)^1/2] for the ECMWF ensemble, averaged over all forecasts in the verification dataset for (top left) the analysis time, and at forecast times (top right) 72 h, (bottom left) 120 h, and (bottom right) 360 h, shown by the color shading. The black contours represent the temporal mean of the geopotential height at 500 hPa for January and February 2015. The heavy dashed line marks the southern boundary of the verification region (30°N).

View raw image

Fig. 4.

Spaghetti diagrams for the temporal mean of the ECMWF ensemble for January and February 2015. Shown are the diagrams for the 5640-gpm isohypse at 500 hPa at (top left) analysis time and forecast lead times of (top right) 72 h, (bottom left) 120 h, and (bottom right) 360 h. The gray contour lines show the temporal mean of the ensemble members, while the black contour line shows the temporal mean of the ECMWF analyses.

View raw image

Fig. 5.

Spaghetti diagrams for the temporal mean of the 360-h (top) ECMWF, (middle) NCEP, and (bottom) CMC ensemble forecasts for January and February 2015. Shown by gray contour lines is the 5640-gpm isohypse at 500 hPa for the ensemble members and by black contour lines for the ECMWF analyses. (Results are not shown for the remaining ensembles, because they do not provide forecasts at 360-h lead time.)

View raw image

Fig. 6.

Zonal anomalies of the time-mean flow for January and February of (left) 2012 and (right) 2015. Shown by color shading are the zonal anomalies (gpm), and by black contours the temporal mean of the geopotential height at 500 hPa. The heavy dashed line marks the southern boundary of the verification region (30°N). The computation of the time flow is based on ECMWF analyses.

View raw image

Fig. 7.

Spatial distribution of the temporal mean of for the ECMWF ensemble for January and February 2015. Shown by color shades is the temporal mean of for (top left) the analysis time and forecast times of (top right) 72 h, (bottom left) 120 h, and (bottom right) 360 h. Black contours show the temporal mean of the geopotential height at 500 hPa for January and February 2015. The heavy dashed line marks the southern boundary of the verification region (30°N).

View raw image

Fig. 8.

The joint probability distribution of the E dimension and the explained variance for the ECMWF ensemble in the NH extratropics at (top left) the analysis time, and forecast times of (top right) 72 h, (bottom left) 120 h, and (bottom right) 360 h. The bin increments are 0.25 for the E dimension and 0.005 for the explained variance. The maximum possible value of the E dimension is 24 for the analysis and 49 for the forecasts.

View raw image

Fig. 9.

Graphical illustration of the relationship between the E dimension and the minimum of the explained variance in the NH extratropics. The training data are represented by triangles, the test data by open circles, and the fitted linear regression function is shown by the straight black line. The test data would fall on the straight line if the linear model was perfect. Shown are the distributions for (top left) the analysis time, and forecast times of (top right) 72 h, (bottom left) 120 h, and (bottom right) 360 h. The legends show the average correlation values for the training dataset ( training) and the test dataset ( test), as well as the average values of a and b.

View raw image

Fig. 10.

As in Fig. 9, but for the relationship between the E dimension and the 5th percentile of the explained variance in the NH extratropics.

View raw image

Fig. 11.

As in Fig. 9, but for the relationship between and the 95th percentile of in the NH extratropics. The legends show the average correlation values for the training dataset ( training) and the test dataset ( test), as well as the average values of c and d.

	All Time	Past Year	Past 30 Days
Abstract Views	0	0	0
Full Text Views	126	41	5
PDF Downloads	95	30	1

GENERAL CONDITIONS

Author:

A. J. HENRY

No. III

BAROMETRIC PRESSURE

VERIFICATIONS

MISCELLANEOUS PHENOMENA

Editorial Type:: Article

Article Type:: Research Article

An Assessment of the Performance of the Operational Global Ensemble Forecast Systems in Predicting the Forecast Uncertainty

Carlee F. Loeser

Carlee F. LoeserTexas A&M University, College Station, Texas

Search for other papers by Carlee F. Loeser in
Current site
Google Scholar
PubMed

Michael A. Herrera

Michael A. HerreraTexas A&M University, College Station, Texas

Search for other papers by Michael A. Herrera in
Current site
Google Scholar
PubMed

, and

Istvan Szunyogh

Istvan SzunyoghTexas A&M University, College Station, Texas

Search for other papers by Istvan Szunyogh in
Current site
Google Scholar
PubMed

Print Publication:: 01 Feb 2017

DOI:: https://doi.org/10.1175/WAF-D-16-0126.1

Page(s):: 149–164

Received:: 11 Jul 2016

Accepted:: 22 Oct 2016

Published Online:: Feb 2017

Displayed acceptance dates for articles published prior to 2023 are approximate to within a week. If needed, exact acceptance dates can be obtained by emailing amsjol@ametsoc.org.

Download PDF

Full access

Abstract

This study investigates the efficiency of the major operational global ensemble forecast systems of the world in capturing the spatiotemporal evolution of the forecast uncertainty. Using data from 2015, it updates the results of an earlier study based on data from 2012. It also tests, for the first time on operational ensemble data, two quantitative relationships to aid in the interpretation of the raw ensemble forecasts. One of these relationships provides a flow-dependent prediction of the reliability of the ensemble in capturing the uncertain forecast features, while the other predicts the 95th percentile value of the magnitude of the forecast error. It is found that, except for the system of the Met Office, the main characteristics of the ensemble forecast systems have changed little between 2012 and 2015. The performance of the UKMO ensemble improved in predicting the overall magnitude of the uncertainty, but its ability to predict the dominant uncertain forecast features was degraded. A common serious limitation of the ensemble systems remains that they all have major difficulties with predicting the large-scale atmospheric flow in the long (longer than 10 days) forecast range. These difficulties are due to the inability of the ensemble members to maintain large-scale waves in the forecasts, which presents a stumbling block in the way of extending the skill of numerical weather forecasts to the subseasonal range. The two tested predictive relationships were found to provide highly accurate predictions of the flow-dependent reliability of the ensemble predictions and the 95th percentile value of the magnitude of the forecast error for the operational ensemble forecast systems.

© 2017 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (http://www.ametsoc.org/PUBSCopyrightPolicy).

Corresponding author e-mail: Carlee F. Loeser, cl75696@gmail.com

Abstract

Corresponding author e-mail: Carlee F. Loeser, cl75696@gmail.com

Keywords: Ensembles; Forecast verification/skill; Numerical weather prediction/forecasting

1. Introduction

The goal of ensemble forecasting is to predict the probability distribution of the atmospheric state given all sources of the forecast uncertainty. Ensemble verification techniques verify either the predicted probabilities of weather events or selected statistical parameters of the predicted probability distribution of the state vector (e.g., Talagrand et al. 1997). In the latter case, the focus is usually on the verification of the first- and second-order moments of the probability distribution. The prediction of the first moment is the ensemble mean forecast, while the predictions of the second-order central statistical moments are the entries of the ensemble covariance (dispersion) matrix. The diagonal entries of this matrix represent the predicted variance of the state vector components, while the off-diagonal entries represent the covariance between pairs of the state vector components. The range of this matrix is the linear (vector) space spanned by the ensemble perturbations, the vectors that represent the difference between the ensemble members and the ensemble mean (e.g., Szunyogh 2014). This vector space can be considered a flow-dependent prediction of the space of forecast uncertainty. The efficiency of an ensemble in capturing the forecast uncertainty can be measured by computing the projection of the vector that represents the difference between a proxy for the true state, such as an analysis, and the ensemble mean on that space. There are a number of papers that have investigated the performance of operational ensemble forecast systems along these lines (e.g., Molteni and Buizza 1999; Wei and Toth 2003; Buizza et al. 2005; Herrera et al. 2016). Perhaps the most important difference between these studies was that they chose the components of the state vector in the computation of the verification metrics differently.

This paper presents the latest results of the investigations of our research group into the efficiency of global forecast ensembles in capturing the spatiotemporal evolution of the forecast uncertainty. In these investigations, the diagnostics have been computed for local state vectors, a choice motivated by the data assimilation papers by Ott et al. (2004) and Szunyogh et al. (2005). A local state vector is defined at each grid point of the ensemble dataset. The components of the local state vector are model gridpoint variables in a highly localized volume of the atmosphere, which is centered at the location associated with the local state vector. The investigations of our group started with an analysis of data generated by a research forecast system that was based on the model component of the Global Forecast System (GFS) of the National Centers for Environmental Prediction (Kuhl et al. 2007; Satterfield and Szunyogh 2010, 2011). It continued with a recent study by Herrera et al. (2016), in which the diagnostics of the earlier papers were applied to global ensemble forecast data from the world’s leading operational numerical weather prediction centers for January–February 2012.

The present paper updates the results of Herrera et al. (2016) based on data from January–February 2015, which allows for an assessment of the progress of the last three years. It also documents our first attempt to validate two prognostic relationships found by Satterfield and Szunyogh (2010, 2011) between the raw ensemble forecasts and the actual forecast uncertainty. One of these relationships provides a tool for the routine prediction of the reliability of the ensemble in capturing the uncertain forecast features. The other one predicts the 95th percentile value of the forecast error. In what follows, we explain the local diagnostics adapted from Herrera et al. (2016) and the performance of the predictive schemes of Satterfield and Szunyogh (2010, 2011) (section 2), describe the operational ensemble data that we analyze (section 3), present the verification results (section 4), and offer our conclusions (section 5).

2. Local diagnostics and predictive schemes

A forecast ensemble samples the flow-dependent multivariate probability distribution of the present and future atmospheric states given the sources of forecast uncertainty. We verify the ensemble-based predictions of the first and second central moments (mean, variances, and covariances) of the probability distribution of the local atmospheric states.

a. Local state vectors

Let be the index that identifies the horizontal location of a grid point. At location , we define the components of the local state vector by the gridpoint values of the two horizontal wind components and temperature in a rectangular box centered at . The wind and temperature components of are scaled such that the Euclidean norm of the local state vector has a dimension of square root of energy [for the definition of the scaling factors see Talagrand (1981), Buizza et al. (1993), and Oczkowski et al. (2005)].

Let

be the representation of the local true state. Because ensemble verification techniques consider the true state a single realization of the random variable that represents the probability distribution of the atmospheric state given all sources of forecast uncertainty,

can be written as

where

is the (unknown) true mean of the probability distribution of the local state and

is the difference between a realization of the random variable and the mean of the distribution from which it was drawn. (Notice that the mean of

is 0.) The ultimate goal of ensemble prediction is to predict the probability distribution of the atmospheric state. In this paper, our focus is on the predictions of

and the second central moments (variances and covariances) of

. To be precise, we verify whether a sample (time series) of ensemble-based predictions of

and the second central moments of

are consistent or not with the related sample (time series) of

b. Local ensemble perturbations

For a K-member ensemble of local state vectors

, the ensemble of local ensemble perturbations

, is defined by

where

is the local ensemble mean:

The ensemble-based prediction of the variances and covariances of the forecast uncertainty are described by the local ensemble covariance matrix:

where the local ensemble perturbations are represented by column vectors and the superscript T denotes the matrix transpose. The ensemble mean

is the ensemble-based prediction of

, while

is the prediction of the covariance matrix of

c. Diagnostics

Because the true state

is unknown, forecasts are verified against a proxy

for the true state. We assume that the error in the proxy can be described by the random variable

; that is,

1) Bias

The relationship

cannot be verified for a single ensemble forecast and location, because an accurate independent estimate of

is not available in a realistic situation. In other words, no practical technique exists to quantify the error in a prediction of the spatiotemporally evolving mean of the probability distribution of the state. A verifiable, necessary, condition for Eq. (6) to hold is

where

is the expected value for forecasts started at different times. Under the standard hypothesis of ensemble forecasting that the processes that govern the evolution of

are ergodic, Eq. (7) can be written as

Making the additional (common) assumption that

is an unbiased estimate of

[

], substituting for

in Eq. (8) from Eq. (5), and rearranging the resulting equation leads to

Introducing the notation

Eq. (9) can be written in the equivalent form

The mean

for location

can be estimated by computing the average of a sample of

for a sufficiently long verification time period. The result is a map of the systematic error (bias) of the ensemble mean forecasts. The values of

on the map can be averaged over the locations

to obtain a single-number M for the characterization of the bias.

2) Variance

Under the assumption that Eq. (6) is satisfied, the trace

, is a prediction of the variance of

. That is, ideally, the relation

would be satisfied for each forecast and location. Similar to the situation for the mean, this condition cannot be verified for a single ensemble forecast and location. Taking the expected value of Eq. (12) and making use of the ergodic hypothesis lead to

Under the assumption that the ensemble satisfies Eq. (6),

and the right-hand side of Eq. (13) can be expanded as

The second term of the last part of Eq. (15) is twice the covariance between

and

. For a properly chosen, high quality verification dataset, this correlation can be assumed to be zero. Hence, Eq. (13) can be written as

where

. The second term on the right-hand side of Eq. (16) is the variance of the error in the verifying data. Because this term does not grow with the forecast lead time, while both

and

grow rapidly [see Herrera et al. (2016) for an estimate of the growth rates for the different TIGGE ensembles], its contribution can be neglected, except for the shortest (shorter than about 24–36 h) lead times. Hence, introducing the notations

and

, Eq. (16) can be written in the equivalent form

where

represents the variance in the ensemble and

represents the forecast uncertainty at location

The two sides of Eq. (17) can be estimated by computing averages of and for a sufficiently large sample of ensemble forecasts and verification data. These estimates can be averaged over the locations to obtain two scalar quantities, and , for comparison. The relation is often referred to as the spread–skill relationship, because characterizes the ensemble spread in state space, while can be considered an estimate of the mean-square error (skill) of the deterministic forecasts based on the ensemble mean.

3) Covariance

The ensemble perturbations

, in a linear sense, span a linear vector space

for each location

. The ensemble captures the uncertain forecast features in the local neighborhood

, if the magnitude

of the projection

onto

is equal to the magnitude

. Because the ensemble typically captures only part of the forecast uncertainty,

We call the ratio

the explained variance. Under the assumption that the error

in the verifying data has no projection on

, and the contribution of

is negligible, the explained variance can be estimated by

where

and

is the projection of

onto

. The smaller the estimated value of

, the lower the efficiency of the ensemble in capturing the forecast uncertainty. Unlike the diagnostic relationships discussed earlier, this relationship can be verified for a particular forecast and location. We will take advantage of this property of

in section 2d.

In practice, can be computed by projecting onto the set of normalized eigenvectors associated with the largest eigenvalues of , which provides a convenient orthonormal basis for the computations. The mean magnitude of can be estimated by averaging over a sample of forecasts. The resulting local estimates can be further averaged over the locations to obtain a single scalar measure for the characterization of the mean projection. This scalar measure always satisfies the relation . The larger the difference between and , the poorer the performance of the ensemble in capturing the spatial structure of the forecast uncertainty.

d. The predictive linear relationships

The ensemble dimension (E dimension) is a measure of the steepness of the eigenvalue spectrum of : the smaller the E dimension, the steeper the spectrum. Satterfield and Szunyogh (2010) found the E dimension to be a good linear predictor of the lower bound of the explained variance. In addition, Satterfield and Szunyogh (2010) found a strong linear relationship between the ensemble spread and the 95th percentile value of the forecast error. We show that these two linear relationships also hold for the ECMWF ensemble.

1) The minimum explained variance

The E dimension (Patil et al. 2001; Oczkowski et al. 2005) is

where

are the eigenvalues of

. The subscript

in the notation

indicates that the E dimension is computed for local volumes. It takes its smallest possible value of 1 when the ensemble variance is associated with a single pattern of uncertainty (eigenvector), and its largest possible value of K − 1 when the ensemble variance is evenly distributed between K − 1 different patterns of forecast uncertainty. Satterfield and Szunyogh (2010) found that the minimum value

of the explained variance

that the ensemble was guaranteed to capture satisfied, to a good approximation, the linear relationship

where a and b are empirical scalars that can be determined from a sample of ensemble and verification data.

2) The 95th percentile value of the forecast error

According to Eq. (12), the spatiotemporally varying ensemble spread

is a predictor of the root-mean square of the forecast uncertainty. Hence, the larger the ensemble spread, the larger the expected magnitude of the forecast uncertainty. What limits the practical quantitative forecast value of this relationship is that for large values of the spread, the magnitude of the forecast error can vary within a wide range (e.g., Fig. 4 in Satterfield and Szunyogh 2011). Satterfield and Szunyogh (2011) found a potentially more useful quantitative relationship between the ensemble spread and the magnitude of the forecast error by noticing that, to a good approximation, the exceptionally large values of the magnitude of the forecast error depended linearly on the ensemble spread. More precisely, they found the relationship

where

is the 95th percentile value of

given

. The parameters c and d are empirical scalars that can be determined from a sample of ensemble and verification data. If Eq. (22) held for the operational ensembles, it would provide a practical formula for the quantitative prediction of the 95th percentile value of the forecast error.

3. The forecast and verification data

We examine ensemble forecast data distributed through the THORPEX Interactive Grand Global Ensemble (TIGGE) by the forecast centers.

a. TIGGE

TIGGE includes operational global model forecasts from 10 major numerical weather prediction centers (Bougeault et al. 2010; Swinbank et al. 2016), but we process forecast data only from the

European Centre for Medium-Range Weather Forecasts (ECMWF),
National Centers for Environmental Prediction (NCEP),
Met Office (UKMO),
Japan Meteorological Agency (JMA),
Korean Meteorological Administration (KMA), and
Meteorological Service of Canada [Canadian Meteorological Centre (CMC)].

Data from the remaining four centers, the China Meteorological Administration (CMA), Météo-France, the Australian Bureau of Meteorology (BoM), and Brazil’s Centro de Previsão de Tempo e Estudos Climáticos (CPTEC), were unavailable, or had quality issues for the investigated time period. Table 1 summarizes the main features of the ensembles of the different centers. Because Herrera et al. (2016) provided a detailed overview of the ensemble generation techniques of the different centers, we give only a brief summary of the most important changes to those techniques between 2012 and 2015.

Table 1.

The main parameters of the investigated ensemble forecast systems.

KMA has moved from using bred vector initial condition perturbations to using perturbations generated by an ensemble transform Kalman filter (ETKF). They now also use a combination of stochastic kinetic energy backscatter (SKEB) and random parameter (RP) schemes to simulate the effects of random model errors. UKMO also switched to a combination of SKEB and RP schemes, reduced the number of ensemble members from 14 to 11, and shortened the maximum forecast time from 360 to 168 h. These changes to the UKMO ensemble were made as part of the Met Office’s strategy to wind down its focus on week 2 forecasting (R. Swinbank 2014, personal communication). JMA has increased the frequency of ensemble forecasts from once to twice daily and extended the maximum forecast time from 216 to 264 h.

b. The components of the local state vector

Following Satterfield and Szunyogh (2010, 2011) and Herrera et al. (2016), we choose to be 12.5° × 12.5° horizontally and extend from 1000 to 200 hPa vertically. (In the midlatitudes, the horizontal dimension of is about 1000 km × 1000 km.) We choose a horizontal dimension of this size because we focus on error growth at synoptic scales. Because the TIGGE archive uses a horizontal resolution of 2.5° × 2.5° and has eight pressure levels between 1000 and 200 hPa, the dimension of the local state vector is 3 × 5 × 5 × 8 = 600.

In our diagnostic calculations, is defined by gridpoint values of the operational ECMWF control analyses in the TIGGE database. For the verification of the ECMWF model forecasts, the operational NCEP control analyses are used as the proxy . We do not use the ECMWF analyses in the verification of the ECMWF forecasts to ensure that and remain statistically independent. The statistical independence of these two random variables is a necessary condition for Eq. (15) to hold. The only drawback to this approach is that it makes a strict quantitative comparison of the statistics for the ECMWF ensemble and the other ensembles at short forecast times impossible. We note, however, that Herrera et al. (2016) investigated the sensitivity of the diagnostic relationships used here to the choice of the verification data by computing the diagnostics for the UKMO ensemble using both ECMWF and NCEP analyses as proxy for the true state, and found that the results were only very weakly sensitive to the choice of the verification data.

4. Results

We present results first for the evolution of , , , and for the different ensembles, then for the application of the two predictive relationships to the ECMWF ensemble.

a. Comparison of VS, TV, and TVS

We compute , , and by averaging , , and over all 0000 and 1200 UTC forecasts for January and February, and , , and by averaging , , and over the Northern Hemisphere extratropics (30°–75°N). We display the values of all diagnostics, including those of the bias, as functions of forecast time. It is important to note while comparing the evolution of the diagnostics for the ensembles of the different centers that the maximum forecast lead time is not the same for all of them.

The evolution of , , and for ECMWF is displayed in the left panel of Fig. 1. As expected, for all lead times, indicating that the ensemble misses some patterns of forecast uncertainty. However, is only slightly smaller than after about 144 h and maintains this desirable behavior at the later lead times. While at analysis time, grows faster initially than , leading to an overestimation of the magnitude of the forecast uncertainty captured by the ensemble. In other words, the magnitude of the patterns of uncertainty that are captured by the ensemble are overinflated in an attempt to better represent the overall magnitude of the forecast uncertainty. Overall, the results for ECMWF are very similar to those reported by Herrera et al. (2016) for 2012.

The results for NCEP are shown in the right panel of Fig. 1. For this ensemble, is visibly smaller than at all lead times. Compared with the behavior of the ECMWF ensemble, the consistency between , , and is weaker, and the convergence of to is slower for the NCEP ensemble. In general, these particular diagnostics suggest that the performance of the NCEP ensemble is poorer than that of the ECMWF ensemble. Because Herrera et al. (2016) reported similar differences in the performance of the two ensembles, the new results suggest that ECMWF has managed to maintain its advantage over NCEP.

The results on the evolution of , , and for the remaining four TIGGE ensembles are shown in Fig. 2. The differences between the patterns of behavior of the different ensemble systems are the largest at analysis time. These differences reflect substantial differences between the techniques for the generation of the initial condition perturbations. The CMC ensemble behaves similarly to the NCEP ensemble, in that asymptotes to quickly, while remains smaller than at all forecast times. The CMC, KMA, and UKMO ensembles overestimate the magnitude of the uncertainty captured at the longer lead times even more than the NCEP ensemble. The JMA ensemble behaves differently than the others, especially at analysis time, at which underestimates . Because of the more rapid initial growth of , catches up with by forecast time 12 h and remains larger until about forecast time 144 h. This is a feature of the JMA ensemble that was also observed by Herrera et al. (2016) and can be attributed to the use of the right singular vectors as initial condition perturbations. In essence, in the JMA ensemble, the magnitude of the initial perturbations is tuned to an unrealistically small value to compensate for the unrealistically rapid initial growth of the singular vectors.

A unique aspect of the evolution of for the KMA ensemble is that it is larger than until about forecast time 48 h. Despite this overinflation of at the early forecast times, underestimates beyond the forecast time 96 h. In contrast to the behavior of the KMA ensemble, for the UKMO ensemble, matches well at all forecast times. However, for that ensemble, significantly underestimates at all forecast times. The large discrepancy between and shows that the UKMO ensemble fails to capture some important features of uncertainty, and it compensates for that deficiency by overinflating the magnitude of the captured uncertainty. Because Herrera et al. (2016) did not observe a similar behavior for the UKMO ensemble, our results suggest that the changes implemented at UKMO degraded the performance of the UKMO ensemble in this respect. Part of this degradation is not unexpected, because the reduction of the ensemble members inevitably leads to a reduction of the number of state space directions (patterns) in which the ensemble can capture the uncertainty. (We recall that the number of UKMO ensemble members was reduced from 14 to 11.) The effect of this degradation of the performance of the raw ensemble on the Met Office ensemble forecast products is unclear, because those products are based on twice as many ensemble members, with the additional members coming from an ensemble run prepared by a 6-h time lag.

b. The evolution of M²

Figures 1 and 2 show a slow but general growing trend of the bias with increasing forecast time, for all ensemble systems. Herrera et al. (2016), who found a similar growth of the bias for the 2012 data, concluded that this result was due to growing errors in the prediction of the low-frequency transients. The similarity between the shapes of the curves of in Figs. 1 and 2 of the present paper and Fig. 3 in Herrera et al. (2016) suggests that the growth of the bias in 2015 was due to the same process. Figure 3, which shows the spatial distribution of for the ECMWF ensemble, provides strong support to this conjecture.^¹ Figure 3 shows that, similar to the situation in 2012, the locations of the maxima of are aligned with the large-scale waves present in the time-mean flow. These local maxima of the bias occur because the ensemble predicts an overly zonal flow at the long lead times (Fig. 4). An additional similarity with the results for 2012 is that this undesirable behavior is shown by the ensembles of all forecast centers (Fig. 5).

While the qualitative patterns of behavior of are very similar between 2012 and 2015, there is an important quantitative difference between the results of the two years: at long lead times, the values of are lower in 2015 than in 2012 (about 588 versus 683 J kg⁻¹ for ECMWF and 650 versus 847 J kg⁻¹ for NCEP at 360-h lead time). The CMC ensemble, the only other ensemble that provides forecasts for the 360-h lead time, shows a similar pattern of behavior ( was reduced from 803 to 599 J kg⁻¹). One possibility is that the reduction of is due to the progress made by the centers between 2012 and 2015, while another is that it is due to differences between the flow regimes of the two years. To confirm or rule out the latter possibility, next, we compare the zonal anomalies of the time-mean flow for the two years (Fig. 6).

The positive anomalies (ridges) are stronger in the North Atlantic region and over eastern Europe in 2012, as well as in the northeast Pacific region in 2015. The region where is the largest coincides with the location of the strongest positive zonal anomalies in both years: in 2012, it is in the North Atlantic region (Herrera et al. 2016), while in 2015, it is in the northeast Pacific region (bottom-right panel in Fig. 3). This observation suggests that, because the ensemble forecasts have difficulties with maintaining the large-scale ridges, the systematic error in the long-term ensemble mean forecasts is the largest where the strongest ridge is located. This relationship suggests that the difference between the flow regimes of 2012 and 2015 does contribute to the differences between the values of in the long forecast range. The fact that is the lowest for the ECMWF ensemble and the highest for the NCEP ensemble in both years suggests that differences between the quality of the ensemble systems may also play a role in determining the value of . However, the differences in the values of for the ensembles of the different centers are small compared to the reduction that could be potentially achieved by maintaining the large-scale zonal anomalies in the long-range forecasts.

c. Results for the predictive linear relationships

The explained variance , which was defined by Eq. (18), determines the efficiency of the ensemble in capturing the patterns of forecast uncertainty. The spatial distribution of the explained variance for the ECMWF ensemble averaged over all of the forecasts is displayed in Fig. 7. As expected based on the results of the earlier studies, the explained variance initially grows rapidly with increasing forecast time in the Northern Hemisphere storm track regions (top-right and bottom-left panels in Fig. 7); then it grows slowly at all locations, reaching values between 0.75 and 1 by the 360-h lead time.

Kuhl et al. (2007) and Satterfield and Szunyogh (2010) observed a negative correlation between and : a lower E dimension indicated a higher explained variance, especially at forecast times longer than 3 days. In other words, when and where the ensemble spread was dominated by a few patterns, those patterns provided an efficient representation of the structure of the forecast uncertainty.

As was done in the previous studies, we prepare estimates of the joint probability distribution function (JPDF) of the explained variance and the E dimension. To obtain estimates of the JPDF, we compute the relative frequencies of the values of the explained variance and the E dimension for discrete bins of the values for all locations of all forecasts in the NH extratropics. We define 200 × 200 bins by using an increment of 0.25 for the E dimension and 0.005 for the explained variance. The number of occurrences in each bin is normalized by , where n = 305 856 is the total number of data points. Figure 8 shows the resulting estimates of the JPDF at analysis time and at forecast lead times of 72, 120, and 360 h. At analysis time, the maximum value of the E dimension is about 24. This number reflects that ECMWF generates its 50 ensemble perturbations in pairs of negative and positive perturbations; thus, half of the perturbations are linearly dependent on the other half, which makes the maximum of the E dimension . In the first 72 h, both the E dimension and the explained variance tend to grow. While the explained variance keeps growing as forecast time increases, albeit at a slower rate than in the first 72 h, the largest values of the E dimension decrease (cf. the two right panels in Fig. 8). The distributions at the 72- and 120-h lead times “lean backward,” indicating a negative correlation between the two variables, as was expected based on the results of Kuhl et al. (2007) and Satterfield and Szunyogh (2010).

To find an approximate quantitative relationship between the explained variance and the E dimension, we fit a function of the form of Eq. (21) to the data pairs at each lead time. For the function fitting, we divide the data pairs randomly into training datasets and test datasets. Seventy-five percent of the data points are assigned to the training dataset, with the remaining 25% assigned to the test dataset. The data are ordered by values of E dimension and divided into 100 bins of equal numbers of data separately for the training and the test datasets. For each bin, we calculate the mean of the E dimension and the minimum value of the explained variance and perform a linear regression on the E dimension and the explained variance values from the training dataset. The linear regression provides the estimates of the parameters a and b. We use these values of a and b to predict the minimum of the explained variance in the test dataset based on the corresponding values of the E dimension.

The squared correlation values between the minimum of the explained variance and the mean E dimension are calculated for the training and test sets for each forecast lead time. This entire process, which started with randomizing the data, is repeated 100 times with varying training and test periods to provide a robust analysis. The values for each iteration are averaged together, and the results are listed in the first two rows in Table 2. While these correlation values represent the average over the 100 iterations, the graphical illustration of the results in Fig. 9 is for only a single random iteration. In Fig. 9, the dark triangles represent the training data, and the open circles represent the test data. Notice that the correlation values at the analysis time and at forecast time 360 h are low compared with the values at lead times of 72 and 120 h.

Table 2.

The values for the regressions in section 4c.

Satterfield and Szunyogh (2010) speculated that outliers were the likely cause of a general overprediction of the minimum explained variance. This motivates us to investigate the correlation between the E dimension and the 5th percentile value of the explained variance rather than its minimum. The results are shown in Fig. 10, with the corresponding average values of listed in the third and fourth rows in Table 2. The correlation between the E dimension and the 5th percentile of the explained variance is much higher for both the training and the test datasets than between the E dimension and the minimum of the explained variance.

Next, we investigate whether or not Eq. (22) holds for the ECMWF ensemble. The training and test datasets are constructed similarly to that already described for the estimation of the parameters a and b in Eq. (21). The available data are divided into 100 bins of equal number of values, and a linear regression is performed for each forecast lead time. This process is repeated for 100 randomly selected data pairs, and the results are summarized in the last two rows of Table 2, as well as in Fig. 11. The correlations are high at all forecast lead times, with the highest value of 0.98 for the test data at lead times of 72 and 120 h, indicating that is a good predictor of the 95th percentile value of the forecast error at all lead times.

5. Conclusions

We carried out diagnostic calculations to identify new directions for research and development to improve the design of ensemble forecast systems and the interpretation of the already available ensemble forecast information. The two main specific objectives of the paper were to update the results of Herrera et al. (2016) based on more recent operational ensemble forecast data and to validate two predictive linear relationships found by Satterfield and Szunyogh (2010, 2011) for a research ensemble.

Our main findings regarding the performance of the operational ensemble forecast systems are as follows:

The main characteristics of the ensemble systems of the different centers have not changed significantly between 2012 and 2015. The only exception is the UKMO ensemble, which was redesigned in 2014, shifting the focus from week 2 to week 1 predictions. The performance of this ensemble was improved in predicting the magnitude of the forecast uncertainty and was degraded in predicting the patterns of forecast uncertainty.
With respect to the performance measures of the present study, the ECMWF ensemble continues to provide the highest quality forecasts.
All ensembles have major difficulties with predicting the large-scale atmospheric flow in the long (longer than 240 h) forecast range. These difficulties are due to the inability of the ensemble members to maintain large-scale waves in the forecasts, which presents a stumbling block in the way of extending the skill of numerical weather forecasts to the subseasonal range.

The results show that the current technology of ensemble forecasting can provide reliable prediction of the first and second central moments of the probability distribution of the atmospheric state in the medium forecast range (from about 48–72 to 240 h). The ensembles, however, have serious difficulties at the shorter and longer forecast times. At the shorter forecast times, they cannot capture all uncertain forecast features ( for all ensembles). Therefore, they underestimate the magnitude of the forecast uncertainty ( ) unless they also overinflate the magnitude of the correctly captured patterns of uncertainty ( ). The ensembles that employ such overinflation are NCEP, CMC, KMA, and UKMO.

It is somewhat disappointing to see that the increasing level of integration of the ensemble forecasting and data assimilation systems has not yet resulted in better short-range ensemble performance. This result shows the need for continued efforts to improve the ensemble generation techniques. Closing the gap between and at the short forecast times may not be possible without using many more ensemble members, at least in the short forecast range.

The poor performance of the ensembles in maintaining the large-scale variability of the long-range forecasts is most likely due to deficiencies of the models rather than to shortcomings of the ensemble generation techniques. This may be the result of inadequate representation of tropical processes (e.g., MJO), atmosphere–ocean interactions, stratospheric variability, etc., in the models. The ensemble-based diagnostics of the present study can be used in the future to assess the forecast effects of improvements in the model.

The predictive relationships of Satterfield and Szunyogh (2010, 2011) hold well for the operational ensembles and could be utilized for the routine operational prediction of

the reliability of ensemble forecasts in capturing the local structure of the forecast uncertainty and
a near-worst-case scenario (95th percentile value) of the forecast error.

While it is widely accepted that ensembles can be used for the prediction of the forecast uncertainty, it is rarely recognized that the performance of the ensembles is also flow dependent. The first relationship provides a tool to predict this flow dependence. The second relationship can provide particularly valuable information for situations in which the ensemble spread is large. In such situations, the magnitude of the forecast error can vary across a wide range. The relationship provides a forecast of the top of that range. We also verified the two relationships for the other ensembles used in this study. The results (not shown) were similar to those for the ECMWF ensemble. We note that while we do not have a theoretical explanation for the second relationship, there is hope that the latest theoretical developments (Van Schaeybroeck and Vannitsem 2016) will lead to such an explanation.

Acknowledgments

This study was supported by the National Science Foundation (Grant ATM-AGS-1237613). We thank the three anonymous reviewers for their careful reading of our manuscript and their many helpful suggestions. Their comments helped us greatly improve the presentation of our ideas and results.

REFERENCES

Bougeault, P., and Coauthors, 2010: The THORPEX Interactive Grand Global Ensemble (TIGGE). Bull. Amer. Meteor. Soc., 91, 1059–1072, doi:10.1175/2010BAMS2853.1.
- Crossref
- Search Google Scholar
- Export Citation
Buizza, R., J. Tribbia, F. Molteni, and T. Palmer, 1993: Computation of optimal unstable structures for a numerical weather prediction model. Tellus, 45, 388–407, doi:10.1034/j.1600-0870.1993.t01-4-00005.x.
- Crossref
- Search Google Scholar
- Export Citation
Buizza, R., P. L. Houtekamer, Z. Toth, G. Pellerin, M. Wei, and Y. Zhu, 2005: A comparison of the ECMWF, MSC, and NCEP global ensemble prediction systems. Mon. Wea. Rev., 133, 1076–1097, doi:10.1175/MWR2905.1.
- Crossref
- Search Google Scholar
- Export Citation
Herrera, M. A., I. Szunyogh, and J. Tribbia, 2016: Forecast uncertainty dynamics in the THORPEX Interactive Grand Global Ensemble (TIGGE). Mon. Wea. Rev., 144, 2739–2766, doi:10.1175/MWR-D-15-0293.1.
- Crossref
- Search Google Scholar
- Export Citation
Kuhl, D., and Coauthors, 2007: Assessing predictability with a local ensemble Kalman filter. J. Atmos. Sci., 64, 1116–1140, doi:10.1175/JAS3885.1.
- Crossref
- Search Google Scholar
- Export Citation
Molteni, F., and R. Buizza, 1999: Validation of the ECMWF Ensemble Prediction System using empirical orthogonal functions. Mon. Wea. Rev., 127, 2346–2358, doi:10.1175/1520-0493(1999)127<2346:VOTEEP>2.0.CO;2.
- Crossref
- Search Google Scholar
- Export Citation
Oczkowski, M., I. Szunyogh, and D. J. Patil, 2005: Mechanisms for the development of locally low-dimensional atmospheric dynamics. J. Atmos. Sci., 62, 1135–1156, doi:10.1175/JAS3403.1.
- Crossref
- Search Google Scholar
- Export Citation
Ott, E., and Coauthors, 2004: A local ensemble Kalman filter for atmospheric data assimilation. Tellus, 56A, 415–428, doi:10.1111/j.1600-0870.2004.00076.x.
- Crossref
- Search Google Scholar
- Export Citation
Patil, D. J., B. R. Hunt, E. Kalnay, J. A. Yorke, and E. Ott, 2001: Local low-dimensionality of atmospheric dynamics. Phys. Rev. Lett., 86, 5878–5881, doi:10.1103/PhysRevLett.86.5878.
- Crossref
- Search Google Scholar
- Export Citation
Satterfield, E., and I. Szunyogh, 2010: Predictability of the performance of an ensemble forecast system: Predictability of the space of uncertainties. Mon. Wea. Rev., 138, 962–981, doi:10.1175/2009MWR3049.1.
- Crossref
- Search Google Scholar
- Export Citation
Satterfield, E., and I. Szunyogh, 2011: Assessing the performance of an ensemble forecast system in predicting the magnitude and the spectrum of analysis and forecast uncertainties. Mon. Wea. Rev., 139, 1207–1223, doi:10.1175/2010MWR3439.1.
- Crossref
- Search Google Scholar
- Export Citation
Swinbank, R., and Coauthors, 2016: The TIGGE project and its achievements. Bull. Amer. Meteor. Soc., 97, 49–67, doi:10.1175/BAMS-D-13-00191.1.
- Crossref
- Search Google Scholar
- Export Citation
Szunyogh, I., 2014: Applicable Atmospheric Dynamics: Techniques for the Exploration of Atmospheric Dynamics. 1st ed. World Scientific, 588 pp.
Szunyogh, I., E. Kostelich, G. Gyarmati, D. Patil, B. Hunt, E. Kalnay, E. Ott, and J. Yorke, 2005: Assessing a local ensemble Kalman filter: Perfect model experiments with the NCEP global model. Tellus, 57A, 528–545, doi:10.1111/j.1600-0870.2005.00136.x.
- Crossref
- Search Google Scholar
- Export Citation
Talagrand, O., 1981: A study of the dynamics of four-dimensional data assimilation. Tellus, 33, 43–60, doi:10.1111/j.2153-3490.1981.tb01729.x.
- Crossref
- Search Google Scholar
- Export Citation
Talagrand, O., R. Vautard, and B. Strauss, 1997: Evaluation of probabilistic prediction systems. Proc. Workshop on Predictability, Reading, United Kingdom, ECMWF, 1–25.
Van Schaeybroeck, B., and S. Vannitsem, 2016: A probabilistic approach to forecast the uncertainty with ensemble spread. Mon. Wea. Rev., 144, 451–468, doi:10.1175/MWR-D-14-00312.1.
- Crossref
- Search Google Scholar
- Export Citation
Wei, M., and Z. Toth, 2003: A new measure of the ensemble performance: Perturbation versus error correlation analysis (PECA). Mon. Wea. Rev., 131, 1549–1565, doi:10.1175//1520-0493(2003)131<1549:ANMOEP>2.0.CO;2.
- Crossref
- Search Google Scholar
- Export Citation

Figure 3 can be directly compared to Fig. 10 in Herrera et al. (2016).

Share Link

Copy this link, or click below to email it to a friend

Email this content

or copy the link directly:

https://journals.ametsoc.org/view/journals/wefo/32/1/waf-d-16-0126_1.xml

Link copied successfully

An Assessment of the Performance of the Operational Global Ensemble Forecast Systems in Predicting the Forecast Uncertainty

Abstract

Abstract

1. Introduction

2. Local diagnostics and predictive schemes

a. Local state vectors

b. Local ensemble perturbations

c. Diagnostics

1) Bias

2) Variance

3) Covariance

d. The predictive linear relationships

1) The minimum explained variance

2) The 95th percentile value of the forecast error

3. The forecast and verification data

a. TIGGE

b. The components of the local state vector

4. Results

a. Comparison of VS, TV, and TVS

b. The evolution of M²

c. Results for the predictive linear relationships

5. Conclusions

Acknowledgments

REFERENCES

Share Link

AMS Publications

Get Involved with AMS

Affiliate Sites

Follow Us

Contact Us

Sections

References

Figures

Cited By

Metrics

Related Content

GENERAL CONDITIONS

No. III

BAROMETRIC PRESSURE

VERIFICATIONS

MISCELLANEOUS PHENOMENA

An Assessment of the Performance of the Operational Global Ensemble Forecast Systems in Predicting the Forecast Uncertainty

Abstract

Abstract

1. Introduction

2. Local diagnostics and predictive schemes

a. Local state vectors

b. Local ensemble perturbations

c. Diagnostics

1) Bias

2) Variance

3) Covariance

d. The predictive linear relationships

1) The minimum explained variance

2) The 95th percentile value of the forecast error

3. The forecast and verification data

a. TIGGE

b. The components of the local state vector

4. Results

a. Comparison of VS, TV, and TVS

b. The evolution of M2

c. Results for the predictive linear relationships

5. Conclusions

Acknowledgments

REFERENCES

Share Link

AMS Publications

Get Involved with AMS

Affiliate Sites

Follow Us

Contact Us

b. The evolution of M²