## 1. Introduction

One of the most recent advances in medium-range numerical weather prediction has been the development of weather prediction systems based on a probabilistic approach, whereby not only is a single forecast of the future atmospheric state provided but an estimate of the probability density function of forecast states is predicted by the application of ensemble/Monte Carlo strategies. Practically, the prediction of the time evolution of the probability density function of forecast states has been based on a finite number of numerical integrations.

Since December 1992, global ensemble prediction systems have been part of the operational suites at the European Centre for Medium-Range Weather Forecasts (ECMWF) (see, e.g., Molteni et al. 1996) and the National Centers for Environmental Prediction (NCEP) (see, e.g., Tracton and Kalnay 1993). Since 1998, an ensemble prediction system has been operational also at the Canadian Meteorological Centre (CMC) (see, e.g., Houtekamer et al. 1996). A recent overview and assessment of the ECMWF, NCEP, and CMC systems is given by Buizza et al. (2003a). Additionally, global and regional ensemble systems have been under development at other institutions, such as Météo-France (J. Nicolau 2001, personal communication), the U.K. Met Office (Harrison et al. 1999), the Regional Meteorological Service of Emilia-Romagna (Molteni et al. 2001; Marsigli et al. 2001), The Royal Dutch Meteorological Institute (KNMI) (see, e.g., Hersbach et al. 2000), and the Bureau of Meteorology Research Centre (BMRC), Melbourne, Australia, as described in the present study.

Research in ensemble prediction started at BMRC in 1996. Since May 2000 the BMRC global ensemble prediction system has been run daily in research trials. Since July 2001 it has been run in a real-time operational mode by the Bureau of Meteorology (BoM) National Meteorological and Oceanographic Centre (NMOC); it is scheduled to become officially operational on 1 November 2004.

While the three global ensemble systems operational at ECMWF, NCEP, and CMC have been designed to provide probabilistic forecasts for both the Northern and Southern Hemispheres, their performance has been assessed over the Northern Hemisphere (see, e.g., Buizza 1997; Toth et al. 1997, 1998). The aim of the present study is to provide an assessment of the performance of the ECMWF Ensemble Prediction System (EC-EPS) and the preoperational BMRC Ensemble Prediction System (BM-EPS) over the Southern Hemisphere annulus (20°–60°S), denoted subsequently by SH, and over the Australian region, denoted subsequently by AU; these results focus on the 500-hPa geopotential height field; BM-EPS and EC-EPS daily forecasts from 2 April to 31 August 2001 (152 cases) are compared.

This study has focused on only the verification at 500 hPa in providing an initial evaluation and comparison of these two systems in the Southern Hemisphere. The recent intercomparison study of the performance of the ensemble prediction systems of ECMWF, NCEP, and CMC in the Northern Hemisphere, presented by Buizza et al. (2003a), has similarly focused on 500 hPa. Quantitative verification of the EC-EPS for the Northern Hemisphere has evolved from initial evaluations predominantly in terms of 500 hPa (Molteni et al. 1996) to more comprehensive validations of, for example, quantitative precipitation forecasts for the continental eastern United States (Mullen and Buizza 2001). A more extensive verification of the BM-EPS validating quantitative precipitation forecasts in the Australian region is presently being conducted (E. Ebert 2003, personal communication) to support and encourage wider operational utilization of the full range of the products available from the BM-EPS.

The paper is organized as follows. After a brief description of the two ensemble systems and of the accuracy measures used to assess the ensemble performance (section 2), deterministic and probabilistic EPS forecasts of 500-hPa geopotential height fields are compared (sections 3 to 8). Some conclusions are then finally drawn (section 9).

## 2. Description of the EC-EPS and BM-EPS

Ensemble prediction systems can differ not only in the way they simulate the effect of initial uncertainties (i.e., errors in the analysis) and the character of model prediction errors, but also through the size of the ensemble and ability to represent the probability distribution function of analysis and forecast states. As far as the simulation of initial uncertainties is concerned, both the EC-EPS and the BM-EPS have followed the same methodology and define the initial perturbations along the directions of the phase-space of the system characterized by the maximum linear growth of total perturbation energy during a finite time interval. These directions are called the singular vectors of the linear propagator of the model equations (Lorenz 1965; Farrell 1982, 1989; Buizza and Palmer 1995). More comprehensive generation of singular-vector-based perturbations, which are consistent with estimates of analysis error statistics, have been described by Ehrendorfer and Tribbia (1997), Palmer et al. (1998), Barkmeijer et al. (1999), and Hamill et al. (2003); from a practical point of view these are more complex and costly to calculate. Palmer et al. (1998) discuss in some detail the properties and appropriateness of the total energy metric for singular-vector calculations and suggest that of the simpler metrics (e.g., enstrophy, kinetic energy, or total energy), total energy is the most appropriate for the predictability problem. An initial evaluation by Barkmeijer et al. (1999) of the use of Hessian singular vectors based on static background error covariance, rather than total energy singular vectors, within the EC-EPS did not improve ensemble prediction performance; the use of a flow-dependent covariance in this context, as demonstrated in simulated assimilation studies within an ensemble Kalman filter assimilation system by Hamill et al. (2003) suggests a very interesting further development of these strategies. The use of singular vectors is different from the methodologies followed at NCEP and at CMC. At NCEP, the initial perturbations are generated using what are described as bred vectors; the perturbations are generated by comparing pairs of ongoing data assimilation systems started from randomly perturbed and unperturbed initial states (Toth and Kalnay 1993). At CMC, each perturbed initial condition is generated by running the CMC data assimilation with perturbed observations and with different model configurations (Houtekamer et al. 1996). Further and more recent discussion of the NCEP and CMC systems (and also the EC-EPS) are included in Buizza et al. (2003a). See Szunyogh et al. (1997), Palmer et al. (1998), and Errico and Langland (1999) for detailed discussions of the similarities and differences between singular vectors and bred vectors.

Differing strategies have been developed to simulate the influence of model errors. The EC-EPS simulates the effect of random model errors by introducing random perturbations within the model physical parameterization schemes (Buizza et al. 1999). In the approach implemented at CMC, each EPS ensemble member is integrated using a different combination of model parameterization schemes (Houtekamer et al. 1996). Within the BM-EPS there is no inclusion of such parameterizations of random model error.

### a. The EC-EPS

Since 21 October 2001, the EC-EPS has been operational with 1 unperturbed and 50 perturbed 10-day model integrations with a spectral triangular truncation T_{L}255 (where subscript L denotes use of linear grid resolution in physical space) and 40 vertical levels (L40). In physical space, the EC-EPS has a grid spacing at midlatitudes of approximately 80 km. See Simmons and Hollingsworth (2002) and references therein for a recent description of the main characteristics of the ECMWF model and physical parameterization schemes. Prior to March 2001, the operational EC-EPS was running with 1200 UTC as starting time; since then ECMWF has been running the EPS twice a day, at 0000 and 1200 UTC (the 0000 UTC EC-EPS used in this study was not part of the ECMWF operational suite at the time, but was part of an effort to develop an ensemble-based system for severe weather prediction). The results discussed in this paper refer only to the 0000 UTC runs of the EC-EPS, as this is the daily starting time used in the initial implementation of the BM-EPS.

*d*th-day EC-EPS initial perturbations are generated using the initial-time singular vectors growing between days

*d*and

*d*+ 2, and the final-time singular vectors growing between day

*d*− 2 and

*d*(Barkmeijer et al. 1999). In the current configuration, these two sets of singular vectors are linearly combined to define the

*j*th initial perturbations:

**f**

^{EC}

_{0}

_{L}255L40 interpolated version of the high-resolution T

_{L}511L60 ECMWF analysis) provided by the ECMWF four-dimensional variational data assimilation (4DVAR) scheme (Mahfouf and Rabier 2000). The scaling factors

*α*^{EC}

_{j,k}

*β*^{EC}

_{j,k}

Northern Hemisphere (NH) singular vectors are defined so that they maximize growth in the total energy norm north of 30°N, and

Southern Hemisphere (SH) singular vectors are defined so that they maximize growth in the total energy norm south of 30°S.

*t*for each perturbed member is defined by the numerical integration of the nonlinear perturbed model equations:

**A**

^{EC}(

*t*) and

**P**

^{EC}(

*t*) denote the time

*t*tendencies due to the nonparameterized and the parameterized physical processes, respectively. Random model errors due to the parameterized physical processes are simulated by randomly perturbing the tendency due to the parameterized physical processes by the factor

*r*

_{j}, as described in Buizza et al. (1999). Currently, the random numbers are defined so that the same random number is used in the vertical, the same random number is used inside a 10° box, and the random numbers are regenerated every 6 h. This setting introduces certain spatial and temporal coherent structures in the model-induced perturbations.

### b. The BM-EPS

The BM-EPS was run in an initial research trial from May 2000 until March 2001 with 1 unperturbed and 32 perturbed 10-day model integrations on a daily basis and with 0000 UTC as starting time. The system was configured with a spectral triangular truncation T79 and 19 vertical levels and with perturbations only applied in the Southern Hemisphere. At the beginning of April 2001 the BM-EPS trial was upgraded in spectral resolution to T_{L}119L19 and with the use of semi-Lagrangian time integration rather than the previously used Eulerian scheme. In physical space, this BM-EPS has a grid spacing at midlatitude of approximately 160 km. Since 5 July 2001 the BM-EPS has run in operational trial mode in the BoM NMOC; in December 2001 this operational trial was extended to include twice-daily running from 0000 and 1200 UTC with perturbations in both hemispheres. The results discussed in this paper refer to the 0000 UTC runs of the BM-EPS from 2 April to 31 August 2001.

The assimilation systems supporting the EC-EPS and BM-EPS are substantially different with respect to the analysis algorithms used and also have nontrivial differences in observational data usage. The ECMWF assimilation system uses a 4DVAR system (Mahfouf and Rabier 2000) and together with conventional data utilizes a rather complete range of remotely sensed data including, for example, much of the data from the then-available two National Oceanic and Atmospheric Administration (NOAA) Advanced Television Infrared Observation Satellite (TIROS) Operational Vertical Sounders (ATOVS) [both the Advanced Microwave Sounding Unit (AMSU-A) and the High Resolution Infrared Radiation Sounder (HIRS/3) radiances]; additionally, marine wind speed estimates from retrievals from two of the Special Sensor Microwave Imagers (SSM/I) onboard polar-orbiting satellites of the Defense Meteorological Satellite Program were in use by ECMWF at this time. The BoM assimilation system utilized three-dimensional multivariate statistical interpolation within its data assimilation system (Seaman et al. 1995); the analyses relevant to these studies used common conventional data but somewhat less remotely sensed data than those used by ECMWF; the BoM system utilized only ATOVS (AMSU-A) radiances within a one-dimensional variational data assimilation (1DVAR) retrieval scheme (Harris and Kelly 2001) from *NOAA-15* until mid-June 2001, and from then onward additionally used *NOAA-16* AMSU-A radiances. The BoM assimilation did not access any of the SSM/I data. The more sophisticated assimilation system and the more comprehensive data utilization of the ECMWF system is considered to be providing more accurate initial conditions than are available from the BoM system.

Apart from using substantially different data assimilation and model prediction implementations, the notable differences between the EC-EPS and the BM-EPS are as summarized in the following discussion.

The *d*th-day BM-EPS initial perturbations are based only on the initial-time singular vectors growing between days *d* and *d* + 2; final time evolved singular vectors growing between day *d* − 2 and *d* are not used in the current BM-EPS approach. The initial unperturbed analysis is provided by the BMRC multivariate statistical interpolation analysis scheme (Seaman et al. 1995); this BM-EPS control analysis has a resolution of T_{L}119L19 and is interpolated from the operational BoM T_{L}239L29 Global Assimilation and Prediction System (GASP) assimilation analyses.

The BM-EPS singular vectors are computed at resolution T42L19, also with only simplified physics, namely linearized surface stress, in the tangent forward and adjoint model versions. The BM-EPS singular vector decomposition utilizes the Arnoldi Package (ARPACK) software (Lehoucq et al. 1998); this software uses a variant of the Lanczos eigenvector decomposition algorithm, known as the Implicitly Restarted Lanczos Method. The major differences between the BM-EPS and the EC-EPS are summarized in Table 1. The formulation of the BM-EPS prediction model has been described in Bourke et al. (1995); the present model implementation differs primarily with respect to its use of semi-Lagrangian time stepping and the mass-flux convective parameterization (Tiedtke 1993).

*d*th-day BM-EPS initial perturbations are defined as

**f**

^{BM}

_{0}

_{L}119L19 interpolated version of the high-resolution operational T

_{L}239L29 BoM analysis). Here the individual perturbations are obtained as linear combinations of the component singular vectors through an orthogonal phase-space rotation followed by an amplitude scaling factor. The rotation, following the strategy of the EC-EPS and as described in Molteni et al. (1996), is designed to maximize geographical spread of each of the ensemble perturbations; the amplitude scaling is defined as a constant factor chosen such that the BM-EPS spread is similar to the error of the BM-EPS control forecast at

*t*= 2 days. With the total energy normalization of the singular vectors set at unity, the scaling factor was set in the present studies at 0.5.

*t*for each perturbed member is defined by the numerical integration of the nonlinear perturbed model equations:

**A**

^{BM}(

*t*) and

**P**

^{BM}(

*t*) denotes the time

*t*tendencies due to the nonparameterized and the parameterized physical processes, respectively.

### c. Definitions: Ensemble mean and ensemble spread

^{XX}denotes either BM or EC, and

*N*

_{XX}is the total number of ensemble members.

### d. Definitions: Accuracy measures

The accuracy of deterministic forecasts such as the control, individually perturbed, or the ensemble mean forecasts is assessed by computing the average root-mean-square error or the average anomaly correlation between the forecast and the verifying analysis inside a specified region. The relationship between the ensemble spread around the control forecast and the skill of the control forecast is investigated, as is the spread around the ensemble mean and the skill of the ensemble mean forecast.

The accuracy of probabilistic forecasts of the occurrence or nonoccurrence of the event, namely 500-hPa geopotential height anomaly larger/smaller than defined thresholds, is assessed using the Brier score and the Brier skill score (Brier 1950), and the area under the relative operating characteristic (ROC) area Mason 1982; Stanski et al. 1989; Wilks 1995). The Brier score measures the mean-square error of probability forecasts for a binary event; that is, it measures the squared difference between a forecast probability of an event *p* and its occurrence *o,* expressed as 0 or 1 depending on whether the event has occurred or not; the Brier skill score compares the Brier score of one ensemble forecast with the Brier score of a probability forecast given by a climatological distribution. A perfect forecast has a Brier skill score of 1, and a forecast less skillful than climatology has a negative Brier skill score. The ROC area is a measure of the capacity of an ensemble system to discriminate between the occurrence and nonoccurrence of an event using hit and false alarm rates. A perfect score would have an ROC area of 1, while a forecast that cannot discriminate between the occurrence and nonoccurrence of an event would have a score of 0.5. The Brier score and skill score and the ROC area are measures of the quality of probabilistic forecasts of single categorical events.

Rank histogram statistics are also analyzed, and the percentage of cases when the analysis falls outside the range of values spanned by the ensemble members is computed (see Talagrand et al. 1998 and Buizza et al. 1998).

This set of accuracy measures is being used to provide the initial comparison of the accuracy and skill of the two ensemble systems and to help to understand the reasons for any detected difference. Results are presented for two regions: AU (latitude from 10° to 45°S, longitude from 100° to 160°E) and SH (latitude from 20° to 60°S). The verification domain of 20° to 60°S, for the Southern Hemisphere annulus, has been chosen to reduce the impact on scores of differences in assimilation and prediction for the Antarctic continental area; satellite radiance data are typically more difficult to use satisfactorily in the assimilation systems over the ice-covered ocean and over high cold terrain. Additionally, the prediction of the radiative balance of the cold surface temperatures and the typical low-level temperature inversion over the high continent in wintertime can be very sensitive to specific details of model parameterizations. EC-EPS forecasts have been verified against ECMWF operational analyses as represented by the T_{L}255L40 interpolated version of the high-resolution T_{L}511L60 ECMWF analyses, that is, equivalent to the control prediction initial condition resolution; the BM-EPS forecasts have been verified against the BoM operational T_{L}239L29 analyses. Both sets of verifying analyses and model predictions of 500-hPa geopotential height used in this study have been specified on a regular latitude–longitude grid with 2.5° spacing.

## 3. EC-EPS and BM-EPS initial perturbations

The three fastest growing initial-time singular vectors, represented by the streamfunction shown for the BM-EPS [at sigma level (*p*/*p**) = 0.5] and EC-EPS [at sigma level (*p*/*p**) ≅ 0.5], are shown in Figs. 1a–c and in Figs. 1e–g, respectively, for the initial date/time of 0000 UTC 15 July 2001; here, although the respective system singular vectors have differing normalizations, the contour count is set to be equivalent. These first three leading singular vectors of each system are seen to have similar patterns, with variance maxima similarly located. It is not expected that the individual singular vectors from each system will have an exact one-to-one correspondence in structure or growth rates. A detailed intercomparison of the equivalence of the independently generated singular vectors has been previously made (A. Doerenbecher 1998, personal communication); a close correspondence in terms of growth rates and structures was found between the separate EC-EPS and BM-EPS singular vectors, when the vertical-level resolution and location (particularly of the lowest model levels) were made close to equivalent in the two systems.

The relationship of these initial-time singular vectors to the overall baroclinic instability of the flow is indicated by an evaluation of the Eady index, which is an estimate of the growth rate of the most unstable Eady mode (see Hoskins and Valdes 1990), given by *σ* ≡ 0.31 (*f*/*N*) (∂*u*/∂*z*), where the static stability, *N,* and the wind shear, ∂*u*/∂*z,*have been estimated from the 300- and 850-hPa potential temperature and wind field data. The Eady index is shown for the BM-EPS (Fig. 1d) and for the EC-EPS initial conditions (Fig. 1h). The geographical location of the first three leading singular vectors is seen to coincide quite well with the high baroclinicity (Eady index values) in the southern Indian and Atlantic Oceans for both sets of singular vectors, although the correspondence is limited and does not extend to the high baroclinicity seen in both Eady plots over southern South America and the high latitudes of the southeast Pacific (see below for further comment).

Figure 2 shows (a)–(d) four BM-EPS and (e)–(h) four EC-EPS initial perturbations of 500-hPa geopotential height at 0000 UTC 15 July 2001; Figs. 2a–h have all been plotted with the same contour interval of 3 m. The comparison of the two sets of figures indicates that the BM-EPS initial perturbations are smaller in both amplitude and scale, with EC-EPS initial perturbations showing local maxima approximately a factor 3 larger (about 50 m compared to about 17 m) than in the BM-EPS case. The difference in local amplitude is due to three main factors: (i) BM-EPS uses only initial singular vectors while EC-EPS uses both initial and evolved singular vectors; (ii) BM-EPS and EC-EPS initial perturbations are scaled differently, with EC-EPS using an analysis error estimate to set the perturbation amplitudes while BM-EPS uses a fixed factor as indicated in section 2b; and (iii) BM-EPS initial perturbations are defined using 16 singular vectors while the EC-EPS initial perturbations are defined using 25 singular vectors. The BM-EPS initial perturbations have smaller horizontal scales because they are generated using only initial-time singular vectors, which are known to have a much smaller scale than final-time singular vectors (Buizza and Palmer 1995). It is useful to note that the perturbations generated from linear combination of the 16 initial-time BM-EPS singular vectors, namely, perturbations 1, 3, 5, and 7, and shown in Figs. 2a to 2d, do show structure in the high latitudes of the South American region approximately matching areas of instability suggested by the Eady index of Fig. 1d, which was not seen in the first three leading singular vectors of Figs. 1a–c; (perturbations 2, 4, 6, and 8 match 1, 3, 5, and 7 pairwise but are of opposite sign).

## 4. BM-EPS and EC-EPS ensemble spread

Figure 3 shows the 152-case probability density function (pdf) of ensemble spread around the control forecast for the BM-EPS (solid) and the EC-EPS (dash) for the Southern Hemisphere (bottom), for the Australian region (top), and at forecast day 3 (left), day 5 (middle), and day 7 (right). Here each system's spread is calculated in terms of root-mean-square distance relative to its own control over SH and AU; the percentage of cases with ensemble spread inside a specific interval, with intervals defined as multiples of 7 m, are represented. The most notable difference between the two systems is the enhanced spread of the EC-EPS, particularly at day 7; at earlier times, and particularly for the AU, the spread of the BM-EPS is slightly skewed to lower spread. The results show that the divergence between the members of each ensemble grows at different rates. Figure 4 shows the time evolution of the 152-case ensemble spread for SH and AU (for each system relative to its own control), that is, the time evolution of the magnitude of the mean spread of the pdf of the ensembles. The differing amplitude of the initial perturbations is evident in Fig. 4 at time *t* = 0; the BM-EPS growth rate during the first 2–3 days is greater than that of the EC-EPS. This is to be expected given that the BM-EPS perturbations are defined in terms of initial singular vectors, whereas the EC-EPS perturbations are comprised equally of fast growing initial and more slowly growing evolved singular vectors. The BM-EPS average spread, which is initially about a factor of 3 smaller than the EC-EPS average spread for both SH and AU, is comparable to EC-EPS spread from around day 1.5 to day 4. Subsequently, the EC-EPS spread grows at a faster rate than the BM-EPS spread for both regions.

Apart from the fact that the initial perturbations are different, other possible explanations for the different growth rates are that the EC-EPS ensemble members are run at higher resolution, and that the resulting spread benefits from the fact that stochastic model perturbations act as a further source of divergence. Buizza et al. (1999) indicated that the impact of stochastic model perturbations on the ensemble spread is rather small, with stochastic physics increasing the ensemble spread by about 3% to 5%. By contrast, Buizza et al. (1998) documented an ensemble spread increase when the EC-EPS changed resolution from T63 to T_{L}159 and from 19 to 31 vertical levels, especially in the 5- to 10-day forecast range. These previous results would suggest that the different growth in ensemble spread in the 5- to 10-day range could be due to the different resolution in the two systems. Another small difference in the present study affecting EC-EPS spread in the Southern Hemisphere is the additional Northern Hemisphere perturbations of the EC-EPS; in the medium range and beyond there is interhemispheric propagation of perturbations.

The faster growth rate of the BM-EPS for the SH relative to AU in the first few days is consistent with the geographical distribution of the initial singular vectors, which predominantly have variance maxima at latitudes poleward of Australia.

Similar conclusions could have been drawn by considering anomaly correlation instead of root-mean-square error as a measure of spread (not shown), or by considering the ensemble spread around the ensemble mean (i.e., by considering the ensemble standard deviation; not shown).

## 5. BM-EPS and EC-EPS skill

Figure 5 shows the 152-case probability density function of root-mean-square error of the BM-EPS (solid) and the EC-EPS (dash), for the Australian region (top) and for the Southern Hemisphere (bottom), at forecast day 3 (left), day 5 (middle), and day 7 (right). Here each system's root-mean-square error is calculated in terms of root-mean-square distance relative to its own analysis over SH and AU; the percentages of cases with ensemble root-mean-square error inside specific intervals, with intervals defined in terms of multiples of 7 m, are represented. Results show that for all forecast times the EC-EPS distribution curve is to the left of the BM-EPS curve, indicating the error of the EC-EPS forecasts is smaller, although this differentiation is less marked for AU.

The time evolution of the 152-case root-mean-square error of the control (Fig. 6a) and of the root-mean-square error of the ensemble mean (Fig. 6b) of each system is shown for all forecast times for both AU (top) and SH (bottom), illustrating further the differences seen in Fig. 5. The overall superior skill of the EC-EPS, by a margin of approximately 1 day, is evident for both the control and ensemble mean forecasts. The reduction in 10-day error provided by the ensemble mean is seen in Figs. 6a and 6b to be of the order of 25%.

The time evolution of the mean forecast skill of the ensemble (Fig. 6c) is shown for all forecast times for both AU (top) and SH (bottom) for each system; here the root-mean-square error of each individual forecast has been averaged over all ensemble members and all 152 cases. The mean errors of the two ensembles (Fig. 6c) are in closer agreement than the other measures shown in Figs. 6a and 6b, particularly for AU. This shows that the probability distribution of errors can be more similar than is suggested by the single measures of control or ensemble mean error.

Figure 6d shows, for the AU and SH regions, the percentage of cases from each ensemble where the root-mean-square prediction error is less than the root-mean-square error of their respective control predictions; it is seen that for both regions the BM-EPS has a higher percentage of members that perform better than the control. Paradoxically, it is also seen in Fig. 6d that at time zero there are a number of members in the BM-EPS with lower error than the control; at time zero these errors are of the order of a few meters and very small relative to the magnitude of prediction error at 24 h and beyond. They are attributable to the use of a postprocessed T_{L}239L29 operational analyses to verify the BM-EPS initial conditions represented at T_{L}119L19. For the verification of the EC-EPS fields we have used the ECMWF control at *t* = 0 as the verifying analyses, rather than the directly postprocessed high-resolution T_{L}511L60 ECMWF analyses.

The implication of Fig. 6d may be that within the BM-EPS, where the underlying control performance is clearly less satisfactory than that of the EC-EPS, there is clearly scope to enhance prediction performance, and that a relatively greater percentage of perturbations do indeed do this. A further comment on the results seen in Fig. 6d is that the BM-EPS's simpler perturbation strategy may be contributing to these results, although the scope of the present studies does not permit any further clarification of this aspect.

Figure 7 shows the time series of the root-mean-square error of 5-day forecasts for the SH, of the control and ensemble mean forecasts and the best and worst ensemble members of the BM-EPS (top) and EC-EPS (bottom); these time series are of the 5-day running mean scores. Additionally the time means of the 152-case root-mean-square of the errors shown in Fig. 7 are presented in Table 2 together with the standard deviations (in brackets). The results again reflect the fact that the EC-EPS forecasts are in general characterized by smaller error; there is a larger absolute difference between the BM-EPS best member and the BM-EPS control than seen between the corresponding EC-EPS best member and its control, although the reduction in absolute error for the best member is approximately 10% in both the EC-EPS and BM-EPS. The equivalent time series for AU are shown in Fig. 8; the 152-case time means of these quantities shown in Fig. 8 are also presented in Table 2. Here it is seen for both BM-EPS and EC-EPS that the gain of 33% in absolute performance, in the 152-case mean, of the best ensemble member relative to the control for AU, is markedly better than seen for SH. It is also notable that there is more variability in performance in the BM-EPS in comparison to the EC-EPS in both the SH and AU regions, as evident in Figs. 7 and 8 and as quantified in the standard deviations presented in Table 2. The EC-EPS is providing a more consistent level of performance, which is interpreted as being primarily associated with the EC-EPS having a more consistent quality in the accuracy of its initial conditions.

A possible explanation of the larger gap in absolute terms between the best member and the control forecast in the BM-EPS compared with the EC-EPS for the SH is the following: There is some evidence that one reason for the poorer skill of the BM-EPS control forecast compared to the EC-EPS control forecast is the fact that the BoM analysis is less accurate; a number of unpublished experiments involving the transplant of ECMWF initial conditions into the BoM prediction system, at T_{L}239L29 resolution, have yielded significant positive impact on prediction performance in the Southern Hemisphere. Assuming then that the BoM analysis is less accurate, it appears plausible for the potential error reduction induced by the initial perturbations in the BM-EPS to be larger.

It is interesting to compare the difference of the magnitude of the ensemble mean forecast error and the control forecast error. Figure 9 shows for the two systems and for each forecast day the average over all cases of the difference between the root-mean-square error of the ensemble mean and the root-mean-square error of the control for SH (bottom) and for AU (top). The ensemble mean forecast is a smoother field than the control forecast, and it could be expected to have a smaller root-mean-square error than the control forecast. The difference between the error of the ensemble mean and the error of the control forecast is a measure of the effectiveness of the filtering induced by the ensemble averaging in defining a single deterministic forecast (namely the ensemble mean) with an error smaller than the control forecast. The error reduction of the ensemble mean is seen to begin earlier and to be greater for the SH domain than for the AU domain. For the SH, the two curves are very similar, while for AU slightly larger negative values are obtained for the BM-EPS. This indicates that the effect of the filtering of small unpredictable scales by taking the mean of the ensemble is positive for both systems over both regions, with the beneficial effect over the AU domain being slightly larger for the BM-EPS.

## 6. Spread–skill relationships

In earlier studies of a perfect ensemble (Buizza 1997), the link between ensemble spread and control forecast skill was examined with respect to the following relationships:

ensemble spread distribution is similar to control error distribution;

smaller ensemble spread corresponds to better control prediction skill; and

the verifying analysis is within the range spanned by the ensemble forecasts.

Figure 10 shows, for both AU (top) and SH (bottom), the difference between the ensemble spread of 500-hPa geopotential height prediction around the control forecast and the control error for the two systems. These results indicate that for both the SH and AU regions the BM-EPS spread is smaller than the average error of the control forecast, while the EC-EPS spread is larger than the control error up to forecast day 5 in both regions and then becomes slightly smaller than the control error. (The negative deficit for BM-EPS at time zero is spurious and arises because the verifying analyses are the operational T_{L}239L29 analyses postprocessed to the verifying grid, whereas the control model fields are postprocessed from the T_{L}119L29 resolution. For the verification of the EC-EPS fields we have used the ECMWF control at *t* = 0 as the verifying analyses.)

Table 3 summarizes the percentage of cases with ensemble spread smaller than control forecast error. An ensemble system with spread comparable to control error on average would have 50% cases with spread lower than the control error. Thus, the ensemble system with the percentage of cases closer to 50% would perform better according to this measure. Table 3 indicates that with respect to the equivalence of spread and control error, the EC-EPS is superior, with the BM-EPS displaying ensemble spread significantly less than the control error.

Table 4 summarizes the correlation between the root-mean-square ensemble spread relative to the control and the control forecast skill. A higher correlation value represents a stronger spread–skill relationship, which is interpreted as better according to this measure. In Table 4 the EC-EPS is superior at later time ranges for SH, whereas the BM-EPS has better performance on this measure for AU.

## 7. Skill of ensemble probabilistic forecasts of geopotential height anomalies

Figures 11a and 11b, respectively, show the area under the ROC-area curve and the Brier skill score (BSS) for both the BM-EPS (solid) and EC-EPS (dash) probability forecasts of two events over the SH, namely, positive 500 hPa geopotential height anomaly with magnitude larger than 50 m and negative 500 hPa geopotential height anomaly with magnitude larger than 50;thm. Details of the evaluation of these scores are given in Buizza and Palmer (1998). Results indicate that both ensemble systems are skillful for the whole 10-day forecast range (positive BSS and ROC area larger than 0.5). Comparing one to the other, the EC-EPS shows higher skill scores. Generally speaking, the EC-EPS forecasts valid for day *d* have skill scores similar to the BM-EPS forecasts valid for day (*d* − *x*), with *x* ranging between 1.5 and 2.0 days.

Figures 12a and 12b are analogous to Figs. 11a and 11b, but for AU. Results indicate again that both ensemble systems are skillful for the whole 10-day forecast range, and that the EC-EPS is more skillful than the BM-EPS with *x* (as defined above) also ranging between 1.5 and 2.0 days. Similar conclusions can be drawn using either the ROC-area curve or the BSS; both are used here since they measure different ensemble characteristics (e.g., when BSS < 0 it is still possible for the ROC area to be greater than >0.5, indicating that the system can still discriminate between the occurrence and nonoccurrence of an event.)

As seen in the control and ensemble root-mean-square error of Figs. 6a and 6b, the deterministic prediction skill gain of the EC-EPS over the BM-EPS is of the order of 1 day; however, as seen in Figs. 11 and 12 in terms of probabilistic verification, the gain of the EC-EPS is seen to be from 1.5 to 2.0 days. This outcome is consistent with results reported by Buizza et al. (2003b), who indicate that when the EC-EPS resolution was increased from T_{L}159 to T_{L}255 (with 50 members in each ensemble) a larger positive impact was detected on the probabilistic products than on the skill of the EPS control (cf. resolutions in the present studies of T_{L}119 and T_{L}255 for BM-EPS and EC-EPS, respectively); these authors state that one of the most striking results from their assessment of increased resolution is that the accuracy of the probabilistic forecasts has been improved more than the accuracy of the deterministic forecasts. Buizza et al. (1998) also found that increased ensemble size (at resolutions of T106) contributes to increased performance of probabilistic scores.

## 8. Rank histogram statistics

Table 5 shows the percentage of times, over and above the expected value, that the verifying analysis value lies outside the forecast range spanned by the ensemble forecasts, for the BM-EPS and EC-EPS. In comparing the probability of the verifying analysis lying outside the forecast range, if the analysis is considered as a random member chosen from the same distribution, then the expected percentage of cases where the analysis lies outside the ensemble would be 100*[2/(*N*_{ens} + 1)], that is, 5.9% and 3.8%, respectively, for the BM-EPS and EC-EPS. Except at day 3 for SH, the BM-EPS is displaying a larger “excess-over-expected” percentage of outliers for both SH and AU, with the difference becoming larger as the forecast duration increases.

For an earlier version of the EC-EPS (T106) and for the Northern Hemisphere, Buizza et al. (1998) showed similar excess outlier percentages to Table 5. They noted that this is not an unexpected outcome given both the effects of model error and the limitations of the strategies for generating initial perturbations. Furthermore, they showed that increased ensemble size contributes to improved performance with respect to this rank histogram diagnostic; for example, increasing the EC-EPS from 32 to 50 members was found to reduce the outliers by 4% at day 5 in the Northern Hemisphere. In the present intercomparison the number of outliers in the BM-EPS 32-member ensemble is 6% larger than that of the 50-member EC-EPS, which is interpreted in part as a further indication of the reduced model error evident in the EC-EPS.

An alternative way of looking at the excess outlier percentage numbers is to scale them by the expected outlier percentages; so, for example, at day 5 for BM-EPS the excess outliers are 3.4 times as many as the expected number, while for EC-EPS they are 3.6 times. Scaling in this way brings the 32- and 50-member numbers into closer alignment, both for these BM-EPS and EC-EPS Southern Hemisphere results and the EC-EPS Northern Hemisphere results of Buizza et al. (1998).

## 9. Conclusions

This work has documented the performance in the Southern Hemisphere of the European Centre for Medium-Range Weather Forecasts (ECMWF) and the Australian Bureau of Meteorology (BoM) Ensemble Prediction Systems. The EC-EPS has been providing daily operational probabilistic forecasts since 1992, while the BM-EPS has been run in real time in BoM schedules since July 2001. Table 1 lists the main differences between the two ensemble systems; notable differences are that the 50-member EC-EPS utilizes approximately double the resolution in both the horizontal and vertical in comparison to the 32-member BM-EPS.

Attention has been focused on 500-hPa geopotential height forecasts. The accuracy of deterministic and probabilistic forecasts have been assessed using different accuracy and skill measures over the SH annulus between 20° and 60°S and over Australia (latitude from 10° to 45°S, longitude from 100° to 160°E). Forecasts for 152 cases, from 2 April to 31 August 2001, have been examined.

Results have indicated that the two systems differ both in ensemble spread characteristics and in ensemble skill. The probability distributions of spread, as seen in Fig. 3, show that the most notable difference between the two systems is the enhanced spread of the EC-EPS, particularly at day 7; at earlier times, and particularly for the Australian region (AU), the spread of the BM-EPS is slightly skewed to lower spread. Considering the growth rate of ensemble spread as seen in Fig. 4, the BM-EPS has initially a smaller spread than the EC-EPS and a subsequent faster growth of the divergence among the different ensemble members up to forecast day 2– 3. This initial difference in the ensemble spread is attributed to the fact that the BM-EPS initial perturbations are generated using only initial-time singular vectors while the EC-EPS uses a mixture of initial-time and final-time singular vectors. The initial-time singular vectors have a substantially larger growth rate than do the final-time evolved singular vectors (Barkmeijer et al. 1999) and thus a more rapid growth of ensemble spread occurs during the first 2–3 days within the BM-EPS, where all the perturbation energy is in these faster growing modes. Both ensemble systems reach a similar value of ensemble spread at around forecast days 2–3, although the BM-EPS initial perturbations are scaled to a 3-times-smaller initial amplitude than the EC-EPS initial perturbations. After the first 3 days, there is a time period during which the two ensemble systems have a similar level of spread, to about forecast day 5. After this period, the ensemble spread of the EC-EPS grows at a faster rate. There are several reasons for this differing longer-term behavior; these include the impact of the higher resolution and larger ensemble size utilized in the EC-EPS, the impact of the stochastic physics, and the impact of the slower growing final-time singular vectors used in the EC-EPS initial perturbations. Given the differing strategies of generating perturbations and the differing number and the vertical resolution of the singular vectors, it is perhaps surprising that the spread probability distributions are as similar as has been found in the present study.

Considering the skill of the two ensemble systems, almost all accuracy measures indicate that the EC-EPS performance is better. This skill difference is shown both by deterministic-type products (e.g., forecasts given by the control and by the ensemble mean forecasts) and by probabilistic-type forecasts (probability of geopotential height anomalies) over both the SH and AU.

Overall, EC-EPS deterministic products are 1 day more skillful than the corresponding BM-EPS products (see Figs. 6a and 6b). This difference is attributed primarily to the superior quality of the ECMWF analyses rather than model differences or model resolution. Analysis transplant experiments using ECMWF analyses in the BMRC forecast system (unpublished) have shown improved performance; such results are consistent with those of Lalaurette et al. (2003, viz., their Fig. 18), which show that the magnitude of 500-hPa deterministic forecast errors in the Southern Hemisphere can be affected much more by the starting analyses than the model resolution. Within the period of this study, the operational BoM (T239L29) forecast verifications compared with BM-EPS control (T119/L19), in 500-hPa predictions for the SH region for August 2001 (31 cases), is only minimally improved by the increase in resolution; that is, increase in resolution from that of the BM-EPS to something similar to the EC-EPS does not resolve the poorer performance of the BM-EPS control relative to the EC-EPS control as seen in Fig. 6a. Likewise, the same August 2001 verification for the ECMWF (T511/L60) deterministic predictions compared with EC-EPS control (T255/ L40) shows only minimal improvement from the increase in resolution. Model parameterization differences between the two systems have not been evaluated but are thought to be less important than the impact of differing initial conditions in these present studies.

The EC-EPS probabilistic products are between 1.5 and 2.0 days more skillful than the corresponding BM-EPS products (see Figs. 11 and 12). Both larger ensemble size and ensemble resolution of the EC-EPS are known in other studies of the EC-EPS to enhance probabilistic skill in the Northern Hemisphere (Buizza et al. 1998, 2003b). The more recent of these two studies, in particular, shows as a function of increased system resolution a greater relative improvement in probability scores rather than in deterministic scores. It is to be expected that some of the improvements in probabilistic skill identified in Buizza et al. (2003b) would be applicable to the BM-EPS in the Southern Hemisphere with commensurate ensemble size and resolution.

The 33% skill gain, on average, of the best ensemble member relative to the control forecast for the AU region for 5-day prediction for both ensemble prediction systems (see Table 2; Fig. 8) is markedly larger than seen for SH; this emphasizes that the present perturbation strategies do indeed yield ensemble members with marked increase in skill over the control and especially so in the Australian region. It is clearly more unlikely that perturbations over the whole hemispheric domain can realize improved prediction simultaneously over all regions within the hemisphere. The occurrence of a greater percentage of members of the ensemble better than the control, as seen in the BM-EPS (see Fig. 6d), may be due to the poorer control performance, although it may also be a function of the differing perturbation strategies.

In assessing the existence of a correlation between spread and skill (see Table 4), it has been found that the BM-EPS is showing the strongest correlations for the AU region, particularly at day 5; the EC-EPS is seen to be superior in this regard at the later time periods for the SH region.

A difference between the two ensemble prediction systems that may be impacting positively over the Australian region in the BM-EPS is the use of singular vectors for the latitude range poleward of 20°, rather than poleward of 30° as used in the EC-EPS.

The present analysis of the BM-EPS represents the first major diagnosis of its performance since its development and implementation; this comparison with the very well established EC-EPS is very informative and points to a number of issues within the BM-EPS that warrant attention. Clearly, enhancing the level of performance of the base control forecast will provide a better framework for the ensemble system. Resolving the more marked shortfall in probabilistic skill, relative to deterministic skill, of the BM-EPS compared with the EC-EPS, and reducing the percentage of outliers are further immediate issues that need to be considered. The implications from EC-EPS studies are that enhanced resolution and ensemble size will alleviate some of these deficiencies.

The results discussed in this paper consider only the respective performance levels of the two ensemble prediction systems in terms of 500-hPa geopotential height ensemble prediction; the issues of, for example, the magnitude of spread from the control, and the spread– skill relationships may have different characteristics for other predicted fields.

## Acknowledgments

Roberto Buizza would like to thank the Bureau of Meteorology Research Centre for arranging and funding the visit to Melbourne in June 2001. This work is part of the very successful and ongoing collaboration between ECMWF and BMRC. The authors acknowledge the helpful suggestions and clarifications by the two reviewers.

## REFERENCES

Barkmeijer, J., R. Buizza, and T. N. Palmer, 1999: 3D-Var Hessian singular vectors and their potential use in the ECMWF Ensemble Prediction System.

,*Quart. J. Roy. Meteor. Soc***125****,**2333–2351.Bourke, W., T. Hart, P. Steinle, R. Seaman, G. Embery, M. Naughton, and L. Rikus, 1995: Evolution of the Bureau of Meteorology's Global Assimilation and Prediction System. Part 2: Resolution enhancements and case studies.

,*Aust. Meteor. Mag***44****,**19–40.Brier, G. W., 1950: Verification of forecasts expressed in terms of probability.

,*Mon. Wea. Rev***78****,**1–3.Buizza, R., 1994: Sensitivity of optimal unstable structures.

,*Quart. J. Roy. Meteor. Soc***120****,**429–451.Buizza, R., 1997: Potential forecast skill of ensemble prediction and spread and skill distributions of the ECMWF Ensemble Prediction System.

,*Mon. Wea. Rev***125****,**99–119.Buizza, R., and T. N. Palmer, 1995: The singular-vector structure of the atmospheric global circulation.

,*J. Atmos. Sci***52****,**1434–1456.Buizza, R., and T. N. Palmer, 1998: Impact of ensemble size on ensemble prediction.

,*Mon. Wea. Rev***126****,**2503–2518.Buizza, R., T. Petroliagis, T. N. Palmer, J. Barkmeijer, M. Hamrud, A. Hollingsworth, A. Simmons, and N. Wedi, 1998: Impact of model resolution and ensemble size on the performance of an Ensemble Prediction System.

,*Quart. J. Roy. Meteor. Soc***124****,**1935–1960.Buizza, R., M. Miller, and T. N. Palmer, 1999: Stochastic representation of model uncertainties in the ECMWF Ensemble Prediction System.

,*Quart. J. Roy. Meteor. Soc***125****,**2887–2908.Buizza, R., P. L. Houtekamer, Z. Toth, G. Pellerin, M. Wei, and Y. Zhu, 2003a: Assessment of the status of global ensemble prediction.

*Proc. ECMWF Seminar on Predictability of Weather and Climate,*Reading, United Kingdom, ECMWF, 127–154.Buizza, R., D. S. Richardson, and T. N. Palmer, 2003b: Benefits of increased resolution in the ECMWF ensemble system and comparison with poor-man's ensembles.

,*Quart. J. Roy. Meteor. Soc***129****,**1269–1288.Ehrendorfer, M., and J. J. Tribbia, 1997: Optimal prediction of forecast error covariances through singular vectors.

,*J. Atmos. Sci***54****,**286–313.Errico, R. M., and R. Langland, 1999: Notes on the appropriateness of “bred modes” for generating initial perturbations used in ensemble prediction.

,*Tellus***51A****,**431–441.Farrell, B. F., 1982: The initial growth of disturbances in a baroclinic flow.

,*J. Atmos. Sci***39****,**1663–1686.Farrell, B. F., 1989: Optimal excitation of baroclinic waves.

,*J. Atmos. Sci***46****,**1193–1206.Hamill, T. M., C. Synder, and J. S. Whitaker, 2003: Ensemble forecasts and the properties of flow-dependent analysis-error covariance singular vectors.

,*Mon. Wea. Rev***131****,**1741–1758.Harris, B. A., and G. Kelly, 2001: A satellite radiance-bias correction scheme for data assimilation.

,*Quart. J. Roy. Meteor. Soc***127****,**1453–1468.Harrison, M. S. J., T. N. Palmer, D. S. Richardson, and R. Buizza, 1999: Analysis and model dependencies in medium-range ensembles: Two transplant case-studies.

,*Quart. J. Roy. Meteor. Soc***125****,**2487–2515.Hersbach, H., R. Mureau, J. D. Opsteegh, and J. Barkmeijer, 2000: A short-range to early-medium-range ensemble prediction system for the European area.

,*Mon. Wea. Rev***128****,**3501–3519.Hoskins, B. J., and P. J. Valdes, 1990: On the existence of storm tracks.

,*J. Atmos. Sci***47****,**1854–1864.Houtekamer, P. L., L. LeFaivre, J. Derome, H. Ritchie, and H. L. Mitchell, 1996: A system simulation approach to ensemble prediction.

,*Mon. Wea. Rev***124****,**1225–1242.Lalaurette, F., L. Ferranti, A. Ghelli, O. Saetra, and H. Bottger, 2003: Verification statistics and evaluation of ECMWF forecasts in 2001–2002. ECMWF Tech. Memo. 414, Reading, United Kingdom, 49 pp.

Lehoucq, R. B., D. C. Sorenson, and C. Yang, 1998:

*ARPACK Users Guide: Solution of Large-Scale Eigenvalue Problems with Implicitly Restarted Arnoldi Methods*. Society for Industrial and Applied Mathematics, 160 pp.Lorenz, E. N., 1965: A study of the predictability of a 28-variable atmospheric model.

,*Tellus***17****,**321–333.Mahfouf, J. F., and F. Rabier, 2000: The ECMWF operational implementation of four-dimensional variational assimilation. II: Experimental results with improved physics.

,*Quart. J. Roy. Meteor. Soc***126****,**1171–1190.Marsigli, C., A. Montani, F. Nerozzi, T. Paccagnella, S. Tibaldi, F. Molteni, and R. Buizza, 2001: A strategy for high-resolution ensemble prediction. II: Limited-area experiments in four Alpine flood events.

,*Quart. J. Roy. Meteor. Soc***127****,**2095–2116.Mason, L., 1982: A model for assessment of weather forecasts.

,*Aust. Meteor. Mag***30****,**291–303.Molteni, F., R. Buizza, T. N. Palmer, and T. Petroliagis, 1996: The ECMWF Ensemble Prediction System: Methodology and validation.

,*Quart. J. Roy. Meteor. Soc***122****,**73–119.Molteni, F., R. Buizza, C. Marsigli, A. Montani, F. Nerozzi, and T. Paccagnella, 2001: A strategy for high-resolution ensemble prediction. I: Definition of representative members and global-model experiments.

,*Quart. J. Roy. Meteor. Soc***127****,**2069–2094.Mullen, S. L., and R. Buizza, 2001: Quantitative precipitation forecasts over the United States by the ECMWF Ensemble Prediction System.

,*Mon. Wea. Rev***129****,**638–663.Palmer, T. N., R. Gelaro, J. Barkmeijer, and R. Buizza, 1998: Singular vectors, metrics, and adaptive observations.

,*J. Atmos. Sci***55****,**633–653.Seaman, R., W. Bourke, P. Steinle, T. Hart, G. Embery, M. Naughton, and L. Rikus, 1995: Evolution of the Bureau of Meteorology's Global Assimilation and Prediction System. Part 1: Analysis and initialisation.

,*Aust. Meteor. Mag***44****,**1–18.Simmons, A. J., and A. Hollingsworth, 2002: Some aspects of the improvement in skill of numerical weather prediction.

,*Quart. J. Roy. Meteor. Soc***128****,**647–677.Stanski, H. R., L. J. Wilson, and W. R. Burrows, 1989: Survey of common verification measures in meteorology. World Weather Watch Tech. Rep. 8, WMO, 114 pp.

Szunyogh, I., E. Kalnay, and Z. Toth, 1997: A comparison of Lyapunov and optimal vectors in a low-resolution GCM.

,*Tellus***49A****,**200–227.Talagrand, O., R. Vautard, and B. Strauss, 1998: Evaluation of probabilistic prediction systems.

*Proc. ECMWF Seminar on Predictability,*Reading, United Kingdom, ECMWF, 1–26.Tiedtke, M., 1993: Representation of clouds in large-scale models.

,*Mon. Wea. Rev***121****,**3040–3061.Toth, Z., and E. Kalnay, 1993: Ensemble forecasting at NMC: The generation of perturbations.

,*Bull. Amer. Meteor. Soc***74****,**2317–2330.Toth, Z., E. Kalnay, S. Tracton, R. Wobus, and J. Irwin, 1997: A synoptic evaluation of the NCEP ensemble.

,*Wea. Forecasting***12****,**140–153.Toth, Z., Y. Zhu, T. Marchok, S. Tracton, and E. Kalnay, 1998: Verification of the NCEP global ensemble forecasts. Preprints,

*12th Conf. on Numerical Weather Prediction,*Phoenix, AZ, Amer. Meteor. Soc., 286–289.Tracton, M. S., and E. Kalnay, 1993: Operational ensemble prediction at the National Meteorological Center: Practical aspects.

,*Wea. Forecasting***8****,**379–398.Wilks, D. S., 1995:

*Statistical Methods in the Atmospheric Sciences*. Academic Press, 467 pp.

(*Continued*)

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(*Continued*)

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(*Continued*)

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(a)–(d) Four initial perturbations of 500-hPa geopotential height for the BM-EPS for 0000 UTC 15 Jul 2001; contour interval of 3 m. (e)–(h) As in (a)–(d), but for EC-EPS

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(a)–(d) Four initial perturbations of 500-hPa geopotential height for the BM-EPS for 0000 UTC 15 Jul 2001; contour interval of 3 m. (e)–(h) As in (a)–(d), but for EC-EPS

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(a)–(d) Four initial perturbations of 500-hPa geopotential height for the BM-EPS for 0000 UTC 15 Jul 2001; contour interval of 3 m. (e)–(h) As in (a)–(d), but for EC-EPS

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(*Continued*)

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(*Continued*)

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(*Continued*)

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

Probability density function of ensemble spread around the control forecast for the BM-EPS (solid) and EC-EPS (dash), for (top) the Australian region and for (bottom) the Southern Hemisphere, at (left) forecast day 3, (middle), day 5, and (right) day 7. The probability density function has been constructed using 500-hPa geopotential height predictions for 152 daily cases

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

Probability density function of ensemble spread around the control forecast for the BM-EPS (solid) and EC-EPS (dash), for (top) the Australian region and for (bottom) the Southern Hemisphere, at (left) forecast day 3, (middle), day 5, and (right) day 7. The probability density function has been constructed using 500-hPa geopotential height predictions for 152 daily cases

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

Probability density function of ensemble spread around the control forecast for the BM-EPS (solid) and EC-EPS (dash), for (top) the Australian region and for (bottom) the Southern Hemisphere, at (left) forecast day 3, (middle), day 5, and (right) day 7. The probability density function has been constructed using 500-hPa geopotential height predictions for 152 daily cases

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

Root-mean-square ensemble spread around the control forecast of the BM-EPS (solid) and EC-EPS (dash), for (top) the Australian region and (bottom) the Southern Hemisphere region. (The rms has been computed over all grid points in each region and over all ensemble members)

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

Root-mean-square ensemble spread around the control forecast of the BM-EPS (solid) and EC-EPS (dash), for (top) the Australian region and (bottom) the Southern Hemisphere region. (The rms has been computed over all grid points in each region and over all ensemble members)

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

Root-mean-square ensemble spread around the control forecast of the BM-EPS (solid) and EC-EPS (dash), for (top) the Australian region and (bottom) the Southern Hemisphere region. (The rms has been computed over all grid points in each region and over all ensemble members)

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

Probability density function of ensemble skill for the BM-EPS (solid) and the EC-EPS (dash), for (top) the Australian region and (bottom) the Southern Hemisphere at (left) forecast day 3, (middle) day 5, and (right) day 7. The probability density function has been constructed using 500-hPa geopotential height values for 152 cases

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

Probability density function of ensemble skill for the BM-EPS (solid) and the EC-EPS (dash), for (top) the Australian region and (bottom) the Southern Hemisphere at (left) forecast day 3, (middle) day 5, and (right) day 7. The probability density function has been constructed using 500-hPa geopotential height values for 152 cases

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

Probability density function of ensemble skill for the BM-EPS (solid) and the EC-EPS (dash), for (top) the Australian region and (bottom) the Southern Hemisphere at (left) forecast day 3, (middle) day 5, and (right) day 7. The probability density function has been constructed using 500-hPa geopotential height values for 152 cases

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(a) The 152-case root-mean-square error of the control forecast of the BM-EPS (solid) and EC-EPS (dash) over (top) the Australian region and (bottom) the Southern Hemisphere. (b) As in (a), but for the ensemble mean forecast. (c) The root-mean-square error of each individual forecast averaged over all ensemble members and all 152 cases. (d) As in (a), but the percentage of all members over the 152 cases where the root-mean-square prediction error is less than the root-mean-square error of the control prediction. Forecast errors have been computed using 500-hPa geopotential height data

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(a) The 152-case root-mean-square error of the control forecast of the BM-EPS (solid) and EC-EPS (dash) over (top) the Australian region and (bottom) the Southern Hemisphere. (b) As in (a), but for the ensemble mean forecast. (c) The root-mean-square error of each individual forecast averaged over all ensemble members and all 152 cases. (d) As in (a), but the percentage of all members over the 152 cases where the root-mean-square prediction error is less than the root-mean-square error of the control prediction. Forecast errors have been computed using 500-hPa geopotential height data

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(a) The 152-case root-mean-square error of the control forecast of the BM-EPS (solid) and EC-EPS (dash) over (top) the Australian region and (bottom) the Southern Hemisphere. (b) As in (a), but for the ensemble mean forecast. (c) The root-mean-square error of each individual forecast averaged over all ensemble members and all 152 cases. (d) As in (a), but the percentage of all members over the 152 cases where the root-mean-square prediction error is less than the root-mean-square error of the control prediction. Forecast errors have been computed using 500-hPa geopotential height data

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(*Continued*)

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(*Continued*)

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(*Continued*)

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

Time series of day-5 root-mean-square error of the control (solid), ensemble mean (dash), best perturbed member (dotted), and the worst perturbed member (chain dash), for the (upper) BM-EPS and (lower) EC-EPS. Values plotted are 5-day running mean errors and refer to 500-hPa geopotential height forecasts over SH

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

Time series of day-5 root-mean-square error of the control (solid), ensemble mean (dash), best perturbed member (dotted), and the worst perturbed member (chain dash), for the (upper) BM-EPS and (lower) EC-EPS. Values plotted are 5-day running mean errors and refer to 500-hPa geopotential height forecasts over SH

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

Time series of day-5 root-mean-square error of the control (solid), ensemble mean (dash), best perturbed member (dotted), and the worst perturbed member (chain dash), for the (upper) BM-EPS and (lower) EC-EPS. Values plotted are 5-day running mean errors and refer to 500-hPa geopotential height forecasts over SH

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

As in Fig. 7, but for AU

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

As in Fig. 7, but for AU

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

As in Fig. 7, but for AU

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

Average of the difference between the root-mean-square error of the ensemble mean and the root-mean-square error of the control forecast of the BM-EPS (solid) and EC-EPS (dash) for (upper) AU and (lower) SH. A positive (negative) value indicates that the ensemble mean error is larger (smaller) than the control error. The average has been computed using 500-hPa geopotential height forecast data over 152 cases

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

Average of the difference between the root-mean-square error of the ensemble mean and the root-mean-square error of the control forecast of the BM-EPS (solid) and EC-EPS (dash) for (upper) AU and (lower) SH. A positive (negative) value indicates that the ensemble mean error is larger (smaller) than the control error. The average has been computed using 500-hPa geopotential height forecast data over 152 cases

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

Average of the difference between the root-mean-square error of the ensemble mean and the root-mean-square error of the control forecast of the BM-EPS (solid) and EC-EPS (dash) for (upper) AU and (lower) SH. A positive (negative) value indicates that the ensemble mean error is larger (smaller) than the control error. The average has been computed using 500-hPa geopotential height forecast data over 152 cases

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

Difference between the average ensemble spread around the control forecast and the error of the control forecast of the BM-EPS (solid) and EC-EPS (dash) for (upper) AU and for (lower) SH. A positive (negative) value indicates that the ensemble spread is larger (smaller) than the control error. The average has been computed using 500-hPa geopotential height forecasts for 152 cases

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

Difference between the average ensemble spread around the control forecast and the error of the control forecast of the BM-EPS (solid) and EC-EPS (dash) for (upper) AU and for (lower) SH. A positive (negative) value indicates that the ensemble spread is larger (smaller) than the control error. The average has been computed using 500-hPa geopotential height forecasts for 152 cases

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

Difference between the average ensemble spread around the control forecast and the error of the control forecast of the BM-EPS (solid) and EC-EPS (dash) for (upper) AU and for (lower) SH. A positive (negative) value indicates that the ensemble spread is larger (smaller) than the control error. The average has been computed using 500-hPa geopotential height forecasts for 152 cases

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(a) ROC area of the BM-EPS (solid) and EC-EPS (dash) probability forecasts for SH for (top) the event “500-hPa geopotential height positive anomaly with magnitude larger than 50 m” and for (bottom) the event “500-hPa geopotential height negative anomaly with magnitude larger than 50 m.” (b) As in (a), but for Brier skill score instead of the ROC area

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(a) ROC area of the BM-EPS (solid) and EC-EPS (dash) probability forecasts for SH for (top) the event “500-hPa geopotential height positive anomaly with magnitude larger than 50 m” and for (bottom) the event “500-hPa geopotential height negative anomaly with magnitude larger than 50 m.” (b) As in (a), but for Brier skill score instead of the ROC area

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(a) ROC area of the BM-EPS (solid) and EC-EPS (dash) probability forecasts for SH for (top) the event “500-hPa geopotential height positive anomaly with magnitude larger than 50 m” and for (bottom) the event “500-hPa geopotential height negative anomaly with magnitude larger than 50 m.” (b) As in (a), but for Brier skill score instead of the ROC area

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(a) As in Fig. 11a, but for AU. (b) As in Fig. 11b, but for AU

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(a) As in Fig. 11a, but for AU. (b) As in Fig. 11b, but for AU

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

(a) As in Fig. 11a, but for AU. (b) As in Fig. 11b, but for AU

Citation: Monthly Weather Review 132, 10; 10.1175/1520-0493(2004)132<2338:POTEAT>2.0.CO;2

List of the principal differences between the BM-EPS and the EC-EPS

Root-mean-square and std dev (in parentheses) of 152 cases of day-5 root-mean-square forecast error for the control, ensemble mean, best ensemble member, and worst ensemble member for the BM-EPS and EC-EPS

Percentage of cases with root-mean-square spread relative to the control smaller than root-mean-square error of the control forecast at forecast days 3, 5, and 7 over the SH and AU for the BM-EPS and EC-EPS. Values refer to 500-hPa geopotential height field forecasts

Correlation of root-mean-square spread relative to the control with root-mean-square error of the control forecast at forecast days 3, 5, and 7 over the SH and AU for the BM-EPS and EC-EPS. Values refer to 500-hPa geopotential height field forecasts

Average excess over expected percentage (over 152 cases) of outliers {i.e., percentage of times the analysis value lies outside the forecast range spanned by the ensemble forecasts minus the ex pected value 100 * [2/(N_{ens} + 1)] at forecast days 3, 5, and 7 over the SH and AU for the BM-EPS and EC-EPS}