## 1. Introduction

Due to the chaotic nature of the weather, small errors in the analysis will grow and limit the predictability (Lorenz 1963) of atmospheric flow. Initial errors and the fact that a numerical model is only an approximation of the true atmosphere give rise to uncertainties in all numerical weather forecasts. To estimate these uncertainties, many forecast centers produce ensemble forecasts. The idea of using ensemble forecasting to improve medium-range forecasts has been known for many years (Leith 1974) and since the early 1990s forecast centers generate ensemble forecasts. An ensemble forecast comprises a number of numerical forecasts, integrated from different initial conditions to represent the initial uncertainties in the analysis. To represent the model uncertainties, some forecast centers use perturbed model physics in the ensemble member simulations.

Techniques to construct initial perturbations for ensemble prediction have a number of objectives. One is to span most of the probable range of developments in the atmosphere; another is to reflect the uncertainties in the analysis. Various perturbation methods focus on these different features. The European Centre for Medium-Range Weather Forecasts (ECMWF) uses the singular vector (SV) method to achieve maximum perturbation growth rate for a given optimization time (Lorenz 1965; Palmer 1993). The National Centers of Environmental Prediction (NCEP) uses the Ensemble Transform (ET) technique (Wei et al. 2008), which is an improved version of the breeding vector (BV) technique (Toth and Kalnay 1993, 2008) to perturb the initial conditions. The breeding perturbations use the previous ensemble to obtain the growing component of the analysis error. The breeding vectors are closely related to the Lyapunov vectors of a dynamical system (Kalnay et al. 2002).

Other initial perturbation approaches, such as the ensemble Kalman filter technique using perturbed observations (Houtekamer et al. 1996) and the ensemble transform Kalman filter (Wei et al. 2006) are aimed at sampling the analysis uncertainties. Note, however, that both SV and BV can be modified to partly take the analysis uncertainties into account.

Numerous studies have been made to compare the singular vector and breeding vector methods using a spectrum of different simplified models (Anderson 1997; Trevisan and Legnani 1995; Smith et al. 1999; Bowler 2006; Houtekamer and Derome 1995; Trevisan et al. 2001; Wei and Frederiksen 2004; Descamps and Talagrand 2007). One conclusion from several of the studies is that different breeding vectors tend to be similar; they converge to the same structure. This is obvious in a low-order system but can also be seen in the eigenvalue spectra for breeding vectors in a NWP model [Wang and Bishop (2003); see section 6]. To overcome this problem, NCEP recently started using ET, which gives orthogonal perturbations.

One property of singular vectors is that the perturbation tends to initially grow faster than the forecast error. This may be a result of singular vectors structures being on average more baroclinic than the average analysis error (Bontemps and Källén 1999). This characteristic could also be related to the super-exponential growth demonstrated by singular vectors (Wei and Frederiksen 2004). By using evolved singular vectors, the growth of spread during the first 2 days is rendered more similar to the slower growth of forecast error (Barkmeijer et al. 1999).

The structure of the singular vectors is dependent of the choice of norm (inner product). To obtain perturbations consistent with analysis error statistics, Palmer et al. (1998) argue that singular vectors should be computed with an analysis error covariance norm as the initial time norm. Recent work by Lawrence et al. (2007, manuscript submitted to *Quart. J. Roy. Meteor. Soc.*) suggests that total energy as initial time norm gives similar baroclinic singular vector structures as using an analysis error covariance norm. Also ET is dependent of the choice of norm and NCEP use an inverse analysis error variance norm (Wei et al. 2008).

The full ensemble prediction systems of NCEP and ECMWF have been compared in a number of studies (e.g., Buizza et al. 2005; Froude et al. 2007; Wei and Toth 2003). These investigations compare the performance of the entire forecast system (i.e., the assimilation system, the forecast model, and the perturbation method). Buizza et al. (2005) concluded that it was difficult to distinguish the performance of the perturbation methods and that of the other components. Nevertheless, they found that most verification measures indicated that the ECMWF EPS demonstrates the overall best performance, but this should not be considered as a proof of a superior performance of the SV-based initial perturbations. Finally, Buizza et al. (2005) proposed an experiment to compare the different perturbation methods with the same analysis and forecast system.

In the present study, the breeding method and the singular vector method are compared using the same unperturbed analysis and the same forecast model. The numerical experiments use the ECMWF Integrated Forecast System (IFS). This provides an opportunity to compare BV and SV in the same model environment, both by looking at the different skill scores and by comparing the dynamics of the perturbation evolution. In this first investigation, the simplest possible version of the breeding vector ensemble prediction system (simple BV-EPS) as well as regional rescaled breeding vectors (masked BV-EPS) are used. Future plans include implementation of ET perturbations (Wei et al. 2008). In the next section, the operational EPS at ECMWF and the implemented breeding system are described, and in section 3 the model and the experiment setup are presented. The perturbation distributions are described in section 4. Thereafter, section 5 deals with the results. In section 6, the orthogonality of the perturbations is evaluated by studying the variance spectra. The study is concluded by a summary as well as a comparison with the results of Buizza et al. (2005), in the light of recent results from NCEP (Wei et al. 2008).

## 2. Initial perturbation methodologies

In this section, the two methods (singular vectors and breeding vectors) used in this paper are described. Both are aimed at sampling the fastest-growing components of initial errors. While the breeding approach attempts to simulate the development of growing errors in the analysis cycle (Toth and Kalnay 1997), singular vectors are perturbations that can produce the fastest linear growth in the near future (48 h).

### a. ECMWF singular vector approach

*P*is the surface pressure and

_{s}*T*(300 K) and

_{r}*P*(800 hPa) are a reference temperature and pressure perturbation, respectively.

_{r}A total of 50 singular vectors are calculated separately for each hemisphere, aimed to find evolved perturbations localized to the extratropics for each hemisphere (30°–90° latitude). A global computation of singular vectors would yield too few singular vectors in the summer hemisphere; therefore, two separate sets of singular vectors are computed in each hemisphere. For the extratropics, in addition to initial singular vectors the initial perturbations are also based on 48-h linearly evolved SV, computed 48 h prior to the ensemble starting time. This is done to represent slower-growing large-scale structures (Barkmeijer et al. 1999; Puri et al. 2001).

For the tropics, the singular vectors are sensitive to the representation of diabatic physics in the tangent linear model. Singular vectors computed for the entire tropical region sometimes represent spurious structures that do not grow in the nonlinear forecast model and the perturbations relevant for tropical cyclone motion are not necessarily among the leading few singular vectors. Therefore, the singular vector computation in the tropics has been limited to the vicinity of tropical depressions and tropical cyclones (Barkmeijer et al. 2001).

In our study the different sets of singular vectors are combined to nine perturbations, using Gaussian sampling, which is used operationally since 2004 (Leutbecher and Palmer 2008). The standard deviation of the sampled Gaussian depends on analysis error estimates provided by the ECMWF four-dimensional variational data assimilation (4DVAR) scheme (Fisher and Courtier 1995). To yield adequate ensemble dispersion the perturbations are scaled by an empirically determined parameter. To ensure that the ensemble is centered on the analysis, a plus–minus symmetry is adopted (Leutbecher and Palmer 2008).

### b. Breeding vector approach

*x*

^{f}

_{i,j,k}) is defined as the difference between the state vector for a perturbed forecast and an unperturbed forecast. The difference is calculated for a certain forecast step (cycle length). The analysis perturbation to be added to the new analysis is calculated asso that the perturbation is normalized with respect to the total energy [Eq. (1)] and where the rescaling factor

*α*is an empirically determined value of the initial perturbation amplitude, set to yield a sufficient dispersion of the ensemble. For the very first ensemble, an arbitrary perturbation is added to the atmosphere. Hereafter, the breeding cycle is repeated to generate initial perturbations (breeding vectors). After a few cycles, statistics of the breeding vector structure (e.g., variance distribution, spatial and multivariate correlations, etc.) should become independent of the initial perturbation used to initialize the very first ensemble. By starting from different initial conditions for the very first initialization, several independent breeding cycles can be obtained.

*c*

_{i,j}of latitude and longitude, also known as

*mask*. It is defined aswhere

*e*

_{i,j}is the vertically integrated analysis uncertainty measured in total energy and

*y*

^{a}

_{i,j}is the total energy of the forecast perturbation (

*x*

^{f}

_{i,j,k}) vertically integrated. Both the analysis uncertainty and the forecast perturbation are horizontally smoothed (truncated to T21 in our case) before the vertical integration and the total energy calculation. To investigate the sensitivity of the truncation additional tests have been performed with T7, showing no improvement in probabilistic scores (z500 and t850) for the Northern Hemisphere (NH) and rather the opposite for u850 in the tropics. Figure 1 shows one example of the mask

*c*

_{i,j}to be applied to a forecast perturbation.

*c*

_{i,j}, the perturbation field is smoothed horizontally to maintain most of the dynamical balance in the breeding vectors (Toth and Kalnay 1997).

The perturbations are finally rescaled to obtain a sufficient dispersion in the same manner as for simple breeding.

Some details of the rescaling differ in our system when compared to the mask used by NCEP. First, we use an estimate of the analysis error provided from the 4DVAR assimilation system (Fisher and Courtier 1995) instead of using a constant-in-time estimate of the analysis uncertainties (Toth and Kalnay 1997). Second, we have used the total energy norm to measure the size of the perturbations. Third, we truncate the regional rescaling mask at T21 instead of T7 as used at NCEP.

## 3. Model and data

For the experiments the ECMWF IFS-model is used with resolution T* _{L}*255L40 and model cycle 31rl (for details see www.ecmwf.int). For each method 18 ensemble members are used. The ensemble systems have been running once a day (0000 UTC) between 1 December 2005 and 15 January 2006.

To generate the breeding ensemble, nine separate perturbed forecasts with different breeding vectors are run. Each produces a 6-h forecast that is used to initialize the next cycle. The very first ensemble was initialized from a 6-h singular vector ensemble. For the first breeding cycles after the initialization, the perturbations is fast growing, a property inherited from the singular vectors. After a few cycles, however, the perturbations find the breeding modes and the growth rate stabilizes. The first BV-EPS forecast was initialized after 16 cycles (4 days).

The nine breeding vectors calculated for every 0000 UTC are both added to and subtracted from the analysis, this to obtain a 18-member ensemble centered around the analysis. The reason we employ a plus–minus symmetric ensemble instead of spherical simplex transformed (Wang et al. 2004) for BV-EPS is that the operational SV-EPS at ECMWF uses a plus–minus symmetric ensemble. The initial perturbation amplitude *α*, for both BV-EPS configurations separately, has been chosen to obtain an ensemble spread of the same magnitude for 500-hPa geopotential height as SV-EPS spread for the NH at day 3.

To represent model error, ECMWF uses the stochastic diabatic tendency perturbations (i.e., “stochastic physics”; Buizza et al. 1999). By scaling the tendencies from the physical schemes with a random number, the uncertainties in the parameterizations are simulated. The random values are between 0.5 and 1.5 drawn from a uniform distribution. The same random number is used in a box of 10° × 10° extent and over a period of 6 time steps (4.5 h). This stochastic scheme is used in all ensemble forecast experiments.

## 4. Initial perturbation distribution

In this section, we will investigate the distribution of the perturbations, initially and after 48 h. This is done by calculating the total energy of the difference between the perturbed and the unperturbed forecast (control forecast) at every grid point. Figure 2 shows the meridional average of total energy accumulated with the initial perturbations (step + 0 h) as a mean for the period 1 December 2005–15 January 2006. Figure 2a is for SV-EPS. The perturbation energy is almost equally distributed between the hemispheres but with very small perturbations in the tropics (see section 2). We see that the maximum of the perturbation energy is located in the subtropical jet region on both hemispheres (35°N, 300 hPa and 40°S, 300 hPa). The horizontal distribution over the NH (not shown) indicates that one maximum of the perturbations is located south of Japan (which also contains the maximum of kinetic energy for the atmosphere), another is located in the storm track over the Atlantic Ocean.

Figure 2b shows the same as above but for simple BV-EPS. The most obvious difference is that breeding has much more energy in the initial perturbations. For breeding the maximum amplitude is around 5 times larger compared to SV and most of the perturbation energy is located on the NH. The difference in initial amplitude is necessary to obtain the same amplitude for medium-range forecasts. The perturbations for simple BV-EPS also have a more poleward location compared to SV-EPS. As the case for SV-EPS, the amplitude of the perturbations is very small over the tropics. For the masked BV-EPS (Fig. 2c), the perturbations are more equally distributed over the globe. Compared to the other two systems the masked BV-EPS also has perturbation energy in the tropics. The perturbations are more equally spread between the hemispheres compared to simple BV-EPS but not as uniform as for SV-EPS.

In Fig. 2d the average of an estimate of the analysis error standard deviation over the time period provided by the 4DVAR system is shown. The analysis error is plotted as total energy, but without the influence of humidity. This is used here to make the energy directly comparable with the perturbation. Comparing with the perturbations from the three systems we see that the masked breeding resembles the distribution of the uncertainties in the analysis most accurately. This is due to the fact that the regional rescaling uses the horizontal distribution of the analysis error as a constraint.

Figure 3 shows the mean distribution of perturbation energy after 48 h. At this time the magnitude of the perturbation energy is comparable between the systems; the energy for SV-EPS has grown much faster than for BV-EPS during the first 48 h. For SV-EPS, the energy maximum has moved poleward from the subtropical jet region and is located in the same region as the perturbations for BV-EPS. The distribution is now more similar between the systems. This is also seen in Bontemps and Källén (1999) who used a quasigeostrophic, two-layer, baroclinic model and concluded that the localization and the horizontal structure of breeding vectors and singular vectors were nearly the same at the optimization time for SV.

We have also compared the perturbation distribution after 48 h with the forecast error energy (without humidity). In Fig. 3d the distribution of forecast error for the control forecast is plotted. This has almost the same distribution as the ensemble mean error for all three experiments. We see for the NH that all three experiments are underdispersive. SV-EPS and masked BV-EPS show similarities in the perturbation distribution with a maximum at 300 hPa and 45°N. The simple BV-EPS has a maximum of perturbation energy in the Artic region where it resembles the actual forecast error (Fig. 3d). For the Southern Hemisphere (SH), both breeding systems have too little spread. Evaluating the tropics, all three systems have too little spread but masked BV-EPS is closest to the real forecast error distribution.

To investigate the spatial scales of the perturbations we have calculated the spectrum of perturbations for Z500. In Fig. 4 the spectrum is shown for initial time and after 48 h. Initially we see that SV-EPS has lower amplitude compared to both BV-EPS experiments and a spectral peak at wavenumber 11. Simple BV-EPS has a peak at about wavenumber 8. We can see that masked BV-EPS has larger amplitude in the long waves compared to the other two methods. This could be explained by the regional rescaling, which introduces an enhanced variability on the largest scales. After 48 h the amplitude of the spectral component for the different experiment shows similarities but we can see that masked BV-EPS still has larger amplitude in the long waves than SV-EPS. We conclude that despite the initial difference in amplitude they are rather similar after 48 h, except for the longest waves.

## 5. Spread and probabilistic scores

To compare the results of both perturbation methods, we have used several verification methods. The methods are RMSE for ensemble mean, the spread–RMSE ratio, the anomaly correlation coefficient (ACC), the ranked probability skill score (RPSS), the ignorance skill score (ISS), and the area under the relative operating characteristics (ROC). The results are reported for 500-hPa geopotential height, if nothing else is stated.

To test if the differences in scores between the experiments are statistically significant, we have used a moving-block bootstrap method. The sample of *N* cases is resampled with replacement to obtain *M* different samples of *N* cases. For our calculation we have used *M* = 5000. The resampling uses blocks of *L* consecutive dates to account for the temporal correlation of the score differences. The block size *L* depends on forecast lead time. The block length is set using an AR-2 model following Wilks (1997). The empirical distribution of the difference of the average statistic (e.g., RPSS or RMSE) is computed. Then, the confidence interval is determined as the *x*th and (l00 − *x*)th percentile of the empirical distribution. To determine if the difference between the experiments significantly differs from zero we have used the 99% confidence limit.

### a. Ensemble mean RMSE and ensemble standard deviation

The RMSE of the ensemble mean is a deterministic measure of its performance. In Fig. 5 this is plotted for simple BV-EPS (dashed), masked BV-EPS (dash–dotted), and SV-EPS (solid). For the NH (Fig. 5a) the SV-EPS is slightly better than both breeding systems for the first 8 days (the difference is statistically significant between SV-EPS and simple BV-EPS for days 1–4, and between SV-EPS and masked BV-EPS for days 1–2). For the last days the result for masked BV-EPS approaches the RMSE for SV-EPS and yields the same result for days 8–10. For the SH (Fig. 5b) all three systems give the same RMSE until day 7. Hereafter simple BV-EPS has slightly higher RMSE, but the difference is not statistically significant.

The spread (also shown in Fig. 5) of the ensemble is the standard deviation of the ensemble members. For a “perfect ensemble,” the spread of the ensemble will be equal to the RMSE of the ensemble mean for all lead times (Palmer et al. 2006). For the NH (Fig. 5a) the spread for SV-EPS is initially (after 12 h) almost equal to the RMSE, but the super-exponential effect (Trevisan et al. 2001) causes the perturbation to grow faster than the RMSE during the optimization time (48 h). Hereafter the spread grows slower than the RMSE, and becomes smaller than the RMSE after 4 days. Both sets of BV-EPS results have a much larger spread initially than the RMSE but the difference decreases and the spreads become smaller than RMSE after 3.5 days. For long forecast lengths the spread becomes equal for all ensembles with a too low spread. This problem may be attributed to the lack of variability in the model. In addition to the simple BV-EPS experiment used here, we have also run one with the same initial amplitude as SV-EPS (not shown). This resulted in too low ensemble dispersion for the breeding experiment in the medium range.

The spread compared to RMSE differs substantially between the two hemispheres. For the SH the spread for SV-EPS is larger than the RMSE for the first five days. For simple BV-EPS the opposite holds true, the spread is too low. One reason for the low spread on the SH is that the breeding vectors are pointing toward the most “active” areas on the globe. During winter these are on the NH. Using masked breeding, which distributes the initial perturbations more equally over the globe, has partly solved this problem. As mentioned above, the initial perturbations for breeding are constructed to yield optimal spread on the NH; therefore, it may not be optimal on the SH. This could be due to perturbations growing faster on the winter hemisphere and therefore being favored in the breeding cycle. The problem could be solved by calculating one rescaling factor for each of the hemispheres, and changing them with season (Toth and Kalnay 1997). For SV-EPS, singular vectors are calculated for both hemispheres and hereafter combined to a perturbation. That gives, by construction, an almost equal spread between the hemispheres initially.

### b. Anomaly correlation coefficient

The ACC is the spatial mean of the correlation of anomalies in the analysis and ensemble mean with respect to the climatological mean. Employing this measurement of performance (not shown), we recognize that for the NH the SV-EPS has a somewhat better performance than the simple BV-EPS, but the difference is not statistically significant. Masked BV-EPS is better than simple BV-EPS and yields the same results as SV-EPS for the last forecast days. For the SH the opposite holds true. Both BV-EPS techniques perform better than SV-EPS and the difference is statistically significant between masked BV-EPS and SV-EPS for days 3–5.

### c. Probabilistic skill scores

To measure the probabilistic skill of the ensemble systems we have used the RPSS (Candille and Talagrand 2005), the ROC area (Wilks 2006), and the ISS (Roulston and Smith 2002). RPSS for the NH shows that the SV-EPS is less than 6 h better than the simple BV-EPS, but the different is statistically significant until day 7. Masked BV-EPS gives results in between SV-EPS and simple BV-EPS but the difference is only statistically significant around day 4. For the SH the results are almost equal for masked BV-EPS and SV-EPS (except the first 2 days where masked BV-EPS is significant better than SV-EPS) and slightly worse for simple BV-EPS (statistically significant until day 5). The ROC area and ISS have been calculated for anomalies in Z500 greater than one standard deviation (not shown). Evaluating the ROC area, SV-EPS and masked BV-EPS show almost equal results and simple BV-EPS is slightly worse for both the NH and the SH. For ISS, SV-EPS shows the best scores for both hemispheres and almost all evaluated forecast times. This could partly be explained by the higher ensemble spread for SV-EPS that yields more accurate simulations of the probability distribution function. In addition to the evaluation of anomalies of greater than one standard deviation, we have also evaluated anomalies <−1 standard deviation. The results follow the same line as for the anomalies greater than one standard deviation.

### d. Other parameters

In addition to Z500, we have also evaluated the scores for the temperature for 850 hPa, the wind components for 850 and 200 hPa, and the geopotential height for 1000 hPa. The RPSS for T850 (NH) is shown in Fig. 6a. We see that SV-EPS performs better compared to both sets of BV-EPS. The difference is statistically significant between SV-EPS and masked BV-EPS between days 3 and 6 and between SV-EPS and simple BV-EPS between days 0 and 6. The slightly better performance for SV-EPS holds true for all other parameters on the NH. For the SH the difference is larger for T850 between SV-EPS and simple BV-EPS (Fig. 6b) compared to the NH and is statistically significant between days 1 and 10. Comparing SV-EPS and masked BV-EPS on the SH, SV-EPS has significant better scores between days 3 and 6 and masked BV-EPS is significantly better at days 1 and 2.

In Table 1, the ROC area and ISS are given for T850 anomaly greater than one standard deviation. The overall performance is slightly better for SV-EPS, but the differences are larger evaluating ISS than the ROC area. Regarding ISS, all three experiments are significantly different for each other for SH and for NH SV-EPS is significantly better than both masked BV-EPS and simple BV-EPS at days 3, 5, and 7.

### e. Tropics

Because of the collapse of geostrophy in the equatorial region, the mass field here exerts only minor influence on the weather. Therefore, the ensemble systems have been compared for the zonal wind component (*u*) at 850 hPa instead of *z* at 500 hPa for the tropical region. The overall performance is best for masked BV-EPS because of a larger ensemble spread (Fig. 7) for the first 7 forecast days (discussed in the section 4). By construction, tropical singular vectors are only computed in the vicinity of tropical cyclones. Therefore low initial spread is expected for the tropics (see section 2). The spread for SV-EPS approaches the spread for masked BV-EPS and becomes equal after day 7 when perturbations from the midlatitudes have propagated into the tropics. For simple BV-EPS the spread is lower compared to the other two.

Figure 8 shows RPSS for u850 in the tropics. Up to day 6 masked BV-EPS (also here) performs significantly better than SV-EPS, but the results approach each other with increasing forecast time. Simple BV-EPS has the worst performance both looking at RMSE and RPSS. However, it should be possible to change the performance of the breeding system in the tropics by changing norm and breeding cycle length (Chikamoto et al. 2007).

## 6. Variance spectra

The number of modes spanned by the ensemble can be examined using the eigenvalue spectra of the covariance matrix of the perturbations (Wang and Bishop 2003). The eigenvalue spectrum shown in Fig. 9 is a mean between 1 December 2005 and 15 January 2006 for Z500 and is calculated for NH. For breeding, the results of simple BV-EPS is used (both breeding experiments look similar). Both BV-EPS and SV-EPS use paired members; therefore, only nine different modes are linearly independent initially.

For the initial time step, we see (Fig. 9a) that the variance for the breeding perturbations is larger compared to SV-EPS (as seen in Fig. 5a) but confined to the first modes. This is a sign of the nonorthogonality of the breeding vectors. Singular vectors are by definition orthogonal in the total energy space (flat eigenvalue spectra), but they do not have to be orthogonal for Z500.

Figure 9b shows the variance spectrum after 48 h. At this time step, the total spread is almost equal for both systems. By studying the spectrum we see that the spread of the SV-EPS is distributed over more modes (flatter spectra) compared to the BV-EPS. This holds true also for time step +96 and +120 h (not shown), where the spectra are similar to the one at +144 h (Fig. 9c). Here the total spread is larger for SV-EPS. In the spectra we see that this difference manifests itself in the higher modes, implying that the SV-EPS distribute the spread over more modes compared to BV-EPS also for medium-range forecasts. After 240 h (Fig. 9d) the spread spectrum are equal for both ensembles.

The fact that the singular vectors spread on more modes may be one reason underlying the better performance of the SV-EPS system. This could be resolved for BV-EPS by using orthogonal breeding vectors such as ET (Wei et al. 2008).

In addition to the experiment described above, we have also run an experiment without using the stochastic physics for the perturbed forecasts. This was done to determine if the stochastic physics affects the linear dependence between breeding vectors. By comparing the eigenvalue spectra for the initial time (not shown), we found that using stochastic physics gives somewhat flatter eigenvalue spectra. Using stochastic physics, all perturbed forecasts are running on somewhat different models, which implies that the different breeding vectors diverge, and a higher level of orthogonality is maintained. We have not investigated the impact of the stochastic physics on the verification scores and we refer to previous studies (e.g., Buizza et al. 1999).

## 7. Summary and conclusions

In this paper, we have compared the breeding technique and the singular vector technique using the ECMWF Integrated Forecast System (IFS). For the breeding system, we have adhered to the general ideas advanced by Toth and Kalnay (1997). The purpose of the investigation has been to compare the breeding and singular vector approaches in the same model environment. The data used is for a 46-day period between 1 December 2005 and 15 January 2006.

- The overall results show that the SV-EPS performs somewhat better compared to both breeding experiments in the NH. For BV-EPS, we can see a slight improvement compared to simple breeding in the NH using masked breeding. But the largest improvement is on the SH and in the tropics by using the masked breeding. This confirms the previous results by Toth and Kalnay (1997) as well as those by Wang and Bishop (2003). For the SH SV-EPS and masked BV-EPS yield almost identical results (except studying the ignorance skill score where SV-EPS yields the best scores). Both breeding systems have a too little spread on the SH, but better for masked. For the tropics, evaluating the performance for the zonal wind component at 850 hPa, the forecast skill is best for masked BV-EPS for the first 6 forecast days. One reason for the better performance of masked BV-EPS when compared to SV-EPS in the tropics is the fact that singular vectors are not used to perturb the tropics in the ECMWF EPS, except for special localized occurrences of tropical cyclones. The masked BV-EPS performs better compared to simple BV-EPS because the perturbations are more equally spread over the globe using the masking approach.
- For the NH there is no difference in lead time reaching ACC 0.6 for the ensemble mean between SV-EPS and masked BV-EPS. In the study from Buizza et al. (2005) the difference is 18 h between ECMWF and NCEP (which used masked breeding vectors). This implies that the dominating part of the difference in results reported by Buizza et al. (2005) probably comes from differences in the data assimilation and forecast models between the centers, rather than from the choice of initial perturbation methods.
- The major problem with the breeding system is the difference in spread between the hemispheres. This could be solved by rescaling the perturbations separately for the hemispheres (Toth and Kalnay 1993). Another disadvantage of breeding vectors is that the initial spread needs to be very large compared to the analysis error. This is caused by the initial spread being optimized to give a sufficient spread for medium-range forecasts. However, the need of large initial amplitude also indicates that all perturbations are not efficient and grow slowly. For singular vectors, the initial super-exponential growth makes the spreads comparable for BV-EPS and SV-EPS after 3 days. For longer lead times, the spread of the three experiments are of the same magnitude (for the NH).
- To calculate the initial condition, a larger amount of computer resources is needed for calculating singular vectors compared to breeding vectors. But compared to the total computer resources needed for the ensemble, the initial condition is a small fraction (about 5% for the operational SV-EPS at ECMWF). It is also worth noting that one additional cost for BV-EPS is to run extra cycles between the forecasts. In our case, we have had to run three additional 6-h ensemble forecasts with 10 members each day.
- By evaluating the variance spectra for the BV-EPS and SV-EPS we found that the breeding vectors have the major part of the variance in a few modes. This could lead to an increase of the number of ensemble members in a BV-EPS not improving the skill of the ensemble to the same degree as in a singular-vector based EPS. However, more research is needed to verify if this indeed is the case and whether it may be feasible to overcome this problem using orthogonal breeding vectors. Recent research at NCEP shows that it is possible to improve the EPS using an orthogonal breeding system, such as ET (Wei et al. 2008). Using simplex transformation instead of a paired ensemble could also give better results (Wang et al. 2004).
- One issue, not investigated in this paper, is the choice of norm for the breeding vectors. The norm could have an effect when orthogonalizing, but also when rescaling the breeding vectors. This needs further investigation and could be a source for improvements of the breeding system.

After this study we can conclude that the difference in RMSE and the anomaly correlation coefficient for the ensemble mean is smaller than the difference documented by Buizza et al. (2005). We also think that there are further possibilities to improve our breeding system, in particular the rescaling and orthogonalization techniques. The rescaling needs to be further investigated both with respect to geographical areas and norm dependence. Orthogonalization can be done either using the Gram–Schmidt technique or some other ensemble transforms techniques. Further studies will explore these different techniques and compare results both in terms of probabilistic and average performance skill scores.

## Acknowledgments

We would like to express our gratitude to Dr. Jonas Nycander, Stockholm University, for valuable discussions and ideas during the preparation of this paper. We would also address our appreciations to the Swedish Meteorological and Hydrological Institute (SMHI) for allowing us to use their computer allocation at ECMWF. We also thank User Support, ECMWF, for their help with technical issues during the implementation of the breeding system.

## REFERENCES

Anderson, J. L., 1997: The impact of dynamical constraints on the selection of initial conditions for ensemble predictions: Low-order perfect model results.

,*Mon. Wea. Rev.***125****,**2969–2983.Barkmeijer, J., , R. Buizza, , and T. N. Palmer, 1999: 3D-Var Hessian singular vectors and their potential use in ECMWF Ensemble Prediction System.

,*Quart. J. Roy. Meteor. Soc.***125****,**2333–2351.Barkmeijer, J., , R. Buizza, , T. N. Palmer, , K. Puri, , and J-F. Mahfouf, 2001: Tropical singular vectors computed with linearized diabatic physics.

,*Quart. J. Roy. Meteor. Soc.***127****,**685–708.Bontemps, B., , and E. Källén, 1999: Singular vectors and breeding modes: A comparative study. DM Rep. 81, Department of Meteorology, Stockholm University, 38 pp.

Bowler, N., 2006: Comparison of error breeding, singular vectors, random perturbations and ensemble Kalman filter perturbation strategies on a simple model.

,*Tellus***58A****,**538–548.Buizza, R., , M. Miller, , and T. Palmer, 1999: Stochastic representation of model uncertainties in the ECMWF Ensemble Prediction System.

,*Quart. J. Roy. Meteor. Soc.***125****,**2887–2908.Buizza, R., , P. L. Houtekamer, , Z. Toth, , G. Pellerin, , M. Wei, , and Y. Zhu, 2005: A comparison of the ECMWF, MSC and NCEP Global Ensemble Prediction Systems.

,*Mon. Wea. Rev.***133****,**1067–1097.Candille, G., , and O. Talagrand, 2005: Evaluation of probabilistic prediction systems for a scalar variable.

,*Quart. J. Roy. Meteor. Soc.***131****,**2131–2150.Chikamoto, Y., , H. Mukougawa, , T. Kubota, , H. Sato, , A. Ito, , and S. Maeda, 2007: Evidence of growing bred vector associated with the tropical intraseasonal oscillation.

,*Geophys. Res. Lett.***34****.**L04806, doi:10.1029/2006GL028450.Descamps, L., , and O. Talagrand, 2007: On some aspects of the definition of initial conditions for ensemble prediction.

,*Mon. Wea. Rev.***135****,**3260–3272.Fisher, M., , and P. Courtier, 1995: Estimating the covariance matrices of analysis and forecast error in variational data assimilation. Tech. Memo. 220, ECMWF, 28 pp.

Froude, L. S. R., , L. Bengtsson, , and K. I. Hodges, 2007: The prediction of extratropical storm tracks by the ECMWF and NCEP ensemble prediction systems.

,*Mon. Wea. Rev.***135****,**315–333.Houtekamer, P. L., , and J. Derome, 1995: Methods for ensemble prediction.

,*Mon. Wea. Rev.***123****,**2181–2196.Houtekamer, P. L., , L. Lefaivre, , J. Derome, , H. Ritchie, , and H. Mitchell, 1996: A system simulation approach to ensemble prediction.

,*Mon. Wea. Rev.***124****,**1225–1242.Kalnay, E., , M. Corazza, , and M. Cai, 2002: Are bred vectors the same as Lyapunov vectors? Preprints.

*Symp. on Observations, Data Assimilation and Probabilistic Prediction,*Orlando, FL, Amer. Meteor. Soc., 173–177.Leith, C., 1974: Theoretical skill of Monte Carlo forecasts.

,*Mon. Wea. Rev.***102****,**409–418.Leutbecher, M., , and T. N. Palmer, 2008: Ensemble forecasting.

,*J. Comput. Phys.***227****,**3515–3539.Lorenz, E. N., 1963: Deterministic nonperiodic flow.

,*J. Atmos. Sci.***20****,**130–141.Lorenz, E. N., 1965: A study of the predictability of a 28-variable atmospheric model.

,*Tellus***17****,**321–333.Palmer, T. N., 1993: Extended-range atmospheric prediction and the Lorenz model.

,*Bull. Amer. Meteor. Soc.***74****,**49–66.Palmer, T. N., , R. Gelaro, , J. Barkmeijer, , and R. Buizza, 1998: Singular vectors, metrics and adaptive observations.

,*J. Atmos. Sci.***55****,**633–653.Palmer, T. N., , R. Buizza, , R. Hagedorn, , A. Lawrence, , M. Leutbecher, , and L. Smith, 2006: Ensemble prediction: A pedagogical perspective. Newsletter 106, ECMWF, 44 pp.

Puri, K., , J. Barkmeijer, , and T. N. Palmer, 2001: Ensemble prediction of tropical cyclones using targeted diabatic singular vectors.

,*Quart. J. Roy. Meteor. Soc.***127****,**709–731.Roulston, M. S., , and L. A. Smith, 2002: Evaluating probabilistic forecasts using information theory.

,*Mon. Wea. Rev.***130****,**1653–1660.Smith, L. A., , C. Ziehmann, , and K. Fraedrich, 1999: Uncertainty dynamics and predictability in chaotic systems.

,*Quart. J. Roy. Meteor. Soc.***125****,**2855–2886.Toth, Z., , and E. Kalnay, 1993: Ensemble forecasting at NCEP: The generation of perturbation.

,*Bull. Amer. Meteor. Soc.***74****,**2317–2330.Toth, Z., , and E. Kalnay, 1997: Ensemble forecasting at NCEP and the breeding method.

,*Mon. Wea. Rev.***125****,**3297–3319.Trevisan, A., , and R. Legnani, 1995: Transient error growth and local predictability: A study in the Lorenz system.

,*Tellus***47A****,**103–117.Trevisan, A., , F. Pancotti, , and F. Molteni, 2001: Ensemble prediction in a model with flow regimes.

,*Quart. J. Roy. Meteor. Soc.***127****,**343–358.Wang, X., , and C. Bishop, 2003: A comparison of breeding and ensemble transform Kalman filter ensemble forecast schemes.

,*J. Atmos. Sci.***60****,**1140–1158.Wang, X., , C. Bishop, , and S. Julier, 2004: Which is better, an ensemble of positive–negative pairs or a centered spherical simplex ensemble?

,*Mon. Wea. Rev.***132****,**1590–1605.Wei, M., , and Z. Toth, 2003: A new measure of ensemble performance: Perturbation versus Error Correlation Analysis (PECA).

,*Mon. Wea. Rev.***131****,**1549–1565.Wei, M., , and J. Frederiksen, 2004: Error growth and dynamical vectors during southern hemisphere blocking.

,*Nonlinear Processes Geophys.***11****,**99–118.Wei, M., , Z. Toth, , R. Wobus, , Y. Zhu, , C. H. Bishop, , and X. Wang, 2006: Ensemble Transform Kalman filter-based ensemble perturbations in an operational global prediction system at NCEP.

,*Tellus***58A****,**28–44.Wei, M., , Z. Toth, , R. Wobus, , and Y. Zhu, 2008: Initial perturbations based on the Ensemble Transform (ET) technique in the NCEP Global Ensemble Forecast Systems.

,*Tellus***60A****,**62–79.Wilks, D. S., 1997: Resampling hypothesis tests for autocorrelated fields.

,*J. Climate***10****,**65–82.Wilks, D. S., 2006:

*Statistical Methods in the Atmospheric Sciences*. 2nd ed. Academic Press, 627 pp.

The ROC area and ISS for T850, anomaly >1.0 std dev.