a. NCEP Ensemble Forecast System

The SAWS has been downloading subsets of the NCEP EFS on a daily basis since 2000. The operational configuration at NCEP used for this study has been in effect since May 2000 and consists of 23 ensemble members per day out to 16 days ahead. At 0000 UTC the suite consists of a high-resolution (T170L42, T254L64 since April 2003) control run up to 7 days, truncated to T62L28 for the remaining 9 days; a low-resolution (T62L28) control run; plus five pairs of perturbed integrations derived from the breeding cycle (Toth and Kalnay 1993; 1997). At 1200 UTC the high-resolution control run extends only to 3.5 days, and there are another five pairs of independently bred perturbations. The full ensemble set thus consists of 23 members per day. Since March 2004 the ensembles have been available four times daily at a T126L28 resolution (more detail on the NCEP EFS is available online at http://wwwt.emc.ncep.noaa.gov/gmb/ens/index.html). Because of bandwidth constraints the SAWS has, up to the writing of this paper, only downloaded the coarser 2.5°-resolution datasets. Although the model output assessed here is at this lower resolution, the model performance has been continuously improving through upgraded model physics, resolution, and data assimilation, and these effects are automatically manifest in the coarser-output sets.

b. Verification methods

Atmospheric variables such as pressure level geopotential heights and sea level pressure are compared against the ensemble control run analysis. These continuous data are verified using root-mean-square error and anomaly correlation coefficient scores. To calculate the anomalies used for the correlation calculations, monthly climatological fields, derived from NCEP–National Center for Atmospheric Research reanalysis data (Kalnay et al. 1996), are converted to daily values by performing a linear interpolation between the two nearest monthly values, and are subtracted from the forecast fields.

Surface variables such as rainfall and maximum–minimum temperatures are verified against station data from the SAWS database. There are 96 temperature stations across South Africa and between 1500 and 2000 rainfall stations (Fig. 1). At the 2.5° EFS forecast resolution, roughly 30–200 rainfall stations fall into each model box, with the lower values corresponding to the sparsely populated arid areas in the western interior of South Africa. This verification is done using two approaches. The first consists of model forecast values of temperature and rainfall probabilities being interpolated to the 96 forecast station locations and the second (used mainly in rainfall verification) of a value constructed for each model box using an average of the stations within the box. The EFS is designed for box-average probabilistic quantitative precipitation forecasts (PQPFs) and not point values, and is therefore not expected to perform well at individual points. A further caveat to note here is that convective-type rainfall in the summer rainfall areas of South Africa has a particularly high spatial variability and station measurements are sometimes not representative of the area—in this case the 2.5° model grid box. This also highlights the reverse problem of deriving a point forecast of rainfall from model grid boxes and will be addressed in future research. In a study over Australia (which has similar conditions to South Africa), Ebert et al. (2003) suggest that the differences between forecast and observed rainfall fields are much greater than errors in verification data (from representativeness), suggesting that we may proceed to verify PQPFs with caution. This verification is done using the equitable threat score (ETS; Ebert 2001), which has the advantage of measuring the fraction of observed and/or forecast events that were correctly predicted but adjusting the score for hits associated with random chance.

The skill of probabilistic forecasts is verified using the Brier skill score (BSS; Brier 1950; Wilks 1995). This skill score measures the squared probability error relative to climatology. Murphy (1973) decomposed the Brier score into reliability (agreement between forecast probability and observed frequency), resolution (ability of the forecast probabilities to distinguish between events and nonevents), and uncertainty (depends on climatology and has a maximum value of 0.25 when the observed frequency is 50%). A further measure of the ability of forecasts to distinguish between events and nonevents is shown using the relative operating characteristic (ROC; Mason 1982). The BSS looks at performance stratified by forecast probabilities and the ROC performance based on the observations. The ability of the spread of the EFS to represent the variability of the real atmosphere is measured using the rank histogram (Talagrand et al. 1997; Hamill 2001). This measure is particularly useful in determining whether the EFS has errors in its mean and spread. Again, as pointed out by Hamill (2001), errors in observations (mostly through representativeness) may introduce false signals in this histogram. More on this will follow in the discussion.

c. Forecast products

The products from the EFS can be divided into two groups. The first is the set of products provided to the forecasters that is used as guidance in compiling their medium-range forecasts. The second group consists of computer-generated products that are disseminated to the public and/or specialized users. These are each described separately below.

a. Continuous variables

This section focuses predominantly on the 0000 UTC 500-hPa height forecasts for the period January 2001–December 2004 and over a domain covering the South Atlantic Ocean, southern Africa, and the southwestern Indian Ocean. Results for sea level pressure concur on the whole with the 500-hPa heights and are thus not repeated in the results. The ensemble average (10 perturbation members) has a lower RMSE than the low-resolution (high resolution) control run after 5 (6) days (Fig. 3). However, the ensemble-average RMSE exceeds the RMSE obtained by using climatology (the limit of skill) after day 8, only 1 day after the control runs reach this threshold. This shows that the benefit of the ensemble technique over the control run to make skillful forecasts of the instantaneous 500-hPa fields is realized on average for only 1 day in this region. Notwithstanding, when taking the best ensemble member at each forecast case (identified a posteriori), the RMSE is smaller than that of climatology up to day 12. Bourke et al. (2004) also note the advantage of using the Bureau of Meteorology EFS, with a 33% relative gain of the best ensemble member to the high-resolution control over Australia at day 5. A tally of the ensemble members shows that the high-resolution control run is the most accurate more often than any of the other ensemble members for the first 3 days of the forecast. The low-resolution control is usually most accurate about half of the amount of time that the high-resolution control is most accurate, illustrating the level of improvement in skill that can be realized by increasing the model resolution. From forecast day 4, one of the perturbed ensemble members is usually the most accurate. By the end of the forecast period (day 16), any ensemble member has an equal chance of being the best. In essence, the ensemble technique is successful in producing at least one forecast member that can be used to make a skillful forecast for this region up to 12 days ahead. This suggests that the breeding method is able to capture the uncertainty in the initial conditions such that the probability distribution of the ensemble forecast envelope does cover the observed events out to this lead time.

As already mentioned, upgrades to the Global Forecast System (GFS) model at NCEP have led to improvements in the model performance. This is evident in the ensemble-average, best-member, and control runs of the EFS. Generally, the skill of the best member has improved by almost 2 days to 14 days and the bias has been reduced by 60% when comparing the scores for 2001 against those of 2004 (not shown). It is also worth noting that the ensemble mean is slightly worse than the control run of the same resolution, for the first few days (also noted in Szunyogh and Toth 2002). This is because the ensemble suite is generated using the breeding method (Toth and Kalnay 1993, 1997), which adds perturbations to the control analysis, in the form of stochastic noise, with the aim of reducing model systematic error in the ensuing forecasts. The uncertainty estimate mask that determines the size of these perturbations has only recently been upgraded at NCEP and was known to produce inflated perturbations for the Southern Hemisphere. This would partially explain the problem with the ensemble mean. Additionally, the perturbed ensemble-member forecasts would be initially disadvantaged relative to the control forecasts if verified against the control analysis. However, the ensemble-generating technique is designed to capture the uncertainty in the initial conditions and hopefully provide a more useful forecast probability distribution. The benefit of this becomes clearer later in this paper, particularly with the 850–500-hPa-thickness probability forecasts.

The anomaly correlation coefficient (ACC) scores are similar to the RMSE scores (Fig. 4). One notable difference is that over the smaller domain (37.5°–17.5°S and 10°–40°E) the ensemble-average ACC scores remain poorer than the high-resolution control throughout the forecast period, whereas the ensemble average beats the high-resolution control from day 6 in the large domain. Furthermore, the score of the best-ensemble member at each case is higher over the small domain, than the large domain. These scores indicate that locally the correct type of weather system is being simulated by the model by (at least) some members and the poorer score for the ensemble average could be caused by the smoothing effect of combining forecasts of the same weather system, but located at different positions.

A useful forecast system must be able to capture as much of the range of natural variability as possible. One way to measure this is to plot the standard deviation of the ensemble-member fields from their own time-averaged value for the full verification period at each forecast lead time (Fig. 5). During the first week of the forecast the variance decreases, but then begins to increase during the second week. The control runs have a lower variance than that observed, with the low-resolution control considerably lower than the high-resolution control. This is consistent with the constrained atmospheric variability, in this region in particular, caused by the finite resolution of the models (Stratton 1999; Tennant 2003). The perturbation breeding method introduces additional variability and results in the first 3 days of the perturbed ensemble members having a higher variance than observed. The ensemble average obviously underestimates the variance significantly, making this field unsuitable for event forecasting in this region despite its apparent advantage over individual ensemble members in terms of RMSE and ACC between forecast days 5 and 8.

b. Bias correction

The most basic way to correct model systematic biases is by subtracting the long-term mean error of the forecasts (Richardson 2001). This is usually done independently for each grid point and forecast lead time. However, this omits two other rather important dependencies, namely those related to the season and to the circulation regime. To address this issue, Atger (2003) proposed a spatially and temporally dependent bias correction. To introduce this sort of bias correction in an operational environment, Eckel and Mass (2005) adopted a 14-day running-mean bias calculation. Cui et al. (2005) also discuss these methods. This study follows such a bias-correction technique with the following additions. The running mean was tested for 30, 14, and 7 days, and the 14-day running mean was found to be optimal. Bias correction was done independently for each control ensemble member but as a group for the perturbed ensemble members. This was necessitated by the different resolutions (and hence biases) of the control runs (Szunyogh and Toth 2002) and possible differences introduced by the initial perturbations on the perturbed members. There is no reason to expect the bias of any particular perturbed ensemble member to be different from the others, so the same average bias of all the perturbed members was subtracted from each member.

The bias-correction procedure was performed as follows. Starting on 1 January 2001, a bias for each forecast lead time for forecasts valid for the 14-day window period (ending 31 December 2000) was calculated. This was done by calculating the mean difference between the forecasts and the observed fields multiplied by a factor alpha (estimated empirically at 0.33; i.e., 33% of the forecast error could be attributed to the forecast bias). All the forecasts valid over the window period were corrected by subtracting the bias factor. The process was then repeated for 2 January 2001 (with the 14-day window extending from 17 December 2000 to 1 January 2001) and each day in turn until 1 January 2005. During this process each particular forecast case was bias corrected 14 times as the window moved across the time. This iterative process provided more stability to the bias-correction process. The latest forecast (simulating an operational environment) was bias corrected using the mean difference between the standard forecasts and the latest set of iteratively bias-corrected forecasts over the 14-day window. In this way the most up-to-date bias information was used to correct the current forecast. These forecasts (with only one bias-correction step) were saved separately and used in the verification process. The main advantage of this bias-correction method is that the bias can be calculated from a relatively short period and thus be implemented easily in an operational environment.

Over the southern Africa domain, the GFS model exhibits an increasing area-average negative bias with forecast lead time (Fig. 6). The magnitude of the bias is dependent on model resolution, with a larger bias associated with the lower resolution. As expected from results of other studies (Atger 2003; Cui et al. 2005), the bias-correction method is successful in reducing the magnitude of the bias considerably throughout the forecast period. Talagrand diagrams also confirm this bias reduction with a more even distribution (not shown). The forecast bias cannot be totally removed using these sorts of correction methods in a real-time forecasting sense, as it is not possible to fully anticipate the systematic-error component of future forecast errors based on past errors. Furthermore, efficient bias correction does not always necessarily lead to an improvement in forecast skill, but the method discussed here does also improve skill somewhat (Fig. 4). This improvement is also evident in the increase in variability after bias correction (Fig. 5). This probably occurs when the adaptive bias correction shifts the model forecast away from the model climate (with constrained variability) toward reality with more variability.

It is interesting that the ensemble perturbation members have a smaller bias than the low-resolution control run at forecast days 2 and 3 (shown by the ensemble-average curve in Fig. 6). The only differences between these runs are the perturbations added to the initial conditions, suggesting that this bias is influenced by the initial conditions. Buizza et al. (2005) state that a successful EFS should capture the effect of both initial condition and model uncertainties on forecast errors. To investigate this, the spatial variation of the difference between the perturbed ensemble members and the control run is shown in Fig. 7. Here, we see that the ensemble-breeding method has an overall spatial pattern where the perturbed ensemble members (on average at time zero) tend to have higher values over the land areas and South Atlantic anticyclone, and negative in the midlatitude westerlies for both the 500-hPa height and sea level pressure fields relative to the control analysis. These are clearly very small values but certainly spatially coherent. Although the cause of this is not clear, some further investigation revealed that these patterns seem to be related to the interpolation of the high-resolution analysis (from NCEP’s Global Data Assimilation System) to create the analysis for the lower model resolution of the ensemble set. This lower-resolution analysis is used for the breeding of the ensemble perturbations. After 24 h, these same patterns amplify to amounts that neatly offset the traditional model bias in the region of a negative bias in the subtropics and a positive bias in the midlatitudes of the region. Such a pattern where the Southern Hemisphere jet stream is displaced toward the equator by many general circulation models is familiar in the region (Tennant 2003). Furthermore, the bias-correction method, although successful in reducing the bias spatially, does leave some of the spatial pattern of the bias behind (Fig. 8). These highlight the need for regime-dependent bias correction, since model performance can be linked to correctly simulating circulation regimes (Chessa and Lalaurette 2001).

c. Probability of precipitation and temperature change forecasts

A clear bias in the EFS is evident over southern Africa in terms of inflated quantitative precipitation probabilities. The root of this problem appears to be that the NCEP GFS model overestimates rainfall amounts, climatologically speaking, by up to 300% over the summer rainfall areas of South Africa, especially along the eastern escarpment at 30°E (Fig. 9). Surprisingly, there is little improvement in the high-resolution forecast over the low-resolution forecast, as usually such biases tend to be related to model resolution. Of further note is that this bias becomes greater for rainfall amounts greater than 5 mm, especially for the longer forecast lead times (Fig. 10). The bias score used here, calculated as the number of forecast cases divided by the number of observed cases, is thus very large (up to 50), especially given the small number of observed cases of >20 mm over the 2.5° × 2.5° model grid box. This would partially explain why the high-resolution model has such a strong bias, because large rainfall amounts tend to be more easily generated by higher-resolution models (Mullen and Buizza 2002). Furthermore, forecast lead time appears to have little bearing on the magnitude of the bias (except for high-resolution control at 5 days), suggesting a problem with the model physics (possibly the precipitation parameterization schemes) over this region. In contrast to the summer rainfall areas, rainfall is underestimated in the winter rainfall region of the Western Cape Province (Figs. 9 and 11). In this region we have large-scale rain-bearing frontal systems that are enhanced by local topography, a feature probably not adequately captured by the current resolution of the NCEP global model.

Talagrand diagrams (Hamill 2001) confirm the findings above (Fig. 12). The summer rainfall region has a clear bias of forecasting too much rain too often (left-skewed diagram), where the observation is less than the driest ensemble member a third of the time. The winter rainfall region on the other hand has a U-shaped diagram indicating a lack of variability in this region, although the diagram does exhibit a slight left-sided bias. Again, these patterns do not change significantly with forecast lead time.

Quantitative precipitation forecast (QPF) fields were calibrated by adjusting the event threshold (i.e., the precipitation forecast amount defining a “yes” or “no” forecast event), so that the forecast frequency matched the observed frequency for the verification period (July 2000–June 2005). For this study, a cross-validation technique was used where each year was withheld from the calculations while the other 4 yr were used to calculate the forecast frequencies. Roebber et al. (2004) suggest that an increase in model resolution, postprocessing of model data, and combining high-resolution with ensemble techniques are practical ways to improve rainfall prediction. These approaches concur with the findings in this study in the following way.

Over the summer rainfall area, ETSs of the calibrated high-resolution control forecasts are improved for light rainfall amounts (Fig. 10). Over the winter rainfall area, the calibration of the high-resolution control was not as successful, except for 5-day forecasts of larger rainfall amounts over the Western Cape Province (Fig. 11). This is probably because the model does not capture the orographic augmentation of rainfall in this region adequately, resulting in a largely systematic bias that can easily be corrected. It is noteworthy that the ETS for highly simplified ensemble probability forecasts (assuming a deterministic yes forecast when the ensemble probability exceeds 50%) generally beats the control run scores for rainfall amounts less than 20 mm, and even more so at the longer lead times (Fig. 10). This demonstrates the utility of the EFS to perform QPFs for light rainfall events. Unfortunately, as found by Legg and Mylne (2004), the calibration of the ensemble QPF, particularly in the summer rainfall region, causes a significant deterioration of the ETS score for heavier rainfall events by reducing the probabilities too far as seen from the negative bias in Fig. 10. Perhaps, this requires a different interpretation of the probabilities for the more extreme events, as these would hardly ever exceed 50% in practice. The negative bias here is dependent on a deterministic decision of what probability should be considered to be a yes forecast.

Equivalent results were obtained from frequency adjustment of the ensemble PQPF values. An increase in the BSS and a marginal increase in the ROC were evident for the first few days of light-rain (<2 mm) forecasts over the summer rainfall area, but for heavier events (>10 mm) the ROC score deteriorated. Over the winter rainfall region, fairly good improvements were made to the BSS and ROC score during the first week. This is attributed to the successful correction of the systematic bias in model-simulated rainfall in this region as mentioned above.

Probability forecasts of temperature changes of 2° and 5°C are skillful for week 1 over most of South Africa and for week 2 over parts of the interior (not shown). Coastal temperatures are probably not resolved sufficiently by the model resolution and are less skillful than those over the interior. Maximum temperatures are more skillful than minimum temperatures, pointing again to the lack of resolution of subgrid-scale processes at night.

d. Forecaster guidance—Probability of events

A useful EFS product for event forecasting is the probability of the 850–500-hPa-thickness field falling below 4200 and 4100 gpm. These synoptic situations are good indicators of extreme cold weather and possible snowfall in South Africa, both of which are considered high-impact events in the region. Over the northern parts of the country the thickness fields almost never fall below 4100 gpm, but do fall below 4200 gpm around 10 times per winter season. Given the high altitude of the escarpment (2000–3000 m) and interior plateau (1500 m), surface temperatures are usually below freezing overnight over large areas. In the southern parts of the country, thickness fields below 4200 gpm occur almost half the time (corresponding to a maximum uncertainty value in the Brier score decomposition), but occurrences of values less than 4100 gpm match those of 4200 gpm over the northern parts. The results discussed below for the 4100-gpm events in the Western Cape Province correspond roughly to the 4200-gpm events in the northern regions of South Africa.

The EFS is able to capture these events in South Africa adequately for the first 7 days of the forecast, as shown by positive BSSs and ROC scores in excess of 0.5 (Figs. 13 and 14). Although reliability and resolution deteriorate during the second week, the forecasts still retain reasonable resolution throughout the forecast period. This suggests that calibration may be able to improve the skill of these forecasts.

The first calibration method tested here is the bias correction described above, where the 850–500-hPa-thickness fields were adjusted at each grid point and new thickness probability fields calculated. This method was successful in improving the BSS, resolution, and reliability considerably during the second week (Fig. 13). Rare events show a significant improvement in the BSSs, but this is not reflected in the ROC scores, which are related to forecast resolution (Fig. 14).

The second calibration method, where the event threshold is adjusted to match the forecast probability with the observed frequency, is more successful in improving both the BSS and ROC score for the 4200-gpm events (Fig. 13). The reliability is much improved (as the calibration is intended to do) and there is also a small increase in resolution during the second week. For the more rare 4100-gpm event, the BSS is again improved and the ROC score is worse than the raw output (Fig. 14). Overall, the first calibration method is more successful as it addresses the bias in the physical patterns more directly. However, neither calibration method does much for the resolution of the more extreme (rarer) events and is consistent with findings in Legg and Mylne (2004) where the skill in predicting extreme events often deteriorates after bias correction.

Forecast uncertainty is another useful forecaster guidance tool. This is indicated by shading the ensemble spread (categorized into a strong, medium, or weak signal and no predictability) under the control prognostic 500-hPa height and sea level pressure fields. It is determined using a basic definition of calculating the ensemble standard deviation around the ensemble mean and dividing by the observed field standard deviation for that time of year at each grid point (Scherrer et al. 2004). Values in the ranges 0%–33%, 33%–66%, 66%–100%, and >100% correspond to strong, medium, weak, and no signal, respectively. Periods of strong atmospheric instability may also lead to a large ensemble spread but not necessarily increased forecast uncertainty. Toth et al. (2001) proposed a relative measure of predictability based on the position of the forecast value in terms of climatological distribution to provide a quantitative probability of forecast uncertainty. The intention at the SAWS for now, however, is that forecasters use these uncertainty fields as a qualitative rather than a quantitative measure of forecast confidence. Notwithstanding, further development and refinement of this process is under way at the SAWS.