a. Predictability from WF EOFS

Here we investigate the skill of the WF EOFS in simulating or hindcasting the Hastenrath rainfall series throughout the year. We use the term “simulation” when using predictor SSTs measured simultaneously with the rainfall and “hindcast” when the predictor SST is measured before the rainfall season begins, and real-time forecasts are not being considered. We assess the simulation skill of the WF SST EOFS for periods of the year for which the correlation analyses suggest that the EOFs are useful. For most of these assessments, standard correlation and LEPSOB or LEPSOBFC are used. In most cases, the variation in LEPS scores is similar to the correlations, except that LEPS skill values are generally not far from 0.7 times the correlation value. To minimize artificial skill, the year being hindcast or simulated, and that preceding and following, are excluded from the regression calculation. This is called the “jackknife” method. The number of years before and after the target year that should be excluded depends on the serial correlation of the predicted series. The above choice is sufficient as NEB rainfall shows little interannual persistence (first serial correlation 1912–97 = 0.13); a quasi-biennial peak in the NEB rainfall spectrum (Chu 1984) contributes to this behavior. Thus a hindcast for 1960 would utilize a regression equation calculated using data for 1912–58 and 1962–97. This method gives as many regression equations as years being hindcast or simulated; the coefficients of the predictor variables in the regression equations also vary with time of year.

GISST3 is used to calculate the time series of the WF EOFs, though the EOFs themselves were calculated from MOHSST3. Figure 5a shows correlations between simulated and observed Hastenrath rainfall for periods of 1–6 months in length over 1948–97. Correlations are highest during the rainy season and generally increase with the length of simulation period. For shorter periods, correlation is a maximum for periods centerd around March and April, the height of the rainy season, but for longer periods this shifts slightly to periods centerd on February. The skill for individual months is generally poor with the strong exception of April (r = 0.66). Skill improves considerably for two-month periods so that March–April and April–May have good skill (r = 0.62, 0.63). Three-month periods are better still while six-month skill has correlations >0.6 from November–April to February–July. The corresponding LEPSOB scores have a similar shape but are considerably lower. LEPSOB penalises the assessments when the forecasts have a different distribution from the observations whereas LEPSOBFC does not. LEPSOB is generally lower than LEPSOBFC because the simulations have a smaller standard deviation than the observations. However, for six-month periods there is only a small difference in the standard deviations so that LEPSOB = 0.44 and LEPSOBFC = 0.49.

Figure 5b assesses hindcasts made from predictors based on SST in the month but one before the start of the hindcast period. This is a practical lead period for long lead forecasts. Remarkably, such hindcasts are often more skillful than the simulations in Fig. 5a. This may reflect a delay in the influence of SST on NEB rainfall. If true, this has beneficial implications for seasonal predictability here and possibly elsewhere in the Tropics. Maximum correlation scores for 1- to 6-month periods are in the range 0.74–0.78. April is again the most skillful single month with a correlation as high as r = 0.78. The most skillful three-month period is April–June (r =0.75) (note this is not the wettest period), the most skillful 4-month period is April–July (r = 0.74), and the most skillful 6-month period is April–September (r = 0.75). This is a strong indication that useful forecasts are possible beyond May for periods greater than 3 months long, though the WF EOFs may not be optimal. The LEPSOB and LEPSOBFC skills are also higher than the corresponding LEPS skill of the simulations. Again LEPSOBFC has higher skill (typically by 0.1) than LEPSOB, with values in the range 0.50–0.55 for the most skilful periods 1–6 months long (not shown). From these results, it seems best to use inflated regression in real-time forecasts, which (nearly) achieves equality of forecast and observed variance. This has only been done for real-time forecasts from 1999.

Staying with the idea of using SST from a single month to create the prediction equation and concentrating on the core rainfall season March–May, Fig. 5c shows the correlation skill for hindcasts using the Atlantic and Pacific EOFs separately and combined. Thus, Fig. 5c also shows how skill varies as the delay between SST and the core forecast period increases. SST is measured in individual prior months from the June of the previous year to the last month of the forecast period, May. We first discuss the skill obtained from using Pacific and Atlantic EOFs together. Figure 5c provides additional guidance to that in WF (e.g., their Fig. 8) on how far ahead it is possible to forecast the main rainfall season. Taking a minimum correlation of 0.6 as clearly useful. March–May rainfall is skillfully predictable from SST measured in December onward, though November might be regarded as giving a useful preview. This corresponds to the real-time practice for preliminary forecasts discussed in this paper where SST averaged between November and January was often used. For January SST, the hindcast correlation is 0.65 (as in Fig. 5b, three-month curve), but peaks for February SST at r = 0.71. This result indicates that “updated” forecasts, or nowcasts, made in early March for March–May rainfall using February SST may give modest extra guidance, as has been done in real-time forecasting since 1988. Note how the skill declines when SST from within the forecast period is used.

Considering the Atlantic and Pacific Oceans separately, consistent with Eq. (4) of WF, the Atlantic WF EOF shows considerably more skill than does the Pacific EOF. Assuming 45 degrees of freedom, the Pacific EOF on its own does not have significant skill (r ≧ 0.29) until February. Nevertheless, Fig. 5c shows that when combined in a stepwise linear regression equation, the combined skill of the two oceanic predictors, measured by linear correlation, exceeds that from either predictor individually after November. Thus NEB provides an example of predictability in a region where El Niño SST patterns play a significant but lesser role compared to a (nearly) independent SST pattern in another tropical ocean (the Atlantic). However, a small part of this Atlantic influence is an indirect impact of ENSO because ENSO affects tropical Atlantic SSTs (Curtis and Hastenrath 1995; Enfield and Mayer 1997; Sutton et al. 2000). Before November, on average almost all the skill for predicting the following March–May rainfall comes from the Atlantic SSTs, though this may not be true in some strong El Niño years.

Harzallah et al. (1995) suggested that NEB rainfall is usefully predictable from SST six months in advance. Our results indicate that this is perhaps not quite achievable for the core rainfall season March–May. However, an issue is the optimum period over which to average SST. So far we have not used averaging periods of less than one month (e.g., using the last week in January to make a long lead forecast in early February), believing that noise in the data would increase too much. [This may be less of a problem with reduced noise SST analyses now becoming available (Jones et al. 2001) though currently these are only made monthly]. So we have concentrated on finding out if averaging periods of two or three months have any benefit, as has often been used in the real-time forecasts. Any benefit is likely to come from a further reduction on noise, offset by the inclusion of older SST data.

Figure 5d shows how correlation skill varies with hindcast lead time and with the length of the period from which the SST predictors are calculated. The hindcast period is again fixed at March–May, and SST is measured over a variety of prior periods ending at the month shown. Only the results for the combined Atlantic and Pacific WF predictors are shown. Other methods of averaging the SST could be imagined and have been used in some real-time forecasts, for example, by giving more weight to the more recent months of a three-month averaging period. However, it is clear from Fig. 5d that there is no advantage in averaging SST beyond a month, though one and two months give similar skill, and three months marginally worse skill except for hindcasts using SST ending in October. Figure 5d indicates again that it seems inadvisable for March–May forecasts to be based on predictors using SST measured before November, though there is some skill. Additional information can be obtained by taking a closer look at the probability distribution of the hindcasts (Table 1). To make this useful for the future, we have used inflated regression as now used routinely in the real-time forecasts. Table 1 shows the skill of hindcasts in the five equiprobable (quint) categories that are routinely made using LEPSOB, which gives similar results to LEPSOBFC as inflated regression is used. A value greater than about 0.5 represents a very useful level of skill while 0.3 is marginal. Table 1 shows the well-known result (Van den Dool and Toth 1991) that the extremes have higher skill than the central categories, and that the average category has no obvious skill. Hindcasts using November SST have useful skill if a hindcast is made for an extreme quint, but not otherwise. Marginally useful skill perhaps exists for hindcast quints 2 and 4 in January and more definitely in February. Thus we must expect the bulk of the skill in seasonal forecasts for NEB to reside in the extremes but not to be solely confined to them. This is fortunate as the wet and dry extremes are societally the most important anomalies to predict.

Finally, we note that although Fig. 4a shows a dipole of tropical Atlantic SST anomalies as a major component of its pattern, the two halves of the dipole fluctuate nearly independently (e.g., Houghton and Tourre 1992;Enfield and Meyer 1997) except on near decadal (Mehta and Delworth 1995; Chang et al. 1997) and perhaps multidecadal (e.g., Folland et al. 1986) timescales. Nevertheless, Fig. 4a reflects the importance of the tropical interhemispheric SST gradient to NEB rainfall, which results from largely independent SST fluctuations in the two tropical hemispheres, of especially large amplitude (not shown) in the NEB rainy season.

a. Model correlations with SST

Figure 6 shows simultaneous correlations between the average rainfall simulated by HadAM2b at the four grid points and SST for 1948–97, calculated for the same months as the observations in Fig. 3. These correlations provide guidance on how the SST patterns influence the model in simulating NEB rainfall. In January–February (Fig. 6a) the pattern is similar to the observations in Fig. 3a except that correlations with the tropical central and eastern Pacific are larger and more significant. This may in part result from the use of ensemble mean data for the model rainfall time series, which removes a large part of the random effects of internal atmospheric variations. Both model and observations show only weak correlations with the north tropical Atlantic. In March–April (Fig. 6b), the model shows strong negative correlations with the tropical north Atlantic and the tropical Pacific and positive correlations with the tropical south Atlantic, similar to observations. The negative correlations with the tropical Pacific are again somewhat stronger than observed, though not so strikingly as in January–February. In May–June (Fig. 6c) correlations weaken in all three ocean basins, but only a little, and less than observed (not shown). At the beginning of the rainfall season (November–December, Fig. 6d), the model shows stronger negative correlations in the tropical central and east Pacific than observed, and stronger positive correlations in the south Tropical Atlantic, though like the observations, only very weak negative correlations are seen in the tropical North Atlantic. Overall, the strengthening of model correlations compared to the observed is mostly in the tropical Pacific. So, in linear terms at least, model rainfall is rather more influenced by tropical Pacific SSTs, and therefore ENSO, than are the observations but the effect is not large. An overstrong influence of ENSO on Atlantic sector tropical climate is a known problem with coupled models with similar atmospheric physics to HadAM2b (S. Tett 1998, personal communication), though NEB is not in the worst affected region.

b. Model simulations of Northeast Brazil rainfall

Figure 7a shows the skill of the model in simulating rainfall on timescales from 1–6 months (we emphasize these are not hindcasts). On the 1-month timescale, the model has positive skill in every month but r > 0.6 only in April–July. On the 2-month timescale r > 0.6 from January–February until July–August reaching 0.80 in April–May. On the 3-month timescale r is slightly higher again most of the year, exceeding 0.7 from January–March until June–August and peaking at r = 0.84 in April–June. Six-month periods show generally the highest skills, all but two 6-month periods (out of 12 possible) having r > 0.6. Peak skill occurs around periods centered on April and May when r = 0.87. Thus the model has exceptional potential when skill is measured this way and could be used throughout NEB rainfall season, even for the less wet months. To make full use of these levels of skill, the model forecasts should be made in standardized units and then converted to observed rainfall using the relationship between the observed standardized and the actual rainfall. In particular, a tendency to overestimate rainfall needs to be corrected.

Figures 7b and 7c directly compare simultaneous 3- and 6-month correlations between the model rainfall and the Hastenrath data for 1948–97 with those between the empirical regression simulations, and hindcasts as in Fig. 5b, and the Hastenrath data. Consistent with Fig. 7a, the model is clearly better in simulation mode than are the empirical simulations, with higher correlations through most of the year. In the dry season the empirical model is not strictly valid and gives near zero correlations. Figures 7b and c also show that HadAM2b is clearly better in simulation mode than the empirical method run in forecast mode as well.

Figure 8 gives a wider-scale picture of the skill of the model in simulating seasonal rainfall in March–May 1947–96 on the model grid scale. This period was chosen as 1997 rainfall data were not available for many areas. The model grid scale is used to show up local gradients, though a larger space scale would be used to maximize predictability for practical use (our NEB region has an effective resolution of 5° × 7.5°). NEB is probably the most skillfully simulated land region in March–May. High skill extends over the Nobre region (see Fig. 1 for definition), but falls off rapidly beyond this area, presumably due to the larger relative impact of internal atmospheric variations (Rowell 1998). Thus the eastern coastal region of Brazil from Cape San Roque southward has much lower skill, as does the southernmost part of the Nordeste. Interestingly, relatively high correlations are seen over parts of southern Africa and the southwest United States. Some regions where rainfall is most affected by El Niño (e.g., Ropelewski and Halpert 1987) do not show particularly high skill, as this is the season of generally lowest ENSO-related simulation skill because of the sometimes rapidly changing SST anomalies. But this is the season of highest ENSO impact on the tropical Atlantic (e.g., Sutton et al. 2000). So the high skill in NEB in March–May, the season of the “ENSO predictability barrier,” is all the more striking. Compared to the results for HadAM1 shown in PFJS, HadAM2b shows rather more skill in many parts of the smaller region shown here due to a combination of improved simulations and verification data.

c. Time series of simulated Hastenrath series for March–May

HadAM2b simulations for 1912–98 are plotted against the observed Hastenrath rainfall series in Fig. 9. The ensemble mean March–May rainfall has a highly significant correlation with the observed rainfall (r = 0.78), though the model rainfall has a wet bias and a lower standard deviation (1951–80 average = 521 mm, standard deviation = 116 mm) when compared to the Hastenrath series (1951–80 average = 444 mm, standard deviation = 155 mm). The correlation is almost as high in early years, r(1912–47) = 0.78, as it is in the more recent years, r(1948–98) = 0.81. The variance of the six ensemble members is only a little more than that of the ensemble mean. An analysis of variance test (Rowell et al. 1995; Rowell 1998) was carried out to determine the fraction of the total variance of the ensemble, which is common to all members. This common variance is explained by the only common factor for the six members, the forcing from SST variability, and is 76% of the total variance in model March–May rainfall over 1912–98. The remaining variance is due to random internal atmospheric variations. Since the model can never be expected to simulate this component of variability in phase with that observed, this limits the correlation skill that can be obtained when the model is compared with observations. The formula of Rowell (1998) shows here that this correlation skill limit would be 0.85. So we can conclude that the small shortfall from perfect skill in simulation mode seen in this region is rather more due to random internal atmospheric variations than to errors in responses to SST forcing. Model forecasts would be expected to show a lower skill since the precise patterns of SST anomalies are not known in advance.

AGCMs have sometimes been found to have highest skill during ENSO events (Graham et al. 2000; Brankovic and Palmer 2000), when the SST forcing imposed on the atmosphere may be particularly large relative to internal variations. To investigate this for NEB, the 87 March–May seasons 1912–98 were stratified according to quints of the March–May SOI calculated using the Trenberth (1984) method. Table 2 shows the correlation and LEPSOB skill for each category. Simulation skill assessed this way without standardization is highest when the SOI is in its middle quint and lowest when the SOI is in the lowest quint, the quint almost invariably associated with El Niño events. When standardization is carried out, this bias disappears. So it is important to standardize the model data. Otherwise the combination of wet bias in the model, and reduced standard deviation compared to the observations, results in poor simulations (and clearly forecasts) in strong El Niño conditions as these are quite often very dry.

Spectral analyses confirm the high skill of the AGCM simulations of March–May rainfall for 1912–98. Sperber and Hameed (1993) showed that March–April 1950–85 NEB rainfall was indeed largely phase locked to a combination of SST variations in the ENSO region in the tropical Pacific and to the tropical Atlantic north–south SST gradient. Both the simulated rainfall and the observed rainfall time series show spectral peaks on the ENSO timescale (around 3–5 yr) and around 11–15 yr. Figure 10 shows that the squared coherency between these series is above the 95% confidence level [using the method of Bloomfield (1976)] on timescales around 2 and 5 yr and 11–15 yr. Statistical simulations from the WF predictors produced in jackknife mode also show spectral peaks near 3–5 and around 11–15 yr. However, the statistical simulations overestimate variability at 11–15 yr and squared coherence at this timescale is lower than for the AGCM simulations, with less near 2 yr, though a significant peak is still seen on quasi-biennial timescales.

d. Correlation of model simulations with atmospheric circulation

A thorough analysis of the observed variations of NEB rainfall and atmospheric circulation can be found in Hastenrath and Greischar (1993a). Here we show a few averaged results from the ensemble of six HadAM2b runs. Figure 11a shows correlations between HadAM2b rainfall over the four Hastenrath grid points and 925-hPa atmospheric circulation. There are very strong local correlations between enhanced rainfall and enhanced westerly (r > 0.8) and northerly components (r > 0.6) of the wind flow. This translates into enhanced wind convergence (r > 0.6) and moisture convergence (r > 0.6) over the NEB coast, and on the southern side of the Atlantic ITCZ. There is enhanced divergence on the northern side of the ITCZ and reduced divergence (increased convergence) on the southern side, representing a southward shift in wet conditions. Over the tropical Pacific, moderately enhanced divergence can be seen (r > 0.4), especially over the central Pacific, reflecting the moderately positive correlation with La Niña widely noted above. It is noticeable that the relationships with local 925-hPa meridional wind and divergence are reversed locally just inland over NEB. The reason is not clear, though may indicate a heat low effect similar to that seen with variations of Sahel rainfall (Rowell et al. 1992).

Figure 11b shows comparable results at 200 hPa. The 200-hPa zonal wind over NEB is negatively correlated with rainfall, with easterly anomalies associated with more rainfall. In wet conditions, southerly winds overlie the surface northerly anomalies over and to the north of the north coast. Wet conditions produce increased 200-hPa divergence over NEB and over a more southerly position of the ITCZ in the tropical Atlantic with reduced divergence on the northern side. These results are generally consistent with previous research and show that the strong relationship between the model’s atmospheric circulation variations near NEB and modelled nordeast (Hastenrath grid points) rainfall are dynamically consistent with the strong model SST–rainfall relationships.

e. Hindcasts from the PROVOST experiments using persisted SST

A set of self-consistent dynamical model hindcasts is available from an extension to the European PROVOST project, also using HadAM2b. Ensembles of nine hindcasts were made for each season from 1982–93 starting from analysed initial atmospheric conditions (Graham et al. 2000). For the March–June hindcasts the observed February SST anomaly field [using the Reynolds and Smith (1994), optimally interpolated SST data] was persisted throughout the hindcast and added to a climatologically changing field. Figure 12 compares the ensemble mean hindcasts in the peak rainfall season, March–May, using standardized rainfall values based on the AGCM climatology for 1951–80 with observed NEB rainfall data. The latter is calculated as an observed average at the four Hastenrath region model grid points using the Hastenrath data. We show standardized hindcasts to remove model biases as discussed above. To put this period into context, the HadAM2b simulations from 1912–98 show that 1982–93 was a particularly predictable period, with a correlation of 0.95 with the observed data and a LEPS skill score of 0.63. This might explain the remarkable skill of the persisted SST hindcasts that had a correlation of 0.96 with the Hastenrath data. The corresponding LEPS skill is also exceptionally high at 0.80. Thus this period appears not to be typical and to overestimate long-term real-time forecast skill.

a. Skill of issued long lead and updated forecasts

When the forecast methods are combined, the issued best-estimate forecast is made in one of five rainfall categories each representing an equiprobable 20% (quint) of the cumulative probability distribution of the rainfall totals for 1951–80. Sometimes a forecast is made “on” a quint boundary, reflecting the judgment of the forecasters that the adjacent quints are about equally likely. The very dry quint is referred to as quint 1 (Q1), the dry quint as quint 2 etc., and the very wet quint is quint 5. The observed Nobre data have been refined compared to those described in Carson (1998) and Ropelewski and Folland (1999). The Nobre index was originally updated from 1981 using data for the (smaller) Hastenrath area for the months February–May. Our analyses are now based on the Hulme (1994) data for the complete Nobre area to 1994, but are still based on February–May Hastenrath data thereafter. This is because of lack of recent data in the western part of the Nobre region (see Fig. 1).

Figures 13a–c show the performance of the real-time forecasts in the form of forecast quints plotted against those observed, and are slightly modified from figures in Carson (1998) and Ropelewski and Folland (1999). The modifications included the correction of errors found in our records of the preliminary forecasts for 1988 and 1990. A reassessment of the quint boundaries has also changed the observed quint for FQ in 1996 from quint 5 to quint 4. Because of strong disagreement between the statistical (dry or very dry) and three AGCM forecasts (wet or very wet) for the preliminary forecast in 1997, a nil best estimate forecast was issued. In Carson (1998) and Ropelewski and Folland (1999) linear regression forecasts are attributed to the issued forecast. Here, we have accepted that an overall nil forecast was issued, but have used the real-time long lead statistical and dynamical forecasts for 1997 when individually assessing them. Table 3 shows the correlation between observed and forecast quints, LEPSCAT skill, bias, and rmse (last two measured in quints) for the six sets of real-time forecasts.

Figure 13d shows a tentative assessment of the potential predictability from the real-time AGCM forecasts alone. Real-time AGCM predictions were made of March–May mean rainfall for the four right-hand grid boxes in Fig. 1 from 1994 onward. The categorical forecasts shown in Fig. 13d were not part of the original issued forecasts, but were calculated later and corrected for the observed bias in the mean and standard deviation of simulated rainfall over the climatological period, 1951–80. At the time the forecasts were issued, the AGCM predictions were only available as anomalies relative to a model climatology that was generally appreciably different from the 1951–80 climatology used for the statistical forecasts. Five years is too short a period to provide statistically reliable skill estimates. Figure 13d shows that the real-time performance of the AGCM has been mixed, with modest skill for the preliminary forecasts, but much better results for the updated forecasts, comparable in skill to the simulations.

The number of issued forecasts varies between 10 and 12, depending on area and the timing of the forecast. Some of the variation in skill between the areas is random. We have summarized the skill using the overall correlation and LEPS scores for (a) long lead forecasts (preliminary FQ and preliminary Hastenrath) and (b) nowcasts (the remaining forecasts). The 21 long lead forecasts defined this way have a correlation of 0.59 between the observed and forecast quints and a LEPSCAT skill of 47%. The 44 nowcasts have a correlation of 0.72 and a LEPSCAT skill of 48%. This arguably indicates a clearly “useful” level of overall skill in both types of forecasts, and is thought to be comparable with the best skill obtained so far over about a decade in real-time seasonal forecasting. These assessments are also consistent with our assessments of the potential predictability of NEB rainy season rainfall.

We now discuss the distribution of forecast and observed quints in more detail for long lead forecasts and nowcasts. Where a forecast was made on a quint boundary, the forecast was allocated to categories on either side of the boundary, with a weight of 0.5 in each category. The full assessment tables are available from the authors. During the period, there was an observed bias to both very wet or very dry conditions compared to the 1951–80 reference period. Combining the three rainfall series together, remembering that only FQ contributes in 1987, the number of observed seasons in quints 1–5 respectively, were 10, 3, 7, 6, and 8, respectively (total 34); 6.8 observed seasons are expected by chance in each quint. This contrasts with Sahel rainfall (Folland et al. 1991; Ward et al. 1993), which has been drier than average in the majority of the seasons for which there have been real-time forecasts since 1986. For preliminary forecasts and updated forecasts together, the total number of forecasts in each category, and the number observed in brackets were Q1: 4.5 (20); Q2: 19.5 (6); Q3: 7.5 (11); Q4: 17.5 (12); Q5: 16 (16). Thus there was a clear tendency to issue too few forecasts of quint 1 (very dry), consistent with a nonsignificant but noticeable wet bias in the forecasts shown in Table 3.

PFJS show that, for a quint table, positive LEPSCAT scores are obtained in all cases for forecasting the correct quint and for making an error of one quint. Negative scores are obtained for errors of 2, 3, and 4 quints. We have analyzed the number of forecasts having errors of 0, 1, 2, 3, and 4 quints compared to their chance number (in brackets), allowing for the numbers of observations. For the 21 truly long lead forecasts, these are 0 quint error 7 (4.2); 1 quint error 10 (6.4); 2 quints error 1.5 (4.6); 3 quints error 2.5 (3.8); and 4 quints error 0 (2.0). In all cases where LEPS scores are positive, the number of forecasts exceeds their chance number and where they are negative, they are less than the chance number. For the 44 nowcasts, comparable statistics are 0 quint error:15.5 (8.8); 1 quint error 22 (12.4); 2 quints error 5 (10.6); 3 quints error 1.5 (7.0); and 4 quints error 0 (5.2). This confirms that the nowcasts have a similar behavior, with more skill than the long lead forecasts when measured in this simple way.

Summarizing, we note that for long lead and nowcasts together, when the expected number of quint errors from 0 to 4 is compared to their chance values, a chi-squared value of 32.7 is obtained. With four degrees of freedom, this is significant far beyond the 99.99% confidence level. The number of independent long lead forecasts is too few to carry out a chi-squared calculation realistically.

b. Performance of real-time regression forecasts

Issued real-time regression forecasts for the three NEB regions have been based on SST averaged over periods preceding the forecast period that varied from one year to another. For the preliminary real-time forecasts, SST was averaged over all or part of the preceding November–January. For the updated forecasts, February was often used but sometimes December, January, and February SST were averaged to provide a better estimate of preseason SST. As described by WF, multiple regression forecasts only started in 1989, so these assessments are only for 1989–98.

Assessments of the real-time regression forecasts are shown in Table 4. Real-time forecast skill is compared against the skill of a set of inflated and noninflated regression hindcasts for 1912–86. We compare hindcasts that are always based on January SST with the preliminary forecasts, and hindcasts always based on February SST with the updated forecasts. Mean LEPS scores are calculated for the best estimate forecast values using continuous and categorical versions of LEPS. The rmse and mean BIAS of the forecasts are expressed as a ratio of the forecast rmse and bias to the standard deviation of the observations.

We first note that differences in skill between preliminary and updated real-time forecasts are reasonably consistent between the skill measures so that the preliminary forecasts are consistently less skillful than the updated forecasts. The correlation skill implies that about 35% of the variance of the observations is explained by the preliminary forecasts, but near 50% by the updated forecasts (when the forecasts are expressed as a best estimate rainfall value). LEPSCONT, which also assesses forecast values is, as expected, systematically less than LEPSCAT, which assesses forecast categories that cover a range of values. Differences in skill between the regions might mainly be ascribed to chance, though on the grounds of areal size Fortaleza and Quixeramobim might in principle be least skillful, and the Nobre region most skilful. A hint of this behavior can be seen in Table 4 for the long hindcast period 1912–86, but is not clear for the decade of real-time forecasts. The bias is similar between regions and the two forecast periods; it is always positive and typically about 20% of observed standard deviations. Note that the bias in the hindcasts is zero, because the jackknife method forces the hindcast values to be equal to the observed values on average. So in this respect the jackknife method can give different results to forecasts in a period independent of the training period and slightly overestimate skill in a completely independent period. The rmse is higher (around 83% of the standard deviation of the observed) for the preliminary forecasts than for the updated forecasts (around 73% of the observed standard deviation), the equivalent respectively of 54 and 62 mm of rainfall over the season on average. Finally, the standard deviations of the forecast values are typically about 70% of the observations, and show no systematic difference between preliminary and updated forecasts.

The skill of the real-time forecasts can be compared to the potential predictability estimated by the noninflated and inflated regression hindcasts for 1912–96. Inflation does not affect correlation skill so this is the same as noninflated correlation. There is no great difference in correlation skill between preliminary forecasts (0.59) and preliminary hindcasts (0.66). For updated real-time forecasts, the mean correlation skill is 0.69 and for hindcasts it is 0.68. This similarity between real-time forecasts and hindcasts suggests that the period 1989–98 as a whole was less predictable than the remarkably predictable PROVOST period of 1982–93. Figure 13 confirms this pattern of behavior in the real-time forecasts; they were indeed very successful over 1989–93 and less successful afterward.

LEPSCONT and LEPSCAT can highlight the benefits to skill, if any, of inflated regression (not used in real-time forecasts in Table 4). LEPSCAT can also provide more appropriate assessments of the categorical form of the forecasts. Concentrating on LEPSCAT, the mean skill for preliminary real-time forecasts was 37% and for updated forecasts was 47%. For the hindcasts, the comparable values for noninflated regression are 56% and 62%. So in contrast to correlation, this suggests that the potential predictability for noninflated regression forecasts issued in quints was greater in the hindcast than during the real-time forecast period (which did not, however, include the successful 1987 and 1988 forecasts, which used discriminant analysis only). A little of this higher potential predictability may be due to the use of the most recent SST month to calculate the hindcast predictors, which Fig. 5 shows to be marginally the best strategy. However, the 1901–80 WF EOFs and the 1951–80 observed quint categories were evaluated over periods overlapping the hindcast period and might slightly exaggerate hindcast skill. The real-time forecasts were made, of course, for an independent period. However, we do not expect this to have more than a minor effect on skill as demonstrated by WF. Turning to the inflated regression hindcast results, the comparable skills are 61% and 63%. This suggests that there is a minor advantage in using inflated regression at the assessed levels of skill for categorical forecasts made in quints. Such a strategy is confirmed by the LEPSCONT skill that assesses the actual hindcast values. On average, LEPSCONT skill is distinctly greater for both preliminary and updated forecasts. For example, 36% for noninflated preliminary hindcasts and 43% for inflated preliminary hindcasts. In addition, preliminary hindcast rmse is much closer to updated hindcast rmse than in the real-time forecast assessments. As expected, rmse is higher for the inflated regression hindcasts than for the noninflated regression hindcasts. So it seems that the difference between preliminary and updated forecasts may be, in part, data dependent.

c. Performance of real-time linear discriminant probability forecasts

We now assess the real-time discriminant analysis probability forecasts. As with the regression forecasts, these were often chosen from several candidate forecasts using SST averaged over different preceding months, so the use of the SST period was not consistent between forecasts. We include use of the LEPSPROB assessment method described in WF. This was not included in PFJS as in principle this can be “played” but this is not considered a problem here. We also use the ranked probability score (Epstein 1969) in the form described by Daan (1984). Finally, we ask the question: if we had chosen the quint with highest forecast probability to be the categorical quint forecast, what would the LEPSCAT skill have been? This is a realistic question as this interpretation is one of those made when putting together the final forecast. The results are shown in Table 5.

The first two scores shown in Table 5 assess the whole probability forecast. Again the updated forecasts are more skillful than the preliminary forecasts. LEPSPROB shows a similar performance for all three regions but the RPS scores are somewhat lower. RPS scores are also very sensitive to high probability forecasts for the oppositely forecast extreme category to that observed. This occurred in 1994 and 1996 and this sensitivity adversely affected the RPS skill of the Nobre preliminary forecasts giving a skill score of 0.07.

An outstanding feature of Table 5, however, is the LEPSCAT skills. This has been used to assess the ability of the discriminant analysis method to pick out the most probable category that is regarded as the best estimate forecast. Used this way, discriminant analysis clearly achieved higher real-time skill than did the regression method in Table 4, which, as explained above, was used in “noninflated mode.” Viewed this way, the discriminant analysis forecasts have probably made the largest contribution to the overall forecast skill, though it must be kept in mind that only these forecasts were made for 1987 and 1988, both of which were very skillful. LEPSCAT gives a strikingly different result from RPS for the preliminary Nobre forecasts. This is because the quint forecast with highest probability was observed seven times from 11 preliminary Nobre forecasts. Furthermore, the seven correctly forecast quints were all extreme (very dry or very wet). This gave very high LEPSCAT score of 0.71. LEPSCAT of course has no sensitivity to high forecast probabilities of other categories as long as these were lower, so skill was not offset by this factor.

Besides its intrinsic ability to make probability forecasts, in principle discriminant analysis does not suffer from the reduction in variance suffered by noninflated regression or the somewhat arbitrary results than can occur when regression forecasts are inflated. We can enquire into the performance of discriminant analysis more closely by enquiring how well were the real-time discriminant analysis forecast calibrated against the observations. To achieve this, we follow the approach of Gilman (1986). Figure 14a shows a plot of the average forecast probabilities for a given quint, binned into increments of 0.25 in the cumulative probability distribution, against the observed frequency of the same quint. To provide sufficient results in each bin, the preliminary and updated forecasts for the three regions (FQ, Hastenrath and Nobre) are assessed together. Quints 2–4 are binned together because of the relatively low numbers of observations of some central quints. Over the period 1987–98, 24 discriminant analysis forecasts were issued for FQ, and 22 for both Hastenrath and Nobre making a total of 68 forecasts (including 1997 when an overall preliminary forecast was not issued).

Ideally the lines plotted for each type of quint forecast should follow the diagonal black line. This joins points for which the average binned forecast probability for a given quint category, and the fraction of such forecasts for which it was observed, are the same. These are the forecasts that are “perfectly calibrated.” Thus for forecasts of quint 1, the point at the bottom left-hand corner gives 0.03 as the observed probability of occurrence and 0.08 the forecast probability of occurrence. The value 0.08 represents the average forecast probability for all quint 1 category forecasts made with probabilities in the range 0–0.25. The observed probability represents the fact that for the 39 such forecasts, quint 1 was in fact observed only once. So for this bin, quint 1 forecasts were well calibrated. There are too few examples of forecast probabilities in the range 0.75–1.00 to plot the results for any quint category.

Averaged over all the forecast quints, calibration is good. However, low sample sizes may be the reason why the point for forecast quints 2–4 shows a rather poor calibration, though the other two points are well calibrated. For quint 1, there are too many observations of quint 1 for forecast probabilities of quint 1 averaging near 0.4. Instead of observing a 40% likelihood, we observe over 70%. For forecasts of quint 5, we observe too few quint 5 outcomes for forecast probabilities near 0.4. The other two points are well calibrated. Overall, forecasts of quint 1 are above the perfectly calibrated line; this implies that more observations of quint 1 actually occurred than the probability forecasts indicated. Similarly, there were fewer observations of quint 5 than the probability forecasts indicated. This accords with the overall moderate wet bias of the real-time forecasts, which results as much from discriminant analysis as from regression. The fact that the high and low probability forecasts for all quints are well calibrated strongly suggests that there is little tendency for discriminant analysis to produce exaggerated forecasts of high or low probability as once was feared. Thus linear discriminant analysis is a quite powerful statistical forecasting method when used wisely, quite apart from the fact that it is well tuned to the intrinsically probabilistic character of the season forecasting problem.

The sample size is too small to determine whether some of the poorer results in Fig. 14a represent genuine deficiencies of the discriminant analysis equations. So jackknife discriminant analysis hindcasts of the FQ, Hastenrath and Nobre series were produced for 1912–86 in the same way as for the regression hindcasts in Fig. 5. Preliminary hindcasts were based on January SST and updated hindcasts on February SST (Fig. 14b). It can be seen that the hindcasts are well calibrated with forecast probability being close to observed probability. Good calibration does not of itself necessarily reflect good skill; it is a necessary but insufficient condition. However, the evidence from Figs. 14a and b indicates that the linear discriminant forecasting system has good skill; the scope for better calibration (as opposed to somewhat better skill) is small.

d. Failure of the 1998 preliminary (long lead) forecast

This forecast deserves a special mention as this real-time long lead forecast was the least successful, despite being influenced by the strongest mature El Niño of the twentieth century. The presumption that such a strong El Niño would result in a straightforward, skillful, forecast, was not borne out (e.g., Nicholls 1999). The observed rainfall was well into the very dry (quint 1) category, at around half the 1951–80 average value. This continues a pattern of extreme droughts observed concurrently with the strong El Niño events of 1958 and 1983. The long lead 1998 forecast (Colman et al. 1998) was made more difficult because of conflicting SST signals from the Atlantic and Pacific Oceans. The strong El Niño event favored a dry season but positive SST anomalies in the Atlantic between 0° and 20°S, and negative anomalies farther southwest, favored above average rainfall. The HadAM2b AGCM (in the same form used for PROVOST) was used to provide the dynamical forecast. This model had been used to produce real-time seasonal forecasts for NEB since 1996. For the updated forecast issued in March, the dynamical forecast was made using the more recent HadAM3 model, which has now replaced HadAM2b.

For the preliminary forecast, using SST up to January, linear regression predicted March–May NEB rainfall to be in the average or wet category. The discriminant analysis forecast was bimodal, however, with above chance probabilities for both dry (quint 2) and wet (quint 4) categories. The nine individual HadAM2b ensemble members all predicted above-average model rainfall, giving an average of 142% of the model climatology. Allowing for model bias, and the error in the model standard deviation over 1951–80, this converts to a quint 5 forecast. A best estimate forecast of above average rainfall was issued largely because of these dynamical forecasts. However, it was noted that HadAM2b predicted below average rainfall over a large area to the east and west of NEB, and also to the north over the ocean. By the time of the updated forecast, the SST signal was more clearly favoring dry conditions due to warming of SST in the north tropical Atlantic relative to average (a fairly common development in the last stages of an El Niño). The empirical forecasts became drier in response, the regression model favoring quint 2. Updated dynamical forecasts from a nine-member ensemble of the new HadAM3 model were very different from the preliminary HadAM2b forecasts. The HadAM3 ensemble mean forecast rainfall was 36% of the HadAM3 1951–80 climatology, firmly in the quit 1 category.

Forecasters were aware during the preliminary forecast of the possible sensitivity of NEB rainfall to strong ENSO events. Because of the small amount of evidence available (only two ENSO events in the last 50 yr were associated with extreme droughts), they decided it was better to believe the dynamical model forecast of very wet conditions. To test if the HadAM2b model was intrinsically at fault, or whether the observed SST changes between January and the forecast period played a strong role, HadAM2b was rerun with simultaneous SST for early 1998, extending the long runs of section 6. In contrast to the real-time forecast, this rerun forecast had no input from real atmospheric data prior to the rainfall season. However the use of simultaneous SST data decreased model rainfall over NEB in March–May only to about the 1951–80 average, though it was simulated to be considerably drier than the average over other parts of Brazil. This suggests that the SST changes between January and the forecast period would have made HadAM2b somewhat drier if they had been known, but by an insufficient amount to rescue the forecast. So the large and beneficial change in the dynamical forecast between the preliminary and the updated forecasts was partly related to the change of model. There are still no long-term assessments of the accuracy of HadAM3 during El Niño conditions in NEB. These are needed to determine if this result reflects a more general improvement during strong ENSOs, or whether internal atmospheric variability not related to SST played a bigger role in 1998 than is usual for NEB rainfall.