a. Raw data

Daily rain gauge data were made available by the National Meteorological Services Agency (NMSA) of Ethiopia and the Zambian Meteorology Service (ZMS) covering four rainy seasons—from June 1996 to September 1999 for Ethiopia and from October 1995 to April 1999 for Zambia. Quality-control checks were applied to eliminate stations with significant gaps in record and other data irregularities. This process left 20 reliable stations for Ethiopia and 35 for Zambia. The locations are shown in Fig. 1.

Daily CCD images covering eastern and southern Africa for the relevant periods were extracted from the TAMSAT satellite archive at the Department of Meteorology at the University of Reading. The CCD for a given pixel represents the number of hours for which that pixel has been colder than a specified threshold temperature T_t. CCD is readily computed from Meteosat TIR imagery and has been used for many years as a proxy for rainfall (Milford and Dugdale 1990; Grimes et al. 1999). In this case CCD images were extracted for T_t = −30°, −40°, −50°, and −60°C.

NWP model analysis fields for a time of 1800 UTC covering the region from 5°S to 23°N and from 20° to 50°E for Ethiopia and from 8° to 18°S and from 22° to 34°E for Zambia were obtained from the European Centre for Medium-Range Weather Forecasts (ECMWF) for the same time period (Simmons and Gibson 2000). The fields used in the estimation process were the two components of horizontal surface wind velocity together with vertical wind velocity at 400 and 700 hPa and surface relative humidity.

b. Data preparation

Algorithm calibration and validation require careful comparison between the algorithm estimates and rain gauge observations. Two problems arise here. The first is the sparseness of the rain gauge data (Fig. 1), and the second is the mismatch in spatial scale between the gauge data (point observations) and the satellite information, which are approximate areal averages over individual pixels. The pixel area is roughly 30 km² for Ethiopia and 35 km² for Zambia.

As in Grimes et al. (2003), these problems have been addressed first by applying the technique of block kriging (Journel and Huijbregts 1978) to the rain gauge data to give areal estimates for each satellite pixel, and second by restricting the calibration and validation procedures only to those pixels that contain at least one rain gauge. In this way the algorithms and ground data are compared at the same spatial scale and the effect of the sparse distribution of rain gauges is minimized by only carrying out comparisons for those pixels with the most reliable ground data. For simplicity, the kriged pixel value was set to zero if the associated gauge recorded zero rain. To avoid confusion, kriged values of pixels containing a rain gauge will be referred to as gauge-pixel rainfall in the remainder of the paper.

The importance of the block kriging step is seen in Fig. 2, which compares raw gauge observations with the kriged gauge-pixel estimates for the Ethiopian pixels containing gauges. In general, the pixel values are higher than the gauge observations for low rainfall and are lower for high rainfall. This is physically reasonable in that high gauge values are most likely to correspond to the most intense part of a storm being located directly over the gauge. The magnitude of the correction is due to the high spatial variability for tropical convective daily rainfall, even at a scale of kilometers (e.g., Flitcroft et al. 1989). For both Zambia and Ethiopia there is an approximately linear relationship between pixel average and gauge rainfall with a slope of 0.5 and an intercept of 3 mm.

The kriging procedure requires the identification of a representative variogram that adequately describes the spatial correlation of the rainfall field. Given the small number of gauges available here, there is some concern as to whether the variogram can be correctly calculated. To make best use of the available data, a daily climatological variogram (Bastin et al. 1984) was computed for each calendar month and each country using data from all four seasons. After removal of zero-rain values, typical numbers of data points going into the calculation of each variogram are then >10 000 for Ethiopia and >40 000 for Zambia. The experimental variograms were modeled using a standard exponential function with range parameters varying between 20 and 50 km. The nugget parameter varied between 0.6 and 0.7 for Ethiopia and between 0.4 and 0.6 for Zambia. These differences could reflect the small number of locations used to compute the variograms, or they could reflect genuine seasonal and geographical variability. However, this level of uncertainty in variogram parameters is not significant for this study because the purpose is only to calculate areal mean rainfall for each gauge pixel, which, by definition will always be within a few kilometers of a gauge. Sensitivity tests have shown that the observed variability in range makes a negligible difference to gauge-pixel values while the observed variability in nuggets serves only to make a small change in the slope of Fig. 2. Because this linear change applies to both calibration and validation data, it makes no difference to results quoted later in this paper.

The kriging procedure also allows the estimation of a standard uncertainty on the gauge-pixel values. The uncertainty for a given pixel depends on the distribution of nearby gauges and the spatial variance of the rainfall field. As a rough guide, the uncertainty was approximately ±30% of the estimated value.

a. General approach

The main aim of this work is to evaluate the TAMANN3 in two geographically distinct regions of Africa and to gain a better understanding of the factors that influence its performance. To assess the performance quantitatively, a cross-validation approach was adopted in which a proportion of rain gauge data was used for calibration and the remainder was held back as an independent validation set; this process was then repeated a number of times with a different subset withheld. The selection of withheld data was carried out in three different ways, as described in section 5. To assess the performance of TAMANN3 relative to other methods, the validation exercise was repeated for two other algorithms. These were a conventional, operational TAMCCD and an MLR approach, using the same inputs as TAMANN3. It is recognized that there are some formal difficulties with the application of the MLR to this dataset in that some of the input variables may not be strictly independent. However, the aim here is to test whether performance improvements of the neural network arise from the additional input information or from the nonlinear architecture of the network structure.

b. The TAMANN3 algorithm

The artificial neural network (ANN) used in this study is a modified version of the TAMANN2 algorithm described by Grimes et al. (2003). TAMANN2 consists of a four-layer feed-forward multiple-layer perceptron. The feed-forward perceptron has a simple architecture comprising a number of layers of network nodes. Nodes in each layer receive inputs only from the nodes in the preceding layer and pass their outputs only to the nodes of the next layer. Thus, for node j in the kth layer the input I_kj is a weighted sum of the outputs o_i of the nodes of the (k − 1)th layer:

View Expanded

where w_(k−1)ij is the weight connecting the output of the node i in the (k − 1)th layer to node j in the kth layer and N_(k−1) is the number of nodes in the (k − 1)th layer. A sigmoid threshold function is applied to I_kj, and this gives the output value of the jth node as o_kj = σ(I_kj), where σ represents the sigmoid function. The weights w_kij are optimized by comparison of the model output (in our case pixel rainfall) with observations (the kriged gauge-pixel values). A more detailed account of the basics of ANN theory can be found in textbooks such as that of Hecht-Nielsen (1990).

TAMANN2 consist of the following four layers: an input layer, two hidden layers, and an output layer. The output layer is a single node representing the target pixel daily rainfall value. The input layer contains the input data, which include target pixel altitude (to allow for the influence of topography on rainfall) and CCD values as well as numerical weather model information pertaining to large-scale weather patterns. To minimize input data requirements, the large-scale weather information is provided in the form of the amplitudes of the first five principal components of the relevant NWP analysis fields covering the region. The full input data parameter set then comprises the daily CCD values for the target pixel at temperature thresholds of −30°, −40°, −50°, and −60°C; the altitude of the target pixel; and amplitudes of the first five principal components of the following model fields calculated for 1800 UTC: surface relative humidity, 850-hPa zonal wind, 850-hPa meridional wind, 400-hPa vertical velocity, and 700-hPa vertical velocity.

With respect to the NWP model fields, 850-hPa winds and relative humidity are selected as indicators of low-level moisture transport; vertical velocities at 400 and 700 hPa are selected as the levels above the boundary layer that might be reasonably be expected to indicate the strength of convection. These two levels have been associated elsewhere with maxima of vertical velocity within the tropical atmosphere (Thompson et al. 1979).

The 30 parameters indicated above were selected from a larger set of trial parameters by an ANN pruning procedure (Weigend et al. 1992) as those that provided the most useful information for determining the required rainfall amount. Details of the selection of variables, the principal component analysis, and the pruning process are all given in Grimes et al. (2003).

The two hidden layers in the network contain respectively six and four nodes. The total number of weights to be identified in training TAMANN2 is 208. The training was carried out using a standard backpropagation algorithm (Lönnblad et al. 1992).

The main improvement of the TAMANN3 algorithm relative to TAMANN2 is in the normalization procedure for the gauge-pixel rainfall used to “train” the network. The network inputs and output are constrained to be in the range [0, 1] to ensure that the model has similar sensitivity to changes in all inputs. In the TAMANN2 version, all inputs were normalized according to

View Expanded

where x is the input value prior to normalization and x_n is the normalized input. Respectively, x_max and x_min are the maximum and minimum values of x. However, if we consider the distribution of daily rainfall for Ethiopia or Zambia we can see that it is highly skewed and is dominated by the large number of zero events (Fig. 3a). This has the effect of reducing discrimination for low rain rates. To avoid this problem, two adjustments were made. First, a proportion of the zero rainfall events was removed from the training process. Empirical trials showed that removal of 90% of zero rainfall observations (10% retention) gave a stable calibration. Second, the nonzero rainfall values were transformed to their empirical probability (Wilks 1995) according to the relation

View Expanded

where p is the measured gauge-pixel value, f (p) is the rainfall frequency distribution, p′ is the normalized value, and p_min and p_max are, respectively, minimum and maximum gauge-pixel values in the training dataset. The normalized nonzero rainfall values now lie in the range [>0, 1], as required, with a much more even distribution than that of the raw data (Fig. 3b). Zero rainfall events are still represented by zero in the normalized dataset. The final output of the ANN is a normalized value between 0 and 1. This approach is similar to the “histogram matching” technique used by others to deal with the highly skewed distribution of daily rainfall (Todd et al. 2001). Calibration of the TAMANN3 model proceeds by optimizing the weights w_kij to minimize the normalized residuals ε′², where

View Expanded

and p̂′ is the estimated normalized rainfall and the summation is over all calibration data. The estimated rainfall p̂ is then computed from p̂′ by means of the cumulative probability distribution in Eq. (3). It should be noted that a side effect of this procedure is that the output rainfall cannot exceed p_max.

As with the original TAMANN2 algorithm, the problem of modeling zero rain areas was addressed by setting to zero all areas with CCD = 0 for T_t = −30°C. The justification for this approach is discussed in Grimes et al. (2003).

c. The TAMCCD algorithm

The TAMCCD algorithm (Milford and Dugdale 1990) is a TIR-only technique and it assumes a simple linear relationship between CCD and rainfall:

View Expanded

where D is pixel CCD and a₀ and a₁ are empirical parameters to be determined.

The first step is to determine the value of T_t, which best distinguishes between rain and no rain. The second step is to determine the optimum values a₀ and a₁ by regressing gauge-pixel rainfall against CCD (see Grimes et al. 2003 for details). To represent intraseasonal changes in the CCD–rainfall relationship, a separate calibration is carried out for each calendar month. The TAMCCD algorithm is used in this work as a typical operational satellite-based approach with which to compare the TAMANN3 algorithm.

d. The MLR algorithm

The MLR algorithm computes an estimated rainfall p̂ by a linear transformation of the same parameters used as inputs to TAMANN3. Thus,

View Expanded

Once more, calibration is carried out by a least squares process, choosing the coefficients b_i to minimize the sum of unnormalized squared residuals defined by

View Expanded

For consistency with TAMANN3, rainfall was set to zero for CCD = 0 at T_t = −30°C.

a. Validation S

Validation S uses all 4 yr of data for calibration, with one station location withheld for validation. The experiment was repeated six times for Ethiopia and five times for Zambia, with a different validation station selected at random in each case. The number of repetitions was chosen as a compromise between the central processing unit (CPU) times needed for training that number of neural networks and the size of the validation sample. For TAMANN3 and MLR, each calibration and validation cycle uses at least 95% of the data for calibration and 5% for validation.

To carry out this procedure rigorously for the TAMCCD method for Ethiopia alone implies 24 separate calibrations (one for each of the 4 months for six datasets, leaving out one gauge location in each case). However, in practice the inclusion or exclusion of one station makes very little difference to the calibration parameters and therefore the calibration was carried out only once for each calendar month and each country using the whole dataset. The calibration parameters for Zambia are given in Grimes et al. (2003); calibration parameters for Ethiopia are summarized in Table 2.

b. Validation T

For validation T, the calibration was run 4 times. Each time, all of the data from a single year was withheld, giving four separate calibrations. This validation is much more data intensive than validation S, with only 75% of the data used for calibration and 25% for validation.

c. Validation ST

For the intermediate validation, 10% of the data were chosen randomly from the 4-yr dataset and were withheld for validation, with the remaining 90% being used for calibration.

d. Spatiotemporal resolution

The rainfall estimates from the three methods have been compared according to the validation protocols described above at the following three different spatiotemporal resolutions: daily rainfall estimates at a pixel resolution of about 30 km² (fine resolution); daily rainfall estimates spatially averaged over all validation pixels (intermediate resolution); and decadal (10 day) rainfall estimates averaged over all validation pixels and 10-day time intervals (coarse resolution). [This reduces the results to the typical time resolution used for agrometeorological applications (Thorne et al. 2001; Herman et al. 1997).]

e. Validation parameters

The TAMANN3, MLR, and TAMCCD estimates have been validated against the gauge-pixel data and the results are presented as scatterplots and statistical tables indicating bias, root-mean-square error (rmse), and correlation coefficient R. To assess the significance of the differences between these parameters, statistical tests were also performed on the residuals given by

View Expanded

where p̂_υ and p_υ are, respectively, modeled and observed gauge pixel values averaged over the appropriate number of pixels. Each of the three possible pairs of algorithms was tested for mean equality (Student’s t test) and variance equality on the residuals (F test) as defined in Eq. (8), thus allowing the significance of differences in bias and rmse to be determined. Because the distribution of residuals may be non-Gaussian, nonparametric tests were also carried out. These were the Wilcoxon rank sum test for equality of mean and the Ansari–Bradley test for equality of variance. In addition, a Z test was used to determine the significance of the differences of both parametric (Pearson) and nonparametric (Spearman) correlation coefficients. All tests were performed at a significance level of 0.05. The parametric and nonparametric significance test results were similar in all cases and, in the interest of brevity, only the parametric statistics and the results of the nonparametric tests have been tabulated.

a. Improvements in TAMANN3 relative to TAMANN2

The TAMANN2 and TAMANN3 algorithms were compared for the Zambian case study, according to the validation protocols described above. At all resolutions, the result was a significant reduction in the bias and rmse of TAMANN3 relative to TAMANN2. As an illustration, results are recorded in Table 7 (described later in the paper) for the ST validation.

b. Validation-S results

Figures 4 and 5 show the validation-S scatterplots of rainfall estimates against kriged gauge-pixel rainfall values for all the three algorithms at fine, intermediate, and coarse resolution for both Ethiopian and Zambian validation datasets. It can be seen that at each resolution the TAMANN3 method more closely approaches the one-to-one line. There is a tendency for all methods to underestimate high-rainfall amounts (>15 mm day⁻¹) but this is less marked for TAMANN3. Also, the MLR method generally overestimates low-rainfall amounts (<10 mm day⁻¹). The difference between the methods is most clearly seen at coarse resolution. Here, TAMANN3 shows a symmetrical distribution about the one-to-one line, whereas MLR tends to overestimate and TAMCCD underestimates, particularly for Ethiopia. This is important because the coarse resolution (averaged over 60 pixel days) is a realistic scale for agrometeorological or hydrological applications. The averaging process unfortunately reduces the maximum observed rainfall to <12 mm day⁻¹ so that it is not possible to compare the algorithms for high-rainfall amounts at this resolution. It is also notable that all algorithms perform better for Ethiopia than for Zambia. This is somewhat surprising because the complexity of the climate in Ethiopia might be expected to reduce the reliability of the rainfall estimates.

These observations are confirmed by the statistical indices in Table 3. Generally, TAMANN3 has the best results for all indicators. In particular, the very low bias for TAMANN3 at all resolutions should be noted. Table 4 shows that the differences in bias between TAMANN3 and the other methods are all significant at the 0.05 level. Differences in rmse are significant in all cases except for Zambia at coarse resolution. Differences between TAMANN3 and the others in terms of correlation coefficient are generally not significant for Zambia or for Ethiopia at coarse resolution. This is in agreement with Fig. 5c, which shows clearly that at coarse resolution, the degree of scatter and the linearity of all methods are similar, with the main difference being in the bias.

c. Validation-T results

The results for validation T at all three resolutions are shown in Fig. 6 and 7, and the statistical indices and significance test results are listed in Tables 5 and 6. This is the most stringent validation test because no contemporaneous data are included in the calibration and 25% of data are reserved for validation in each experiment. At the finescale (individual pixel days), there is a general worsening of the statistical indicators for all methods when compared with validation S, which is most evident for the TAMANN3 algorithm. At intermediate and coarse scales, the performance of all algorithms relative to validation S is generally worse in the case of Ethiopia but better in the case of Zambia. This may be because of the greater degree of spatial averaging at intermediate resolution in the Zambian case (average is over 35 gauge pixels for Zambia and 20 for Ethiopia). It may also be that the interannual variability in calibration parameters is less for Zambia, making the lack of contemporaneous data in the S calibration less important.

The statistical indices in Table 5 still show slightly better results for TAMANN3 except for the fine and intermediate resolution rmse and correlation for Zambia. However, Table 6 indicates that these differences are not significant at the 0.05 level, although the differences in bias are significant in all cases, reflecting, in particular, the low bias of TAMANN3. This is clearly seen for the coarse-resolution case for Ethiopia in Fig. 6, where again there is a tendency for TAMCCD and MLR, respectively, to underestimate and overestimate while the TAMANN3 data are evenly distributed about the one-to-one line.

The advantages of the ANN approach relative to the other two methods are less clear-cut under validation T than under validation S. It may be that the benefits of the neural network are more significant when contemporaneous rainfall data are available for the calibration. It could also be that the smaller calibration dataset reduces the generalization capability of the neural network. These points are considered in section 6e.

d. Validation-ST results

The relevant scatterplots for all of the resolutions for validation ST are shown in Figs. 8 and 9. The statistical indices and significance test results are listed in Tables 7 and 8. TAMANN3 scores highest for all indicators at all resolutions except (marginally) for the correlation for Zambia at coarse resolution. Again, the differences in bias are significant at the 0.05 level in all cases. For other indicators, the differences are generally less significant with decreasing resolution. This is borne out by inspection of Figs. 8 and 9, which generally show a smaller underestimation or overestimation for TAMANN3 than for the other two algorithms.

e. Comparison of validation protocols

In summary, the TAMANN3 algorithm shows a significant reduction in bias relative to the other algorithms in all cases. It also generally performs best for all validation procedures at all scales, although the significance of the differences in terms of residual variance and correlation is sometimes low, particularly at a coarse scale. The differences between the algorithms are most apparent for validation S and least apparent for validation T. This is illustrated in Fig. 10, which plots the rmse for all algorithms and validation protocols at fine resolution for Ethiopia.

It is not clear to what extent the greater superiority of TAMANN3 for validation S is due to the greater quantity of calibration data or to the existence of a greater proportion of contemporaneous calibration information. To address this question, validations S and ST were repeated with the same proportion of data (25%) withheld as for validation T. The results of these experiments (respectively S ′ and ST′) in terms of statistical indices are shown in Tables 9 and 10.

At fine resolution, the strongest effect occurs for TAMANN3 with indicators in S ′ and ST′ generally deteriorating relative to S and ST. This implies that the reduction in the quantity of calibration data is more critical for the neural network, although it should be noted that it still outperforms the other two algorithms, suggesting that the neural network makes better use of the available information. This is illustrated in Fig. 11, which repeats Fig. 10, substituting S ′ and ST′ for S and ST. It should also be noted that validation S ′ still gives better results than T does for all algorithms, confirming the importance of contemporaneous calibration data.

At intermediate and coarse resolutions, the situation is less clear. In comparing Tables 9 and 10 with Tables 3 and 7, it is seen that the bias is worse for TAMANN3 but the values of R and rmse generally improve as the proportion of validation data increases to 25%, presumably because of the greater degree of spatial averaging that goes into each plotted point.