Introduction

Weather data from automated weather stations have become an important component for prediction and decision making in agriculture and forestry. The data collected from such stations are used in predictions of insect and disease damage in crops, orchards, turfgrasses, and forests (Dinelli 1995); in deciding on crop-management actions such as irrigation (Parker and Zilberman 1994); in estimating the probability of occurrence of forest fires (Fujioka 1995); and in many other applications. Both networks of weather stations and individual stations accessed via modem currently are in use (Snyder et al. 1996).

The reliability of the automated weather stations has been improved greatly (Elwell and Klink 1993). Failures of weather stations still happen (Mott et al. 1994), however, and techniques are necessary to fill gaps in data sequences to use the data as input in agricultural models. Several approaches have been pursued by researchers. Mott et al. (1994) proposed developing regression equations from data obtained from neighboring weather stations. Ashraf et al. (1997) used geostatistics to interrelate data from several neighboring weather stations.

If the density of the weather stations is low, the use of neighboring stations becomes problematic because daily weather patterns often are strongly related to local landscape conditions (Blackie and Simpson 1993). Therefore, using local historical weather records may be a viable alternative to using neighboring stations in reconstructing missing data.

The reconstruction of missing weather data is different from weather forecasting because both data collected before the gap and data collected after the gap can be used. Still, the dependencies between the missing and available weather variables are expected to be complex, and advanced data analysis tools are needed to find and to express these dependencies with sufficient accuracy.

Group method of data handling (GMDH) is a term for the group of algorithms developed to simulate observed complex “input–output” dependencies. GMDH provides an automated selection of essential input variables and builds hierarchical polynomial regressions of necessary complexity (Farrow 1984). GMDH algorithms were applied successfully to predict seasonal precipitation (Lebow et al. 1984), monthly mean temperature (Lin et al. 1994; Setoodehnia et al. 1993), and daily minimum and maximum temperature (Abdel-Aal and Elhadidy 1994, 1995). GMDH algorithms are self organizing so that they do not presume any fixed functional dependence between inputs and outputs; this dependence evolves as the modeling with GMDH proceeds.

The objective of this paper was to explore the applicability of GMDH to reconstruct missing daily weather variables when preceding and subsequent daily weather variables were available. Historical weather records for a particular location in Mississippi were used to build and to test GMDH models. The location represents the region for which the so-called GLYCIM crop simulator is used on farms having weather stations (Reddy et al. 1995) at which gaps in data sometimes occur.

Materials and methods

GMDH algorithm

The idea of GMDH is to employ estimates of the output variable obtained from simple regression equations that include small subsets of input variables (Farrow 1984). Although the accuracy of such estimates is low, these estimates may be better predictors of the output variable than are some of the input variables. The best of these estimates are included in the set of input variables, and, again, small subsets of variables from this set are used to build new estimates.

The general functioning of the GMDH algorithm can be understood from the following example. Let the original data consist of the column of the observed values of y and of N columns of the observed values of the independent variables x₁, x₂, . . . , x_N. The primeval equations are quadratic polynomials:

zABx_jCx_kDx²_jEx²_kFx_jx_k(1)

Here A, B, C, D, E, and F are parameters, and x_j and x_k are two independent variables taken from the total list of independent variables. Each iteration consists of three steps. Step 1 consists of obtaining estimates of y using primeval equations. All independent variables x₁, x₂, . . . , x_N are taken two at a time to become x_j and x_k in Eq. (1) and regression polynomials [Eq. (1)] are constructed so that values of z best fit the dependent variable y. The total number of these polynomials is N(N − 1)/2. The resulting columns of z_m values, m = 1, 2, . . . , N(N − 1)/2, contain estimates of y from each polynomial and are interpreted as new improved variables that may have better predictive power than the original x₁, x₂, . . . , x_N. The objective of the next step is to keep only the best of these new variables.

Step 2 consists in screening out the least-effective new variables. There are several selection criteria (Farrow 1984), all based on mean square absolute or relative error of z_m with respect to measured y, and often including a correction that “punishes” a network for excessive complexity. In some GMDH versions, columns z_m are retained that have the criterion value smaller than a prescribed value. In other versions, a prescribed number of the best z_m are retained.

The list of input variables is modified at the end of this step. In some versions of the method, columns of x₁, x₂, . . . , x_N are merely replaced with the retained columns of z₁, z₂, . . . , z_K, where K is the total number of retained columns. In other versions, the best K retained columns are added to columns x₁, x₂, . . . , x_N to form a new set of input variables. The total number N of input variables changes to reflect the addition of z_m values or the replacement of old columns x_N with z_m.

Step 3 tests whether the set of equations can be improved further. The smallest value of the selection criterion obtained at this iteration is compared with the smallest value obtained for a previous iteration. If an improvement is achieved, steps 1 and 2 are repeated, otherwise the iterations stop, and the network is built.

We have worked with a version of the algorithm coded in the commercial software package ModelQuest (AbTech Corp., 1992–96). In this version, singlets, doublets, and triplets are used as primeval equations.

View Expanded

One, two, or three independent variables at a time are used to build the primeval equations. A stepwise regression is employed to diminish, if possible, both the number of coefficients and the number of independent variables in the primeval equations. The number of variables retained in the input list is limited to 15. The criterion of Barron (1984) is used to screen out the new variables and to stop the iterations when the complexity is balanced with the accuracy. The whole dataset is divided into training and evaluation datasets in a 3:1 ratio. The training dataset is used to build a hierarchical polynomial network and to estimate its accuracy, whereas the evaluation dataset is used to estimate the reliability of the network. Both the original input variables x₁, x₂, . . . , x_N and the output variable y are normalized to have a mean value of zero and a variance of one, and the normalized variables participate in network building. After the network is built, the results are denormalized to the original values.

An example of the GMDH network estimating daily minimum temperature at day 4 (Tmin4) is shown in Fig. 1. The original list of input variables included Rad1, Rad2, Rad3, Rad5, Rad6, Rad7, Tmax1, Tmax2, Tmax3, Tmax5, Tmax6, Tmax7, Tmin1, Tmin2, Tmin3, Tmin5, Tmin6, Tmin7, Prec1, Prec2, Prec3, Prec5, Prec6, Prec7, WndR1, WndR2, WndR3, WndR5, WndR6, WndR7, and IDAY for a total of 36 variables. All of them were normalized. Only three variables, Rad5, Tmin3, and Tmin5, were selected at the first iteration, and a new variable z₁ was retained. At the second iteration, Rad6 and Tmin2 were selected as additional variables that formed a primeval equation with the z₁ variable, and a new variable z₂ was retained. At the third iteration, Tmax3 and Wndr5 were selected as additional variables that formed a primeval equation with the z₂ variable, and a new variable z₃ was retained. Last, at the fourth iteration, WndR2 and WndR7 were selected to be coupled with the variable z₃, and new variable z₄ was retained. After the fourth iteration, the number of coefficients in the model was balanced with the model’s accuracy, and the iterations were stopped. The variable z₄ was denormalized to calculate Tmin4 in its original range. All primeval equations were cubic polynomials.

Performance of the GMDH method has been compared with “climatology” and persistence estimates. Here climatology is defined to mean that estimates of a weather variable for a particular day of a particular year were obtained as average values for the same day across all years in the database except for the year for which the estimate was sought. The persistence estimates of a weather variable were obtained as values of the same variable measured the day before the day for which the estimate was sought. A statistical comparison of accuracies of the methods was made using F ratios calculated as

View Expanded

where s²_GMDH, s²_Clim, and s²_Pers are variances of errors in estimates obtained with GMDH in validation datasets, climatology, and persistence, respectively. The F ratios were compared with F statistics F_N−1,M−1, where M is the number of data points in validation datasets, M = 360, and N is the total number of data points in the database, N = 1440. The F statistics with large numbers of degrees of freedom are not tabulated. We calculated F_1439,359, as described by Press et al. (1992, p. 222), using the BETAI subroutine published by these authors. Critical values of F_1439,359 at 0.01 and 0.001 significance levels were 1.33 and 1.15, respectively.

Results

The accuracy and reliability of the polynomial networks are shown in Fig. 2 and Table 1. The best results were obtained for minimum temperatures (Fig. 2). About 88% of variation in Tmin values could be explained using observed weather variables from 3 days before and 3 days after the day in question. Root-mean-square error of the estimates was about 1.3°C. The slope of the regressions of estimated minimum temperatures on measured ones was significantly less than 1. Hierarchical regressions built with GMDH tended to overestimate low values and underestimate high values of daily minimum temperature.

Data on daily maximum temperature and daily wind run shown in Fig. 2 demonstrate similar accuracy levels. The percentage of the variation explained by the GMDH hierarchical regression is about 80%. Root-mean-square error of the maximum temperature and wind run estimates is about 1.6°C and 13 km, respectively. High values are underestimated and low values are overestimated.

Estimates of daily radiation and precipitation had relatively low accuracy. The coefficient of determination (R²) was significant but small, in the range 0.2–0.3. Somewhat better results were obtained when we introduced an indicator variable IP. This variable was set to zero when there was no precipitation on day 4 and was set to one when any precipitation occurred on day 4. With this variable, about 50% of variation in the daily solar radiation and in precipitation could be explained using weather variables from previous and following days (Fig. 2). The evaluation dataset had values of R² near 0.3, significantly less than the value of 0.50 obtained for the training data set. Mean square errors were similar for training and evaluation datasets (Table 1).

Polynomial networks to estimate missing weather variables are shown in Fig. 3. All of the networks have four layers. Table 2 shows how the accuracy of the network progressed as more layers were added. The main improvement in estimation of Tmin4, Tmax4, and WndR4 occurred after the first triplet was found. Therefore, variables included in this triplet were expected to be the most influential predictors. Results from sensitivity analysis (not shown) concurred with this hypothesis. In all cases, the most influential predictors were values of the same weather variables measured the day before and the day after the day of missing datum. The third influential variable was the next-day solar radiation for the minimum temperature, the precipitation on the day before for the maximum temperature, and the next-day maximum temperature for the wind run. The radiation and precipitation estimates gradually improved as new layers were included in the network (Table 2), and no single variable except for the indicator variable could be distinguished as the most influential.

Statistics that show the accuracy of the GMDH estimates are compared in Table 1 with the same statistics for climatology and persistence estimates. The climatology estimates are much less accurate than the GMDH estimates in evaluation datasets. For example, root-mean-square errors of maximum temperature, minimum temperature, and wind run estimates are approximately 2 times larger in climatology estimates than in GMDH estimates. Differences in accuracy between climatology and GMDH estimates of daily radiation integrals and precipitation also are large. Correlation between measured and estimated-from-climatology weather variables is very weak. The ratios of variances of climatology and GMDH estimation errors exceed the critical value F of 1.33 for p < 0.001 (Table 1). Therefore, greater than 0.999 probability exists that the errors in climatology estimates have larger spread than do the errors in GMDH estimates.

Weather variables estimated from persistence are closer to the actual values than those estimated from climatology. All persistence estimates trail the GMDH estimates in accuracy, however. For temperature, radiation integral, and precipitation, the ratios of variances of persistence and GMDH estimation errors exceed the critical value F of 1.15 found for p < 0.01 (Table 1). Therefore, greater than 0.99 probability exists that the errors in persistence estimates have a larger spread than do the errors in GMDH estimates.

Table 3 shows changes in the estimation accuracy as related to number of days after the missing-data day. Comparing data in this table with data in Table 1, one can conclude that there is a very small loss in accuracy as the number of days after the missing-day data decreases from 3 to 1. The accuracy of estimates, however, becomes significantly worse when no data after the missing data day are used, that is, when the estimates represent a pure forecast. Using at least 1 day of data after the missing day is essential in filling gaps in weather data.

Results of estimating average weather variables for two and three missing days are shown in Table 4. The accuracy of the temperature, daily solar radiation, and daily precipitation estimates decreases as the length of the gap in data increases. By contrast, the accuracy of the wind-run estimates increases as the number of missing days increases. The deterioration of temperature estimates is not as significant as that for radiation and precipitation.

Discussion

The performance of the GMDH model developed to predict a missing daily weather variable depended on the variable. Best results were obtained for maximum and minimum temperatures and for wind run. Therefore, the best use of GMDH network estimates (with the inputs available for this study) can be made in models for which temperatures and wind run are the most important environmental inputs, as in, for example, insect population models. The lack of accuracy in precipitation estimates makes our GMDH networks not suitable to be used to make agricultural management decisions such as irrigation scheduling that are based on the amount of precipitation. This conclusion may be specific for both the region of study and the list of predictors, however. The successful use of self-learning algorithms to predict precipitation has been reported for conditions in Germany (Dumais and Young 1995).

GMDH models tended to overestimate observations with low values and to underestimate observations with high values. This result may be related to the fact that low-order polynomials are used as regression functions in elements of the networks. These functions are not suited to express rapid changes in output variables.

We compared our results with results of Abdel-Aal and Elhadidy (1995), who used GMDH for weather forecasts in a location in Saudi Arabia. Their mean absolute errors of minimum and maximum temperature were 0.79° and 0.97°C, respectively. These values were lower compared with our values of 0.96° and 1.18°C. The reason for this difference could be that Abdel-Aal and Elhadidy (1995) used 18 weather variables in their predictive models, whereas we worked with five. Another reason may be that the weather patterns are more stable in the Dhahran location studied by Abdel-Aal and Elhadidy than in the region around Stoneville, Mississippi. The accuracy of the minimum temperature estimates was higher than that of the maximum temperature estimates in both our and the Abdel-Aal and Elhadidy studies.

The day of year was included in the GMDH network to improve the daily solar radiation estimates and was only marginally important. This result concurs with the fact that the recovery of missing values of a weather variable was based mostly on the values of the weather variables the day before and the day after the day of interest. Use of the indicator variable showing presence or absence of rain is justified only if the user can determine whether the day with missing datum was rainy.

Both the coefficients and the structure of GMDH models may be valid only locally. Variables that are most important in estimating missing data may be the same for regions with similar climate, however. We plan to research this question in the near future. Although GMDH models gave much better estimates than climatology did in this study, the comparison with persistence estimates did not show decisive superiority of GMDH for all weather variables. For example, wind-run estimates from GMDH and from persistence had the same accuracy. GMDH should be tested in parallel with other forecast techniques when the estimation of missing data is needed.

In general, our results show that GMDH can be a useful tool in filling gaps in weather data from weather stations installed in the field. Selection of the predictor variables needs to be researched. The successful application of GMDH shows that historical weather data for a particular farm can be used to recover missing weather data important for on-farm decision support systems, including agricultural models.