The original statistical evaluation of the South African hygroscopic flare seeding experiment described by Mather et al. focused on the amount of radar-estimated rain produced by randomly seeded and unseeded convective cloud complexes in 10-min time periods from −10 to +60 min with respect to their time of selection. Statistically significant differences in the quartile values in some of the 10-min time periods were reported, with indications that the response time to seeding increased with increasing cloud system size. It was found also that the rain mass of the seeded cloud complexes in the 10-min period prior to selection was greater than that of their unseeded counterparts; the statistical evaluation did not take this apparent “inadvertent bias” into account, however. The evaluation included all 5 yr of the experiment, although the design was changed after the third year to allow for seeding larger cloud systems with a larger number of flares per convective cloud system.
An independent statistical reevaluation of the South African cloud seeding experiment using hygroscopic flares is reported on here. Several interesting aspects of the results are found that require a physical explanation and, perhaps, a revision to Mather et al.’s seeding hypothesis; nothing was found that contradicts the claim that there is statistical evidence that hygroscopic flare seeding increased the rain mass from the South African convective cloud systems in the experiment, however.
Mather et al. (1997a) (hereinafter referred to as MTSF) tested a method of increasing precipitation from convective clouds by seeding them at cloud base with hygroscopic flares that produce small salt particles of about 0.5-μm mean diameter. According to their seeding hypothesis, the hygroscopic flare seeding at cloud base accelerates the growth of large hydrometeors in the treated clouds, which harvest more of the available supercooled water before it is expelled into the anvils of the cloud system, thereby increasing the efficiency of the rainfall process. MTSF reported the results of microphysical measurements, a randomized seeding experiment, and numerical condensation–coalescence calculations that they claim support the validity of their seeding hypothesis. At the same time, MTSF reported indications that the hygroscopic flare seeding had an effect upon the dynamics of the treated cloud systems—an effect not yet accounted for in their seeding hypothesis. The dynamic effects were associated with an observed lengthening of the treated cloud systems’ lifetimes that was attributed to a strengthening of the coupling of the updraft–downdraft cloud system propagation mechanism, a finding also reported by Bigg (1997).
The formal statistical evaluation of the South African hygroscopic flare seeding experiment focused on the amount of radar-estimated rain mass produced by randomly seeded and unseeded convective cloud complexes in 10-min time periods from −10 to +60 min with respect to their time of selection (Mather et al., 1996, 1997a). Assuming that radar-estimated rain-mass production is proportional to storm size, MTSF reported that the smaller cloud systems (first quartile of rain mass) responded to treatment first, the medium size cloud systems (second quartile of rain mass or median) a little later, and the largest cloud systems (third quartile of rain mass) last. MTSF claimed that most of the radar-estimated rain-mass increases came from the medium-to-large cloud systems, although there was little statistical support for the indicated increases from the large cloud systems (third-quartile results). The formal statistical evaluation did not include an analysis of the cumulative rain mass from decision time onward for seeded versus unseeded cloud systems.
MTSF stated that one of the advantages of evaluating the results in 10-min time periods was that it permitted an assessment of what they called “inadvertent bias” that might have occurred. Inadvertent bias was, in fact, suggested, in that the rain mass in the seeded cloud systems in the 10-min time period prior to selection was greater than it was in their unseeded counterparts; the statistical evaluation did not take this inadvertent bias into account, however, or explain why it was not taken into account. Note also that the evaluation included all 5 yr of the experiment, although the design was changed after the third year by lifting the limit on convective cloud system size and the limit on the number of flares per convective cloud system.
Despite the limited nature of the evaluations and the potential effect on the evaluations of the design change and the suggested inadvertent bias, MTSF stated the remarkable conclusion, “While we have shown that rainfall can be increased from individual storms. . . .” The purpose of this study is to determine whether this claim is, in fact, justified on statistical grounds. The data on radar-estimated rain mass at cloud base used in the evaluation by MTSF were used in this study, the data having been graciously provided by Steffens (1997, personal communication). These analyses will also cite the findings reported by Mather et al. (1997b), hereinafter referred to as MTS, because MTS contains an expanded report of the findings in MTSF.
Bigg (1997) also conducted an independent statistical assessment of the effects of seeding in the South African hygroscopic flare seeding experiment. Whereas the statistical evaluation of MTSF and the statistical evaluation reported herein were based on radar-estimated rain mass, Bigg’s statistical assessment was based on total storm mass (cloud water plus precipitation) in order to gain additional insight into the nature of the seeding effect. Bigg’s assessment confirmed that there was a strong, positive effect of seeding.
Method of analysis
Means analyses are based on the single ratio (SR) of means of the seeded-to-unseeded radar-estimated rain masses (Gabriel 1999b). Quartile analyses are based on the radar-estimated rain-mass differences between the seeded and unseeded first, second (median), and third quartile values, Q1, Q2, and Q3, respectively (Fletcher and Steffens 1996). The analyses of the potential effect of the inadvertent bias are based on the regression ratio (RR) (Gabriel 1999b). All three analyses use rerandomization tests (Gabriel and Feder 1969). The P values and confidence intervals associated with each of the statistical tests are calculated as appropriate. It is emphasized that the evaluations made in this study are exploratory analyses, and, therefore, the P values and confidence intervals are used to measure the strength of the suggested seed versus no-seed ratios and quartile differences and not as measures of statistical significance.
The original statistical evaluation of the South African hygroscopic flare seeding experiment focused on the amount of radar-estimated rain mass produced by randomly seeded and unseeded convective cloud complexes in 10-min time periods from −10 to +60 min with respect to their time of selection and used both quartile and means analyses. Although quartile analyses are presented in both MTSF and MTS, means analyses are presented in MTS only. The results of the current reevaluation are given in Table 1; P values less than or equal to 0.05 are shown in boldface. This convention will apply to all tables in this paper.
The results for each of the time periods given in Table 1 agree, as they should, with the results given in MTSF and MTS. They are the main results of the formal statistical analysis obtained by MTSF and MTS—the results upon which they based their statistical conclusions. These results by themselves provide impressive statistical evidence of the effectiveness of hygroscopic flare seeding in increasing rain from convective cloud systems in South Africa. They are presented again here because MTS is an internal report and is, therefore, not likely to be widely available. It also serves as a baseline for the discussions to follow. Note that the means analysis results for the time period −10 to 0 min are slightly different from those given in MTS, because, in this study, values of rain mass in this time interval are included only if there is simultaneously a valid value of rain mass for the time period 0–60 min. When all values of rain mass for this time period are included, they agree.
Analysis of cumulative rain mass
An evaluation of results in 10-min time periods may be very useful in checking the validity of any differences between seeded and unseeded storms for physical consistency as they evolve from one time period to the next and, perhaps, for examining consistency of effects with the seeding conceptual model; it cannot by itself reveal whether the seeded cloud systems have produced more rain over the lifetime than did their unseeded counterparts, however. It is conceivable that the seeding produced a change in the timing of the evolution of precipitation without changing the overall precipitation produced by the seeded clouds. The ability to examine this issue is limited because the available data include only rain mass totals for the first 60 min after decision time. MTF lead the reader to believe that the 60-min results are, however, a good index of what is to be expected over the lifetime of the storms, stating that “[t]his clearly shows that the seeded storms on average remained larger for a longer time, and the seeding effect probably persisted for longer than an hour after the seeding decision.” This assumption also is supported by MTSF’s exploratory analysis that includes a figure (their Fig. 12) that presents radar-measured rain masses accumulated from decision time onward that are greater than those accumulated during the first 60 min after decision time.
In this reevaluation, the proportional effect of seeding (SR − 1) for radar-estimated rain-mass accumulations over time periods from 0–10 to 0–60 min, increasing by 10-min intervals, is calculated along with one-sided P values and 90% confidence intervals. MTS did provide the single ratio of seeded radar-estimated rain mass to unseeded radar-estimated rain mass but only for the time period 10–60 min and without citing a P value or confidence interval for their result. MTS stated that they did not expect any effect of seeding during the first 10 min after decision time. Note that the result for the time period 10–60 min given in MTS is incorrect, the correct value being 0.67 (Steffens 1998, personal communication).
The current cumulative rain-mass analysis provides direct evidence of the effect of hygroscopic flare seeding on the rainfall from the randomly selected cloud systems. It can be seen from Table 2 that P values less than or equal to 0.05 are obtained for the proportional effect of seeding for rain-mass accumulations from 0–50 min onward. Statistical support for rain-mass accumulation differences in the quartile analysis is indicated only for the first quartile, although it may be that similar results are likely to occur for the second quartile, assuming the indicated trend in the results continues.
Effect of the suggested inadvertent bias
It can be seen from Table 1 that the value of SR − 1 for the time period −10 to 0 min was 0.36, suggesting the existence of inadvertent bias that might have affected the results. Gabriel (1999a) has indicated that calling this phenomenon inadvertent bias is a statistical misnomer; rather, it is the result of the randomization producing a “bad draw” during the experimental unit qualification process. Gabriel called it a “culprit covariate,” which might explain away an otherwise statistically significant result as a type-1 error. In this paper, it is called a “potentially compromising covariate.” An argument could be made that the evaluation need not be adjusted for this potentially compromising covariate because it was not statistically significant. On the other hand, one could argue that it is prudent to adjust the evaluation for any potential impact in order to make the result as unequivocal as possible. In cases where the design of the evaluation was constructed specifically to allow for an assessment of potentially compromising covariates, as was the case in MTSF and MTS, one probably is obliged to adjust the evaluation for any potential effects that are detected.
It was decided to examine the effect of this potentially compromising covariate on the evaluations to see how it affected the results. Before giving these results, it is pointed out that MTS were, in fact, concerned about such an effect, which they called inadvertent bias. Accordingly, they adjusted their means evaluation for the 10–60-min time period by using a double ratio (DR), that is, by dividing SR for the 10–60-min time-period mean by SR for the −10-to-0-min time-period mean. It reduced the proportional effect of seeding from 0.67 to 0.23, but, again, MTS did not provide either a P value or confidence interval for this result. In this study, RR was used instead of DR in adjusting for the effect of the potentially compromising covariate. The RR was used to take advantage of the correlation between the −10-to-0-min time-period rain mass and the rain mass in subsequent time periods to reduce the variance of the test statistic. Gabriel (1999b) has shown that RR is more precise and has a smaller variance than either DR or SR in a single-target experiment. Tables 3 and 4 give the results of the time-interval and cumulative rain-mass evaluations, respectively, with the effect of the potentially compromising covariate taken into account.
As expected, the adjustment for the effects of the potentially compromising covariate reduced the proportional effect of seeding, the values of RR − 1 being considerably lower than those of SR − 1. The P values of the results did not change much, however, because the variances of the test statistic also were reduced. The adjustment also has decreased the differences between the seed and no-seed values in the quartile analyses, but, despite that, the adjustment has strengthened and sharpened the results. It can be seen that the results for the first and second quartiles (small and medium storms) have become stronger, with more of the time categories having P values less than 0.05, and the results for the third quartile (large storms) have become weaker.
Effect of the design change
MTSF reported that the design was changed after the third year by lifting the limit on convective cloud system size and the limit on the number of flares per convective cloud system; the evaluation, however, included all 5 yr of the experiment. To estimate the effect of this design change, evaluations were done on the data for the first 3 yr of the experiment and the last 2 yr of the experiment separately and were compared with the results of the entire 5-yr experiment (see Table 5). Note that there was virtually no evidence that the randomization produced a bad draw during the experimental unit qualification process during the first 3 yr of the experiment, the SR for radar-estimated rain masses in the 10 min prior to decision time being 0.97. The SR for radar-estimated rain masses in the 10 min prior to decision time during the last 2 yr of the experiment was 3.16, however.
It can seen from Table 5 that the results for the first 3 yr of the experiment are essentially the same as that for the entire 5-yr experiment. The results for the last 2 yr of the experiment tend in the same direction but are statistically weaker. This result may be due only to the smaller sample during the last 2 yr of the experiment, that is, 23 in comparison with 95 for the first 3 yr. There was no indication that larger cloud systems, which were made possible by the design change, affected the results during the last 2 yr of the experiment any more than they had during the first 3 yr. Note that the average rain mass in each data subset and the entire dataset was considerably larger than its median value, indicating that the distribution of the rain masses was skewed to the right; therefore, the quartile evaluations (especially Q2) may be more appropriate than the means evaluation. It appears that the effects of seeding on the first and second quartile (presumably small and medium storms) have the strongest P-value support.
Bethlehem versus Carolina
The experiment included two research areas that are separated by about 300 km (Carolina and Bethlehem), and the evaluation pooled the experimental units obtained from both areas. Table 6 gives the results of the evaluation for the Carolina (48 experimental units) and Bethlehem (70 experimental units) cloud systems separately and compares them with the evaluation of the pooled sample. Because the proportional effect of seeding (SR − 1) during the 10 min prior to selection was 0.49 and 0.23 for Carolina and Bethlehem, respectively, the evaluation included an adjustment for the effects of this potentially compromising covariate.
It would appear that the Bethlehem cloud systems are responding much more favorably to seeding than are the Carolina cloud systems, showing a pattern of results very similar to those of the pooled sample. Here again, the suggested effects on the first and second quartile (small and medium storms) appear to have P-value support, but those for the third quartile (large storms) do not.
SMALL versus LARGE storms
MTSF conducted an exploratory analysis on a subset of the experimental units; large cloud systems were rejected by limiting the analysis to those cloud systems whose volumes at decision time did not exceed 750 km3. For this subset, it was reported that the seeded cloud systems lasted longer than the unseeded cloud systems and that the overall seeded distribution of rain mass differed from that of unseeded cloud systems by a significant amount. In view of the results of this exploratory analysis, a separate evaluation was done on the experimental units whose volumes did not exceed 750 km3 at decision time and on those that did, that is, “SMALL” and “LARGE” storms, respectively. Capital letters are used to differentiate the storms classified by volume at selection as SMALL and LARGE from the storms inferred to be small, medium, and large in quartile analyses described in the previous exploratory analyses.
The LARGE storms consisted of 31 experimental units, or about 26% of the total number in the 0–60-min cumulative rain-mass evaluation. The LARGE storms did not favor either seed or no seed, either Carolina or Bethlehem, or either the first 3 yr or the last 2 yr of the experiment. Of the 31 LARGE storms, 18 were seeded (about 31% of the total seeded) and 13 were not seeded (about 23% of the total not seeded). In addition, 13 were Carolina storms (about 28% of the total Carolina storms) and 18 were Bethlehem storms (about 26% of the total Bethlehem storms). During the first 3 yr of the experiment, 24 LARGE storms were selected (about 25% of the 3-yr total) whereas during the last 2 yr of the experiment, seven LARGE storms were selected (about 30% of the 2-yr total). It is interesting to note that, because the LARGE storms make up about 26% of the total number of convective storms selected, they essentially constitute the third quartile (large storms) in the quartile analyses described above, and the small storms constitute the first and second quartiles (small and medium storms).
Because the proportional effect of seeding (SR − 1) during the 10 min prior to selection was 0.08 and 0.49 for the SMALL and LARGE storms, respectively, the evaluation included an adjustment for the effects of this potentially compromising covariate. The proportional effect of seeding (RR − 1) for the SMALL and LARGE storms based on accumulated rain mass as a function of time is shown in Fig. 1; the proportional effect of seeding (RR − 1) for all the storms is included for reference.
It can be seen from Fig. 1 that hygroscopic flare seeding has no effect on the rain-mass production of LARGE storms. On the other hand, the rain production of convective storms that are smaller is significantly increased by seeding. The differences in accumulated rain mass between convective storms that are seeded and not seeded attain P values equal to and less than 0.05 by 30 min after selection. These results confirm the suggestion of the analyses described above, namely, that the small (first quartile) and medium (second quartile) size storms are affected by seeding, but the large (third quartile) storms are not.
An independent statistical reevaluation of the South African cloud seeding experiment that used hygroscopic flares was made. Several interesting aspects of the results were found that require a physical explanation and, perhaps, a revision to MTSF’s seeding hypothesis; nothing was found that contradicts the claim that there is statistical evidence that hygroscopic flare seeding increased the rain mass from the South African convective cloud systems in the experiment, however.
Specifically, the following was found:
The evaluation of cumulative rain mass as a function of time is consistent with MTSF’s findings from their time-interval quartile analysis.
The adjustment for the possible bad draw produced by the randomization during the experimental unit qualification process, referred to by MTSF as inadvertent bias and by this author as a potentially compromising covariate, strengthened and sharpened the suggested effects of seeding rather than diluting them. This adjustment was made by using the regression ratio to take advantage of the correlation between the −10-to-0-min time-period rain mass and the rain mass in subsequent time periods, which reduces the variance of the test statistic.
There was no indication that larger cloud systems, which were made possible by the design change, affected the results during the last 2 yr of the experiment any more than they had during the first 3 yr.
Bethlehem cloud systems responded much more favorably to seeding than Carolina cloud systems did. Hygroscopic flare seeding may not have been as effective on Carolina cloud systems because they may be more maritime in nature than Bethlehem cloud systems are, because of their closer proximity to the Indian Ocean; this explanation, however, must be considered a matter of speculation in the absence of confirming data.
Hygroscopic flare seeding has no apparent affect on cloud systems whose volumes at the time of selection exceed 750 km3. Seeding apparently is too little and too late when applied to storms that already are so large at the time of selection.
It is important that a thorough physical evaluation of hygroscopic flare seeding be carried out to clarify and to validate the seeding hypothesis and, thereby, to determine the full practical potential and limitations of this seeding methodology.
Thanks to Drs. Deon Terblanche and Francois Steffens, principal scientist and statistician for the South African hygroscopic flare seeding experiment, respectively, for providing the data that made this reevaluation possible and for the stimulating exchange of ideas on the issues involved.
Corresponding author address: Dr. Bernard A. Silverman, 7038 E. Peakview Place, Englewood, CO 80111.