a. Selection of cases and overpasses

The PIP-2 initially focused on 27 individual rainfall cases. Each case consists of one or more SSM/I overpasses during the course of its development. Thus each overpass contains an image of a precipitating weather system, while all overpasses associated with the same weather system constitute a case. The cases are distributed throughout a latitude range of 60°N–17°S, over continental and oceanic surfaces, for every season of the year, and from multiple years. Cases have been selected from each of the first three SSM/I satellite platforms (F8, F10, and F11) since the inaugural launch of the F8 platform in July 1987. The cases extend from July 1987 to February 1993. Table 1 identifies the 27 cases, including information on the names of the individuals who recommended the cases, case names, dates, satellite sources, background types, types of validation data associated with the cases, and total number of overpasses selected for the cases. Cases are numbered up to 28, but case 7 is not included since its data were unusable.

A total of 118 overpasses were selected for the initial analysis. Of these, 18 are completely oceanic, 18 are completely continental, and 82 are combined land–ocean. The rainfall targets for the 27 cases were carefully selected by the individual PIP-2 participants identified in Table 1. The target areas were generally intended to surround a storm or storm system (to the extent possible given the orbit swath coverage), including surrounding areas that contained either additional convective elements, stratiform sheets, or clear areas of interest. The targets are variable in size, with the number ofoverpasses selected for a case depending on a number of factors involving duration of the storm system, SSM/I coverage, and validation data availability. Table 2 provides details on all 118 overpasses indicating exact dates, times, locations, target sizes, data sources, and the availability and quality of the corresponding radar or rain gauge validation datasets.

The analysis was conducted in two stages: 1) an initial stage and 2) a reprocessing stage in which the algorithm participants resubmitted their results after a group analysis of the initial results held at a WetNet meeting in Durham, New Hampshire, in February 1995. During the initial analysis, a set of 56 of the highest quality overpasses were selected as the basis for the reprocessing. They are indicated by a “#” symbol in the overpass number column of Table 2. Of these, 7 are completely oceanic, 6 are completely land, and 43 are mixed land–ocean (25 of the original 27 cases were retained—no overpasses were retained for cases 3 and 15, each a single overpass case). A high quality overpass signifies that the SSM/I coverage of the precipitation system was relatively complete, that there were minimal bad data problems with the SSM/I pixels, and that the validation data (if available) were reliable over a portion of the SSM/I target. However, validation data coverage is always limited, invariably truncating important features of the rainfall targets.

b. Selection of validation data

For a selection of overpasses, the individual(s) who recommended the cases indicated in Table 1 also provided validation data processed into rain-rate maps. These datasets are mostly used to help consider how well algorithms differentiate between raining and nonraining areas, and for the raining areas, to help evaluate how the algorithms perform in retrieving both the average rain rate over the validation sectors and the pixel-to-pixel variability. However, as discussed, the validation data cannot be used for absolute calibration because of intrinsic uncertainties in representing the intensity of rainfall.

There are six codes used in Table 1 for the ground validation data: 1) Grnd Rad S (C) indicates S- or C-band ground weather radar; 2) A/C Rad indicates C-band aircraft radar from NOAA P3 aircraft; 3) CP-2 Rad indicates the National Center for Atmospheric Research’s dual polarization-dual frequency (S–X band) multiparameter radar; and 4), 5), and 6) LD, MD, and HD gauge indicate low-, medium-, and high-resolution rain gauge networks, where these descriptors denote gauge densities on the order of 1, 10, and 100 per degree square, respectively. A number of the radar validation datasets are from coastal radar installations, which are troublesome for satellite intercomparison in that satelliteretrieval algorithms often either cannot retrieve or retrieve only poorly over mixed land–ocean pixels. In addition, there are some validation datasets for which it was clear that the data quality was insufficient, plus a few situations where the validation data were so poor that they are not useful for evaluation purposes.

Not all overpasses had validation datasets available for analysis. Of the 27 cases, 23 had at least one overpass with a corresponding validation dataset. Of the 118 total overpasses, 83 had validation datasets. These included 57 radar datasets and 26 rain gauge datasets. Of these, 4 were oceanic, 18 were continental, and 61 were a combination of land and ocean. The requirements for the validation datasets were that they be mapped into the same pixel format as the algorithm results (i.e., the native SSM/I orbit swath format) and converted to the common rain-rate units used by the algorithms (i.e., mm h⁻¹). As will be discussed, a general problem with the validation datasets is that their areal coverages are almost always much smaller than the rainfall target areas selected by the investigators. It should be recognized that the satellite targets were not limited to areas covered by the ground measuring systems. This was by design since the statistics of the algorithm-to-algorithm intercomparisons are more robust when entire rain systems are considered, regardless of whether the algorithm-to-validation intercomparisons can only be done over partial targets.

Each pixel of a validation dataset is associated with a binary flag set to “reliable” or “unreliable,” according to the investigator who provided the dataset. For example, in a rain gauge–based validation dataset, the investigators defined all pixels whose footprint contained a gauge as reliable, whereas all other pixels generated from objective analysis over the grid mesh were set as unreliable. For a radar-based dataset, the investigators defined unreliable pixels at radar ranges where beam broadening and beam height were large. As it turned out for a few overpasses, a large number of the validation pixels were set as unreliable by the data providers. Moreover, it was found that some of the radar-based rain-rate estimates and rain gauge estimates were clearly beyond the fringe of reality, although repeated checks of the quality control and analysis procedures did not reveal the source of the problems. The clearly deficient validation datasets are marked with a “poor” designation in the data quality indicators in Table 2 (four overpasses were designated this way).

As seen in Table 2, a number of the validation datasets are judged “fair,” denoting that the coverage of the rain system captured in the SSM/I overpass was limited and did not capture the focal point of the SSM/I observed rainfall. There are well-understood reasons for these problems. For example, radars have limited range, all but high density rain gauge networks are too sparse to provide robust statistics, and well-distributed gauge networks are not available over the open ocean. In general, the radar-based products provided more uniform coverage than the gauge-based products and generally indicated reasonable appearing rain area coverages over the limited scan domains.

All of the validation pixels were used in the initial intercomparison analysis, including the unreliable pixels. For the reprocessing stage in which a subset of 56 of the best quality overpasses were selected, the associated validation datasets (when present) were made up mostly of reliable pixels in which the validation data coverage was assessed as fair or “good.” It should be emphasized here that the general quality of the validation datasets selected for PIP-2 is not below the data quality of radar–rain gauge data in general. In fact, the overall quality of the PIP-2 validation datasets is likely much higher than a randomly selected group of datasets of this size because of their careful selection and preparation by the investigators. The main issue is that developing a high quality validation dataset at high time and space resolution, concomitant with the type of coverage and data quality that is needed for exacting calibration of satellite algorithms, is a major undertaking. This has been made particularly clear by the ground validation project of TRMM.

The shortcomings with ground validation datasets motivate the use of composite algorithms as an additional means to evaluate algorithm performance through intercomparison. Composite algorithms are formed by assembling the results from a set of algorithms (e.g., according to the solution method, channel input, screening approach, or by simply taking all algorithms together) and assigning to each pixel position the mean or median of all the individual algorithm-derived rain rates for that pixel. A median value approach is sometimes useful over a straight averaging approach, since the latter produces nonzero values of rain rate for any pixel position in which just one algorithm may report rain, thus generating unrealistic rain cover statistics.

c. Rules of engagement

A total of 19 individuals or groups supplied results for 20 different algorithms to the intercomparison. The authors and names of the 20 algorithms are given in Table 3 (affiliations are found at the front of the paper). Table 4 provides a summary of the algorithms, including which SSM/I channels are used for screening and which are used for rain rate conversion, type descriptors of the algorithms according to five descriptor categories, and the principal references describing the algorithms. All 20 algorithms are applicable to ocean, whereas only 17 are applicable to land.

Detailed instructions were sent to all participants to ensure the algorithms were applied in a consistent fashion. The SSM/I datasets used for the analysis consisted of predominantly “Wentz type format” (Wentz 1993), although a few overpasses were derived from the NESDIS format; see Ritchie et al. (1998) for an explanation of the various SSM/I data sources. Each overpass contains an inner target array over which rain-rate results were calculated. These target arrays are at most 105 low-resolution scan lines in length and at most 64 low-resolution pixels wide. SSM/I files for all 118 overpasses were distributed to the different participants via a CD-ROM entitled “PIP-2 SSM/I Brightness Temperatures For Selected Cases.”² After receiving the CD-ROM and applying their algorithms, the participants transferred the algorithm-derived rain-rate results to FSU by ftp.

There were two submissions of results for the project. The first submission, called initial processing, was used to shake down the intercomparison procedures, identify data problems and obvious errors in the algorithms, and examine ways to improve the overall intercomparison analysis. The second submission called reprocessing, involved 56 of the original overpasses (as indicated in Table 2), and represented the final set of results used for the major analysis. A component of the final analysis process was a Final PIP-2 Results Participant Workshop held in June 1995 in Huntsville, Alabama.

d. Differentiation between screening and T_B–RR conversion

The term “screening” refers to the technique of assigning a given pixel a possible-rain or no-rain flag. Pixels flagged as no-rain are automatically assigned azero rain rate, whereas pixels flagged as possible-rain may or may not be assigned nonzero rain rate according to a given algorithm’s specific T_B–RR conversion scheme. The results of the initial processing indicated that various rain-rate screening techniques employed by the different algorithms contributed significantly to algorithm-to-algorithm rain-rate differences. For that reason, the reprocessing included a method of analyzing algorithm-derived rain rates in a manner so that the effects of the different screening techniques could be isolated from the effects of the different conversion techniques. This was accomplished by incorporating two sets of rain-rate results in the reprocessing effort. In the first “principal investigator” set, the algorithm authors used their own specific screens (PI screens), as was done in the initial processing. In the second “common screen” set, the authors used a standardized screen developed for this project based on the operational screen used at the National Environmental Satellite Data Information Service (NESDIS) fine-tuned with selected PIP-2 validation data. This scheme is described in detail in Ferraro et al. (1998). The common screen is applied using one of two approaches. In the first approach, the algorithm uses the subroutine for the common screen in place of the code for the algorithm’s original screen. In the second, the algorithm calculates its rain rate results as always, and then, in a back-end fashion, Boolean“ANDS” its result with the Common screen by zeroing out each pixel where the Common screen reports no rain. The first method is preferred since it allows an algorithm to calculate nonzero rainrates for pixels that the common screen reports as raining but the algorithm’s original screen reports as no-rain. When the back-end method is used, these pixels must remain zero. However, since not all algorithms could be modified to use the first approach, the back-end approach was applied in various cases.

Besides assigning pixels as rain or no-rain, the common screen also identifies bad data pixels and pixels over or near coastlines. These pixels are eliminated from the rain calculations, ensuring that the common screen results use standardized bad data and coastline filters. This procedure enables rain rates to be intercompared for a common set of raining pixels, regardless of residual errors in the common screen due to methodology and fine-tuning with validation data. Thus, differences are removed due to rain-area dissimilarities, such that the remaining discrepancies are only due to differences in the T_B–RR conversion methods.

The two-pass screening approach is useful in pinpointing the source of algorithm differences, as screening and conversion procedures manifest themselves in entirely different ways in the final rain rate maps and the accumulated intercomparison statistics. If a screening procedure allows nonraining pixels to be processed, then spurious rain rates will be retrieved. Over ocean, these will be low rain rates, whereas over land even high rain rates can be retrieved erroneously. If thescreening procedure is too aggressive, then some rain will be missed. Thus errors in the screening procedure can increase or decrease the retrieved rain totals. Similarly, the underlying rainfall retrieval can err in either direction. All combinations of these errors adding or compensating were observed among the PIP-2 algorithms.

e. Stratification procedures

Eight stratifications, briefly described above, are used in the intercomparison analysis to help synthesize the results. The first three stratifications involve discriminating between land and ocean, between screening and T_B–RR conversion, and between the PI and common screen results. The fourth stratification assigns the individual algorithms to four categories associated with their solution method. The four methods are as follows:1) statistical deterministic rain map techniques, which base the relationship between T_Bs (and/or T_B transform variables) and surface rain rates on regression, calibrated empirically with ground measurements; 2) quasi-physical deterministic rain map techniques, which base the relationship between T_Bs (and/or T_B transform variables) and surface rain rates on radiative transfer (RTE) modeling with respect to some type of specified cloud structure, but empirically calibrated with ground measurements; 3) physical deterministic rain map techniques along the same lines as category 2 but without empirical ground calibration; and 4) physical inversion profile techniques, which generate vertical distributions of rain rate by adjusting predefined hydrometeor profiles derived from either aircraft radar measurements or numerical cloud models until calculated T_Bs from a forward RTE model are consistent with measured T_Bs at different channel frequencies, without dependency on empirical ground calibration.

The fifth and sixth stratifications represent a reshuffling of the solution method approach, in which for the fifth stratification the algorithms are assigned to different categories according to the channel input used for the T_B–RR conversion. For land, two categories are used: 1) scattering and 2) mixed. For ocean, three categories are used: 1) emission, 2) scattering, and 3) mixed. In this nomenclature, emission algorithms use only 19 GHz or a combination of 19 GHz with 22 and/or 37 GHz in their conversion schemes, scattering algorithms use 37 or 85 GHz alone or in combination with each other in the conversion, and mixed algorithms involve both 19 and/or 22 GHz together with 85 GHz in the conversion (37 GHz may or may not be present). In the sixth stratification, the algorithms are assigned to one of three categories based on their screening approach. These categories are: 1) “heavy,” 2) “light,” and 3) intrinsic. These names denote screens that tend to be comprehensive, relatively simple, or intrinsic to the T_B–RR conversion scheme, respectively. Table 5 provides a summary of how the 20 algorithms wereclassified for the fourth, fifth, and sixth stratifications involving solution method, channel input, and screening approach.

The seventh stratification differentiates between types of raining systems, using four meteorological classifications for both land and ocean, with the first three classes being common to both categories. The land classes are 1) tropical cyclones, 2) midlatitude cyclones, 3) convective systems, and 4) squall line systems. Ocean classes 1–3 are the same; its fourth class is stratus systems. Table 6 provides the summary of how the 56 reprocessed overpasses were assigned to the meteorological categories (44 were assigned to land categories, 49 to ocean).

The eighth and final stratification for the analysis discriminates between low and high spatial resolution results. SSM/I imagery is composed of five low-resolution channels, 19V, 19H, 22V, 37V, 37H, and two high-resolution channels, 85V and 85H. Each SSM/I overpass consists of a series of scans called B-scans, each of which is 128 pixels wide. Every pixel includes measurements of the two high-resolution 85-GHz channels, with a ground distance of approximately 12.5 km between adjacent pixels and between successive scans.Each odd numbered pixel of every odd numbered scan also includes measurements of the five low-frequency channels (64 pixels per scan). These are the low-resolution pixels (A-scans). Thus, one out of every four pixels is low resolution; see CalVal (1989) for further details on the A/B scan formats.

The algorithm participants were asked to submit results at both resolutions if feasible. Of the 20 algorithms, 10 submitted both low and high resolution, whereas 9 were submitted at low resolution only. For one algorithm (GPROF), only high-resolution files were submitted. To enable the comparison of all algorithms at both resolutions, the results for low-resolution-only algorithms were bilinearly interpolated to a high-resolution grid mesh. For the GPROF results, averaging was used to create the necessary low-resolution files. In addition, all validation products were also submitted at both low and high resolution.

f. Intercomparison procedures

Six basic intercomparison procedures explained below are used to evaluate the retrieval results. The schematic diagram given in Fig. 1 is helpful in understanding the calculations described in procedures 2, 3, and 4.

a. Image analysis

The first salient point concerning the PIP-2 intercomparisons is that validation data coverage is rarely consistent with the spatial extent of the rainfall system target. This is highlighted in Fig. 2 with three overpass examples. The first is a land-based squall line over thecentral United States (case 2/overpass 1), the second a combined land–ocean convective system over the north coast of Australia in the vicinity of Darwin (case 12/overpass 5), and the third a west Pacific tropical storm (Tropical Cyclone Oliver) south of the TOGA COARE study area. The top row of panels shows the all-algorithm rain rate composites from SSM/I (based on median value compositing), while the bottom row of panels shows the validation data maps. The validation data are derived from ground weather radar systems located in Kansas City and Darwin, plus a P3 aircraft used during TOGA COARE. It is evident that radar coverage in each example presents a limited and truncated view of the system of interest. Each of these overpasses is used to emphasize a different weakness with radar-based ground validation intercomparison. In the first example, radar coverage is limited to the central portion of the squall line so that in not covering the northern and southern sectors, the radar does not sample the complete rainfall gradient intrinsic to an elongated multicell squall system. In the second example, since the radar is located on the coast with Melville Island positioned in its northwest sector, much of the coverage area involves coastal pixels insofar as the SSM/I measurements are concerned. Moreover, the radar only provides a truncated view of the convection field that has an intensity range evident in the SSM/I map exceedingthat which the radar has sampled. In the third example, the radar field of view not only truncates the coverage of the storm, it is limited to an outer band region whose rain-rate properties are unlike the focal point of the storm in the inner eyewall area where deep convection and intense rain rates are located.

The three examples presented here are similar to the other radar examples excepting cases 4, 5, and 6, which use radar validation data from the FRONTIERS radar network deployed within the British Isles. For those cases, the spatial sampling is greater, but at the same time much of the radar coverage is in the vicinity of coasts, in which intercomparison to SSM/I retrievals is inhibited by the mixed land–ocean pixel problem. Although examples involving rain gauge–based validation data are not shown, the problems are analogous. The limited scale high and medium density gauge networks (cases 25 and 26) truncate rain system coverage, while the spatially extended low density networks (cases 14, 15, 23, and 24) do not adequately sample the range of rain-rate intensities evident in the SSM/I-based composite analyses.

The above discussion is not intended to castigate ground radar or rain gauge systems, but to point out that there are intrinsic difficulties with using data from such systems to validate satellite rainfall algorithms at high-resolution–instantaneous space–timescales. Although the sampling limitations that PIP-2 experienced at the time the validation data were acquired have been partly mitigated by the installation of the upgraded Next Generation Weather Radar (NEXRAD) radar network, that system only provides coverage over the continental United States. The main point is that in conducting these intercomparisons, in the context of the SSM/I rainfall targets, in no case did a validation dataset provide representative coverage and sampling. Thus, it remains as a problem in statistics to determine if representative case mean intercomparison results can be obtained by accumulating the patchwork of limited-view validation scenes. It will be shown that the statistics derived from the validation data intercomparisons do not exhibit stationarity, unlike the more stable algorithm-to-algorithm intercomparison statistics. This leads to the conclusion, even overlooking known problems with accuracy and precision in the validation measurements, that in assessing high-resolution–instantaneous rain rates, differences between the algorithms are below the uncertainty limits inherent in the validation data. This is due to scale incompatibility between what ground systems “see” and what constitutes the rainfall targets as defined by a collection of scientific observers. Thus the validation data gathered for PIP-2 cannot be used in an objective sense to rank-order satellite algorithms in terms of accuracy and precision.

In Figs. 3a–c and 4a–c, examples of algorithm-derived rain-rate maps are shown for case 11/overpass 2 (winter nor’easter) and case 28/overpass 5 (west Pacific tropical storm–Tropical Cyclone Oliver). The first 20panels of each of these three-part figures show results from the individual algorithms, whereas the last seven panels show the composite algorithms based on solution method. There are two important features apparent in the individual results. First is the dispersion in the rain-area coverage pertaining to differences in how the various algorithms detect light rain. For example, in Fig. 3a compare the BRIS result to the MSFC result, which demonstrates that although both algorithms tend to agree on the focal point of the storm (in this case an occluding midlatitude winter cyclone), the BRIS algorithm generates an extensive light rain background, whereas the MSFC algorithm generates almost no light rain background. Since both these algorithms calculate the oceanic rain-rate magnitudes from scattering-type methods based on the 85-GHz channels (see Table 4 and the appendix), the rain-area coverage differences arise from differences in how the rain detection (screening) is carried out. Other examples of rain-area coverage differences are found throughout the individual algorithm panels in both Figs. 3 and 4.

The second major feature of the individual algorithm results are that there are differences in the maximum rain rates in the focal regions of the storms, bearing on how the different algorithms convert brightness temperatures to rain rates from different mixes of channel inputs, different calibration procedures, different microphysical underpinnings, and different radiative transfer assumptions. For example, in Fig. 4a, considering an intense tropical cyclone, compare the CALVAL result to the GSCAT result, particularly along the main north–south and east–west aligned feeder bands, as well as in and around the cyclone eyewall where the most intense rain rates occur. Whereas the GSCAT algorithm generates maximum rain rates exceeding 24 mm h⁻¹, the maximum values from CALVAL do not exceed 12 mm h⁻¹, or less than half the GSCAT maxima. This isrelated to a known weakness with the CALVAL algorithm, which is an empirically calibrated statistical algorithm whose radar training dataset did not contain a sufficient frequency of high rain rates. By the same token, as will be discussed in later sections, the GSCAT algorithm, whose rain rates are derived from the 85-GHz horizontally polarized T_Bs, is consistently on the high end of the scale vis-a-vis the ocean algorithm group composites. This, by itself, does not denote an accuracy problem with the GSCAT results since the composite results are only a relative measure to gauge the dispersion of the individual algorithms and not an absolute calibration reference. However, it does denote that maximum intensity differences are inherent to these intercomparisons, an issue of algorithm design that needs further examination.

It is also worthwhile to examine the different group composite panels, because there are differences in these panels that relate both to the method of solution and to the mix of screening procedures used for the individual algorithms within the different solution method categories. This is best described by the rain coverage differences evident between the statistical rain map composites (STAT RM maps in upper right-hand panels of Figs. 3c and 4c) and the quasi-physical rain map composites (Q-PHYS RM maps in left-hand middle panels of Figs. 3c and 4c). Note that although both of these solution methods are statistical in nature, because the former group of algorithms uses straightforward empirically formulated regression schemes applied to ground measurements and the latter group makes final empirical calibrations of physically formulated algorithms with ground measurements, there are differences in the rain-area coverages. This is independent of the fact that case 11 is a winter cold-core midlatitude cyclone and case 28 is a warm-core tropical cyclone. Part of this is because the regressions intrinsic to the STAT RM algorithms are designed to pass through zero rain rate at the T_B–RR conversion stage, whereas a number of the remaining algorithms do not produce continuous rain rates through zero. Note that when the STAT RM and Q-PHYS RM groups are combined (STAT), the associated rain area coverages agree well with the composite of the two physically based composites (PHYS).

The differences and similarities between composites have more to do with how specific features of screening methods used by the various algorithms in different groups tend to impose their individual signatures on the composites, differences that tend to disappear as the smaller groups are combined into larger groups and the individual screening impacts on rain-area coverages become more randomized in the composite maps. By contrast, unlike the individual algorithm results, it is not evident that there are systematic differences in the maximum rain rates within the different solution methods in going from case 11 to case 28 (also borne out by individual analysis of the remaining 54 reprocessed overpasses). Thus there is no qualitative evidence that physical methods are superior to empirical methods, the subject of discussion in Kidd et al. (1998).

b. Along-diagonal statistics

Quantitative summaries of algorithm performance are shown in Figs. 5a,b, illustrating the main along-diagonal statistical results for both ocean and land cases, each stratified according to whether PI or common screening has been applied. All four panels in the two figures illustrate four variables: (a) percentages of target pixels processed (%calc), (b) percentages of processed pixels detected as raining (%rain), (c) area-averaged rain rates over the processed target areas (average rain), and (d) rain-only area-averaged rainrates (rain only ave.). Results are presented for the 20 individual algorithms, the composite-all results, and the validation data results. The ocean intercomparison results given in Fig. 5a reveal six important features:

1. there are variations in the percentages of pixels thatindividual algorithms process (%calc), related to differences in how individual algorithms detect bad data and impose coastal masks;

2. there are variations in percentages of processed pixels detected as rain (%rain) by the individual algorithms operating with the PI screens, variations associated mainly with differences among the various screening techniques, and variations that diminish when the common screen is applied;

3. there is a factor of 5–6 difference between minimum and maximum averaged rain rates of individual algorithms (e.g., GSCAT vs CalVal), regardless of which screening procedure is used and regardless of whether all-target or rain-only area averaging is considered;

4. although some algorithms exhibit little change between their PI and common screen results insofar as averaged rain rates, other algorithms exhibit significant differences, highlighted by an approximate 3 to 1 increase in the rain-only area average for the composite-all result in going from PI to common screen (although the all-target area-average changes are small), again emphasizing the influence of screening differences among individual algorithms;

5. the ratios of rain-only area-averaged rain rates to all-target area-averaged rain rates vary from algorithm to algorithm, but there is a generally dramatic increase in the ratio in going from PI screening to common screening, manifested in the composite-all results in which the ratio goes from less than 2 to 1 for the PI screen results to some 5 to 1 for the common screen results, another facet of the underlying screening differences; and

6. for average rain rates, the individual algorithm results are nearly all biased high with respect to the validation results, but since the validation data coverages of the cumulative target area for the 50 ocean overpasses are less than 10% for both the PI and common screen results, these biases are not statistically significant.

Overlooking the differences, it is worth considering that there are underlying consistencies in the results, the most important being the general reduction in percentages of processed pixels assigned to rain in going from PI to common screening, the associated small changes in the all-target area averages resulting from changing the screening procedure combined with the generally large increases in rain-only area-averaged results after the screening procedure is altered. The explanation for these systematic changes is that there are generally significant differences in how individual algorithms detect light rain, differences that have major impact on the rain-area coverage and rain-only area-averaged rain rates but minor impact on total area-averaged rain rates. Otherwise, with the exception of a few of the algorithms (identified in the next section), the individual algorithms tend to preserve their relative relationships to one another in going from PI to common screening, and in how they would be ranked in going from all-target area-averaging to rain-only area-averaging. It remains to be determined why there is such a large range of variability between the individual algorithm’s averaged rain rates, regardless of the averaging area selected. It is evidently not due to the light rain end of the spectrum, otherwise the all-target area averages would exhibit more dramatic changes. This issue is considered in more detail in sections 4d and 4e in conjunction with the large rain-rate end of the spectrum.

For the land case (Fig. 5b), which had only one fewer overpass considered (49 instead of 50), most of the same features observed in the ocean results remain. Only 17 algorithms report rain since the BERG, RSS, and TAMU algorithms do not contain a land module (see Table 4). The most notable change in the intercomparison relationships is that the amount of validation data that survives after the bad data detection and coastal masking procedures of the common screen are applied represents only about half of the original validation pixels that intersect the targets. This is not surprising since the percentages of pixels processed for the individual algorithms undergo a systematic drop of some 30% after the common screen is employed, related to the conservative masking procedure employed for coastal pixels. However, similar to the ocean case, the percentages of pixels processed, regardless of whether the PI or common screen is used, are much smaller than even the smallest percentage value coming from the individual algorithms (in this case the OLSON algorithm). Thus, direct statistical comparisons between the along-diagonal algorithm and validation rain-rate averages are meaningful only to the extent that the validation sample represents the algorithm sample. The other notable difference between land and ocean intercomparison relationships is that the ratio between rain-only and all-target area-averaged rain rates increases from about 2 to 1 to about 8 to 1 in going from PI to common screening (it went from 2–1 to 5–1 for ocean), indicating that screening differences among land algorithms are even more significant in establishing the light rain threshold.

c. Off-diagonal statistics—Individual algorithm analysis

The off-diagonal intercomparison statistics are summarized in Tables 7a–d. Here it is important to recognize that the individual algorithm results are intercompared to both the validation results and the composite-all results, and that the portions of the target areas that are incorporated into the intercomparisons are those in which there is an intersection of valid pixels between an algorithm and its associated comparator variable. Obviously, given the dearth of validation data over the target areas, the composite-all intercomparisons are more statistically robust in terms of sample sizes than the validation data intercomparisons. Each of the fourtables contain two subtables. The upper subtables consider the intercomparisons to the validation data; the lower ones consider the composite-all intercomparisons. Tables 7a and 7b consider the PI and common screen results for the ocean case. Tables 7c and 7d consider the same results for the land case.

In a given subtable, the algorithms are identified under the name column (second column) and ordered (first column) according to their all-target area-averaged rain-rate magnitudes obtained from the along-diagonal analysis. For example, in the upper subtable in Table 7a, the algorithm assigned 1 is the IFA–SAP algorithm, which had indicated the lowest all-target area-averaged rainrate in the top panel of Fig. 5a (i.e., 0.25 mm h⁻¹). The algorithm in the 20th position for the upper subtable is GPROF, based on its along-diagonal all-target area-averaged rain rate of 1.34 mm h⁻¹. In the third column (immediately to the right of algorithm name) are the numerals of a second ordering based on the rain-only area averaging. If an algorithm’s position between the first and second ordering changes by more than four places, an asterisk is placed to the right of the name, thereby denoting that the algorithm is sensitive to this ordering procedure. Except for FER–AVE, the algorithms that are sensitive for the PI screened ocean results, that is, IFA-SAP, MSFC, FSU, and FER–AVE, tend to report the lowest percentages of raining pixels (lowest rain coverages), indicating the tightest screens and thus the greatest susceptibility to changes between all-target area averages to rain-only area averages.

In considering the common screen results, the same dual ordering procedure is used, but in this case (as evident in Table 7b), if an algorithm’s original position in the first ordering based on PI screen results changes by more than four positions in the first ordering of the common screen results, a “#” sign is placed to the right of the numeral in the first column. Note for the ocean case, only the IFA–SAP and BAUER algorithms are sensitive to the screening change. In fact, the IFA–SAP algorithm moves from position 1 for the PI screen results to position 20 for the common screen results, indicative of an algorithm whose PI screen is very tight, but whose T_B–RR conversion screen produces relatively large rain rates. In terms of sensitivity to the all-target versus rain-only ordering procedure, only the MSFC and OLSON algorithms change more than four positions between the two orderings for the common screen results (denoted by asterisks in Table 7b).

Following the name and position order columns are the actual intercomparison statistics. They include the mean rainrates in mm h⁻¹ for the algorithm pixels and associated validation or composite-all pixels lying within the intersection areas (Alg and Val or Com All), the bias, the bias-adjusted rms, the bias ratio (bias rat, given by the ratio of the bias to the validation or composite-all average), the means ratio (means rat, given by the ratio of the algorithm to the validation or composite-all average), the rank of means rat in terms of its absolutedifference from 1.0 (1.0 designates perfect agreement), the bias-adjusted rms ratio (rms rat, given by the ratio of the bias-adjusted rms to the validation or composite-all average), the rank of rms rat in terms of smallest to largest (0.0 designates perfect agreement), the correlation coefficient (CC), and the rank of CC in terms of largest to smallest (1.0 designates perfect agreement).

The last two columns provide a scoring system in which the rank values for the means ratio, bias-adjusted rms ratio, and correlation coefficient are added up yielding a final score (F Scr), with the final score ranked from smallest to largest and denoted by F Rnk. This scheme gives each algorithm an intercomparison performance factor relative to the other algorithms in terms of how its results intercompare to the comparator variates (i.e., the validation or composite-all results). This is the “winner–loser” selection procedure that has been adopted by the PIP-1 project, although it does not engender any type of absolute measure of accuracy or precision of the individual algorithms, since neither validation data nor composite-all results represent absolute calibration standards. However, the scoring procedure is helpful in quantifying an algorithms relative proximity to or distance from a comparator variable.

In considering the ocean case, and in comparing algorithms to the validation data for the PI screen results (upper subtable of Table 7a), all correlations are small lying between 0.0 (BERG) and 0.33 (RSS and FER–AVE), the means ratios extend from 1.1 (IFA-SAP) to 4.49 (TAMU), and the bias-adjusted rms ratios (similar to coefficient of variation) extend from 1.84 (OLSON) to 4.27 (KIDD). The five algorithms with highest scores consist of one physical profile (OLSON), one statistical rain map (CALVAL), and three physical rain map (GSFC, FER-AVE, LIU-CUR) designs. The five algorithms with lowest scores consist of one quasi-physical rain map (GSCAT), two statistical rain maps (BERG, KIDD), and two physical rain map (PETTY, TAMU) designs. Thus, there is no preference for a given solution method to stand out in terms of winners or losers.

The fact that the CALVAL algorithm exhibits a means ratio of 1.74 (with a rank of 5) reveals a serious problem with accepting the validation averages as a measure of ground truth. Note that it was established independently that the CALVAL algorithm underestimates rain rates greater than 5 mm h⁻¹ because its training dataset lacked an appropriate sample of intense rain rates (reinforced by the results in the upper subtable of Table 7b for the common screen intercomparisons in which the CALVAL algorithm indicates the smallest means ratio of all 20 algorithms). Therefore, it follows that the validation measurements for the ocean case underestimate the actual rain rates, assuming that the radar samples that were contained within the training dataset for the CALVAL algorithm were accurately calibrated. Given this, all algorithms with means ratios in the vicinity of 1.74 or less, even though they will score high on the basis of their means ratios, are underestimating the true rainfall.In terms of all algorithms with means ratios significantly greater than 1.74, nothing can be firmly established since there is no way of establishing the true bias of the validation data.

On the premise that the bias error in the validation data must amount to a scale factor between 1.74 and some larger value, a value of 2.0 is assumed to remain consistent with the characteristic bias between the AIP-3 radar validation data and the mean satellite results. Based on this assumption, the adjusted means ratios would then range from 0.55 to 2.25 with 19 of the algorithms indicating adjusted means ratios between 0.55 and 1.9, and 13 of 20 adjusted ratios exceeding 1.0. The significance of this exercise is that 19 of the 20 algorithms would then have bias uncertainties below that of the estimated uncertainty in the validation data, that is, below 100% either high or low.

In making these bias estimates, the adjusted means ratios are highly consistent with the means ratios of the set of 20 comparison algorithms participating in the AIP-3 project. Of these 20, 15 are equivalent to those used in PIP-2 with CALVAL, GPROF, MSFC, OLSON, and RSS from PIP-2 not represented in the AIP-3 set, while five were used in AIP-3 but not in PIP-2 (BA3, FE4, HA1, PR1, and SM2). In the AIP-3–PIP-2 vernacular, the 15 algorithm matchups are as follows: AD1–GSCAT, AO1–MRI, BA1–BRIS, BA2–KIDD, BE1–BERG, CH1–GSFC, FE1–NESDIS, FR1–FER–AVE, KM1–KUM, LI1–LIU–CUR, MZ1–IFA–SAP, PE1–PETTY, SC1–BAUER, SM1–FSU, and WI1–TAMU. [The nine AIP-3 algorithms excluded from the total set of 29 SSM/I entries consist of seven special case entries from multiple-entry groups (AO2, BA0, BA4, BA5, FE2, FE3, and PE2) that did not survive after AIP-3 and two (IA1, IA2) that were severe outliers because of now-known problems.] Using the “All Cruises Combined Dataset,” the AIP-3 means ratios range from 0.71 to 2.0, with 17 of 20 ratios exceeding 1.0 (see p. 50 of Ebert 1996). It should be emphasized in this discussion that the assumption that the CALVAL results can be used to show that the PIP-2 ocean radar validation data are low biased, is subject to debate.

From examination of the lower subtable of Table 7a, in which the PI screen ocean intercomparisons to composite-all results are given, it is evident that final rankings change, although some of the algorithms that scored high in the validation data intercomparisons continue to score high in the composite-all intercomparisons. However, in this case, their high scores are derived from top rankings in the bias-adjusted rms ratio and correlation coefficient columns (correlation coefficients now generally run high, ranging between 0.60 and 0.94 for all but two algorithms), whereas for the validation comparisons, the high scoring algorithms obtained their top rankings in means ratios and bias-adjusted rms ratios (while exhibiting mid to low correlation coefficient rankings). As expected, because all algorithms exhibit means ratios greater than 1.0 for the validation intercomparisons, the ones that ranked high in means ratios for the validation intercomparisons necessarily indicate mid to low rankings in means ratios for the composite-all case.

A more important facet of the lower subtable in Table 7a is that the set of means ratios ranges from approximately 0.45 to 1.65, with 13 of 20 algorithms indicating means ratios between ∼0.7 and 1.3 (note for AIP-3, 15of 20 indicate means ratios between ∼0.7 and 1.3). This would indicate that if the composite-all results for ocean are taken as truth, the bias uncertainties for many algorithms are within approximately ±30%, with PIP-2 precisions generally ranging from 35% to upward of 100%. Although the PIP-2 composite-all results do not represent truth, the distributions of both the PIP-2 composite-all and AIP-3 means ratios are consistent withthat of the PIP-2 adjusted means ratios (i.e., from the validation intercomparisons assuming the scale factor error of 2.0 applies).

Repeating this analysis for the Table 7b ocean common screen results indicates that with a couple of exceptions (BAUER and IFA–SAP), algorithms that scored high or low in the PI screen case continue to score high or low in the common screen case for both the validation and composite-all intercomparisons, although the exact rankings are not preserved (denoting another outcome of screening differences in algorithm intercomparison analysis). As before, the means ratios for the composite-all intercomparisons range betweenapproximately 0.45 and 1.65, with 11 of 20 algorithms exhibiting means ratios between ∼0.7 and 1.3. Also consistent with the PI screen results, using an estimated scale factor error of 2.0 for the validation data, are the adjusted means ratios ranging from 0.53 to 2.19 (compare to the 0.55–2.25 range from the PI screen results).

In examining the land results, it is evident that different sets of algorithms belong to the winners and losers groups. For the intercomparisons to validation data using the PI screen (upper subtable of Table 7c), the FER–AVE, NESDIS, MSFC, GPROF, and BRIS algorithms exhibit the highest final rankings, while the BAUER, KIDD, LIU–CUR, GSCAT, and GSFC algorithms exhibit the lowest. These groups are mostly unperturbed in going to the common screen results (upper subtable of Table 7d), although the GSCAT algorithm migrates from the winners to losers group and the IFA–SAP algorithm falls from a middle-ranked position to the bottom-ranked position. Moreover, the algorithms that scored highest in the validation intercomparisons, tend to score highest in the composite-all intercomparisons (lower subtables in Tables 7c,d). However, in contrast to the ocean case, where highest ranking algorithms achieved that performance by generally obtaining high individual rankings in two of the three ranking categories, the high-ranking land algorithms as a group do not outrank the other algorithms in the individual rankings.

Although a number of additional features noted in the ocean intercomparisons continue to hold for the land intercomparisons, there are various differences. The important ones are:

1. the ordering is volatile for the PI screen results when switching between all-target area averaging and rain-only area averaging (nine algorithms are marked with asterisks denoting position order changes greater than four), but completely stable for the common screen results (no algorithms change position by more than four places between the two orderings);

2. there is additional volatility in the ordering when switching between PI screen results and common screen results as four algorithms are marked with a“#” sign (only two for ocean); and

3. the distribution of means ratios associated with the PI screen validation data intercomparisons range from well below 1.0 (0.24) to around 2.0 (2.04), in contrast to the ocean distribution for which all unadjusted ratios were greater than 1.0, with the total range reducing slightly in moving to the common screen results (0.25–1.68).

These three results lead to two important conclusionsabout the land intercomparisons. The first is that they are more sensitive to screening differences, a feature consistent with the along-diagonal statistical analysis. The second is that there does not appear to be a systematic bias to the validation data, as found with the ocean results. In this context, the distributions of means ratios for the PI and common screen validation intercomparisons are consistent with the associated composite-all intercomparisons, as the range limits are nearly identical in both cases. Supporting the notion that this is not a fortuitous result, but an outcome of the fact that radar systems are generally better calibrated over land than ocean, is the fact that the CALVAL algorithm, which is the one algorithm with a known property of underestimating rain rate, exhibits the lowest validation means ratios—0.24 and 0.25 for PI and common screen results, respectively.

d. Off-diagonal statistics—Composite algorithm analysis

The off-diagonal intercomparison statistics for the composite algorithm analysis were prepared as four two-part tables. The first of these is presented in Tables 8a,b, containing the results for all reprocessed ocean overpasses (all rain systems) based on the PI screen and using low-resolution SSM/I measurements. Table 8a shows the intercomparisons between the validation data and the various algorithm composites [in terms of the 13 different statistical measures discussed in section 3f(4)]. The 11 columns correspond to the different composite algorithms that are used as comparators to the validation data. The first four columns (1–4) contain the solution method groupings, that is, physical rain map (PHY-RM), physical profile (PHY-PRO), the first twocategories combined (PHY), and the combined quasi-physical and statistical rain map categories (STAT). The next three columns (5–7) contain the channel input groupings, that is, emission (EMIS), scattering (SCAT), and mixed emission–scattering (MIXED). The eighth, ninth, and tenth columns contain the screening approach groupings, that is, heavy (HEAVY), light (LIGHT), and intrinsic (NONE). Finally, the 11th and last column contains the all-algorithm composite.

Table 8b shows the intercomparisons of different algorithm composites with the separate solution method, channel input, and screening approach stratifications (using the same 13 statistical variates). As indicated, the sample populations for the composite–composite intercomparisons consist of approximately an order of magnitude more pixels than the validation–composite intercomparisons. Results contained in the tables for the common screen ocean results and for both PI and common screen land results are discussed but not shown. The common screen produces improvements in the % difference, correlation, skill score, and slope factor statistics. Similar tables have been prepared for each of the different meteorological categories used for ocean and land, with the entire set of tables reproduced using high-resolution results. Results from these additional tables are also referred to in the discussion below but not shown. The main results gleaned from this analysis are discussed in the following four sections.

e. Difference histograms

To understand the distributions of rain rate differences with respect to either the validation data or the composite-all algorithm, rain rate difference histograms for the 20 individual algorithms are analyzed separately. An example of a set of these histograms for the FER–AVE algorithm is shown in Fig. 8. Eight panels are prepared for each algorithm, since the histograms consider both validation data (VAL) and the composite-all algorithm (ALL) as comparators, stratify according to ocean and land, and consider both PI and common screen results (PI and CS). The histograms only consider pixel pairs within the intersections of the reprocessed overpasses in which at least one of the rain rates from each pair exceeds 1 mm h⁻¹. In other words, light rain rates have been excluded from the analysis. This is done to preventthe relatively large number of small rain rates from dominating the statistics and to specifically focus on differences between the medium to large rain rates. Presented numerically below each histogram panel are the average values of the pixels from the specific algorithm and the comparator algorithm used to form the differences, as well as the bias, correlation coefficient (corr), and bias-adjusted rms factors between these two sets of pixels. Table 9 summarizes the analysis. The algorithms are grouped into three categories depending on whether the bias is large negative (<−1.5 mm h⁻¹), small to medium (between ±1.5 mm h⁻¹), or large positive (>+1.5 mm h⁻¹). The PI screen results are used to do the grouping with ocean and land categories considered separately. Footnotes indicate changes in grouping assignments that would occur if common screen results are used instead of PI screen results.

It is evident from Table 9 that 14 of 20 ocean algorithms exhibit a large positive bias with respect to the validation data, but only 4 of 17 land algorithms exhibit a large bias (positive or negative). Both results are consistent with what has been noted previously when all intersecting pixels are considered. For the composite-all results, only 5 of 20 ocean algorithms fall outside the small-medium grouping, while for land algorithms, only 4 of 17 fall outside the same grouping. This denotes a high degree of consistency between the algorithms in terms of bias.

Regardless of the general bias consistency between the individual algorithms, careful inspection of the entire set of histogram differences indicates significant rain-rate differences between individual pixels up to and exceeding 20 mm h⁻¹. This was alluded to in section 3a when considering various examples of individual overpass rain-rate maps and demonstrates the main source of rms difference between algorithms, that is,inconsistency at the large rain rate end of the scale. It is also evident from this set of figures that the distribution properties of difference histograms are variable, ranging from Gaussian, to skewed left or right, to multimodal. The kurtosis factors (third moments) are also highly variable, with some histograms exhibiting tight clustering near zero-difference and others exhibiting large spreads.

The foremost inference from the histogram analysis is that the major cause of algorithm differences in regards to T_B–RR conversion stems from discrepancies occurring at large rain rates. It is also clear that in a number of instances, these discrepancies would be missed if the intercomparisons only consider averages and the difference histograms are Gaussian with near-zero means. Therefore, the large rain rate spectra should be given closer attention in the future in seeking convergence among the algorithms and optimal designs for combined algorithms.

f. Fan maps

To illustrate and explain why algorithms behave differently and why the greatest differences among the algorithm composites occur when the channel inputstratification is used, we consider the transform relationships between T_Bs and RRs on a pixel by pixel basis. As described in section 3f, a fan map encompasses the T_B behavior of all pixels for a given algorithm and given overpass (stratified by land or ocean) whose rain rates fall within a specified rain rate range, regardless of which channels are used as input. The mapping consists of one bundle of lines connecting the 19-GHz unpolarized T_Bs (19Us) on the left-side abscissa to the corresponding 37Us on the left-hand ordinate, and another bundle of lines connecting the 19Us on the right-side abscissa (mirror image of left abscissa) to the corresponding 85Us on the right-hand ordinate. Seven rain-rate ranges (categories) are used to generate seven separate fan maps, forming the transform relationships for all raining pixels, for a given algorithm, for the land or ocean area of an overpass.

Figure 9 presents the results for nine algorithms based on the ocean area of case 11/overpass 2 (winter nor’easter). Note that 19U–37U and 19U–85U relationships are given for each algorithm, regardless of which channels are used in the associated T_B–RR conversion procedure. The foremost result from this selection of fan maps is that the maximum rain-rate categories associated with the different algorithms are different. Forexample, compare FER–AVE and LIU–CUR, whose rain rates do not exceed category 4 (7.5–12.5 mm h⁻¹, with GSCAT and NESDIS, whose rain rates extend to category 7 (>25 mm h⁻¹), a difference in maximum rain rates exceeding a factor of 2. Next note the varying positions and thicknesses of the bundles of lines originating from the abscissas and terminating at the ordinates. Were two algorithms to perform similarly, they would exhibit similar line bundle properties for each rainrate category.

For example, compare the scattering algorithm GSCAT, which only uses 85 GHz in its T_B–RR conversion scheme, to the emission algorithm TAMU dominated by 19 GHz in its conversion scheme. The GSCAT fan maps indicate, as they should, that as the rain rates increase and scattering depresses the 85-GHz T_Bs, the right-hand bundles terminate at decreasing values of 85-GHz T_B. However, for the first five categories, the originating points of the 19Us on the right-side abscissas show little variation and varying degrees of thickness, indicating little correlation between rain rate assignment and 19-GHz T_B for rain rates up to 17.5 mm h⁻¹. Examination of the corresponding left-hand bundles oflines connecting the 19-GHz T_Bs with 37-GHz T_Bs indicates that rain-rate assignments up through the fifth category are unrelated to brightness temperatures at both the lower frequencies. In the case of TAMU, the dominant feature is the steady rise of T_B at 19 GHz as rain rate increases (i.e., the thin spreads of the line bundles emanating from both left- and right-side abscissas), associated with the thick spreads and near-invariance in positions of the bundle terminations on the right-hand 85-GHz ordinates for categories 2–6. Although there is clearly an increase in the 37-GHz T_Bs between category 1 and 6 (indicative of the emission signal), the rain rates between 2.5 and 12.5 mm h⁻¹ (categories 2–4) show little relationship to 37-GHz T_B.

There are additional examples of differences evident in Fig. 9 and numerous other examples when all the fan maps generated for the entire PIP-2 intercomparison analysis are considered. However, the main issue here is not to compile all the many differences, but to point out that these differences are prevalent and that they can become significant at large rain rates, which ensures that rms differences will become correspondingly large regardless of algorithm-to-algorithm bias.