• Adeyewa, Z. D., and K. Nakamura. 2003. Validation of TRMM radar rainfall data over major climatic regions in Africa. J. Appl. Meteor. 42:331347.

    • Search Google Scholar
    • Export Citation
  • Arkin, P. A., and B. N. Meisner. 1987. The relationship between large-scale convective rainfall and cold cloud over the Western Hemisphere during 1982–84. Mon. Wea. Rev. 115:5174.

    • Search Google Scholar
    • Export Citation
  • Austin, P. M. 1987. Relation between measured radar reflectivity and surface rainfall. Mon. Wea. Rev. 115:10531069.

  • Bell, T. L., and P. K. Kundu. 1996. A study of the sampling error in satellite rainfall estimates using optimal averaging of data and a stochastic model. J. Climate 9:12511268.

    • Search Google Scholar
    • Export Citation
  • Bell, T. L., and P. K. Kundu. 2000. Dependence of satellite sampling error on monthly averaged rain rates: Comparison of simple models and recent studies. J. Climate 13:449462.

    • Search Google Scholar
    • Export Citation
  • Chang, A. T. C., and L. S. Chiu. 1999. Nonsystematic errors of monthly oceanic rainfall derived from SSM/I. Mon. Wea. Rev. 127:16301638.

    • Search Google Scholar
    • Export Citation
  • Cosgrove, C. M., and M. Garstang. 1995. Simulation of rain events from rain gauge measurements. Int. J. Climatol. 15:10211029.

  • Efron, B. 1990. More efficient bootstrap computations. J. Amer. Stat. Assoc. 85:7989.

  • Gebremichael, M., and W. F. Krajewski. 2004a. Assessment of the statistical characterization of small-scale rainfall variability from radar: Analysis of TRMM ground validation datasets. J. Appl. Meteor. 43:11801199.

    • Search Google Scholar
    • Export Citation
  • Gebremichael, M., and W. F. Krajewski. 2004b. Characterization of the temporal sampling error in space-time-averaged rainfall estimates from satellites. J. Geophys. Res. 109.D11110, doi:10.1029/2004JD004509.

    • Search Google Scholar
    • Export Citation
  • Gebremichael, M., , W. F. Krajewski, , M. Morrissey, , D. Langerud, , G. J. Huffman, , and R. Adler. 2003. Error uncertainty analysis of GPCP monthly rainfall products: A data-based simulation study. J. Appl. Meteor. 42:18371848.

    • Search Google Scholar
    • Export Citation
  • Huffman, G. J. 1997. Estimates of root-mean-square random error for finite samples of estimated precipitation. J. Appl. Meteor. 36:11911201.

    • Search Google Scholar
    • Export Citation
  • Huffman, G. J. Coauthors 1997. The Global Precipitation Climatology Project (GPCP) combined precipitation dataset. Bull. Amer. Meteor. Soc. 78:520.

    • Search Google Scholar
    • Export Citation
  • Huffman, G. J., , R. F. Adler, , M. M. Morrissey, , D. T. Bolvin, , S. Curtis, , R. Joyce, , B. McGavock, , and J. Susskind. 2001. Global precipitation at one-degree daily resolution from multisatellite observations. J. Hydrometeor. 2:3650.

    • Search Google Scholar
    • Export Citation
  • Joss, J., and A. Waldvogel. 1990. Precipitation measurement and hydrology. Radar in Meteorology, Battan Memorial 40th Anniversary Radar Meteorology Conference, D. Atlas, Ed., Amer. Meteor. Soc., 577–606.

  • Joss, J., and R. Lee. 1995. The application of radar-gauge comparisons to operational precipitation profile corrections. J. Appl. Meteor. 34:26122630.

    • Search Google Scholar
    • Export Citation
  • Kessler, E., and B. Neas. 1994. On correlation, with applications to the radar and raingage measurement of rainfall. Atmos. Res. 34:217229.

    • Search Google Scholar
    • Export Citation
  • Krajewski, W. F., and J. A. Smith. 2002. Radar hydrology: Rainfall estimation. Adv. Water Resour. 25:13871394.

  • Krajewski, W. F., , G. J. Ciach, , J. R. McCollum, , and C. Bacotiu. 2000. Initial validation of the Global Precipitation Climatology Project over the United States. J. Appl. Meteor. 39:10711086.

    • Search Google Scholar
    • Export Citation
  • Kummerow, C., , W. S. Olsen, , and L. Giglio. 1996. A simplified scheme for obtaining precipitation and vertical hydrometeor profiles from passive microwave sensors. IEEE Trans. Geosci. Remote Sens. 34:12131232.

    • Search Google Scholar
    • Export Citation
  • Legates, D. R., and G. J. McCabe. 1999. Evaluating the use of the “goodness-of-fit” measure in hydrologic and hydroclimatic model validation. Water Resour. Res. 35:233241.

    • Search Google Scholar
    • Export Citation
  • Leger, C., , D. N. Politis, , and J. P. Romano. 1992. Bootstrap technology and applications. Technometrics 34:378398.

  • Li, Q., , R. L. Bras, , and D. Veneziano. 1996. Analysis of Darwin rainfall data: Implications on sampling strategy. J. Appl. Meteor. 35:372385.

    • Search Google Scholar
    • Export Citation
  • McConnell, A., and G. R. North. 1987. Sampling errors in satellite estimates of tropical rain. J. Geophys. Res. 92:95679570.

  • Morrissey, M. L., and J. E. Janowiak. 1996. Sampling-induced conditional biases in satellite climate-scale rainfall estimates. J. Appl. Meteor. 35:372385.

    • Search Google Scholar
    • Export Citation
  • Nelson, B. R., , W. F. Krajewski, , A. Kruger, , J. A. Smith, , and M. L. Baeck. 2003a. Archival precipitation data set for the Mississippi river basin: Development of a GIS-based data browser. Comput. Geosci. 29:595604.

    • Search Google Scholar
    • Export Citation
  • Nelson, B. R., , W. F. Krajewski, , A. Kruger, , J. A. Smith, , and M. L. Baeck. 2003b. Archival precipitation data set for the Mississippi river basin: Algorithm development. J. Geophys. Res. 108.8811, doi:10.1029/2002JD003158.

    • Search Google Scholar
    • Export Citation
  • Neter, J., and W. Wasserman. 1974. Linear Statistical Methods. Richard D. Irwin, 667 pp.

  • Ramage, K., , I. Jobard, , T. Lebel, , and M. Desbois. 2000. Satellite estimation of 1-day to 10-day cumulated precipitation: Comparison of TRMM, METEOSAT and GPCP products. Proc. of the 2000 EUMETSAT Meteorological Satellite Data Users’ Conf., Bologna, Italy, EUMETSAT, 363–369.

  • Roads, J. Coauthors 2003. GCIP water and energy budget synthesis (WEBS). J. Geophys. Res. 108.8609, doi:10.1029/2002JD002583.

  • Seed, A., and G. L. Austin. 1990. Variability of summer Florida rainfall and its significance for the estimation of rainfall by gages, radar, and satellite. J. Geophys. Res. 95:22072215.

    • Search Google Scholar
    • Export Citation
  • Skomorowski, P., , F. Rubel, , and B. Rudolf. 2001. Verification of GPCP-1DD global satellite precipitation products using MAP surface observations. Phys. Chem. Earth B 26:5–6,. 403409.

    • Search Google Scholar
    • Export Citation
  • Soman, V. V., , J. B. Valdes, , and G. R. North. 1995. Satellite sampling and the diurnal cycle statistics of Darwin rainfall data. J. Appl. Meteor. 34:24812490.

    • Search Google Scholar
    • Export Citation
  • Steiner, M. 1996. Uncertainty of estimates of monthly areal rainfall for temporally sparse remote observations. Water Resour. Res. 32:373388.

    • Search Google Scholar
    • Export Citation
  • Susskind, J., , P. Piraino, , L. Iredell, , and A. Mehta. 1997. Characteristics of the TOVS pathfinder path A dataset. Bull. Amer. Meteor. Soc. 78:14491472.

    • Search Google Scholar
    • Export Citation
  • Wilks, D. S. 1995. Statistical Methods in the Atmospheric Sciences. Academic Press, 467 pp.

  • Willmott, C. J. 1981. On the validation of models. Phys. Geogr. 2:184194.

  • Willmott, C. J. 1982. Some comments on the evaluation of model performance. Bull. Amer. Meteor. Soc. 11:13091313.

  • Willmott, C. J., and D. E. Wicks. 1980. An empirical method for the spatial interpolation of monthly precipitation within California. Phys. Geogr. 1:5973.

    • Search Google Scholar
    • Export Citation
  • Xie, P., and P. A. Arkin. 1997. Global precipitation: A 17-year monthly analysis based on gauge observations, satellite estimates, and numerical model outputs. Bull. Amer. Meteor. Soc. 78:25392558.

    • Search Google Scholar
    • Export Citation
  • View in gallery

    The Mississippi River basin in the United States. The basin is divided into three regions identified by the numbers.

  • View in gallery

    Spatial distribution of rainfall statistics estimated from the (a) GPCP 1DD and (b) MRB 1DD mean, and (c) GPCP 1DD and (d) MRB 1DD std dev. Unshaded areas are regions outside of the Mississippi River basin. Region 3 is shown on the maps.

  • View in gallery

    Spatial distribution of comparison statistics: (a) NBIAS, (b) d, and (c) R2, and the regression parameters (d) a and (e) b

  • View in gallery

    Spatial distribution of difference measures, both in absolute and relative units. The mean refers to the MRB 1DD–estimated mean rain rate.

  • View in gallery

    Time series of daily rainfall for the selected 1DD grid estimated from the MRB 1DD (thick line) and GPCP 1DD (thin line). The selected grid is indicated by the arrow in Fig. 3a.

  • View in gallery

    Distribution of the spatial variability of the validation statistics obtained using the 100 1° grids located in region 3.

  • View in gallery

    GPCP 1DD–MRB 1DD difference as a function of the MRB 1DD–estimated rain rate for region 3.

  • View in gallery

    The mean error, the bias-adjusted RMSD (BA-RMSD), and the RMSD between the GPCP 1DD and the MRB 1DD as a function of the MRB 1DD–estimated rain rate.

  • View in gallery

    Spatial distribution of POD, FAR, and HSS values for a threshold of 0.1 mm day−1.

  • View in gallery

    Categorical statistics as a function of precipitation threshold for region 3. The statistics are POD, HSS, and FAR, from the top down. The statistics are calculated for samples equal to or greater than the threshold.

  • View in gallery

    Seasonal variation of the validation statistics for the GPCP 1DD product, computed by pooling all seasonal 1DD estimates in region 3.

  • View in gallery

    Regional variation of the validation statistics for the GPCP 1DD, computed by pooling all estimates in regions 2 (dark box) and 1 (lighter box).

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 76 76 12
PDF Downloads 44 44 7

A Detailed Evaluation of GPCP 1° Daily Rainfall Estimates over the Mississippi River Basin

View More View Less
  • a IIHR—Hydroscience and Engineering, The University of Iowa, Iowa City, Iowa
  • | b Environmental Verification and Analysis Center, The University of Oklahoma, Norman, Oklahoma
  • | c Science Systems and Applications, Inc., and NASA Goddard Space Flight Center, Greenbelt, Maryland
  • | d NASA Goddard Space Flight Center, Greenbelt, Maryland
© Get Permissions
Full access

Abstract

This study provides an intensive evaluation of the Global Precipitation Climatology Project (GPCP) 1° daily (1DD) rainfall products over the Mississippi River basin, which covers 435 1° latitude × 1° longitude grids for the period of January 1997–December 2000 using radar-based precipitation estimates. The authors’ evaluation criteria include unconditional continuous, conditional (quasi) continuous, and categorical statistics, and their analyses cover annual and seasonal time periods. The authors present spatial maps that reflect the results for the 1° grids and a summary of the results for three selected regions. They also develop a statistical framework that partitions the GPCP–radar difference statistics into GPCP error and radar error statistics. They further partition the GPCP error statistics into sampling error and retrieval error statistics and estimate the sampling error statistics using a data-based resampling experiment. Highlights of the results include the following: 1) the GPCP 1DD product captures the spatial and temporal variability of rainfall to a high degree, with more than 80% of the variance explained, 2) the GPCP 1DD product proficiently detects rainy days at a large range of rainfall thresholds, and 3) in comparison with radar-based estimates the GPCP 1DD product overestimates rainfall.

Corresponding author address: Dr. Witold F. Krajewski, IIHR—Hydroscience and Engineering, The University of Iowa, Iowa City, IA 52242. witold-krajewski@uiowa.edu

Abstract

This study provides an intensive evaluation of the Global Precipitation Climatology Project (GPCP) 1° daily (1DD) rainfall products over the Mississippi River basin, which covers 435 1° latitude × 1° longitude grids for the period of January 1997–December 2000 using radar-based precipitation estimates. The authors’ evaluation criteria include unconditional continuous, conditional (quasi) continuous, and categorical statistics, and their analyses cover annual and seasonal time periods. The authors present spatial maps that reflect the results for the 1° grids and a summary of the results for three selected regions. They also develop a statistical framework that partitions the GPCP–radar difference statistics into GPCP error and radar error statistics. They further partition the GPCP error statistics into sampling error and retrieval error statistics and estimate the sampling error statistics using a data-based resampling experiment. Highlights of the results include the following: 1) the GPCP 1DD product captures the spatial and temporal variability of rainfall to a high degree, with more than 80% of the variance explained, 2) the GPCP 1DD product proficiently detects rainy days at a large range of rainfall thresholds, and 3) in comparison with radar-based estimates the GPCP 1DD product overestimates rainfall.

Corresponding author address: Dr. Witold F. Krajewski, IIHR—Hydroscience and Engineering, The University of Iowa, Iowa City, IA 52242. witold-krajewski@uiowa.edu

Introduction

Several global or regional precipitation products exist that are based on satellite observations (e.g., Xie and Arkin 1997; Huffman et al. 1997; Krajewski et al. 2000; Huffman et al. 2001). Although most satellite-based precipitation products are for monthly rainfall accumulations over large areas, recent efforts have extended satellite-derived precipitation products to higher temporal and spatial scales. The Global Precipitation Climatology Project (GPCP) has been producing global daily precipitation estimates at 1° longitude × 1° latitude resolution for the period of January 1997 to the (delayed) present by combining data from different satellite sensors (Huffman et al. 2001). These estimates, hereinafter referred to as 1° daily (1DD), are important in such applications as hydrology, agriculture, and climate studies.

There are major issues associated with the ability of satellite-based techniques. Errors in satellite rainfall estimation would result from temporal sampling error, instrumental error, and algorithm error. Temporal sampling errors arise from infrequent satellite visits and the variability of rain fields in space and time. Instrumental errors are related to calibration and measurement noise. Algorithm errors are directly associated with the algorithm approximation to the physics that are used to generate estimates of “instantaneous” rain maps. Instrumental and algorithm errors are effectively considered to be retrieval errors. Estimates of these errors are required to provide quantitative confidence on the satellite rainfall products. These estimates are essential to clearly understand the limitations and capability of remotely sensed data, to properly utilize the data, and to improve the remote sensing rainfall algorithms.

Evaluation of the GPCP 1DD, albeit limited, has been conducted in parts of the United States, the European Alps, and Africa. Huffman et al. (2001) performed an evaluation of the GPCP 1DD for the period of January 1997–December 1998 using the Oklahoma Mesonet rain gauge network and presented the validation statistics for one 1° grid. For the entire record, they reported that the GPCP 1DD underestimates rainfall by 3%, and the correlation between the GPCP 1DD and the reference rainfall is 0.76. At the seasonal time scale, they found that the bias reveals a semiannual fluctuation, with underestimation during March–May (MAM) and September–November (SON), nearly unbiased during June–August (JJA), and overestimates during December–February (DJF). Their results show that the mean absolute error generally follows the trend of the mean rain rate, with smaller values during DJF and higher values during SON. In terms of correlation, they found higher values during DJF (0.89) than during JJA (0.65).

Over the European Alps, Skomorowski et al. (2001) evaluated the GPCP 1DD for the period of June–July 1997 using data from 3100 rain gauges archived in the Mesoscale Alpine Program (MAP) data center. They concluded that the GPCP 1DD underestimates rainfall by 15%–19%, and the correlation between the GPCP 1DD and the reference rainfall is 0.56–0.58. In terms of categorical statistics, they reported a probability of detection (POD) of 0.65 and a false-alarm ratio (FAR) of 0.22, at a threshold of 0.1 mm day−1.

Ramage et al. (2000) analyzed the GPCP 1DD over Niger for the period of June–September 1998 using a rain gauge network that consists of 149 gauges. They reported that the GPCP 1DD overestimates rainfall on average by 6%, and the correlation between the GPCP 1DD and the reference rainfall is 0.45. They also assessed the relative performance of the reference gauge network by comparing it to another overlapping independent gauge network. They found a correlation of 0.94 between the two networks with significant scatter for small rain events. Hence, part of the discrepancy between the GPCP 1DD and the gauge network is attributed to the deficiencies in the network itself.

In a recent study, Adeyewa and Nakamura (2003) evaluated the GPCP 1DD for 36 months over continental Africa with respect to the rain gauge analysis products available from the Global Precipitation Climatology Center (GPCC). They showed that the comparison statistics vary over a wide range depending on the season and the region—the bias ranges from −40% to 200%, the correlation ranges from 0.32 to 0.99, and the root-mean-square-error ranges from 10% to 200% of the mean. They also found that the largest discrepancy between the GPCP 1DD and the GPCC product occurs during DJF. It should be noted, however, that the two products are not independent, because the GPCC product is one of the input data sources of GPCP 1DD (Huffman et al. 1997).

This quick survey suggests that the evaluation statistics for the GPCP 1DD exhibit geographical and seasonal variability. It also highlights the difficulty of obtaining rainfall products that are of sufficient quality to validate the GPCP 1DD.

In this paper, we document our efforts in evaluating the GPCP 1DD over the Mississippi River basin using an independent, high-quality dataset. The independent dataset, henceforth referred to as the Mississippi River basin (MRB) dataset or the MRB estimates, documents 5 yr (January 1996–December 2000) of radar-derived rainfall estimates for the Mississippi River basin (Roads et al. 2003; Nelson et al. 2003b). The MRB dataset is constructed from about 50 Weather Surveillance Radar-1988 Doppler (WSR-88D) Next-Generation Weather Radars (NEXRAD) located across the basin (Nelson et al. 2003a). The MRB dataset’s primary product is its hourly estimates of rainfall at the spatial resolution of 4 km × 4 km, which we scaled up to daily 1° × 1° to match the GPCP 1DD resolution.

Our variable of interest is the rain rate at a resolution of 1DD. The temporal period of our analysis is 1997–2000, a period represented in both datasets. We used a number of statistical measures to evaluate the GPCP 1DD. These measures can be categorized as unconditional (continuous) statistics, conditional (quasi continuous) statistics, and categorical statistics. Our analysis covers annual and seasonal time periods. We present the spatial variability of the validation statistics, as well as their aggregated values for three selected regions.

Last, we acknowledge that although much effort was exerted to obtain high-quality estimates from a highly dense network of radars, the MRB dataset estimates are also subject to errors and we do not know much about their error characteristics. We provide a statistical framework to describe how the GPCP 1DD–MRB 1DD difference statistics could be partitioned between the GPCP 1DD error and the MRB 1DD error components. Such a framework provides the potential to further improve the GPCP 1DD error statistics when additional results on validation of the MRB 1DD estimates are obtained. We, and others, are presently addressing this issue in a parallel study.

We organized this paper in six sections. Our starting point provides a description of the MRB and the GPCP 1DD datasets (section 2), as well as the approach followed in this study (section 3). In section 4 we present and decipher the results of the evaluation statistics. In section 5 we provide the framework to partition the GPCP 1DD–MRB 1DD difference statistics into the MRB 1DD error and the GPCP 1DD error statistics. We further partition the GPCP 1DD error into the sampling error and retrieval error components, and estimate the sampling error using a data-based simulation experiment. We close the paper in section 6 with conclusions and recommendations.

Study area and data

Figure 1 presents our study area—the Mississippi River basin in the United States. For the analysis, we divided the basin into three regions, taking the satellite data type and the quality of the data into account. Region 1 (region 2) covers the study area above (below) 40°N latitude. As we will discuss shortly, there are differences between the satellite data type used to obtain the GPCP 1DD products over region 1 and over region 2. We delineated region 3 as the region where we have relatively high confidence in the MRB rainfall estimates. Region 3, shown by a dotted rectangle around the center of the basin, does not suffer from factors that can compromise the MRB estimates, such as, significant snowfall, poor radar density, and beam blockage (Nelson et al. 2003b). The westernmost areas of the basin receive significant amount of snowfall. Furthermore, the density of the radars is inadequate for complete coverage in the westernmost areas of the basin, with some pocket areas, particularly in the northern west, suffering from partial beam coverage. The Appalachians, in the easternmost areas, also experience beam blockage problems. Regions 1, 2, and 3 consist of 219, 216, and 100 1° grids, respectively.

The GPCP 1DD estimates are based on a combination of different satellite data. In the 40°N–40°S region, infrared (IR) estimates from geostationary satellites calibrated with microwave (MW) estimates from the polar-orbiting Special Sensor Microwave Imager (SSM/I) are used. The rainfall estimation technique is based on the popular Geostationary Operational Environmental Satellite (GOES) precipitation index (GPI; Arkin and Meisner 1987). This adaptation, known as the threshold-matched precipitation index (TMPI; Huffman et al. 2001), assigns a constant conditional rain rate to all pixels within a given region that have a brightness temperature (Tb) less than the threshold brightness temperature (Tb0). The GPCP 1DD obtains Tb0 by comparing the 3-hourly coincident (i.e., within ±1.5 h) geostationary IR histogram and the precipitation frequency histogram estimated from SSM/I MW data using the algorithm developed by Kummerow et al. (1996), after accumulating each for the month on a 1° × 1° grid. The conditional rain rate is obtained from regression between the GPCP satellite gauge rainfall estimates (SG) and the number of geostationary IR pixels below Tb0. The SG product provides monthly estimates on a 2.5° × 2.5° grid, and its algorithm involves sequential merging of input estimates from satellite MW, IR, and rain gauge observations (Huffman et al. 1997).

Outside the 40°N–40°S region, instead of geostationary IR data (current geostationary IR datasets do not cover this region) GPCP uses the Television and Infrared Observation Satellite (TIROS) Operational Vertical Sounder (TOVS) data from the polar-orbiting National Oceanic and Atmospheric Administration (NOAA)-12 and NOAA-14 satellites. The algorithm involves three main steps. First, rainfall estimates are made from TOVS data using an empirical relationship between TOVS-derived parameters (cloud-top pressure, fractional cloud cover, and relative humidity profile) and surface rainfall estimates (Susskind et al. 1997). Second, rain-rate estimates that are computed from TOVS data for the region of 39°–40°N and 39°–40°S are compared with corresponding TMPI estimates in terms of the number of rainy days. According to Huffman et al. (2001), the number of rainy days estimated from TOVS is systematically high compared to the TMPI estimates. The number of rainy days estimated from TOVS is then rescaled to match the TMPI estimates by zeroing the smallest rain accumulations of the TOVS estimates. Third, the remaining (nonzero) TOVS rain days are rescaled to add up to the monthly SG. Details of the GPCP 1DD algorithm are given in Huffman et al. (2001).

The MRB dataset that we used to validate the GPCP 1DD was developed as part of the Global Water and Energy Cycle (GEWEX) Continental-Scale International Project (GCIP) and is available for the entire Mississippi River basin (Nelson et al. 2003a, b). The principal inputs of the MRB dataset are the national composite radar reflectivity maps produced by Weather Services International Corporation (WSI). They are based on data from the NEXRAD network that is operated by the National Weather Service, the Federal Aviation Administration, and the U.S. Air Force Air Weather Service and Naval Oceanography Command. These composite maps are a national product termed NOWrad and are available at 15-min and 2 km × 2 km resolution. This product is described in Nelson et al. (2003a). Nelson et al. (2003b) used two primary steps to derive the MRB precipitation estimates from the NOWrad reflectivity inputs. First, they quality controlled the data to account for anomalous propagation of the radar beam, the highly biased radar returns from the melting layer, and differences in the calibration of one radar versus another. Second, they applied a reflectivity–rainfall (Z–R) relationship in which the power-fit parameters are obtained through an optimization technique between the precipitation estimates obtained from some high-quality and high-density rain gauge networks in the region and the corresponding NOWrad reflectivity-based precipitation estimates. Using these steps, they obtained radar-derived rainfall estimates at 15-min and 2 km × 2 km resolution. They aggregated these estimates to a resolution of 15 min and 4 km × 4 km, and then to a resolution of 1 h and 4 km × 4 km, which is the official product. In this study, we used the official product to validate the GPCP 1DD estimates and the intermediate product (15 min, 4 km × 4 km) to estimate the sampling error in the GPCP 1DD estimates.

As acknowledged, radar-based rainfall estimates are subject to a number of error sources, including the natural variability of drop size distribution, variation of reflectivity with height, attenuation of radar signal, and radar hardware miscalibration and noise (Austin 1987; Joss and Waldvogel 1990; Joss and Lee 1995; Krajewski and Smith 2002). In addition, the reflectivity inputs to the MRB estimates are available quantized in levels of large 5-dBZ intervals, which adds to the list of uncertainty sources. The random errors introduced by these sources tend to cancel out as the rainfall estimates are averaged over larger space–time scales but our understanding of this error-averaging process is limited. This was one main reason why Nelson et al. (2003b) developed the official product at a smaller resolution (4 km × 4 km, 1 h) while the input data were available at a higher resolution (2 km × 2 km, 15 min). Averaging the MRB estimates to 1DD further reduces the random error, by increasing the total sample size to ∼610 × 24 = 14 640, where the numbers 610 and 24 correspond to the average number of 4-km pixels in a 1° grid and the number of hours in a day, respectively. Gebremichael and Krajewski (2004a) also demonstrated that the information obtained through radar when used at the appropriate space–time scale can provide a more accurate account of the space–time-averaged rainfall fields.

However, the systematic errors do not average out with an increase in the spatial scale unless they happen to change their signs spatially within a 1° grid. Nelson et al. (2003b) obtained the ZR fit parameters using the following four rain gauge networks: the Oklahoma Mesonet (with 111 stations), the Automated Environmental Monitoring Network in Georgia (with 47 stations), the Goodwin Creek research network (with 32 stations), and the Iowa City Airport stations (with 10 gauges). The gauges were not corrected for undercatch bias. Over areas represented by these stations, the ZR parameters are obtained in such a way that there is no bias in the MRB estimates. It should be noted that the Oklahoma Mesonet, the network in Georgia, and the Iowa City Airport stations lie in region 3.

Nelson et al. (2003b) compared the MRB estimates with those obtained from 112 first-order gauge stations that lie across the MRB region. The stations are spread in regions 1, 2, and 3. The stations are mainly climate stations and are maintained by experts. They are run by National Weather Service and other federal agencies, and their data are compiled by the National Climatic Data Center (NCDC). Nelson et al. (2003b) found that the MRB estimates are unbiased with respect to these stations. In summary, the MRB estimates represent the highest-quality large (spatially and temporally) rainfall estimates for the MRB region. Our present information is that the MRB estimates are unbiased, as evidenced by comparison with the high-quality rain gauge networks. In a parallel study, we are working on quantifying the random error component of the MRB estimates at 1DD scale, which we believe is reasonably small.

Approach

To assess the performance of the GPCP 1DD with respect to the MRB 1DD, we quantified the systematic and random differences between the two estimates using a suite of statistical techniques that include unconditional continuous statistics, conditional (quasi) continuous statistics, and categorical statistics.

Unconditional continuous statistics

Following the recommendations of Willmott (1981, 1982) and Legates and McCabe (1999), we used the following set of unconditional statistics for quantitatively evaluating the nature of differences between the GPCP 1DD and MRB 1DD estimates.

Agreement statistics

Here,
i1520-0450-44-5-665-e1
where RG and RM are the GPCP 1DD and the MRB 1DD estimates, respectively, and E[] denotes the expected value over realizations of the dataset. The statistic R2 measures the degree of collinearity between the two datasets. It describes the proportion of the variance of RG explained by RM. As pointed out by Willmott (1981, 1982), the magnitude of R2 is not consistently related to the accuracy of prediction, that is, where accuracy is defined as the degree to which model-predicted observations approach the magnitudes of their observed counterparts. This stems from the inability of R2 to discern differences in proportionality and/or constant additive differences between the two datasets. In another study, Willmott and Wicks (1980) observed that high values of R2 might, in fact, be misleading, because they are often unrelated to the sizes of the difference between the two datasets. It is also quite possible for small differences between the two datasets to occur with low values of R2.
To circumvent some of the problems associated with R2, Willmott (1981) developed a new statistic called the index of agreement (d),
i1520-0450-44-5-665-e2
where RG = RGE[RM] and RM = RME[RM]. The statistic d specifies the degree to which the deviations of RM about E[RM] correspond—both in magnitude and sign—to the deviations of RG about E[RM]. Here, d is not a measure of correlation or association in the formal sense, but is rather a measure of the degree to which a model’s predictions are error free. As well, d varies between 0 and 1, where a computed values of 1 indicates perfect agreement between the two datasets, and 0 indicates one of a variety of complete disagreements. Unlike R2, d is sensitive to differences between the two datasets, as well as to certain changes in proportionality.

The agreement statistics, that is, d and R2, provide a relative comparison of the performance of one of dataset against another.

Overall bias statistic

The bias ratio (NBIAS),
i1520-0450-44-5-665-e3
measures the overall bias in RG relative to RM, in a dimensionless unit.

Difference statistics

In contrast to the above ratio statistics, difference statistics are measured in the units of actual measurement, and, hence, are easy to interpret. The most popular difference statistics is the overall root-mean-square difference (RMSD), defined as
i1520-0450-44-5-665-e4
Willmott (1981) proposed that the systematic difference could be described by the systematic RMSD (RMSDs), defined by
i1520-0450-44-5-665-e5
whereas the unsystematic error can be described by the unsystematic RMSD (RMSDu), defined by
i1520-0450-44-5-665-e6
in which G is the regressed rainfall derived from the linear relationship G = aRM + b, where a and b are the slope and intercept. Because the system is conservative,
i1520-0450-44-5-665-e7
The squared difference term in RMSD places undue importance on the outliers in the dataset.
The mean absolute deviation (MAD), defined by
i1520-0450-44-5-665-e8
is less sensitive to extreme values than RMSD. The MAD is intuitively more appealing than RMSD, because it avoids the physically artificial exponentiation that is an artifact of the statistical mathematical reasoning from which RMSD is derived. On the other hand, RMSD is generally amenable to more in-depth mathematical statistical analyses than MAD.

Spatial pattern

Comparisons of the spatial pattern of the mean and the standard deviation provide a useful measure of the agreement/disagreement in the spatial pattern of the two datasets.

Conditional (quasi) continuous statistics

To investigate how the difference between the two datasets behaves in various rain-rate ranges, we have applied the evaluation statistics conditioned on the magnitude of rain rate. The conditional statistics used are the parameters a and b in the regression equation G = aRM + b, and the distribution of the GPCP 1DD–MRB 1DD difference, which is conditional upon various rain-rate ranges.

Categorical statistics

The categorical statistics are used to measure the correspondence between the GPCP 1DD– and MRB 1DD–estimated occurrence of events. The categorical statistics used are the POD, FAR, and Heidke skill score (HSS), as defined in Wilks (1995). The POD represents the fraction of cases in which the rain event occurrence is reported by the GPCP 1DD when it was also reported by the MRB 1DD. The FAR is the fractional number of times that the rain event was reported by the GPCP 1DD but was not registered by the MRB 1DD. The POD and the FAR vary from 0 to 1; the skill varies from 1 (perfect skill) to −1 (perfect negative skill). Zero represents no skill relative to chance.

Results and discussion

In the analysis that follows, the rain rate refers to 1-day-accumulated rain depth averaged over a 1° by 1° area (i.e., 1DD). The statistics are based on rainfall estimates for the 4-yr period from 1997 through 2000. We present the spatial and seasonal variability of the statistics and their representative values over three selected regions. The three regions considered are defined in Fig. 1. The seasons considered are DJF, MAM, JJA, and SON.

Multiannual analysis

We begin by examining the spatial variability of the (temporal) mean and the (temporal) standard deviation of rainfall estimated from the MRB 1DD and the GPCP 1DD (Fig. 2). According to the MRB 1DD, the rainfall amount and its variability increase as one goes from the northwest toward the southeast. A similar spatial pattern is displayed by the GPCP 1DD products. In fact, R2 between the spatial GPCP 1DD–estimated mean (standard deviation) and the corresponding spatial MRB 1DD–estimated mean (standard deviation) is 0.94 (0.92). So the MRB 1DD estimates can adequately capture the spatial pattern of the mean and standard deviation of the GPCP 1DD estimates.

Despite the similarity in the spatial pattern, significant differences exist in actual magnitudes of the mean and standard deviation values between the GPCP 1DD and MRB 1DD estimates. It appears that overall the GPCP 1DD estimates are higher and exhibit larger temporal fluctutation than the MRB 1DD.

In Fig. 3 we present the spatial variability of the agreement statistics (d and R2) and the systematic difference measures (NBIAS, a, and b). The R2 values are greater than 0.8 over all locations, indicating that the GPCP 1DD explains most of the temporal variability in the MRB 1DD estimates. The statistic d gives values that are higher than R2 in the northwest, while it is similar to R2 in most other regions. The relatively smaller value of R2 (about 0.80) in the northwest compared to the d values could be due to the smaller rainfall amount in this region. The tendency of R2 to show smaller correlations for smaller sets of values has also been observed by Willmott and Wicks (1980) and Kessler and Neas (1994). In general, the values of R2 and d are exceedingly high at almost all locations, indicating that the GPCP 1DD estimates reflect the temporal fluctuation of the MRB 1DD estimates to a great extent. It should be noted that the agreement statistics results are significantly better than those reported in previous studies (see section 1).

The systematic errors are measured by NBIAS and regression parameters (i.e., the slope and intercept). As NBIAS demonstrates, the GPCP 1DD overestimates the mean rain rate by 5%–100% with respect to the MRB 1DD, depending on the location (see Fig. 3). This is further illustrated through the slope (mostly between 1 and 2) and the intercept (mostly between 0 and 0.2 mm day−1).

In Fig. 4 we present the spatial variability of the total difference measures in absolute units (MAD and RMSD), as well as after normalization by the MRB 1DD–estimated mean (MAD/mean, and RMSD/mean). On the same figure we also present the systematic and random error measures in both absolute (RMSDs and RMSDu) and relative (RMSDs/mean, and RMSDu/mean) units, as well as their ratio (RMSDs/ RMSDu).

The total difference between the GPCP 1DD and the MRB 1DD estimates varies in the range of 50%–200% of the mean in terms of RMSD. However, this difference shrinks to 25%–100% of the mean if the MAD is used instead. The difference between MAD and RMSD results from the fact that the weighting of each difference by its square tends to inflate RMSD, particularly when extreme values are present. The MAD and RMSD statistics follow the trend of rainfall mean or its variability, with values increasing as one goes from the northwest to the southeast. However, when these statistics are normalized by the mean, this trend disappears.

Both the RMSDs and RMSDu amount to 25%–200% of the mean. As can be seen from the plot of RMSDs/RMSDu, the unsystematic errors dominate in the northwest part of the basin, while the systematic errors dominate in the rest parts of the basin. Further investigation is needed to identify the reason for this.

Both Figs. 3 and 4 demonstrate that the evaluation statistics exhibit spatial variability. For example, NBIAS ranges from about 1.0 to 2.0, d ranges from 0.83 to 0.99, and RMSD ranges from 0.1 to 3.5 mm day−1. This suggests that the performance of the GPCP 1DD varies with location, which highlights the need for a validation study over such large areas as opposed to the generalization of results for the entire region based on a small sample of grids.

In Fig. 5 we offer an example of the MRB 1DD and GPCP 1DD precipitation time series for the year 1999 for one selected 1DD grid box located at the center of region 3 (indicated by the arrow in Fig. 3a). As expected, the GPCP 1DD estimates closely approximate the rainfall temporal fluctuation. At small rain rates, the GPCP 1DD and MRB 1DD values are remarkably similar. However, as the rain intensity increases, the GPCP 1DD shows higher values than the MRB 1DD, regardless of the season considered.

Let us now focus on region 3, a region where we have relatively higher confidence in the MRB products. Using the statistics obtained for the 100 1DD grids located in region 3, we summarize the results of Figs. 3 and 4 in Fig. 6. In the box-and-whisker plot of this figure, the values of the 25th and 75th quantiles form the ends of the box. The line inside the box signifies the median value, and the whiskers (the lines drawn from the quantiles) end at the extreme value of the dataset. For 90% of the locations inside region 3, the statistics vary as follows: NBIAS = 1.25–1.73, MAD = 32%–75%, RMSD = 85%–208%, RMSDs = 61%–168%, and RMSDu = 0.45%–1.37%, where the percentage indicates the proportion of the statistics relative to the MRB 1DD–estimated mean rain rate. From the median values, one can deduce that GPCP 1DD typically has the following performance: NBIAS = 1.45, MAD = 48%, RMSD = 130%, RMSDs = 110%, and RMSDu = 74% of the MRB 1DD–estimated mean.

Most of the statistics that are presented thus far fall into the category of unconditional statistics. However, the regression parameters displayed in Figs. 3d and 3e and the time series plot in Fig. 5 indicate that the discrepancy between the two rainfall datasets depends on the magnitude of the rain rate. In Fig. 7 we show the GPCP 1DD–MRB 1DD difference as a function of the MRB 1DD–estimated rain rate, by pooling all 1DD rain-rate estimates for the locations in region 3. The figure clearly shows that the mean and the variance of the GPCP 1DD–MRB 1DD difference increase with increasing rain rate. Therefore, to properly quantify the error we need to compute the error statistics that are conditional on rain rate. Taking the number of samples into account, we divided the MRB 1DD–estimated rain rate into seven categories {[0], (0–0.5], (0.5–1.0], (1.0–2.0], (2.0–4.0], (4.0–10.0], and above 10.0 mm day−1}. For each rain rate category, we calculated the GPCP 1DD–MRB 1DD difference statistics in terms of mean, variance, and a number of quantiles (Table 1). We further show the MRB 1DD–estimated mean rain rate and the sample size for each category. It is evident that the distribution of the difference varies from one category to another. Given the occurrence of rain (according to the MRB 1DD), the GPCP 1DD overestimates in each category that is considered. Both the mean error (i.e., the difference between the mean of the GPCP 1DD and the mean of the MRB 1DD) and the RMSD consistently increase with increasing rain rate. The mean error varies from 0.09 to 7.21 mm day−1 at the MRB 1DD–estimated mean of 0.10 and 17.26 mm day−1, respectively. The standard deviation of the GPCP 1DD–MRB 1DD difference varies from 0.22 to 5.87 mm day−1 at the MRB 1DD–estimated mean of 0.10 and 17.26 mm day−1, respectively. Note that because of the relationship MSD = E[(RGE[RG])2] + E[RGRM]2, the standard deviation of the GPCP 1DD–MRB 1DD difference is sometimes called the bias-adjusted RMSD.

In Fig. 8 we show the variation of the mean error RMSD, and the bias-adjusted RMSD as a function of MRB 1DD–estimated rain rate. Overall, these measures increase with increasing rain rate. There are a couple of obvious reasons for this. As mentioned in section 2, the GPCP 1DD estimate is obtained by multiplying the microwave-adjusted rain rate by the percentage of the pixels with an IR temperature that is colder than the threshold temperature. Any error in the conditional rain-rate estimate results in absolute errors, which increase with increasing rain rate. Another reason involves the sampling error, which is a significant portion of the total error in 1DD estimates, as shown in section 4. Several investigators have concluded that the sampling error (in absolute units) tends to increase with increasing rain rate (e.g., Bell and Kundu 1996; Huffman 1997; Chang and Chiu 1999; Gebremichael and Krajewski 2004b). In general, this is consistent with the fact that the absolute estimation errors tend to be greater for large values of measured positively defined variables than for small values.

An interesting issue worthy of investigation is whether the error behavior can be captured in some compact expression that depends solely on large-scale observables like mean rain rate. Using the data points plotted in Fig. 8, we examined in particular whether the error measures obey the scaling law with respect to the mean rain rate. We applied the Durbin– Watson test, which is a measure of the randomness in the residuals (Neter and Wasserman 1974). An absence of randomness indicates a failure of the scaling law. We used the significance level of 0.05 in the statistical testing and found that RMSD obeys the scaling law, RMSD = 1.14 R0.73m, where Rm is the MRB 1DD–estimated mean rain rate in millimeters per day. The fit statistics are R2 = 0.998, rms error = 0.193, Durbin–Watson statistic = 1.428, and a sample size of 16.

Next, we considered the correspondence between the GPCP 1DD and MRB 1DD–estimated occurrence of events. In Fig. 9 we present the spatial distribution of the categorical statistics in terms of POD, FAR, and HSS using a threshold of 0.1 mm day−1. The GPCP 1DD detects more than 95% of the events reported by the MRB 1DD, with a FAR of less than 20% and a skill score of more than 80%. Overall the GPCP 1DD shows great accuracy in detecting rain above a threshold of 0.1 mm day−1. However, the quantitative usefulness of such a comparison is affected by the sensitivity of the results to the threshold rain rate. In Fig. 10 we use all of the 1DD estimates for locations in region 3 to examine how the rain-rate threshold, over the range of 0.1–16 mm day−1, affects the categorical statistics. It is apparent that the POD remains fairly insensitive to the threshold rate, whereas the FAR increases and the HSS decreases with increasing threshold. Even at a threshold of 4 mm day−1 (which is equivalent to the 85% quantile of the MRB 1DD), however, the GPCP 1DD exhibits good skill: POD = 98%, FAR = 19%, and HSS = 86%. This is highly consistent with Fig. 5. The GPCP 1DD identifies virtually all of the rain events, but is systematically too high at the high end creating the tendency for events to be classified as higher intensity than they actually are.

Seasonal analysis

Thus far our analyses were based on the entire dataset. We also performed the same calculations after segregating the data by season. Using all of the grids located in region 3, we obtained the MRB 1DD–estimated means as 1.53 during DJF, 2.50 during MAM, 2.49 during JJA, and 1.89 during SON, where all units are in millimeters per day. In Fig. 11 we present the GPCP 1DD evaluation statistics as a function of season. In terms of the absolute difference measures (RMSD and MAD), we found the values for MAM and JJA to be higher than those for DJF and SON. This can be explained by the dependence of these measures on the mean rain rate (see Fig. 8). However, a comparison of DJF and SON shows larger RMSD and MAD values for DJF, even though it has a smaller mean rain rate. When these measures are normalized by the mean rain rate, the largest NRMSD occurs for DJF. In terms of a similar measure, NMAD, however, the seasonal differences are small. This suggests that DJF experiences more outliers in which the GPCP 1DD shows large errors with respect to the MRB 1DD. With respect to NBIAS, d, POD, and HSS, the seasonal differences are small. However, DJF produces more false alarms than the other seasons.

In conclusion, the absolute difference measures show seasonal dependence, which can be attributed to the seasonally varying amounts of rainfall. DJF is an exception where the seasonal dependence cannot entirely be explained by the rainfall amount. In terms of the relative difference measure (NRMSD) and the categorical statistics (FAR), the largest values occur for DJF while the other seasons experience similar values. The rest of the statistics show smaller seasonal variations. Overall, one can conclude that the discrepancies between the GPCP 1DD and the MRB 1DD are larger for DJF (winter), while they remain more or less the same for the other seasons. Nelson et al. (2003b) stated that the MRB 1DD estimates are expected to perform least effectively during snowfall events because of the problems associated with deriving the equivalent rain rate. Therefore, the large discrepancy observed during winter between the GPCP 1DD and the MRB 1DD estimates could arise as a result of the limitation in either or both of the estimates.

Region 1 versus region 2

As discussed in section 2, the types of satellite data used to obtain the GPCP 1DD over region 1 (above 40°N) are different from those over region 2 (below 40°N). Over region 2, the GPCP 1DD uses IR estimates from geostationary satellites and MW estimates from polar-orbiting satellites; over region 1, the GPCP 1DD uses TOVS data and MW estimates both from polar-orbiting satellites. In Fig. 12 we compare how the performance of the GPCP 1DD over region 1 differs from that over region 2. The GPCP 1DD–estimated means are 1.01 and 1.83 mm day−1, for regions 1 and 2, respectively. Higher RMAD and MAD values are obtained for region 2, which can be explained by its larger rainfall accumulation. In terms of the normalized difference measures, the index of agreement d, and the categorical statistics, GPCP 1DD performs almost the same over both regions. However, region 2 has a 10% greater bias than region 1. It is not clear at this moment whether the higher bias in region 2 is attributed to the limitation in the GPCP 1DD algorithm or to the limitation in the MRB dataset algorithm.

Error decomposition

Error decomposition framework

It is important to indicate that the difference between the MRB 1DD and the GPCP 1DD estimates is the result of errors in both of these estimates. However, the underlying premise of this study is that the MRB estimates are of high quality (see section 2a), and, hence, the bulk portion of the difference statistics could be attributed to the GPCP 1DD errors. For completeness, in this section we discuss a statistical framework for describing the partitioning of the error into the GPCP 1DD error and the MRB 1DD error components. This framework also provides room to further improve the GPCP 1DD error estimates as more validation statistics are obtained regarding the performance of the MRB 1DD estimates. In the latter part of this section, we decompose the GPCP 1DD error into its components—the sampling and retrieval error components—and perform an extensive analysis to estimate the sampling error.

Let RT be the unknown true daily average rain rate on a 1° × 1° rain rate. We write the error in each estimate as
i1520-0450-44-5-665-e9a
i1520-0450-44-5-665-e9b
where εM and εG represent the error associated with the MRB 1DD and the GPCP 1DD estimates, respectively. The difference between the two estimates is
i1520-0450-44-5-665-e10
The distribution of the quantity Δ is shown in Table 1 in terms of mean, standard deviation, and quantiles. The error εG results from two sources, sampling error εG,S and retrieval error εG,A, which includes both algorithm and instrument errors.
The bias in the GPCP 1DD estimate can be written as
i1520-0450-44-5-665-e11
where E[] denotes the expected value over realizations of the dataset. If the MRB 1DD estimates are unbiased, E[εG] should be the same as E[Δ]. Generally, E[εG] could be higher, lower, or the same as E[Δ], depending on the magnitude of E[εM]. Morrissey and Janowiak (1996) found that for a 3-hourly sampling scheme the bias due to sampling is close to zero. Hence, we may write E[εG] ≈ E[εG,A].
The GPCP 1DD error variance can be written as
i1520-0450-44-5-665-e12
where V[] and C[] denote the variance and the covariance operators. Assuming that εM and εG are uncorrelated, we may write
i1520-0450-44-5-665-e13
This assumption seems plausible because the sensors involved in the GPCP 1DD and MRB 1DD estimates have quite different physical bases. Equation (13) provides the upper bound for the GPCP 1DD error variance. The variance V[εG] should be smaller than V[Δ], the square root of which is shown in Table 1. The difference between V[εG] and V[Δ] is controlled by V[εM], and the larger V[εM] is, the smaller V[εG] will be.
Following Bell and Kundu (2000), we assumed that εG,S and εG,A are uncorrelated. Therefore, we may write
i1520-0450-44-5-665-e14

Sampling error

In this section we estimate V[εG,S] based exclusively on the MRB datasets. We performed data-based resampling experiments to quantify the sampling error. The principle is that the resampling experiments deliver an ensemble of imaginary satellite overpasses over a real dataset to assess the magnitude of error that can be expected in the real satellite overpasses. Resampling is a popular technique used to estimate the sampling error (McConnell and North 1987; Seed and Austin 1990; Cosgrove and Garstang 1995; Soman et al. 1995; Li et al. 1996; Steiner 1996). The basic assumption in this method is that the satellite overpasses are at regular time intervals and make flush (100% coverage at each overpass) visits. At short sampling intervals and small spatial scales, the error that results from this assumption is small. One main difference between our study and those mentioned above is that in this study we use the so-called moving-block-bootstrapping (MBB) resampling technique. As discussed by Bell and Kundu (2000), this technique avoids the problems that are inherent in the resampling techniques used in many other studies. Using the same dataset, Gebremichael and Krajewski (2004b) applied this technique to estimate the sampling error at a variety of satellite sampling intervals, space–time scales, and mean rain rates. We briefly describe the MBB method and present the results below.

As mentioned, the rainfall data are available at 15-min and 4 km × 4 km resolution. These data averaged over area A at time ti are assumed to represent the true 15-min areal rainfall SA(ti). Consider the set RA = {SA(t1), SA(t2), . . . , SA(tn)} in which SA(t1) refers to the true areal rainfall at the first 15-min time period, SA(t2) at the second 15-min time period, and so on, and n refers to the total number of 15-min time periods within the averaging time period T. For T = 1 day, n = 96. If we use all of the elements of RA, we obtain the true T period areal rainfall RAT. If we use only a subset of RA, as per the desired sampling interval Δt, we obtain AT, an estimate of RAT. In our case, T = 1 day, Δt = 3 h, and A = 1° × 1°. We applied the MBB resampling technique to obtain several realizations of {ATRAT}. The GPCP 1DD sampling error variance V[εG,S] is then simply the variance computed from the ensembles of {ATRAT}.

The MBB requires the selection of block length l and the number of bootstrap samples m. Choices for l in applied work have been ad hoc and qualitative (Leger et al. 1992). In this study, we used l = 24 h where a block has to begin at the same hour of the day. That is, if “block 1” starts at 1400 LT on day d1, then “block 2” and all succeeding blocks must start at 1400 LT on the randomly selected days. Because this value of l is also more than twice the e-folding correlation time (for all the spatial scales considered in this study), the correlation between blocks becomes negligible as desired. Efron (1990) argued that 50–200 bootstrap samples are sufficient to estimate the bias and the variance of a statistic. On the other hand, 1000–2000 bootstrap samples are required to estimate a quantile to construct bootstrap confidence intervals. In this study, we used a conservative value of m = 5000.

In Table 2, we presented the resulting V[εG,S] values estimated this way, along with other error statistics at the seasonal time scale. Respectively, V[εG,S] amounts to 89%, 71%, 61%, and 71% of the mean during DJF, MAM, JJA, and SON. Depending on the season V[εG,S] is shown to account for 28%–37% of V[Δ]. The last column of Table 2 shows the values for V[εG,A] + V[εM], which could be considered the upper bound for V[εG,A]. Respectively, V[εG,A] + V[εM] amounts to 157%, 92%, 99%, and 100% of the mean during DJF, MAM, JJA, and SON.

Conclusions and recommendations

In this study we have evaluated the GPCP 1DD for the period of January 1997–December 2000 using radar-based rainfall estimates over the Mississippi River basin that covers 435 1° × 1° grids. We have presented the evaluation results in terms of several statistics that fall into the category of unconditional continuous statistics, conditional (quasi) continuous statistics, and categorical statistics. The analyses have included annual and seasonal time periods. We have presented spatial maps showing the results for the 1° × 1° grids, and a summary of the results for three selected regions. Results of this study reveal the following:

  1. The GPCP 1DD estimates capture the spatial and temporal variability of the MRB 1DD estimates to a high degree, with more than 80% of the variance in the MRB estimates being explained. The index of agreement d and R2 values reported in this study are significantly higher than those reported in most of the previous evaluation studies.
  2. The GPCP 1DD has a very high skill over a large range of rainfall thresholds at distinguishing between rainy and nonrainy days.
  3. In over 90% of the locations in region 3, the GPCP 1DD overestimates rainfall by 25%–73% compared to the MRB 1DD, with the majority overestimating by 45%. Further work is required to investigate the source of this bias. The RMSD (MAD) varies in the range of 85%–208% (32%–75%) of the mean rain rate, with the typical estimate being 130% (48%) of the mean. Typically, RMSDs amounts to 110% of the mean, whereas RMSDu amounts to 74% of the mean.
  4. Analysis of the distribution of the GPCP 1DD–MRB 1DD difference at different rain-rate categories has shown that both the mean error and the RMSD consistently increase with increasing rain rate. The RMSD obeys the scaling law with respect to the MRB 1DD–estimated mean rain rate, RMSD = 1.14R0.73m, where all of the units are in millimeters per day.
  5. In terms of bias, the SSM/I–TOVS algorithms (region 2) result in 10% more bias than the SSM/I–geostationary IR algorithms (region 1). However, in terms of all other statistics, the performance statistics of both algorithms are similar.

So far we have summarized the results based on the GPCP 1DD–MRB 1DD difference statistics. The premise of our study is that the MRB 1DD estimates are of high quality, and, hence, the bulk of the difference statistics is attributed to the GPCP 1DD error statistics. For completeness, we have also presented a statistical framework that partitions the GPCP 1DD–MRB 1DD difference statistics into GPCP 1DD error and MRB 1DD error statistics. We have further partitioned the GPCP 1DD error statistics into sampling error and retrieval error statistics, and we have estimated the sampling error statistics using data-based resampling experiments. Further research is required to assess the impact of the assumptions involved in our error-partitioning framework. Key results pertaining to error partitioning are as follows:

  1. In general, the bias in the GPCP 1DD estimates could be higher, lower, or the same as the bias of the GPCP 1DD–MRB 1DD difference, depending on the magnitude of the bias in the MRB estimates. Initial validation studies carried out by Nelson et al. (2003b) showed that the MRB estimates are unbiased, which would suggest that the mean of the difference could be treated as the bias in the GPCP 1DD products. However, additional studies over large representative areas are required to draw conclusions regarding the bias in the MRB 1DD products.
  2. The GPCP 1DD error variance is always smaller than the difference variance. The difference between the GPCP 1DD error variance and the difference variance is controlled by the MRB 1DD error variance—the larger the MRB 1DD error variance is, the smaller will the GPCP 1DD error variance be.
  3. The GPCP 1DD sampling error variance amounts to 28%–37% of the difference variance, depending on the season. Subtracting the sampling error variance, we have shown results for the sum of the GPCP 1DD retrieval error variance and the MRB 1DD error variance. This sum can also be considered as the upper bound for the GPCP 1DD retrieval error variance.

Our results have further shown that there is no one statistic that summarizes the performance of the GPCP 1DD estimates. Rather, a set of statistics is necessary to adequately describe these rainfall products. Agreement and categorical statistics indicate that the GPCP 1DD agrees well with the MRB 1DD. However, other statistics such as the bias and RMSD indicate that there is a significant difference between the two estimates. We have also shown some cases where similar statistics produce quite different values, as in the cases of d versus R2, and MAD versus RMSD. Part of the discrepancy in these values is attributed to the lack of robustness that is associated with some of these statistics. Following this, we recommend the use of a set of validation statistics in evaluating rainfall products. To facilitate comparison of results among studies and to enhance the usability of the results, we see the need to establish a consistent set of evaluation measures that should be undertaken by researchers. Attempts have been made to accompany rainfall products with corresponding random error estimates, with the latter summarized in terms of root-mean-square error. However, our results show that RMSD results are sensitive to outliers and consequently may fail to properly summarize the random error component. The possibility of employing other statistics such as the MAD needs to be considered, although this is easier said than done.

Last, we point out the need for similar validation studies in different climatic regimes. More information will be obtained if the comparison statistics are partitioned into the GPCP 1DD error and the MRB 1DD error components. Statistical error decomposition frameworks presented in this study and in a similar other study by Gebremichael et al. (2003), depending on whether the reference dataset is radar or gauge based, should be used for this purpose.

Acknowledgments

This research was supported by the NOAA Office of Global Programs through Grant NA57WHO517 to the second author, by NASA through Grants NAG5-4755 and NASA NAG5-9664, and the NASA Earth System Science Fellowship to the first author. The second author also acknowledges the support of the Rose and Joseph Summers professorship.

REFERENCES

  • Adeyewa, Z. D., and K. Nakamura. 2003. Validation of TRMM radar rainfall data over major climatic regions in Africa. J. Appl. Meteor. 42:331347.

    • Search Google Scholar
    • Export Citation
  • Arkin, P. A., and B. N. Meisner. 1987. The relationship between large-scale convective rainfall and cold cloud over the Western Hemisphere during 1982–84. Mon. Wea. Rev. 115:5174.

    • Search Google Scholar
    • Export Citation
  • Austin, P. M. 1987. Relation between measured radar reflectivity and surface rainfall. Mon. Wea. Rev. 115:10531069.

  • Bell, T. L., and P. K. Kundu. 1996. A study of the sampling error in satellite rainfall estimates using optimal averaging of data and a stochastic model. J. Climate 9:12511268.

    • Search Google Scholar
    • Export Citation
  • Bell, T. L., and P. K. Kundu. 2000. Dependence of satellite sampling error on monthly averaged rain rates: Comparison of simple models and recent studies. J. Climate 13:449462.

    • Search Google Scholar
    • Export Citation
  • Chang, A. T. C., and L. S. Chiu. 1999. Nonsystematic errors of monthly oceanic rainfall derived from SSM/I. Mon. Wea. Rev. 127:16301638.

    • Search Google Scholar
    • Export Citation
  • Cosgrove, C. M., and M. Garstang. 1995. Simulation of rain events from rain gauge measurements. Int. J. Climatol. 15:10211029.

  • Efron, B. 1990. More efficient bootstrap computations. J. Amer. Stat. Assoc. 85:7989.

  • Gebremichael, M., and W. F. Krajewski. 2004a. Assessment of the statistical characterization of small-scale rainfall variability from radar: Analysis of TRMM ground validation datasets. J. Appl. Meteor. 43:11801199.

    • Search Google Scholar
    • Export Citation
  • Gebremichael, M., and W. F. Krajewski. 2004b. Characterization of the temporal sampling error in space-time-averaged rainfall estimates from satellites. J. Geophys. Res. 109.D11110, doi:10.1029/2004JD004509.

    • Search Google Scholar
    • Export Citation
  • Gebremichael, M., , W. F. Krajewski, , M. Morrissey, , D. Langerud, , G. J. Huffman, , and R. Adler. 2003. Error uncertainty analysis of GPCP monthly rainfall products: A data-based simulation study. J. Appl. Meteor. 42:18371848.

    • Search Google Scholar
    • Export Citation
  • Huffman, G. J. 1997. Estimates of root-mean-square random error for finite samples of estimated precipitation. J. Appl. Meteor. 36:11911201.

    • Search Google Scholar
    • Export Citation
  • Huffman, G. J. Coauthors 1997. The Global Precipitation Climatology Project (GPCP) combined precipitation dataset. Bull. Amer. Meteor. Soc. 78:520.

    • Search Google Scholar
    • Export Citation
  • Huffman, G. J., , R. F. Adler, , M. M. Morrissey, , D. T. Bolvin, , S. Curtis, , R. Joyce, , B. McGavock, , and J. Susskind. 2001. Global precipitation at one-degree daily resolution from multisatellite observations. J. Hydrometeor. 2:3650.

    • Search Google Scholar
    • Export Citation
  • Joss, J., and A. Waldvogel. 1990. Precipitation measurement and hydrology. Radar in Meteorology, Battan Memorial 40th Anniversary Radar Meteorology Conference, D. Atlas, Ed., Amer. Meteor. Soc., 577–606.

  • Joss, J., and R. Lee. 1995. The application of radar-gauge comparisons to operational precipitation profile corrections. J. Appl. Meteor. 34:26122630.

    • Search Google Scholar
    • Export Citation
  • Kessler, E., and B. Neas. 1994. On correlation, with applications to the radar and raingage measurement of rainfall. Atmos. Res. 34:217229.

    • Search Google Scholar
    • Export Citation
  • Krajewski, W. F., and J. A. Smith. 2002. Radar hydrology: Rainfall estimation. Adv. Water Resour. 25:13871394.

  • Krajewski, W. F., , G. J. Ciach, , J. R. McCollum, , and C. Bacotiu. 2000. Initial validation of the Global Precipitation Climatology Project over the United States. J. Appl. Meteor. 39:10711086.

    • Search Google Scholar
    • Export Citation
  • Kummerow, C., , W. S. Olsen, , and L. Giglio. 1996. A simplified scheme for obtaining precipitation and vertical hydrometeor profiles from passive microwave sensors. IEEE Trans. Geosci. Remote Sens. 34:12131232.

    • Search Google Scholar
    • Export Citation
  • Legates, D. R., and G. J. McCabe. 1999. Evaluating the use of the “goodness-of-fit” measure in hydrologic and hydroclimatic model validation. Water Resour. Res. 35:233241.

    • Search Google Scholar
    • Export Citation
  • Leger, C., , D. N. Politis, , and J. P. Romano. 1992. Bootstrap technology and applications. Technometrics 34:378398.

  • Li, Q., , R. L. Bras, , and D. Veneziano. 1996. Analysis of Darwin rainfall data: Implications on sampling strategy. J. Appl. Meteor. 35:372385.

    • Search Google Scholar
    • Export Citation
  • McConnell, A., and G. R. North. 1987. Sampling errors in satellite estimates of tropical rain. J. Geophys. Res. 92:95679570.

  • Morrissey, M. L., and J. E. Janowiak. 1996. Sampling-induced conditional biases in satellite climate-scale rainfall estimates. J. Appl. Meteor. 35:372385.

    • Search Google Scholar
    • Export Citation
  • Nelson, B. R., , W. F. Krajewski, , A. Kruger, , J. A. Smith, , and M. L. Baeck. 2003a. Archival precipitation data set for the Mississippi river basin: Development of a GIS-based data browser. Comput. Geosci. 29:595604.

    • Search Google Scholar
    • Export Citation
  • Nelson, B. R., , W. F. Krajewski, , A. Kruger, , J. A. Smith, , and M. L. Baeck. 2003b. Archival precipitation data set for the Mississippi river basin: Algorithm development. J. Geophys. Res. 108.8811, doi:10.1029/2002JD003158.

    • Search Google Scholar
    • Export Citation
  • Neter, J., and W. Wasserman. 1974. Linear Statistical Methods. Richard D. Irwin, 667 pp.

  • Ramage, K., , I. Jobard, , T. Lebel, , and M. Desbois. 2000. Satellite estimation of 1-day to 10-day cumulated precipitation: Comparison of TRMM, METEOSAT and GPCP products. Proc. of the 2000 EUMETSAT Meteorological Satellite Data Users’ Conf., Bologna, Italy, EUMETSAT, 363–369.

  • Roads, J. Coauthors 2003. GCIP water and energy budget synthesis (WEBS). J. Geophys. Res. 108.8609, doi:10.1029/2002JD002583.

  • Seed, A., and G. L. Austin. 1990. Variability of summer Florida rainfall and its significance for the estimation of rainfall by gages, radar, and satellite. J. Geophys. Res. 95:22072215.

    • Search Google Scholar
    • Export Citation
  • Skomorowski, P., , F. Rubel, , and B. Rudolf. 2001. Verification of GPCP-1DD global satellite precipitation products using MAP surface observations. Phys. Chem. Earth B 26:5–6,. 403409.

    • Search Google Scholar
    • Export Citation
  • Soman, V. V., , J. B. Valdes, , and G. R. North. 1995. Satellite sampling and the diurnal cycle statistics of Darwin rainfall data. J. Appl. Meteor. 34:24812490.

    • Search Google Scholar
    • Export Citation
  • Steiner, M. 1996. Uncertainty of estimates of monthly areal rainfall for temporally sparse remote observations. Water Resour. Res. 32:373388.

    • Search Google Scholar
    • Export Citation
  • Susskind, J., , P. Piraino, , L. Iredell, , and A. Mehta. 1997. Characteristics of the TOVS pathfinder path A dataset. Bull. Amer. Meteor. Soc. 78:14491472.

    • Search Google Scholar
    • Export Citation
  • Wilks, D. S. 1995. Statistical Methods in the Atmospheric Sciences. Academic Press, 467 pp.

  • Willmott, C. J. 1981. On the validation of models. Phys. Geogr. 2:184194.

  • Willmott, C. J. 1982. Some comments on the evaluation of model performance. Bull. Amer. Meteor. Soc. 11:13091313.

  • Willmott, C. J., and D. E. Wicks. 1980. An empirical method for the spatial interpolation of monthly precipitation within California. Phys. Geogr. 1:5973.

    • Search Google Scholar
    • Export Citation
  • Xie, P., and P. A. Arkin. 1997. Global precipitation: A 17-year monthly analysis based on gauge observations, satellite estimates, and numerical model outputs. Bull. Amer. Meteor. Soc. 78:25392558.

    • Search Google Scholar
    • Export Citation

Fig. 1.
Fig. 1.

The Mississippi River basin in the United States. The basin is divided into three regions identified by the numbers.

Citation: Journal of Applied Meteorology 44, 5; 10.1175/JAM2233.1

Fig. 2.
Fig. 2.

Spatial distribution of rainfall statistics estimated from the (a) GPCP 1DD and (b) MRB 1DD mean, and (c) GPCP 1DD and (d) MRB 1DD std dev. Unshaded areas are regions outside of the Mississippi River basin. Region 3 is shown on the maps.

Citation: Journal of Applied Meteorology 44, 5; 10.1175/JAM2233.1

Fig. 3.
Fig. 3.

Spatial distribution of comparison statistics: (a) NBIAS, (b) d, and (c) R2, and the regression parameters (d) a and (e) b

Citation: Journal of Applied Meteorology 44, 5; 10.1175/JAM2233.1

Fig. 4.
Fig. 4.

Spatial distribution of difference measures, both in absolute and relative units. The mean refers to the MRB 1DD–estimated mean rain rate.

Citation: Journal of Applied Meteorology 44, 5; 10.1175/JAM2233.1

Fig. 5.
Fig. 5.

Time series of daily rainfall for the selected 1DD grid estimated from the MRB 1DD (thick line) and GPCP 1DD (thin line). The selected grid is indicated by the arrow in Fig. 3a.

Citation: Journal of Applied Meteorology 44, 5; 10.1175/JAM2233.1

Fig. 6.
Fig. 6.

Distribution of the spatial variability of the validation statistics obtained using the 100 1° grids located in region 3.

Citation: Journal of Applied Meteorology 44, 5; 10.1175/JAM2233.1

Fig. 7.
Fig. 7.

GPCP 1DD–MRB 1DD difference as a function of the MRB 1DD–estimated rain rate for region 3.

Citation: Journal of Applied Meteorology 44, 5; 10.1175/JAM2233.1

Fig. 8.
Fig. 8.

The mean error, the bias-adjusted RMSD (BA-RMSD), and the RMSD between the GPCP 1DD and the MRB 1DD as a function of the MRB 1DD–estimated rain rate.

Citation: Journal of Applied Meteorology 44, 5; 10.1175/JAM2233.1

Fig. 9.
Fig. 9.

Spatial distribution of POD, FAR, and HSS values for a threshold of 0.1 mm day−1.

Citation: Journal of Applied Meteorology 44, 5; 10.1175/JAM2233.1

Fig. 10.
Fig. 10.

Categorical statistics as a function of precipitation threshold for region 3. The statistics are POD, HSS, and FAR, from the top down. The statistics are calculated for samples equal to or greater than the threshold.

Citation: Journal of Applied Meteorology 44, 5; 10.1175/JAM2233.1

Fig. 11.
Fig. 11.

Seasonal variation of the validation statistics for the GPCP 1DD product, computed by pooling all seasonal 1DD estimates in region 3.

Citation: Journal of Applied Meteorology 44, 5; 10.1175/JAM2233.1

Fig. 12.
Fig. 12.

Regional variation of the validation statistics for the GPCP 1DD, computed by pooling all estimates in regions 2 (dark box) and 1 (lighter box).

Citation: Journal of Applied Meteorology 44, 5; 10.1175/JAM2233.1

Table 1.

Statistics of the GPCP 1DD–MRB 1DD difference (mm day−1): mean, standard deviation (std dev), and quantiles (0.10, 0.25, 0.50, 0.75, and 0.90), conditional on each of the seven MRB 1DD–derived rain intensity categories. The numbers in the parentheses indicate the mean of the MRB 1DD–derived rain rate in each category.

Table 1.
Table 2.

Seasonal variability of the GPCP 1DD sampling error variance V[εG,S], the GPCP 1DD–MRB 1DD difference variance V[Δ], and the sum of the GPCP 1DD retrieval error variance and the MRB 1DD error variance V[εG,A] + V[εM]. Also shown is the MRB 1DD–estimated mean rain rate. Units are mm2 day−2 for the variances, and mm day−1 for the mean.

Table 2.
Save