1. Introduction
Numerous studies have validated remotely sensed precipitation products (e.g., Sharifi et al. 2016; Tan et al. 2017; Wang et al. 2017; Xu et al. 2017; Wu et al. 2014) at local and regional scales. While such studies provide great insight to the validity of the precipitation products for respective locales, comparing different studies is challenging due to the use of different methods and validation datasets. This study attempts a novel approach to validate a remotely sensed precipitation product at a near-global scale to provide critical insights into successes and shortcomings at detecting precipitation.
Remotely sensed precipitation products are crucial for the hydrological sciences community, and are used to inform flood and drought assessment, water resource management, and many other such applications, particularly in areas where ground observations are not available. Satellite-based observational products have been available since the late 1980s with the advent of the Global Precipitation Climatology Project (GPCP; Huffman et al. 1997), with subsequent products that feature higher spatial and temporal resolutions having shown great utility, such as Precipitation Estimation from Remotely Sensed Information Using Artificial Neural Networks–Cloud Classification System (PERSIANN-CCS; Hong et al. 2004), Climate Prediction Center morphing method (CMORPH; Joyce et al. 2004), TRMM Multisatellite Precipitation Analysis (TMPA; Huffman et al. 2007), Global Satellite Mapping of Precipitation (GSMaP; Kubota et al. 2007), Climate Hazards Group Infrared Precipitation with Stations (CHIRPS; Funk et al. 2015), and the Global Precipitation Measurement (GPM) mission’s Integrated Multisatellite Retrievals for GPM (IMERG) products (Huffman et al. 2020). With the advent of the latest products, satellite-based precipitation products have been used in the application of the Global Flood Monitoring System (Hong et al. 2007; Wu et al. 2012), Landslide Hazard Assessment for Situational Awareness (Kirschbaum and Stanley 2018), as well use in the energy and agriculture sectors (e.g., Kirschbaum et al. 2017).
Generally, satellite retrievals from microwave sensors produce a radiative signature that is highly correlated with precipitation, while infrared retrievals can produce estimates at higher temporal frequency by measuring cloud-top radiance (Hou et al. 2014; Funk et al. 2015). While multisensor products like IMERG attempt to balance the higher accuracy of passive microwave with the high temporal sampling of the infrared, these products still may miss or falsely indicate precipitation. This study focuses on the IMERG products from the Global Precipitation Measurement (GPM) satellite constellation, the GPM Core Observatory satellite was launched as a partnership between NASA and JAXA in February 2014.
While previous studies have validated aspects of remotely sensed precipitation products, the methods and results are variable. Studies that use ground-based observation stations, such as rain gauge networks, tend to focus on the local-to-regional scale (e.g., Laurent et al. 1998; Maghsood et al. 2020; Pombo et al. 2015). While studies that use other forms of ground-based radar meteorology tend to focus on the regional-to-continental scale (e.g., Kim et al. 2014; Liao and Meneghini 2009; Schwaller and Robert Morris 2011; Tan et al. 2016).
In addition to the aforementioned traditional methods to validate precipitation, a relatively new approach is to use independent hydrologic observations that do not depend on radar and gauge networks to validate precipitation products. The approach of hydrologic ground validation (Hydro-GV) is a unique method to characterize the errors of remotely sensed precipitation products. Cao et al. (2018) used snow water equivalent (SWE) estimates from NASA Airborne Snow Observatory and a hydrologic model to show that the Multi-Radar/Multi-Sensor System (MRMS) underestimates the amount of precipitation needed to reproduce the observed snowpack as part of the Olympic Mountain Experiment (OLYMPEX) in the Olympic Mountains and Chehalis River basin. Similarly, Henn et al. (2018) demonstrated a Hydro-GV method that uses a hydrologic land surface model to determine whether precipitation products can reproduce observed streamflow in well observed catchments in the Sierra Nevada, ultimately finding that there is less observed precipitation than what is needed to replicate observed streamflow. Wu et al. (2017) produced a similar study in the Iowa–Cedar River basin comparing nine precipitation products ability to replicate observed streamflow finding that biases in streamflow is strongly correlated to biases in precipitation. A conceptually similar approach by Brocca et al. (2013) used observed soil moisture time series at three sites in Europe to estimate precipitation with a water budget, finding that the method reproduced precipitation accumulation satisfactorily, with daily estimates of precipitation having correlations and Nash–Sutcliffe efficiencies greater than 0.7 at a majority of the test sites. In this work, we seek to avoid the use of a numerical model to reduce the level of uncertainty introduced into this study, and therefore we focus on precipitation occurrence rather than magnitude. These novel Hydro-GV approaches provide methods for validation that can extend validation from regional to global, which is beyond the traditional realm of gauge- and radar-based techniques.
This study aims to determine whether changes in satellite-based soil moisture observations can be used as a proxy for the presence of precipitation, and if such a conceptual framework can be applied to globally validate IMERG products. Using IMERG in concert with the Soil Moisture Active Passive (SMAP) mission (Entekhabi et al. 2010) and MRMS (Zhang et al. 2016), this study demonstrates a continental- and global-scale validation of the IMERG products using soil moisture evolution as a surrogate for precipitation detection. Section 2 presents an overview of the method and data used. Section 3 highlights the results of our selected skill metrics (e.g., POD and FAR), followed by a discussion of the results in section 4.
2. Methods
This study carries out a retrospective CONUS-scale validation of the IMERG products with SMAP (Entekhabi et al. 2010) and MRMS using a categorical skill method to characterize errors, with a subsequent global validation using SMAP. The first step is to evaluate IMERG over CONUS with a high-resolution, radar-based precipitation product, MRMS. Following a successful validation of the method, our method is implemented to validate IMERG using SMAP on an overpass-to-overpass basis. After confirming that the IMERG validations with MRMS and SMAP produce comparable results, an expansion of the domain to produce a near-global-scale validation is presented. The goal of this work is to develop a conceptual approach for a global-scale analysis; however, there are areas where the validation technique cannot be applied, such as oceans, inland freshwater, snow/ice covered regions, dense forest regions, and high-latitude regions. While the method does have a large global-scale footprint—approximately 40% of global land areas are suitable for the application of this method—this near global-scale method does not provide a full global validation. Of the land areas that are suitable for this validation method, the global agricultural regions are largely encompassed by our domain, a critical region for IMERG applications.
a. Data
IMERG is a gridded precipitation dataset that merges and interpolates satellite microwave precipitation estimates with an observed gauge adjustment (Huffman et al. 2020). There are three products produced by the IMERG algorithm: two near-real-time products, “Early” and “Late,” that are produced with a climatological gauge adjustment; and a “Final” product that uses a monthly gauge adjustment to further reduce bias. The IMERG products are available on a 0.1° resolution grid that spans from ±60° latitude, with a temporal resolution as high as 30 min. While this study analyzes the precipitation estimates from all three IMERG V06B products, the results will primarily focus on the Early product as this product is most relevant for applications such as flood and landslide forecasting.
SMAP provides morning and evening [i.e., 0600 and 1800 local time (LT)] estimates of near-surface (i.e., upper 50 mm) volumetric soil moisture with near-global coverage every 1–3 days (Tan et al. 2017) using passive microwave brightness temperatures. This study uses the enhanced product that interpolates the native 36-km resolution product to a 9-km product via the Backus–Gilbert algorithm (Tan et al. 2017) and meets the stated mission goal of an unbiased root-mean-square error of 0.04 mm3 mm−3. This study uses the 0600 LT overpasses, as the 0600 LT overpasses are more accurate than the 1800 LT overpasses (Tan et al. 2017). This study only considers overpass intervals that are 5 days or less. Beyond that timeframe, noise from multiple weather events could lead to errors in detection of precipitation. Additionally, we adhere to the quality control flags produced with the SMAP data to mask out any regions and overpasses that have been flagged as being poor quality. SMAP quality control flags can be due to dense vegetation or terrain that makes sensing the surface difficult, frozen soil which has a similar dielectric constant to dry soil, and static or transient water that lowers brightness temperature (Entekhabi et al. 2010; O’Neill et al. 2020). Please note that an overpass interval is defined as the interval between valid SMAP overpasses on a specific point on the surface, while there should be a relatively consistent time discretization at a given location, invalid overpasses can lead to longer overpass time periods than expected at a given location.
MRMS uses 180 operational weather radars across the United States and Canada to create a continental-scale radar mosaic (Zhang et al. 2016). The radar portion of MRMS consists of S-band dual-polarization Weather Surveillance Radar-1988 Doppler (WSR-88D) radars and C-band single-polarization weather radars. Additionally, MRMS integrates approximately 7000 hourly rain gauge estimates to correct biases in radar precipitation estimates. MRMS has a high spatial (1 km) and temporal (2 min) resolution, making the product particularly useful for validating independent precipitation estimates. To match the temporal resolution of SMAP, the MRMS data are aggregated up to daily precipitation totals.
Due to the different temporal domains for each product, we conduct our CONUS-scale analysis between IMERG and MRMS for the years 2017–19 and the global-scale analysis with IMERG and SMAP for the years 2016–19.
To match the differing spatial domains, SMAP and MRMS are remapped to the IMERG grids. SMAP uses a nearest neighbor interpolation to replicate the IMERG gridding. While the MRMS data are screened to remove any data with a radar QPE quality index (RQI) less than 20, then conservatively remapped to keep MRMS precipitation totals constant from the native grid to the desired IMERG grid.
b. Validation technique
A contingency table is initially applied for the categorical classification of presence and absence of IMERG precipitation, see Table 1. For comparison with SMAP, an increase (decrease) in soil moisture from one overpass to the next with the presence (absence) of IMERG precipitation during that timeframe denotes a hit (correct rejection). Conversely, a soil moisture decrease (increase) with the presence (absence) of precipitation can be categorized as a false alarm (miss). When comparing MRMS with IMERG over CONUS, the SMAP increase (decrease) is changed to the presence (absence) of precipitation in MRMS.
Contingency table design for using SMAP to classify IMERG precipitation.
3. Proof of concept at Walnut Gulch
As a proof of concept to verify that SMAP can be used and effect measure of soil moisture, the validation method is applied at the SMAP core validation site (CVS) of Walnut Gulch. The Walnut Gulch CVS is located in Arizona with a semiarid climate and exhibits vast elevational change. Using satellite-based estimates of soil moisture and precipitation inherently has a point-to-gridbox issue when comparing to ground-based in situ observations. To take this impact of scale into account, we will use the Walnut Gulch CVS to provide potential insights as to how scale impacts our validation method. We take a multistep approach that includes the following:
-
Compare soil moisture changes for overpass intervals from SMAP estimates to in-situ observations at the 30 sites located within Walnut Gulch.
-
Apply the validation method to assess POD and FAR at the 30 sites located within Walnut Gulch using in situ precipitation.
-
Assess POD and FAR at the 30 sites located within Walnut Gulch using IMERG precipitation.
-
Assess POD and FAR of the SMAP grid box encompassing Walnut Gulch using IMERG precipitation.
This multistep approach will allow us to assess the skill at both landscape scale of SMAP and the point scale of in situ observations to determine if the method can produce comparable skill regardless of scale.
We acknowledge that some precipitation events will not be captured by SMAP; the same can be said for in situ observations of soil moisture (Fig. 1). Examining an excerpt of the observations in August and September of 2016, we see that smaller precipitation events are not captured by SMAP or in situ observation, but larger precipitation events do have a detectable signal with VSM.
(top) SMAP observations (red) and IMERG observations (green) at the Walnut Gulch CVS. (middle) In situ soil moisture observations (gray) and mean in situ precipitation observations (purple) at the Walnut Gulch CVS. (bottom left) Fraction of the precipitation gauges that observed precipitation during times of soil moisture increase and the associated POD with the increase of VSM. (bottom right) Mean observed precipitation during times of soil moisture increase and the associated POD with the increase of VSM.
Citation: Journal of Hydrometeorology 23, 1; 10.1175/JHM-D-21-0035.1
Comparing changes in VSM from one overpass to the next, we use in situ measurements of soil moisture across the 30 observations sites at Walnut Gulch to validate SMAP for the encompassing grid cell, we find PODs ranging from 0.390 to 0.686 and FARs ranging from 0.160 to 0.604. These results highlight two items of note: 1) SMAP observations at the landscape scale can provide results indicative of the point scale, and 2) the range of results show that there is sublandscape heterogeneity in soil moisture changes that SMAP cannot represent. These results provide confidence in SMAP’s ability to capture changes in soil moisture while highlighting that subgrid heterogeneity portends to some local uncertainty.
Locating the nearest in situ precipitation observation to each in situ soil moisture and then implementing the validation method, we find PODs from 0.224 to 0.510 and FARs ranging from 0.303 to 0.556 at each site. By substituting IMERG observations for in situ precipitation, POD ranges from 0.433 to 0.818 and FAR ranges from 0.430 to 0.604. The skill ranges produced with both in situ and IMERG precipitation overlap and demonstrate further validity of the method. Note that the SMAP interval timing is being used for the in situ observations, thus at shorter temporal scales the in situ observations should show increased skill as the effects of weather may dampen skill at the SMAP overpass timeframe.
Examining precipitation on an event-by-event basis (Fig. 1), two distinct findings are made: 1) the greater the spatial extent of the precipitation, the greater likelihood that it will be detected, and 2) the greater the precipitation volume, the greater likelihood that it will be detected. A larger spatial precipitation footprint highlights the role of subgrid heterogeneity and the ability for precipitation to be detected.
Applying our validation method of IMERG with SMAP at Walnut Gulch, we see a POD of 0.617 and a FAR of 0.404. In comparison to the skill scores produced when using solely in situ data, the SMAP–IMERG POD is greater than all 30 observations site and the FAR is better than 25 of the sites. We hypothesize that this increased skill using the remotely sensed products is likely due to the spatial heterogeneity of precipitation within SMAP and IMERG grid cells that can be captured by SMAP and IMERG but is missed by point-scale measurements.
Ultimately, this proof of concept leads to greater confidence that SMAP can provide a robust validation at the landscape scale, while highlight the potential uncertainties that can be ascribed to subgridcell processes.
4. Evaluation over CONUS: Results and discussion
Following the approach described in the methods, we first discuss the results of the IMERG–MRMS comparison for CONUS, then IMERG–SMAP comparisons over both CONUS and global domains.
a. CONUS IMERG validation with MRMS
As seen in Figs. 2a and 2b, the spatial coverage of valid MRMS sensing regions is primarily limited to the southwestern United States, central United States, and the Great Lakes region. It is of note that regions masked out due to SMAP QC flags have also been masked out for the IMERG–MRMS comparison, leading to less valid data in the southeastern United States.
Validation of IMERG Early with (a),(b) MRMS and (c),(d) SMAP; skill metrics depicted are (left) POD and (right) FAR. Areas with no color are masked out due to lack of data coverage or SMAP quality control flags.
Citation: Journal of Hydrometeorology 23, 1; 10.1175/JHM-D-21-0035.1
Comparing IMERG and MRMS precipitation on the same temporal discretization as the SMAP overpass timing that is used in the IMERG–SMAP comparisons, we find a CONUS mean POD of 0.72, with only a slight degradation in skill over the southwest United States testing FAR, we find a CONUS mean of 0.22, with the region of highest FAR being in the southwest United States again.
Modifying the technique to compare results on a daily basis, rather than between SMAP overpass timing, we see POD and FAR of 0.62 and 0.31, respectively. When altering the time discretization to daily there is a slight decrease in skill when compared to the POD and FAR results when using SMAP temporal discretization; although we experience diminished skill, IMERG still performs well enough to warrant confidence moving forward. This is supported by a previous IMERG–MRMS validation by Tan et al. (2017), that found a POD and FAR of 0.67 and 0.47, respectively. The skill metric values are of a similar scale but reflect different spatial and temporal domains for the respective analyses.
b. CONUS IMERG validation with SMAP
For ease of description and brevity, this study primarily focuses on the results using the IMERG Early product. Although not shown, the analysis was also carried out using the IMERG Late and IMERG Final products, these two products produce very similar results to IMERG Early and therefore do not warrant a separate discussion.
Figures 2c and 2d depict the POD and FAR results using SMAP to validate IMERG Early for CONUS. Averaging over the domain of valid MRMS coverage, there is a CONUS mean of 0.59 for POD and 0.48 for FAR; expanding the domain to the entire CONUS-scale yields similar areal means for the skill metrics. While the POD results follow a similar spatial pattern of skill as the IMERG–MRMS results do—i.e., higher skill in the central United States and decreased skill in the southwestern United States—the FAR results show very few similarities to the previous results shown. We discuss some potential causes for this below.
c. The role of RQI in assessing validation
When comparing the two validations over CONUS, it is worth discussing the role MRMS RQI and the potential impact of skill in regard to validation. RQI is an index to assess errors of beam blockages and beam spreading/ascending with distance from the beam source (Zhang et al. 2016). Notably, these issues arise in areas of complex topography (see lack of coverage over mountainous regions in Fig. 3a) and at large distances. In an assessment of RQI coverage (Fig. 3b), we find that 63% of our domain (areas masked by MRMS and SMAP coverage) has an RQI greater than 80 and less than 13% of our domain has an RQI less than 60.
(a) Map of RQI coverage for validation domain. RQI values are binned by tens with RQI values of 100 being a single bin. (b) Percent coverage of the domain for each RQI bin. (c) POD skill scores for each RQI bin for IMERG–MRMS validation (dark blue line with circles) and IMERG–SMAP validation (red line with squares). (d) FAR skill scores for each RQI bin for IMERG–MRMS validation (dark blue line with circles) and IMERG–SMAP validation (red line with squares).
Citation: Journal of Hydrometeorology 23, 1; 10.1175/JHM-D-21-0035.1
While the premise of RQI was to assess errors for precipitation quantity, when assessing the IMERG validation with MRMS skill scores for binned RQI values (Figs. 3c,d), there is a relatively constant POD and FAR across all bins of RQI. This provides confidence in extending the validation to levels of RQI as low as 20 due to MRMS’s ability to detect precipitation events, although there was no assessment of quantity at these lower RQI values. By extending the RQI values to this level, we enhance the spatial footprint of the validation over topographically complex regions where IMERG and SMAP both have observations in greater quantity.
Using the same binning of RQI for the IMERG validation with SMAP, there is a relatively consistent skill in POD (Fig. 3c) and FAR (Fig. 3d) across all RQI ranges. Although there is a slight degradation in POD when RQI is less than 60, the mean POD for each RQI bin does not decrease below 0.52.
Upon analysis for the dependence of skill with RQI, the results show that MRMS is capable of detecting precipitation at RQI values as low as 20 with consistent skill in terms of validating IMERG. While IMERG validation with SMAP shows diminished skill in comparison to the IMERG–MRMS results at all RQI bins, there is no dependence of skill with RQI for the IMERG–SMAP validation.
d. Shortcomings of the 2 × 2 contingency table method
Comparison of the two validation methods highlights areas of concern for validating IMERG with SMAP. Notably, two key issues are found: 1) POD in the arid southwestern United States is too low, and 2) FAR results using SMAP show decreased skill in almost all regions of CONUS. Investigation into these issues find that SMAP estimates are more volatile in arid regions and overpass intervals categorized as false alarms are dominated by light precipitation and minute changes in volumetric soil moisture (VSM).
Figure 4a shows the coefficient of variation (CV; the ratio of the standard deviation to the mean) of VSM as a function of mean VSM. The larger CV values at lower mean SMAP values highlight that the detection method is more volatile in arid regions, and there is a significant (p < 0.05) anticorrelation between the two variables.
(a) Coefficient of variation (CV) of SMAP VSM (cm3 cm−3) as a function mean SMAP VSM. (b) Gridcell median IMERG precipitation (mm) and SMAP change for false alarm overpass intervals, with histograms depicting the distributions lying outside the axes; red-dashed lines represent IMERG thresholds of 1 and 2 mm, while blue-dashed lines represent SMAP VSM change thresholds of −0.04, −0.02, and −0.01 cm3 cm−3.
Citation: Journal of Hydrometeorology 23, 1; 10.1175/JHM-D-21-0035.1
Examining all overpass intervals that result in a false alarm (Fig. 4b), a clearer picture becomes more evident as to why this technique gives such poor performance. Of the false alarm overpasses, 92% of the domain sees a median precipitation value of less than 1 mm, while 51% of overpass intervals deemed a false alarm have a median decrease in VSM that has a magnitude less than 0.02 cm3 cm−3. From comparing changes in VSM from SMAP and IMERG precipitation for false alarm overpasses, it is clear that most false alarm overpasses have light precipitation and declines in VSM are within the SMAP uncertainty range.
Inherently, the over categorization of false alarms makes physical sense. For example, light precipitation could occur shortly after a SMAP overpass and wet the soil, before the next SMAP overpass, evaporation could erase this wetting signal and show a decline in VSM by the time the next SMAP overpass occurs. It is of note that this could be an issue particularly related to SMAP, as Shellito et al. (2016) notes that SMAP drying rates following precipitation are twice the rate observed with in situ measurements.
Following an analysis of the results and an investigation into two key issues that were found, it is apparent that an alteration of the method is needed to gain enough confidence in the method for CONUS to expand to a global-scale domain. To account for noise in the SMAP observations, particularly in arid regions, and the potential effects of weather that can erase wetting signals from light precipitation events, we introduce a 3 × 3 contingency table method to extract increased signal from the observational data.
e. Improved technique: A 3 × 3 contingency table method
Since trace amounts of precipitation and decreases in SMAP VSM that are within the bounds of uncertainty are driving the poor performance, we propose a different method to potentially extract more signal than our original method. To address this issue, we employed a 3 × 3 contingency table (Table 2) to reduce the amount of noise in our skill scores.
A 3 × 3 contingency table designed for using SMAP to classify IMERG precipitation. Note the bolded cells (a, c, i, and k) are considered to be the cells where the signal is most present. The cells on the far right and bottom represent column (m, n, o), row (d, h, l), and table (p) sums. SMAP is units of volumetric soil moisture (cm3 cm−3) and IMERG is units of millimeters.
Following the categorization technique described by Stanski et al. (1989), we use a 3 × 3 contingency table to validate IMERG with SMAP. The method uses three forecast bins: 1) ΔSMAP < −0.01 cm3 cm−3, 2) −0.01 cm3 cm−3 ≤ ΔSMAP ≤ 0 cm3 cm−3, and 3) ΔSMAP > 0 cm3 cm−3; and three observed bins: 1) IMERG = 0 mm, 2) 0 < IMERG ≤ 1 mm, and 3) IMERG > 1 mm. The second bin in each of the respective groupings was chosen by considering the typical error estimates for both IMERG and SMAP as a way to capture the noise in the analysis. However, the size of these bins can greatly reduce the availability of data, so the trade-off between noise suppression and sample size was carefully considered.
When selecting binning, the ability to utilize as many of the overpass intervals is vital. From the analysis of false alarm overpasses previously presented (Fig. 4b), when using an IMERG threshold of 1 mm, SMAP thresholds of −0.01 and −0.02 cm3 cm−3 would capture 93% and 94% of the domain’s median false alarms components, respectively. Due to the fact that 48% of false alarm overpasses have VSM decreases between −0.01 and −0.02 cm3 cm−3, the IMERG threshold can capture the vast majority of these false alarms, thus a less restrictive VSM threshold supports the potential for more overpasses to be categorized as signal.
In both Eqs. (3) and (4), the alphabetic variables used refer to the number overpass intervals that are categorized into those respective categories in Table 2. The denominator of both Eqs. (3) and (4) can be described as the sum of overpass intervals with precipitation (l) and the sum of overpass intervals with observed drying (m), respectively. In Eq. (4) the e term can be considered “noise” as it represents the number of overpasses that experienced drying but received minimal precipitation.
f. CONUS IMERG validation with SMAP using the 3 × 3 contingency table
Figure 5 depicts the POD and FAR results using SMAP to validate IMERG Early for CONUS using the 3 × 3 contingency table method. This new method yields areal means of 0.62 for POD and 0.41 for FAR. Both skill metrics show improved consistency with the MRMS results as compared to the 2 × 2 contingency table method.
Validation of IMERG Early with SMAP using the 3 × 3 contingency table method, skill metrics depicted are (a) POD and (b) FAR. Areas with no color are masked out due to lack of data coverage or SMAP quality control flags.
Citation: Journal of Hydrometeorology 23, 1; 10.1175/JHM-D-21-0035.1
Revisiting the previous analysis to describe the performance between the validation methods, we see improvements to the validation technique by altering the method. The SMAP validation method using the 3 × 3 contingency table produces POD and FAR estimates that are more consistent with the MRMS-based POD and FAR over 15% and 8% of the domain, respectively when compared to the original 2 × 2 contingency table analysis using SMAP and IMERG Early. POD results show that 48% of the domain exhibits performance that less than 0.1 worse than the MRMS-based validation, while only 11% of the domain exhibits POD performance that is more than 0.2 worse than the MRMS-based validation. Additionally, FAR results show improvement as now 50% of the domain for the SMAP validation produces FAR values within 0.2 of the MRMS-based validation. Notably, when comparing the 2 × 2 and 3 × 3 SMAP validation results, we see that more than 80% of the domain sees increased FAR skill relative to the MRMS reference.
Furthermore, our method now produces skill scores of POD and FAR that are comparable to Tan et al. (2017), which report a POD and FAR of 0.67 and 0.47, respectively. Additionally, the 3 × 3 contingency table method produces areal mean skill scores that are more comparable to the daily validation of IMERG Early using MRMS (see Table 3). These new results provide confidence that a global-scale validation of IMERG with SMAP is possible and valid.
Summary of POD and FAR mean values for each validation method. CONUS and global means are provided for the IMERG validation with SMAP, and CONUS means for SMAP overpass timing and daily precipitation are provided for IMERG validation with MRMS. Note that both metrics range from 0 to 1, with 1 being perfect for POD and 0 being perfect for FAR.
5. Global scale: Results and discussion
a. Global IMERG validation with SMAP
Using this 3 × 3 categorization technique at the global scale, nearly 60% of overpasses are found to be “signal” (i.e., the meaningful components of Table 2) with 19.01% being correct rejections, 23.86% being misses, 5.23% being false alarms and 11.36% being hits. Of the forecast binning, 23.82% of the overpasses fall within the noise range (−0.01 cm3 cm−3 ≤ ΔSMAP ≤ 0 cm3 cm−3), and 20.45% of overpasses fall within the respective observation noise range (0 < IMERG ≤ 1 mm). Although approximately 40% of the overpass intervals are considered noise, a majority of the overpass intervals are within the signal categorizations, and while cells e (7.70%) and j (1.58%) are considered noise, they are used in the calculations of POD and FAR.
Figures 6a and 6b illustrates the global-scale results utilizing the 3 × 3 contingency table method, and we find global mean POD is 0.64 and the global mean FAR is 0.40. POD shows rather homogenous values across a majority of the globe with an area of enhanced skill stretching from the Arabian Peninsula through western China and Mongolia. There are two regions with low PODs that are of concern, eastern mainland Asia and the desert regions of Australia. While desert regions around the globe show increased POD, the lack of skill in the Australian deserts is confounding.
(left) POD and (right) FAR for respective comparisons between IMERG (a),(b) Early, (c),(d) Late, and (e),(f) Final and SMAP using the 3 × 3 contingency table method. Areas with no color are masked out due to lack of data coverage or SMAP quality control flags.
Citation: Journal of Hydrometeorology 23, 1; 10.1175/JHM-D-21-0035.1
The spatial pattern of global FAR is more complex, there does appear to be decreased skill in the tropics and increased skill in the subtropics and midlatitudes. Again, while the world’s deserts exhibit low FAR values, particularly through northern Africa and the Middle East, the Australian desert regions once again behave in an opposite manner by exhibiting elevated FAR values.
Table 3 provides a summary of POD and FAR results for both contingency table methods, and it is worth noting that by moving to the 3 × 3 contingency table for validation with SMAP, we now see comparable skill to the initial validation with MRMS. The switch to the 3 × 3 contingency table method improved the results and our ability to capture an IMERG signal in SMAP. This change allows for a better signal to be produced in our analysis, while still using a majority of the data.
Figures 6c–f depict the global-scale results utilizing the 3 × 3 contingency table method with IMERG Late and Final products. These products produce similar results as to the IMERG Early results, this largely results from IMERG processing that scales the precipitation rather than modifying the spatial pattern of occurrence. Due to this, the ability of SMAP to be used as a validation tool to detect precipitation occurrence for all IMERG products is valid.
b. Limitations and uncertainties
A validation method, such as this one, can only be as good as the reference data quality. In this instance, SMAP has shortcomings that limits this method’s potential to validate IMERG. This study is largely limited by the SMAP sensing errors, as areas with dense vegetation, complex topography and seasonally or annually frozen soil cannot be thoroughly vetted, recalling that we utilize the SMAP QC flags to screen the data. Additionally, SMAP overpass intervals of varying length (i.e., 1–3 days or more) provide the opportunity for weather to alter the local water balance before subsequent SMAP observations. This is most critical when there is light precipitation shortly after a SMAP overpass, potentially allowing for multiple days of evaporation to erase the precipitation signature from the soil moisture.
While the 3 × 3 contingency table method largely shows promising results with POD, there is still a fair amount of inconsistency between the FAR obtained via SMAP versus that obtained via MRMS (Fig. 6b). One aspect of the results that remains confounding is the lack of marked improvement in the FAR results while POD exhibits vast improvement across the domain. Digging deeper into potential causes for this, there were no relationships between diminished FAR improvement and soil type, soil porosity, vegetation type, and precipitation characteristics. The best potential hypothesis for the lack of improved consistency is that it is an artifact of the method itself. The categorization (Table 2) introduced to eliminate noise from the 2 × 2 contingency table does just that for POD calculations, removing the noise categorization for IMERG and only using SMAP noise that is a scale of magnitude less than signal l in Table 2. While with FAR [Eq. (4)] calculations using the 3 × 3 contingency table method, the overpass intervals with traces amounts of IMERG precipitation are included in the calculation, while minute changes in SMAP are screened out. This leaves numerous overpass intervals that have trace amounts of precipitation while experiencing decreases in soil moisture. In fact, more overpasses are categorized in Table 2 as e (noise) than i (signal), the two values that make up the numerator of the FAR calculation. In total, while this method aids in the categorization of detection, the ability to accurately identify absence has room for improvement. This highlights a need to advance this method to adopt a total water budget approach and find new variables that could enhance the ability to identify absences of precipitation at a global scale.
The general goal of this study was to determine whether changes in satellite-based soil moisture observations can be used as a proxy for the presence of precipitation; the method and results provide confidence that changes in SMAP observed soil moisture can be used as a method to detect precipitation. Future work, as discussed more in the conclusions, will need to address the deficiencies that have been noted. Likely sources of enhanced confidence will come from the inclusion of a hydrologic modeling framework that will allow for a baseline skill assessment accounting for all water fluxes and allowing for the potential inclusion of other variables that could both enhance validation and be observable by satellites, such as snowpack and terrestrial water storage.
6. Conclusions
Validating IMERG is a needed and worthwhile endeavor to build confidence in global precipitation estimates. Through a validation with MRMS for CONUS, there is confidence that IMERG provides an accurate representation of precipitation absence/presence. Pivoting the analysis to use SMAP changes as a proxy for precipitation occurrence, we found a mismatch between SMAP-based and MRMS-based metrics over the continental domain. Modifying our analysis to use a 3 × 3 contingency table to discern more signal from the noise, we see marked improvements in our SMAP-based skill metrics that increases our confidence in the global-scale expansion of our novel Hydro-GV analysis.
The ability to use a globally consistent dataset to validate IMERG is a novel attempt at a global ground validation technique of IMERG and ultimately provides the first credible attempt at global ground validation of IMERG. The potential utility of this ground validation framework to provide insight into IMERG skill for areas that are not gauged and underobserved is critical in advancing global precipitation measurements. The early results from this method seem promising, while the authors understand there are potential shortcomings from uncertainties in SMAP and IMERG that need to be addressed, namely, the noise in overpass-to-overpass changes in SMAP and the frequency at which light precipitation can evaporate before the following SMAP observation.
While conceptually similar to the SM2Rain product (Brocca et al. 2013; Ciabatta et al. 2018) in the sense that both studies use soil moisture as a tool for validating precipitation, there are fundamental differences. The SM2Rain methodology makes assumptions about the local water budget (e.g., evaporation and runoff are negligible) and analytically accounts for a variable soil drainage. This study takes a more simplistic approach by only accounting for changes in soil moisture and using a range of SMAP and IMERG data to screen out potential influences of evaporation and drainage. Due to an implicit treatment of the water balance, the method is employed to detect the presence of precipitation and not to estimate precipitation, such as SM2Rain. Both studies show the potential utility of soil moisture in assessing global precipitation through differing means.
Future work to evolve this method will be multipronged. An investigation of the spatial properties for the skill metrics could be done to determine if surface characteristics (e.g., vegetation, soil, topography) have an impact on skill, and if the skill metrics have potential seasonality. It would also be interesting to determine how skill changes with various precipitation types (e.g., synoptic versus mesoscale, rain versus mixed phase).
Future developments will also include incorporating a hydrologic model as an attempt to fill the gaps between SMAP overpasses. This method will allow for a more detailed representation of how light precipitation impacts VSM on the sub-SMAP overpass time scale, as well as determine how soil drying impacts the frequency of false alarms. Last, using both observations and hydrologic model output, we can attempt to estimate precipitation quantity in a similar framework as the SM2Rain methodology by Brocca et al. (2013). The latter effort will expand our analysis from focusing on presence/absence of precipitation to a more useful quantity that could be for operational in the future.
Validating IMERG globally with SMAP has shown there is inherent skill in the method and provides confidence in IMERG precipitation estimates. Furthering the method for constant systematic validation of precipitation detection at a near-global scale beyond the typical CONUS scale that many validation studies use for IMERG is a critical next step. The ability to apply this method to evaluate IMERG with SMAP as soon as observations become available, latencies of 4 and <50 h, respectively, can lead to the detection of IMERG issues on the order of days to weeks, and potentially increase the viability of IMERG applications.
Acknowledgments.
This research was supported by the NASA Global Precipitation Measurement Mission Ground Validation Program. This support is gratefully acknowledged.
Data availability statement.
All data used in this study are publicly available. IMERG data and SMAP data can both be accessed via the NASA Earth Data portal (https://disc.gsfc.nasa.gov/) and MRMS data can be accessed via the Precipitation Measurement Missions Ground Validation Data portal (https://pmm-gv.gsfc.nasa.gov/).
REFERENCES
Brocca, L., T. Moramarco, F. Melone, and W. Wagner, 2013: A new method for rainfall estimation through soil moisture observations. Geophys. Res. Lett., 40, 853–858, https://doi.org/10.1002/grl.50173.
Cao, Q., T. H. Painter, W. R. Currier, J. D. Lundquist, and D. P. Lettenmaier, 2018: Estimation of precipitation over the OLYMPEX domain during winter 2015/16. J. Hydrometeor., 19, 143–160, https://doi.org/10.1175/JHM-D-17-0076.1.
Ciabatta, L., and Coauthors, 2018: SM2RAIN-CCI: A new global long-term rainfall data set derived from ESA CCI soil moisture. Earth Syst. Sci. Data, 10, 267–280, https://doi.org/10.5194/essd-10-267-2018.
Entekhabi, D., and Coauthors, 2010: The Soil Moisture Active Passive (SMAP) mission. Proc. IEEE, 98, 704–716, https://doi.org/10.1109/JPROC.2010.2043918.
Funk, C., and Coauthors, 2015: The Climate Hazards Infrared Precipitation with Stations—A new environmental record for monitoring extremes. Sci. Data, 2, 150066, https://doi.org/10.1038/sdata.2015.66.
Henn, B., M. P. Clark, D. Kavetski, A. J. Newman, M. Hughes, B. McGurk, and J. D. Lundquist, 2018: Spatiotemporal patterns of precipitation inferred from streamflow observations across the Sierra Nevada mountain range. J. Hydrol., 556, 993–1012, https://doi.org/10.1016/j.jhydrol.2016.08.009.
Hong, Y., K. L. Hsu, S. Sorooshian, and X. Gao, 2004: Precipitation estimation from remotely sensed imagery using an artificial neural network cloud classification system. J. Appl. Meteor., 43, 1834–1853, https://doi.org/10.1175/JAM2173.1.
Hong, Y., R. F. Adler, F. Hossain, S. Curtis, and G. J. Huffman, 2007: A first approach to global runoff simulation using satellite rainfall estimation. Water Resour. Res., 43, W08502, https://doi.org/10.1029/2006WR005739.
Hou, A. Y., and Coauthors, 2014: The Global Precipitation Measurement Mission. Bull. Amer. Meteor. Soc., 95, 701–722, https://doi.org/10.1175/BAMS-D-13-00164.1.
Huffman, G. J., and Coauthors, 1997: The Global Precipitation Climatology Project (GPCP) Combined Precipitation Dataset. Bull. Amer. Meteor. Soc., 78, 5–20, https://doi.org/10.1175/1520-0477(1997)078<0005:TGPCPG>2.0.CO;2.
Huffman, G. J., and Coauthors, 2007: The TRMM Multisatellite Precipitation Analysis (TMPA): Quasi-global, multiyear, combined-sensor precipitation estimates at fine scales. J. Hydrometeor., 8, 38–55, https://doi.org/10.1175/JHM560.1.
Huffman, G. J., and Coauthors, 2020: NASA Global Precipitation Measurement (GPM) Integrated Multi-satellitE Retrievals for GPM (IMERG). Algorithm Theoretical Basis Doc., version 06, 35 pp., https://gpm.nasa.gov/sites/default/files/2020-05/IMERG_ATBD_V06.3.pdf.
Joyce, R. J., J. E. Janowiak, P. A. Arkin, and P. Xie, 2004: CMORPH: A method that produces global precipitation estimates from passive microwave and infrared data at high spatial and temporal resolution. J. Hydrometeor., 5, 487–503, https://doi.org/10.1175/1525-7541(2004)005<0487:CAMTPG>2.0.CO;2.
Kim, J. H., M. L. Ou, J. D. Park, K. R. Morris, M. R. Schwaller, and D. B. Wolff, 2014: Global Precipitation Measurement (GPM) ground validation (GV) prototype in the Korean Peninsula. J. Atmos. Oceanic Technol., 31, 1902–1921, https://doi.org/10.1175/JTECH-D-13-00193.1.
Kirschbaum, D. B., and T. Stanley, 2018: Satellite-based assessment of rainfall-triggered landslide hazard for situational awareness. Earth’s Future, 6, 505–523, https://doi.org/10.1002/2017EF000715.
Kirschbaum, D. B., and Coauthors, 2017: NASA’S remotely sensed precipitation: A reservoir for applications users. Bull. Amer. Meteor. Soc., 98, 1169–1184, https://doi.org/10.1175/BAMS-D-15-00296.1.
Kubota, T., and Coauthors, 2007: Global precipitation map using satellite-borne microwave radiometers by the GSMaP project: Production and validation. IEEE Trans. Geosci. Remote Sensing, 45, 2259–2275, https://doi.org/10.1109/TGRS.2007.895337.
Laurent, H., I. Jobard, and A. Toma, 1998: Validation of satellite and ground-based estimates of precipitation over the Sahel. Atmos. Res., 47–48, 651–670, https://doi.org/10.1016/S0169-8095(98)00051-9.
Liao, L., and R. Meneghini, 2009: Validation of TRMM precipitation radar through comparison of its multiyear measurements with ground-based radar. J. Appl. Meteor. Climatol., 48, 804–817, https://doi.org/10.1175/2008JAMC1974.1.
Maghsood, F. F., H. Hashemi, S. H. Hosseini, and R. Berndtsson, 2020: Ground validation of GPM IMERG precipitation products over Iran. Remote Sensing, 12, 48, https://doi.org/10.3390/rs12010048.
O’Neill, P. E., S. Chan, E. G. Njoku, T. Jackson, R. Bindlish, and J. Chaubell, 2020: SMAP enhanced L3 radiometer global daily 9 km EASE-grid soil moisture, version 4. NASA National Snow and Ice Data Center Distributed Active Archive Center, accessed 13 November 2019, https://doi.org/10.5067/NJ34TQ2LFE90.
Pombo, S., R. P. de Oliveira, and A. Mendes, 2015: Validation of remote-sensing precipitation products for Angola. Meteor. Appl., 22, 395–409, https://doi.org/10.1002/met.1467.
Schwaller, M. R., and K. Robert Morris, 2011: A ground validation network for the global precipitation measurement mission. J. Atmos. Oceanic Technol., 28, 301–319, https://doi.org/10.1175/2010JTECHA1403.1.
Sharifi, E., R. Steinacker, and B. Saghafian, 2016: Assessment of GPM-IMERG and other precipitation products against gauge data under different topographic and climatic conditions in Iran: Preliminary results. Remote Sensing, 8, 135, https://doi.org/10.3390/rs8020135.
Shellito, P. J., and Coauthors, 2016: SMAP soil moisture drying more rapid than observed in situ following rainfall events. Geophys. Res. Lett., 43, 8068–8075, https://doi.org/10.1002/2016GL069946.
Stanski, H., L. Wilson, and W. Burrows, 1989: Survey of common verification methods in meteorology. WMO/TD-358, World Meteorological Organization, 114 pp.
Tan, J., W. A. Petersen, and A. Tokay, 2016: A novel approach to identify sources of errors in IMERG for GPM ground validation. J. Hydrometeor., 17, 2477–2491, https://doi.org/10.1175/JHM-D-16-0079.1.
Tan, J., W. A. Petersen, P. E. Kirstetter, and Y. Tian, 2017: Performance of IMERG as a function of spatiotemporal scale. J. Hydrometeor., 18, 307–319, https://doi.org/10.1175/JHM-D-16-0174.1.
Wang, Z., R. Zhong, C. Lai, and J. Chen, 2017: Evaluation of the GPM IMERG satellite-based precipitation products and the hydrological utility. Atmos. Res., 196, 151–163, https://doi.org/10.1016/j.atmosres.2017.06.020.
Wu, H., R. F. Adler, Y. Hong, Y. Tian, and F. Policelli, 2012: Evaluation of global flood detection using satellite-based rainfall and a hydrologic model. J. Hydrometeor., 13, 1268–1284, https://doi.org/10.1175/JHM-D-11-087.1.
Wu, H., R. F. Adler, Y. Tian, G. J. Huffman, H. Li, and J. Wang, 2014: Real‐time global flood estimation using satellite‐based precipitation and a coupled land surface and routing model. Water Resour. Res., 50, 2693–2717, https://doi.org/10.1002/2013WR014710.
Wu, H., R. F. Adler, Y. Tian, G. Gu, and G. J. Huffman, 2017: Evaluation of quantitative precipitation estimations through hydrological modeling in IFloodS river basins. J. Hydrometeor., 18, 529–553, https://doi.org/10.1175/JHM-D-15-0149.1.
Xu, R., F. Tian, L. Yang, H. Hu, H. Lu, and A. Hou, 2017: Ground validation of GPM IMERG and TRMM 3B42V7 rainfall products over southern Tibetan Plateau based on a high-density rain gauge network. J. Geophys. Res. Atmos., 122, 910–924, https://doi.org/10.1002/2016JD025418.
Zhang, J., and Coauthors, 2016: Multi-Radar Multi-Sensor (MRMS) quantitative precipitation estimation: Initial operating capabilities. Bull. Amer. Meteor. Soc., 97, 621–638, https://doi.org/10.1175/BAMS-D-14-00174.1.
