Adjusting ISCCP Cloud Detection to Increase Consistency of Cloud Amount and Reduce Artifacts
1. Introduction
The International Satellite Cloud Climatology Project (ISCCP) is a widely cited and well used source of historical, global cloud information (Mayernik et al. 2015). The product has been used for many different applications, including investigations of cloud properties by cloud type (Fu et al. 1990; Hahn et al. 2001; Liao et al. 1995; Shouguo et al. 2004) and region (Calbó and Sanchez-Lorenzo 2009; Curry et al. 1996; Kiehl 1994; Li et al. 2006, 2004), global atmosphere and land surface model validation (Reichle et al. 2010), understanding the role of clouds on Earth’s surface radiation budget (Lohmann et al. 2006; Rossow et al. 2002; Zhang et al. 2004), and tropical mesoscale cloud dynamics (Tselioudis et al. 2013). While these studies largely focus on essential climate variables (Bojinski et al. 2014), the product has also been compared with many other cloud property datasets to better understand differences in derived cloud properties due to variations in cloud property algorithms and evolving instrument capabilities that span many multispectral passive and active satellite instruments (Stubenrauch et al. 2013). ISCCP has also contributed to the science of satellite intercalibration whereby data from multiple instruments are rigorously combined to enable the delivery of global, multidecadal environmental satellite data records (Brest et al. 1997; Chander et al. 2013; Goldberg et al. 2011; Knapp 2008a; Stone et al. 2013).
In 2014, the ISCCP production was transitioned to the National Centers for Environmental Information (NCEI) as part of NOAA’s former Climate Data Records Program (Bates et al. 2016). The baseline requirements for NCEI’s stewardship were to (i) reproduce the ISCCP D-series product with higher-resolution B1U data (Knapp 2008b) and (ii) extend the period of record by maintaining regular product updates. The latest version of ISCCP data, now known as the ISCCP H-series CDR product, were released publicly (Young et al. 2018). The first installment of data released by NCEI covered the period from July 1983 to December 2009. Ongoing updates followed and now ISCCP H-series data extend through June 2017 (Rossow et al. 2016).
Continued ISCCP H-series updates and improvements are planned in the form of a future reprocessing of ISCCP. These improvements include possible correction of issues that inhibit its use for evaluating long-term trends due to satellite viewing geometry data artifacts. Some artifacts have been previously identified in the legacy ISCCP D data, such as a dependency on the systematic changes in satellite zenith angle (Evan et al. 2007; Norris and Evan 2015). This effect is demonstrated in Fig. 1, using the ISCCP H data. The local correlation of the monthly mean cloud amount versus time for each grid cell over the ISCCP H period of record (July 1983–June 2017) is shown in Fig. 1a, hereafter called the local temporal correlation. From this calculation, one would expect to see where cloud patterns are changing. However, the correlation values are dominated by patterns of decreasing clouds that align with satellite coverage that was present in the 1980s. This confirms that the artifacts pointed out by Evan et al. (2007) are also present in the ISCCP H data. This was anticipated, however, since the goal of the initial production of the ISCCP data at NCEI was to reproduce the original ISCCP D data as closely as possible. The effect is noticeable in the boundaries of the geostationary satellite coverage present over the Indian Ocean. In Fig. 1b, we provide the mean cloud amount for February 1985. This is a year with minimal satellite coverage: only three geostationary and one polar orbiting satellite. There are similarities between Figs. 1a and 1b. Similar boundaries can be seen in the Indian Ocean and near the Falkland Islands where the cloud amount changes near the extents of geostationary satellite coverage. These similarities are important because Fig. 1a derives from the entire ISCCP period of record while Fig. 1b is a mean value for just one month. This suggests that there is a VZA dependence in the cloud amount and that these dependencies affect both spatial analyses (Fig. 1b) and temporal time series (Fig. 1a).
(a) Linear correlation of cloud amount vs time at each ISCCP grid cell. (b) Mean cloud amount (%) for February 1985.
Citation: Journal of Atmospheric and Oceanic Technology 38, 2; 10.1175/JTECH-D-20-0045.1
To resolve this issue, Norris and Evan (2015) proposed an empirical correction of the ISCCP data by performing a least squares best fit between cloud anomalies and artifact factor anomalies, leaving the residuals from the best-fit line as the newly corrected data. Unfortunately, the problem with such empirical corrections is twofold. First, the empirical correction adjusts cloud amount but cannot correct derived cloud parameters (e.g., cloud-top temperature). Thus, biased data cannot be easily fixed. Second, Norris and Evan (2015) note that their correction makes long-term studies impossible. Thus, it is desirable to adjust the ISCCP cloud algorithm at the initial cloud detection step such that VZA dependence is decreased and allows downstream products such as cloud-top temperature to be consistent with the derived cloud amount.
This paper investigates an adjustment to the ISCCP cloud algorithm and its effect on reducing these satellite view zenith angle dependencies. It does not produce a completely new record yet but paves the way for a future reprocessing that will include this adjustment along with some other changes. The goal, herein, is to adjust the cloud detection threshold such that cloud amount values from separate satellites (with different view zenith angles) are more consistent than the current ISCCP product. The next section summarizes the current ISCCP algorithm and our new adjustment, followed by a section which analyzes the results of the adjustment.
2. ISCCP cloud detection
a. Current algorithm
The data collection for ISCCP began in 1983. At that time, and for the next two decades, the only globally contiguous set of channels from satellite imagers were those that sensed the visible (VIS; approximately 0.65 μm) window and infrared (IR) window (approximately 11 μm) wavelengths. The ISCCP program developed a cloud detection algorithm using these two channels (Rossow and Garder 1993). There are two basic steps: first, it determines cloud-cleared (or clear-sky) radiances; second, it determines cloudy pixels based on some deviation from the clear-sky radiance. This is performed for both channels because clouds, for the most part, increase visible reflectance and decrease IR radiance, compared to clear-sky values. The original ISCCP algorithm is detailed by Rossow and Garder (1993) and is only summarized here, following Rossow and Schiffer (1999).
To determine the clear-sky radiance, pixel-level radiance data for a given location are collected over the course of a month at the same time of day. Initial cloud tests are applied to these observations in order to remove the effects of most clouds, producing a set of mostly cloud free pixels. This approach allows the initial cloud tests to not necessarily identify all clouds and in some cases to remove too many pixels. The statistical distribution of these clear pixels are then used to estimate a clear-sky radiance.
Clouds are assumed, in ISCCP, to decrease the IR radiance (i.e., clouds are colder) and to increase the VIS radiance (i.e., brighter). ISCCP defines a threshold as the change in radiance (from clear sky) that must be exceeded to suggest a cloud is present. It is used to compare any given observation (viz., a pixel) to the derived clear-sky radiance. Pixels that are colder (in the IR) or brighter (in the VIS) are more likely affected by clouds. Thresholds are defined based on surface type and are provided in Table 1; more details are provided by Rossow and Garder (1993). Pixels that are more than twice the threshold from the clear-sky radiance are labeled as cloudy. Pixels that are only one to two threshold steps away are labeled marginally cloud, representing the uncertainty level of the cloud mask (since it represents borderline cloud pixels). Last, pixels that are not more than one threshold colder or brighter than the clear-sky radiance are labeled as clear. This is summarized in Table 2. The thresholds represent the certainty of the clear-sky radiance for given scenes. For instance, open ocean areas (i.e., those away from coastlines and sea ice boundaries) have clear-sky radiances that can often be determined more accurately, so have the lowest threshold. Conversely, the clear-sky radiances over rough terrain and regions of snow and ice have more heterogeneities such that clear-sky radiances are less certain, thus they have larger thresholds. In summary, ISCCP at its most basic level performs two steps: Calculate the clear-sky radiances and identify cloudy pixels as those that have large departures from the clear sky.
Cloud thresholds for IR (TIR) and VIS (TVS) channels based on the ISCCP surface types.
ISCCP cloud mask definitions as deviations from clear-sky radiance (Rcs). Thresholds are defined in Table 1.
When discussing ISCCP, it is important to understand that geostationary satellites (GEOs) are processed individually, as are the data from low-Earth-orbiting (LEO) satellites. The five GEO positions used in ISCCP are the Geostationary Operational Environmental Satellite (GOES) West (GOW) positioned at 135°W, GOES East (GOE) at 75°W, European Meteorological Satellite (Meteosat) (MET) at 0°, Indian Satellite (INS) position at 63°E, and the Japanese Geostationary Meteorological Satellite (GMS) located at 140°E. These three-letter identifiers represent the official ISCCP names for satellite positions. They do not refer to specific satellites, but to the traditional location of the geostationary satellites available to ISCCP. The GEO and LEO data are then merged together for a global dataset, where pixel-level data are gridded to 1° equal-area grid cells for which statistics are reported. Herein, we use the cloud amount (CA) variable from ISCCP H (Young et al. 2018). It is calculated as the percentage of pixels in a 1° grid cell that are flagged as cloudy. While missing data in ISCCP are filled using interpolation, the analysis herein ignores filled grid cells and only uses values from observations. Analysis of results in this paper use the statistical tools of linear regression, expressed through Pearson correlation, slope, and offset, as well as root-mean-square (rms) error differences and bias (defined as the mean difference between two variables).
The problem is that the current ISCCP cloud detection algorithm is not intraconsistent. That is, differences occur when viewing the same region from two separate satellites. Specifically, there is variation in cloud amount when comparing different satellites at different view zenith angles (VZA). This is demonstrated by comparing values of cloud amount from near-nadir GMS observations with off-nadir observations from GOW. Figure 2 provides the comparison for October 2000, where each point represents a 1° grid cell that was observed simultaneously from both satellites. The mean VZA for GMS is 28° while the mean GOW VZA is far off nadir at 70°. Although the observations are nearly coincident, there is vast disagreement in the cloud amount between the two satellites. The bias error is 20%, and the slope is 0.59 with an offset of 46%. For times when GMS sees a mostly clear cell (CA < 20%), the GOW satellite labeled the cell as mostly cloudy (CA > 80%) 5% of the time. The converse rarely happened; that is, the off nadir detecting few clouds while the near nadir detected mostly cloudy. In fact, the cloud algorithm rarely estimates the near-nadir cloud amount as less cloudy than the off nadir. The off-nadir grid cell is 23 times more likely to be too cloudy than too clear. In short, the difference between off-nadir and near-nadir cloud amount is large.
Cloud amount (%) from original ISCCP threshold between near-nadir GMS grid cells (VZA < 30°) and coincident GOW grid cells (off nadir) for October 2000. Magenta line is the linear regression fit.
Citation: Journal of Atmospheric and Oceanic Technology 38, 2; 10.1175/JTECH-D-20-0045.1
This same comparison (Fig. 2) is expanded to compare every GEO satellite with its neighboring satellite for October 2000. Table 3 provides the RMS and bias differences for all combinations of near-nadir observations and neighboring satellites. That is, the satellites in the first column provide near-nadir cloud amount for comparisons with off-nadir observations from adjacent satellites to the east and west. The locations of these matchups for the month are shown in Fig. 3a. Below each satellite position over the equator, each GEO has near-nadir observations that are matched with off-nadir observations from the adjacent satellites. The values in Table 3 will not be discussed specifically, but will be used as a reference point for the VZA-adjusted algorithm. Suffice it to say: The GOW–GMS bias (as discussed with respect to Fig. 2) is not unique to that pair.
Cloud amount bias and rms differences (%) between near-nadir satellites (in the first column) and the off-nadir views from neighboring satellites using the original ISCCP thresholds for October 2000.
(a) Gridcell matchup locations for the near-nadir GEO to off-nadir GEO (cf. Table 3) for ISCCP satellite positions: GOW (magenta), GOE (cyan), MET (yellow), INS (green), and GMS (orange). (b) Matchup gridcell locations for near-nadir LEO to off-nadir GEO with the same GEO color coding (cf. Table 5).
Citation: Journal of Atmospheric and Oceanic Technology 38, 2; 10.1175/JTECH-D-20-0045.1
b. VZA-adjusted algorithm
The goal of the adjustment was twofold: minimize the changes to the algorithm to decrease chances for any side effects and increase the cloud amount intraconsistency. Also, given that much work went into deriving the ISCCP algorithm and the thresholds, we endeavor not to change the overall performance of the cloud algorithm. However, a change that is related to view zenith angle appears necessary in order to decrease the differences noted in Fig. 2 and Table 3.
There are three effects that can lead to the vast differences in cloud amount illustrated in Fig. 2: parallax, limb darkening and longer pathlengths. The parallax effect is caused by clouds being three-dimensional. That is, when viewing a region from an angle, some of the clear regions are hidden from view by nearby clouds. The cloud amount only appears to change because less of the clear-sky regions are visible. Limb darkening occurs at large VZA because of the gaseous emission that is above (usually colder than) the emitting surface, thus lowering the sensed radiance [e.g., see Joyce et al. (2001), who develop a correction of this effect]. These colder observations might appear as cloud. These effects can lead to an exaggeration of the cloud amount while the last effect could be a detection of actual clouds: longer pathlengths. The larger view zenith angle leads to increased pathlengths through the atmosphere and any potential cloud. That is, the longer pathlengths from large VZA make clouds more detectable (more reflectance in the VIS and emission in the IR). This allows thinner clouds to be detected when viewed from oblique angles. Whatever the true cause may be, the result is clear from Fig. 2 and Table 3: significant overestimation of clouds from off nadir when compared to near nadir.
The result of such heterogeneities is an inability to perform spatial comparisons (over regions where the VZA would vary) as well as an inability to perform temporal comparisons (during times when the VZA has a secular change). So we strive to make the off-nadir cloud amount consistent with nadir observations in spite of the chance that some of the off-nadir cloud amount increase is real. Thus, we sacrifice sensing thin clouds at the limb in order to improve spatial and temporal consistency.
Cloud detection threshold for the original ISCCP (dashed line) and the new VZA-adjusted threshold (solid) as a function of VZA, which is twice the threshold for open water is shown (cf. Tables 1 and 2).
Citation: Journal of Atmospheric and Oceanic Technology 38, 2; 10.1175/JTECH-D-20-0045.1
The cloud amount resulting from this VZA-adjusted threshold for GMS near nadir versus GOW off nadir for October 2000 is shown in Fig. 5, to be compared with Fig. 2 which uses the original cloud algorithm. The new cloud amount values agree much better than the previous matchups. The bias has decreased to 0.1%. The linear regression slope and offset are now 0.97% and 3%, respectively. The rms is 13%, which is down from 20%. The occurrence of CA overestimation (off-nadir CA > 80% when near-nadir CA < 20%) decreases from 5% to 0.06%. The ratio of off-nadir overestimating cloud amount drops from 23 to 1.1. In summary, the comparisons are much more consistent than the original ISCCP threshold. Likewise, Table 4 provides the rms and bias differences between the various GEO satellites using the VZA-adjusted thresholds (for comparison with Table 3). For most satellite pairs (7 of 10), the rms is smaller and the bias is closer to zero than the original ISCCP threshold. However, this result is for one month of data for a few satellites. The following section describes the impact of the new threshold for a variety of conditions.
As in Fig. 2, but using the VZA-adjusted cloud threshold.
Citation: Journal of Atmospheric and Oceanic Technology 38, 2; 10.1175/JTECH-D-20-0045.1
As in Table 3, but using the new VZA-adjusted threshold. Boldface values are those that have lower rms or whose bias is closer to zero.
3. Results from the new detection threshold
The satellite artifacts decrease when using the VZA-adjusted thresholds. Figure 6a provides the new map of monthly averaged cloud amount for February 1985 for comparison with Fig. 1b (which is the original ISCCP H data). For cloud amount, it is now difficult to see the GEO boundaries with the new threshold. The distinct satellite boundary lines in the Indian Ocean and near the Falkland Islands are decreased. The change in the cloud amount caused by the new threshold is provided in Fig. 6b. The decreased cloudiness is as large as 30% in some areas. The largest decreases align with regions of large local temporal correlation (cf. Fig. 1a). Thus, it is possible that the changes may reduce the artifacts in the spatial long-term correlations.
(a) Mean cloud amount for February 1985 using the VZA-adjusted threshold (cf. Fig. 1b) and (b) the cloud amount difference for February 1985 between the original threshold and the VZA-adjusted threshold.
Citation: Journal of Atmospheric and Oceanic Technology 38, 2; 10.1175/JTECH-D-20-0045.1
The meridionally averaged cloud amount (from 10°S to 10°N) is provided in Fig. 7a for February 1985. The satellite boundaries are highlighted by vertical dashed lines. Whereas the original ISCCP thresholds produced large differences at these boundaries the new cloud amount (orange line) has very little change in cloud amount across the boundaries. Figure 7b shows the same plot but averages the entire year of 1985. Again, little to no dependence on satellites is apparent as compared to the significant decrease in clouds in the GEO gap over the Indian Ocean. While this INS gap is filled in later years with relocated satellites, its presence early on sets up a secular change in VZA for the region.
(a) Meridional mean cloud amount (CA) for from February 1985 for 10°S–10°N using the original ISCCP cloud threshold (black line) and the new VZA-adjusted threshold (orange). The dashed lines represent satellite boundaries showing the gap over the Indian Ocean (at 63° and 73°E) and the GMS-GOW (165°W) and GOW-MET (54°W) boundaries. (b) As in (a), but for the entirety of 1985.
Citation: Journal of Atmospheric and Oceanic Technology 38, 2; 10.1175/JTECH-D-20-0045.1
The comparisons of near-nadir GEO cloud amount with off nadir is limited because the region of Earth where comparisons are possible is limited to tropical regions directly below the satellites (see Fig. 3a). Thus we also perform some comparisons between near-nadir LEO cloud amount and off-nadir GEO cloud amount, which has a wider variety of surface conditions (Fig. 3b). The following discussion inspects the data by comparing GEO data with LEO data, plotting global temporal trends, and showing the impact of the VZA-adjusted threshold on ISCCP-derived cloud types.
a. Using low-Earth orbiters to investigate intraconsistency
Comparisons between GEO and LEO provide a wider variety of conditions to investigate the impact of the new threshold. Near-nadir (VZA < 25°) observations of cloud amount from LEO were compared with off-nadir observations from GEO (VZA > 45°). The linear regression slope and offset as well as the mean bias are shown in Table 5 for 1985. For all matched grid cells (N > 709 000), the slope increases to 0.88, the offset decreases to 8% and the bias decreases from 11% to 0%. The following breaks down the comparisons for different conditions in order to identify what inconsistencies remain.
Linear regression coefficients and bias for cloud amount (%) comparisons between near-nadir LEO (VZA < 25°) observations and GEO off nadir (VZA > 45°) for original and VZA-adjusted threshold for all matchups in 1985 separated by various conditions.
For land versus water cells, the original ISCCP threshold showed slightly better agreement over land than water. The VZA-adjusted matchups have higher correlation. While the bias for water decreases to near zero, the land value overshoots zero, going from 6% to −6%. A smaller adjustment over land might be warranted to have bias closer to zero.
Zonally, the comparison mirrors the land versus water comparisons. The tropics, which have 89% of the cells over water had some remaining bias (and a positive offset) while the midlatitudes (which are 47% land) had resulting slope, offset and bias values similar to the land results.
Seasonally, the impact of the threshold adjustment shows no obvious dependency on season (at least when comparing Northern Hemisphere summer to winter).
Ice and snow regions occur mostly related to the Northern Hemisphere winter (snow over North America, sea ice on Hudson Bay, etc.) and the Himalayas (from the regions of coverage in Fig. 3b). The performance of the VZA adjustment shows significant overcorrection. The bias becomes −19%, the slope is worse and the offset is now negative. This is by far the worst subset. The snow/ice regions have the largest thresholds (Table 3). It appears that smaller adjustments should be made for snow and ice.
For the night data, the VZA adjustment showed marked improvement with bias decreasing from 9% to −3% and the offset decreasing as well. However, the daytime data did not show such large improvements. The offset did show a large decrease from 31% to 15%, but that is still a pretty high value compared to other offsets in the table. More work is needed to see if the daytime cloud amount can be made more consistent.
Last, the statistics are provided by satellite for the three GEO positions available in 1985: GOW, MET, and GMS. Overall, the performance is relatively uniform with improvements for all statistics (slopes are closer to 1, offsets are closer to zero and biases are nearly eliminated). More analysis is needed, though, for instruments with narrower bandwidths (which have less contamination from gaseous absorption).
b. Temporal cloud trends
One of the more prominent artifacts in the ISCCP data is the long-term trend in cloud amount for each grid cell (Fig. 1a). The ISCCP period of record (1983–2017) shows large CA decreases in regions where high view zenith angles were present in the 1980s. In lieu of a complete reprocessing of the ISCCP data, we instead inspect the impact of the VZA adjustment by processing pentad subsets (viz., every fifth year) of the period of record: 1985, 1990, 1995, …, 2015. Figure 8a shows the ISCCP period of record mean cloud amount. The local temporal correlation of cloud amount versus time are provided again in Fig. 8d. The mean cloud amount and correlation for the pentad subset years are provided in Figs. 8b and 8e, respectively. The patterns of both the cloud amount and local temporal correlation are similar to the full period of record. In fact, the correlation between the mean cloud amounts (Fig. 8a vs Fig. 8b) is 0.998. Likewise, the correlation between the local temporal correlations over the full period of record maps (Fig. 8d) and that for the pentad subset (Fig. 8e) is 0.84 (explaining 71% of the variance). It appears that the pentad subset is sufficiently representative of the full ISCCP period of record to investigate the impact of the VZA threshold adjustment.
(a) Mean ISCCP cloud amount for the full period of record; (b) as in (a), but only using the subset pentad (every fifth year) using the ISCCP cloud amount; (c) as in (b), but using the VZA-adjusted threshold; (d) local temporal correlation of cloud amount for ISCCP from the full period of record (as in Fig. 1a); (e) as in (d), but using the subset pentad; and (f) as in (e), but using the proposed VZA-threshold.
Citation: Journal of Atmospheric and Oceanic Technology 38, 2; 10.1175/JTECH-D-20-0045.1
The VZA-adjusted mean cloud amount and its local temporal correlation is shown in Figs. 8c and 8f, respectively. Overall, the satellite boundaries are reduced in the mean cloud amount (Fig. 8c). Likewise, the large regions of negative correlation near satellite boundaries are reduced. For example, there is a cloud amount minima in the southern Indian Ocean in all the maps. In the first two (Figs. 8a,b), it is isolated between the GEO coverages; it is not apparent if it is a real minima or an artifact of being in the gap between the GEOs. The VZA-adjusted threshold decreases the clouds on the MET limb east of Africa. From the reprocessed data, the minima is now part of a clearer region extending from Africa, over Madagascar and eastward. The cloud patterns appear more realistic using the new threshold. The VZA-adjusted threshold significantly decreases the satellite coverage artifacts that are present in the original data.
The local temporal correlation map (Fig. 8f) also has fewer artifacts. The maps using the original threshold (Figs. 8d,e) both show large negative correlations at the satellite limbs in the eastern Pacific, north of South America and the Indian Ocean. These are decreased when using the VZA-adjusted threshold. However, there appear to be some new artifacts with positive correlation in the southern Indian Ocean but they are smaller than those in the original data. Central Asia has an area of negative correlation that was not present in the original, appearing in the gap between the GEOs. This could be a combination of the trends noted above: possible overcorrection over snow and ice surfaces as well as a slight overcorrection for land areas. Nonetheless, the regions affected by satellite VZA artifacts are smaller with this VZA adjustment.
The overall global mean cloud trend appears to be significantly affected by the VZA-adjusted threshold. Figure 9 shows the global monthly mean time series for both thresholds as well as the difference (which are only available for months during each pentad). The difference is largest (approximately 12%) during 1985 when there were only three GEO satellites and 1 LEO. The difference decreases through time largely because the global coverage of the satellites improves (e.g., the Indian Ocean GEO gap was filled in 1999). The mean difference over the last few pentads is about 5%. The impact on the overall ISCCP cloud record is that the early years will likely see large decreases, the magnitude of which will depend on the number of satellites in coverage. As a result, it does seem likely that the mean global cloud amount trend will change.
(a) Time series of globally averaged monthly cloud amount (CA) from both the original ISCCP thresholds (crosses) and the new VZA-adjusted threshold (squares) for pentad years. (b) Time series of cloud amount differences between the original and VZA-adjusted cloud amount.
Citation: Journal of Atmospheric and Oceanic Technology 38, 2; 10.1175/JTECH-D-20-0045.1
c. Impacts on cloud type
The ISCCP H cloud products include a 2D histogram which provides the cloud distribution of a grid cell as a function of cloud-top pressure (Pc) and optical depth (τ). Clouds are binned into low (Pc < 680 hPa), middle (440 < Pc < 680 hPa) and high (10 < Pc < 440 hPa) and are either thin (0 < τ < 3.55), medium (3.55 < τ < 22.63) or thick (τ > 22.63). Given that the VZA-adjusted threshold appears to decrease clouds at the satellite limbs, the following provides a summary of how the frequency histograms change.
For this analysis, we used the GEO to GEO matchups for all satellites operating in October 2000 (this uses the same matchup points in Fig. 3a). The distribution of the clouds from the near-nadir satellite is provided in Table 6 where the matchups have a mean cloud amount of 60% with low–medium, high–thin, and low–thin clouds being the most prevalent (in that order). The distributions of these same cells from the off-nadir satellites is provided in Table 7, where the mean cloud amount increases to 70%. The same three types are still the most prevalent, but the order has changed with the thin clouds appearing more prevalent. Overall, the thin cloud amount increases by 12.4%, which accounts for all of the increase in the overall cloud amount (the medium and thick clouds had changes of −1.4% and 0.4%, respectively). With the VZA adjustment applied (Table 8), the distribution is much closer to the original (cf. Table 6). Each distribution is within 2% of the original except for low–medium clouds. These low–medium clouds amount dropped by 3.2% for off nadir and instead of recovering, decreased more with the VZA adjustment. This decrease can be explained by the overcorrection of the threshold adjustment over land (an overcorrection implies that the new threshold is mislabeling warm clouds as clear thereby decreasing low clouds). The overall cloud amount is 55.8% (4.2% less than the original), which happens to be the same amount of decrease to low–medium clouds. It appears that VZA adjustment cloud distribution is very similar to the near-nadir observation except for one cloud type. However, this result is for just 1 month.
GEO cloud amount (%) by cloud type for October 2000 based on near-nadir observations when and where coincident off-nadir GEO observations were also available.
As in Table 6, but for the off-nadir observations corresponding to the same set of cells in Table 6 (differences are presented in parentheses).
As in Table 7, but using the cloud amount from the off-nadir GEO using the VZA-adjusted thresholds (differences from Table 6 are provided in parentheses).
The impact of the new threshold is also seen in the mean cloud distribution for all of 1985. Table 9 provides the global Pc–τ distribution for all of 1985. The same distribution is shown in Table 10 for the new threshold. Over an entire year, the impact is limited to the thin clouds. The low–thin and high–thin clouds decrease by 2.6% and 2.4%, respectively. The medium and thick clouds have changes smaller than 0.2%. It appears that the impact of this correction for 1985 is to reduce thin clouds with little to no impact on thicker clouds. This would, of course, vary for other years depending on the number of satellites available.
Cloud amount (%) by cloud type for 1985 using the original ISCCP threshold.
4. Summary
The ISCCP cloud algorithm has provided cloud property information for decades. This paper investigated the ISCCP H data that were produced at a new location, by a new science team, with new input data but using the same ISCCP code. However, ISCCP H data are not without some artifacts that affect long-term trends and mean spatial cloud patterns. The goal, herein, was to determine if an adjustment to the ISCCP cloud detection algorithm could be made that would reduce previously identified artifacts. The results include the following:
Satellite artifacts discovered in ISCCP D (legacy production) are also present in ISCCP H (NCEI production).
The patterns in regional long-term cloud amount trends are similar to a monthly cloud amount map from a time when only three geostationary satellites were operating.
Cloud amount inconsistencies and discontinuities in time and space were verified to be related to differences in the detection of clouds at different view zenith angles.
A simple change to the cloud detection threshold vastly improved the satellite-to-satellite consistency of the retrieved cloud amount. The VZA-adjusted cloud detection threshold produces more spatially consistent data as demonstrated by spatial maps and satellite intercomparisons.
The improvement to cloud amount consistency was not ubiquitous. There are circumstances where it likely overcorrected (i.e., removed too much cloud). This appears to be the case over snow and ice regions and to a lesser extent over land.
The impact of the adjustment was a removal of thin clouds at high view zenith angles, generally decreasing the high–thin and low–thin cloud amounts.
Processing a temporal sampling (pentad sampling of every fifth year) is sufficient to investigate the impact of the change on the full ISCCP period of record.
At a global scale, the overall decrease is 12% less clouds in the 1980s and decreases to a 5% reduction after 2000 (when all five geostationary satellites were operating). Regionally, the adjustment removed most of the artifacts and should make studies of temporal changes in clouds, or relationships to long-term climate indices, possible.
It is now clear that a reprocessing of ISCCP using a cloud detection threshold that varies with VZA will improve spatial consistency in the data as well as provide better long-term stability for detecting regional trends.
Acknowledgments
The authors acknowledge Bill Rossow and Jessica Matthews for reviewing a previous version of this manuscript. We are also grateful for the constructive reviews provided by two anonymous reviewers and Joel Norris. A. Inamdar was supported by NOAA through the Cooperative Institute for Climate and Satellites–North Carolina under Cooperative Agreement NA14NES432003.
Data availability statement
The original ISCCP H Cloud dataset for the complete period of record is available at https://www.ncdc.noaa.gov/isccp with metadata describing it at https://data.nodc.noaa.gov/cgi-bin/iso?id=gov.noaa.ncdc:C00956. The VZA-adjusted data for pentad subset years are available from the corresponding author and will only be available until the next ISCCP reprocessing.
REFERENCES
Bates, J. J., J. L. Privette, E. J. Kearns, W. Glance, and X. Zhao, 2016: Sustained production of multidecadal climate records: Lessons from the NOAA Climate Data Record program. Bull. Amer. Meteor. Soc., 97, 1573–1581, https://doi.org/10.1175/BAMS-D-15-00015.1.
Bojinski, S., M. Verstraete, T. C. Peterson, C. Richter, A. Simmons, and M. Zemp, 2014: The concept of essential climate variables in support of climate research, applications, and policy. Bull. Amer. Meteor. Soc., 95, 1431–1443, https://doi.org/10.1175/BAMS-D-13-00047.1.
Brest, C. L., W. B. Rossow, and M. D. Roiter, 1997: Update of radiance calibrations for ISCCP. J. Atmos. Oceanic Technol., 14, 1091–1109, https://doi.org/10.1175/1520-0426(1997)014<1091:UORCFI>2.0.CO;2.
Calbó, J., and A. Sanchez-Lorenzo, 2009: Cloudiness climatology in the Iberian Peninsula from three global gridded datasets (ISCCP, CRU TS 2.1, ERA-40). Theor. Appl. Climatol., 96, 105–115, https://doi.org/10.1007/s00704-008-0039-z.
Chander, G., T. J. Hewison, N. Fox, X. Wu, X. Xiong, and W. J. Blackwell, 2013: Overview of intercalibration of satellite instruments. IEEE Trans. Geosci. Remote Sens., 51, 1056–1080, https://doi.org/10.1109/TGRS.2012.2228654.
Curry, J. A., J. L. Schramm, W. B. Rossow, and D. Randall, 1996: Overview of Arctic cloud and radiation characteristics. J. Climate, 9, 1731–1764, https://doi.org/10.1175/1520-0442(1996)009<1731:OOACAR>2.0.CO;2.
Evan, A. T., A. K. Heidinger, and D. J. Vimont, 2007: Arguments against a physical long-term trend in global ISCCP cloud amounts. Geophys. Res. Lett., 34, L04701, https://doi.org/10.1029/2006GL028083.
Fu, R., A. D. Del Genio, and W. B. Rossow, 1990: Behavior of deep convective clouds in the tropical Pacific deduced from ISCCP radiances. J. Climate, 3, 1129–1152, https://doi.org/10.1175/1520-0442(1990)003<1129:BODCCI>2.0.CO;2.
Goldberg, M., and Coauthors, 2011: The Global Space-Based Inter-Calibration System. Bull. Amer. Meteor. Soc., 92, 467–475, https://doi.org/10.1175/2010BAMS2967.1.
Hahn, C. J., W. B. Rossow, and S. G. Warren, 2001: ISCCP cloud properties associated with standard cloud types identified in individual surface observations. J. Climate, 14, 11–28, https://doi.org/10.1175/1520-0442(2001)014<0011:ICPAWS>2.0.CO;2.
Joyce, R., J. Janowiak, and G. Huffman, 2001: Latitudinally and seasonally dependent zenith-angle corrections for geostationary satellite IR brightness temperatures. J. Appl. Meteor., 40, 689–703, https://doi.org/10.1175/1520-0450(2001)040<0689:LASDZA>2.0.CO;2.
Kiehl, J., 1994: On the observed near cancellation between longwave and shortwave cloud forcing in tropical regions. J. Climate, 7, 559–565, https://doi.org/10.1175/1520-0442(1994)007<0559:OTONCB>2.0.CO;2.
Knapp, K. R., 2008a: Calibration of long-term geostationary infrared observations using HIRS. J. Atmos. Oceanic Technol., 25, 183–195, https://doi.org/10.1175/2007JTECHA910.1.
Knapp, K. R., 2008b: Scientific data stewardship of international satellite cloud climatology Project B1 global geostationary observations. J. Appl. Remote Sens., 2, 023548, https://doi.org/10.1117/1.3043461.
Li, Y., R. Yu, Y. Xu, and X. Zhang, 2004: Spatial distribution and seasonal variation of cloud over China based on ISCCP data and surface observations. J. Meteor. Soc. Japan, 82, 761–773, https://doi.org/10.2151/jmsj.2004.761.
Li, Y., X. Liu, and B. Chen, 2006: Cloud type climatology over the Tibetan Plateau: A comparison of ISCCP and MODIS/TERRA measurements with surface observations. Geophys. Res. Lett., 33, L17716, https://doi.org/10.1029/2006GL026890.
Liao, X., W. B. Rossow, and D. Rind, 1995: Comparison between SAGE II and ISCCP high-level clouds: 1. Global and zonal mean cloud amounts. J. Geophys. Res., 100, 1121–1135, https://doi.org/10.1029/94JD02429.
Lohmann, S., C. Schillings, B. Mayer, and R. Meyer, 2006: Long-term variability of solar direct and global radiation derived from ISCCP data and comparison with reanalysis data. Sol. Energy, 80, 1390–1401, https://doi.org/10.1016/j.solener.2006.03.004.
Mayernik, M. S., S. Callaghan, R. Leigh, J. Tedds, and S. Worley, 2015: Peer review of datasets: When, why, and how. Bull. Amer. Meteor. Soc., 96, 191–201, https://doi.org/10.1175/BAMS-D-13-00083.1.
Norris, J. R., and A. T. Evan, 2015: Empirical removal of artifacts from the ISCCP and PATMOS-x satellite cloud records. J. Atmos. Oceanic Technol., 32, 691–702, https://doi.org/10.1175/JTECH-D-14-00058.1.
Reichle, R. H., S. V. Kumar, S. P. Mahanama, R. D. Koster, and Q. Liu, 2010: Assimilation of satellite-derived skin temperature observations into land surface models. J. Hydrometeor., 11, 1103–1122, https://doi.org/10.1175/2010JHM1262.1.
Rossow, W. B., and L. C. Garder, 1993: Cloud detection using satellite measurements of infrared and visible radiances for ISCCP. J. Climate, 6, 2341–2369, https://doi.org/10.1175/1520-0442(1993)006<2341:CDUSMO>2.0.CO;2.
Rossow, W. B., and R. A. Schiffer, 1999: Advances in understanding clouds from ISCCP. Bull. Amer. Meteor. Soc., 80, 2261–2287, https://doi.org/10.1175/1520-0477(1999)080<2261:AIUCFI>2.0.CO;2.
Rossow, W. B., C. Delo, and B. Cairns, 2002: Implications of the observed mesoscale variations of clouds for the Earth’s radiation budget. J. Climate, 15, 557–585, https://doi.org/10.1175/1520-0442(2002)015<0557:IOTOMV>2.0.CO;2.
Rossow, W. B., and Coauthors, 2016: International Satellite Cloud Climatology Program H-series. National Centers for Environmental Information, accessed 1 November 2019, https://doi.org/10.7289/V5QZ281S.
Shouguo, D., S. Guangyu, and Z. Chunsheng, 2004: Analyzing global trends of different cloud types and their potential impacts on climate by using the ISCCP D2 dataset. Chin. Sci. Bull., 49, 1301–1306, https://doi.org/10.1360/03wd0614.
Stone, T. C., W. B. Rossow, J. Ferrier, and L. M. Hinkelman, 2013: Evaluation of ISCCP multisatellite radiance calibration for geostationary imager visible channels using the moon. IEEE Trans. Geosci. Remote Sens., 51, 1255–1266, https://doi.org/10.1109/TGRS.2012.2237520.
Stubenrauch, C. J., and Coauthors, 2013: Assessment of global cloud datasets from satellites: Project and database initiated by the GEWEX radiation panel. Bull. Amer. Meteor. Soc., 94, 1031–1049, https://doi.org/10.1175/BAMS-D-12-00117.1.
Tselioudis, G., W. Rossow, Y. Zhang, and D. Konsta, 2013: Global weather states and their properties from passive and active satellite cloud retrievals. J. Climate, 26, 7734–7746, https://doi.org/10.1175/JCLI-D-13-00024.1.
Young, A. H., K. R. Knapp, A. Inamdar, W. Hankins, and W. B. Rossow, 2018: The International Satellite Cloud Climatology Project H-series climate data record product. Earth Syst. Sci. Data, 10, 583–593, https://doi.org/10.5194/essd-10-583-2018.
Zhang, Y., W. B. Rossow, A. A. Lacis, V. Oinas, and M. I. Mishchenko, 2004: Calculation of radiative fluxes from the surface to top of atmosphere based on ISCCP and other global data sets: Refinements of the radiative transfer model and the input data. J. Geophys. Res., 109, D19105, https://doi.org/10.1029/2003JD004457.
