The Advanced Clear Sky Processor for Oceans (ACSPO) generates clear-sky products, such as SST, clear-sky radiances, and aerosol, from Advanced Very High Resolution Radiometer (AVHRR)-like measurements. The ACSPO clear-sky mask (ACSM) identifies clear-sky pixels within the ACSPO products. This paper describes the ACSM structure and compares the performances of ACSM and its predecessor, Clouds from AVHRR Extended Algorithm (CLAVRx). ACSM essentially employs online clear-sky radiative transfer simulations enabled within ACSPO with the Community Radiative Transfer Model (CRTM) in conjunction with numerical weather prediction atmospheric [Global Forecast System (GFS)] and SST [Reynolds daily high-resolution blended SST (DSST)] fields. The baseline ACSM tests verify the accuracy of fitting observed brightness temperatures with CRTM, check retrieved SST for consistency with Reynolds SST, and identify ambient cloudiness at the boundaries of cloudy systems. Residual cloud effects are screened out with several tests, adopted from CLAVRx, and with the SST spatial uniformity test designed to minimize misclassification of sharp SST gradients as clouds. Cross-platform and temporal consistencies of retrieved SSTs are maintained by accounting for SST and brightness temperature biases, estimated within ACSPO online and independently from ACSM. The performance of ACSM is characterized in terms of statistics of deviations of retrieved SST from the DSST. ACSM increases the amount of “clear” pixels by 30% to 40% and improves statistics of retrieved SST compared with CLAVRx. ACSM is also shown to be capable of producing satisfactory statistics of SST anomalies if the reference SST field for the exact date of observations is unavailable at the time of processing.
The Advanced Clear Sky Processor for Oceans (ACSPO), developed at the National Environmental Satellite, Data, and Information Service (NESDIS), generates clear-sky ocean products, such as clear-sky radiances (CSRs), sea surface temperatures (SSTs), and aerosols from measurements in the atmospheric transparency windows in visible (VIS), near-infrared (NIR), and thermal infrared (TIR) spectral ranges at a sensor’s pixel resolution (see Table 1 for a list of acronyms used in this paper). Initially, ACSPO was developed to replace the operational Main Unit Task (MUT) system, which continues to operationally process data of the Advanced Very High Resolution Radiometer (AVHRR) (McClain et al. 1985; Ignatov et al. 2004). Recently the scope of ACSPO applications has been extended to the Meteosat Second-Generation Spinning Enhanced Visible and Infrared Imager (MSG SEVIRI), which is used as a proxy for the future Advanced Baseline Imager (ABI) onboard the Geostationary Operational Environmental Satellite R Series (GOES-R; Shabanov et al. 2009).
The ACSPO clear-sky mask (ACSM) is a module whose purpose is to identify clear-sky pixels within the ACSPO products. This paper describes the algorithm and performance of the ACSPO version 1.10, introduced on 15 May 2009, as it applies to AVHRR data processing. The AVHRR/3 instrument, flown on board the NOAA-15, -16, -17, -18, and -19 and MetOp-A satellites, measures the top-of-atmosphere (TOA) reflectance in three solar reflectance bands centered at 0.63 (Ch1), 0.83 (Ch2), and 1.61 μm (Ch3A), as well as brightness temperature (BT) in three TIR bands centered at 3.7 (Ch3B), 10.8 (Ch4), and 12 μm (Ch5). Only one of channels 3a and 3b is transmitted to the ground at any given time. For instance, on the midmorning satellites NOAA-17 and MetOp-A Ch3B is “on” (and hence Ch3A is “off”) on the dark side of the earth, whereas on the sunlit part of the orbit, these positions are switched over automatically to a “Ch3A on/Ch3B off” mode (for more information, see the NOAA KLM user’s guide online at http://www.ncdc.noaa.gov/oa/pod-guide/ncdc/docs/klm/cover.htm). On the afternoon satellites NOAA-16, -18 and -19, Ch3B is on all the time. The AVHRR data are available in two formats with different spatial resolutions. In the global area coverage (GAC) format, the AVHRR scan is comprised of 409 fields of view (pixels) of 4-km size at nadir. In the local area coverage (LAC) mode, available on National Oceanic and Atmospheric Administration (NOAA) satellites, and in the global full-resolution area coverage (FRAC) format (enabled on MetOp-A), every scan includes 2048 pixels of 1-km size at nadir. On the NOAA satellites GAC data are produced during onboard data processing. In the case of MetOp-A only FRAC data are transmitted to the ground, and GAC data are operationally generated from FRAC data at the NOAA/NESDIS Office of Satellite Data, Processing, and Distribution (OSDPD) with the same algorithm as used onboard NOAA satellites.
ACSM builds upon the Clouds from AVHRR Extended Algorithm (CLAVRx; Heidinger et al. 2002; Heidinger 2004), which traces back to CLAVR-1 (Stowe et al. 1999), which in turn has grown out of the MUT. While the focus of CLAVRx has been mostly on cloud detection and typing both over sea and land at a pixel resolution, the goal of ACSM is to detect and screen out the ocean pixels, useless for clear-sky products, while preserving as many useful pixels as possible. Achieving this goal requires closer consideration of cloud effects on the specific products (e.g., Cayula and Cornillon 1996; Martins et al. 2002; Pellegrini et al. 2006), in our case SST and CSR. For this reason the emphasis in ACSM has been made on using simulations with clear-sky TIR radiative transfer model (RTM) and retrieved SST rather than on exploiting radiative properties of clouds. Another difference between CLAVRx and ACSPO is that ACSPO less relies on using reflectance properties of clouds in the visible spectral range. Reflectance-based cloud tests are not applicable at night and often fail at big view zenith angles and in the glint area. Extensive using of these tests deteriorates day/night consistency and spatial uniformity of clear-sky masking results. The performance of TIR ACSM tests in the daytime is comparable to the performance of the most effective CLAVRx reflectance tests. As discussed in sections 2 and 5e, the ACSPO version 1.10 inherits two CLAVRx reflectance-based tests but, as shown in section 6, these tests add very little to the overall ACSM performance.
In general, clear-sky identification, based on top-of-atmosphere TIR observations, is possible because cloud-induced variations in BT go far beyond the boundaries of the clear-sky domain. While detection of large radiative contrasts, caused by cold clouds on the background of the warm sea surface, is a relatively simple task, a real challenge is to discriminate between clear-sky pixels and low-contrast (warm low stratus, semitransparent, subpixel) clouds. A capability of online RTM simulations is critically important from this standpoint. Accordingly, ACSPO incorporates the Community Radiative Transfer Model (CRTM), version 1.1 (user’s guide available online at http://www.star.nesdis.noaa.gov/smcd/spb/CRTM/crtm-code/CRTM_UserGuide-beta.pdf). The inputs for the CRTM are numerical weather prediction (NWP) information, such as the (AVHRR-based) 0.25° daily high-resolution blended SST (DSST; Reynolds et al. 2007) and the 6-h 1° atmospheric fields from National Centers for Environmental Prediction (NCEP) Global Forecast System (GFS) (http://nomad3.ncep.noaa.gov/pub/gfs/rotating/). Clear-sky BTs and their derivatives with respect to SST are computed at the nodes of the GFS grid and bilinearly interpolated to all ocean AVHRR pixels. The accuracy and precision of online clear-sky RTM simulation is about 0.5 K (Liang et al. 2009; Liang and Ignatov 2010, manuscript submitted to J. Geophys. Res.). DSST is also bilinearly interpolated to AVHRR pixels, which provides the first-guess SST field TR. Deviation of retrieved SST, TS, from TR—ΔTS = TS − TR—is used in ACSM as a cloud predictor. In addition, as shown in section 3, accounting for ΔTS further improves accuracy and precision of BT simulations.
The most traditional approach to operational cloud masking implies classification of pixels into a few discrete categories from “clear” to “cloudy” using a set of cloud tests in which cloud predictors, constructed from observed radiances, are compared against thresholds, predefined as functions of observational conditions (e.g., Saunders 1986; Saunders and Kriebel 1988; Stowe et al. 1999; Heidinger 2004; Derrien and Le Gleau 2005; Dybbroe et al. 2005). Typically, the following types of cloud tests are used:
various “gross” tests, which cut off apparent cloud manifestations by unrealistically cold BT in TIR channels or retrieved SST, or by high reflectances in VIS channels in the daytime;
a set of spectral tests, which exploit relationships between BTs and reflectances, observed in different channels; and
one or more texture tests, which detect subpixel cloud by higher spatial variability in BT or reflectances.
The predictors for the spectral tests (i.e., functions of observed BTs or reflectances, used as cloud indicators) are typically derived from (or justified by) the notions on radiative properties of clouds, while the corresponding thresholds are established either empirically or from offline RTM simulations.
An alternative to the multiple thresholding is the Bayesian approach (Uddstrom et al. 1999; Murtagh et al. 2003; Merchant et al. 2005, 2009a). The algorithm of Merchant et al. (2005), which became operational for GOES-12 (Maturi et al. 2008; Merchant et al. 2009a), uses RTM simulations and NWP information to construct the a posteriori probability of the pixel being clear-sky as a function of observed BTs and a local texture parameter. The algorithm reduces to a single test in which the above probability is calculated and compared against the predefined threshold. This reduction in the number of cloud tests is achieved at the expense of using a large amount of a priori information, including a Gaussian multivariate statistical distribution of NWP variables and a probability density function (PDF) of BTs over cloudy areas.
The way ACSM assimilates online RTM simulations requires less a priori information. The cloud-masking problem is posed as testing observed BTs for adequacy with CRTM. The model is considered adequate to observations if, first, it fits the observations with a predefined accuracy and, second, the values of model variables, at which the accuracy is achieved, are within the predefined range (e.g., Bard 1973). In the context of clear-sky identification, this means that in clear-sky areas CRTM is expected to fit the observed BTs within predefined uncertainty intervals. A priori information, needed for the adequacy check, includes NWP expectations of those atmospheric and surface variables, which are used as the input for CRTM, and the limits of allowable variations in those variables, which participate in the process of fitting observations with CRTM. In ACSPO version 1.10 TS is the only variable retrieved from TIR observations. As shown in section 3, although the accuracy of approximation of observed BTs with this single variable is limited, it is useful for clear-sky identification purposes.
The main cloud predictors in ACSM are the RMS residual of BT approximation and ΔTS. Using ΔTS as a cloud predictor offers several important advantages. First, TS is supported with detailed a priori information in the form of both climate and NWP fields. Second, TS is estimated as a combination of BTs, designed to minimize the sensitivity to the atmospheric transmission variations, thus allowing direct comparison against the reference SST field. Third, the cloud tests based on TS directly address cloud contamination within the SST product. Furthermore, TS has been used among other cloud predictors in many cloud-masking algorithms (e.g., Cayula and Cornillon 1996; Heidinger 2004; Derrien and Le Gleau 2005; Dybbroe et al. 2005; Pellegrini et al. 2006, to name a few). It should be noted, however, that posing too conservative restrictions on ΔTS in the cloud tests can result in rejecting real SST variations and in artificial forcing retrieved TS field to TR. In ACSPO this effect is minimized by accounting for real characteristics of accuracy of both TS and TR. As described in sections 4 and 5, the restrictions, imposed on “realistic” TS variations, account for global biases in ΔTS, which are estimated within ACSPO prior to ACSM, and the estimated local TR errors, which are a part of the DSST product.
Since the CRTM adequacy check does not guarantee 100% identification of clear-sky pixels, the ACSM includes additional tests to filter out the pixels, subject to residual clouds.
The goal of ACSM is to distinguish between “clear” pixels, usable for SST and “cloudy” pixels, useless or unreliable for SST. Respectively, the performance of ACSM can be characterized with the rates of misclassifications of clear pixels as cloudy and cloudy pixels as clear. Both types of misclassifications affect the quality of the SST product. Misclassifications of clear pixels as cloudy reduce the amount of pixels, available for the further SST analysis. Misclassifications of cloudy pixels as clear reduce mean ΔTS value (bias), increase the standard deviation (STD) of ΔTS distribution, and cause this distribution to deviate from a Gaussian shape. In this paper, the performance of ACSM and CLAVRx is evaluated and compared in terms of amount of ocean pixels, classified as clear, and statistics of ΔTS. These characteristics are accumulated from processing GAC AVHRR observations, made on 100 orbits of each of four AVHRR-carrying platforms—NOAA-16, NOAA-17, NOAA-18, and MetOp-A—during the time period 1–7 August 2008. This time period is long enough to ensure significance of global SST statistics. [For variability of SST and BT statistics during other time periods, the reader is referred to http://www.star.nesdis.noaa.gov/sod/sst/micros/ (Liang and Ignatov 2010, manuscript submitted to J. Geophys. Res.) and http://www.star.nesdis.noaa.gov/sod/sst/squam/ (Dash et al. 2010). The long-term time series of SST and BT statistics presented therein show that variations in these statistics are caused mainly by ACSM changes between subsequent ACSPO versions rather than by seasonal trends.]
2. The flowcharts of CLAVRx and ACSM
In ACSPO development, CLAVRx (version 4, 12 May 2006) was initially adopted as a first-cut cloud mask. This version of CLAVRx is a modification of the operational version delivered soon after the launch of MetOp-A. CLAVRx development continues and the results shown in this paper may not be representative of the performance of the latest versions. CLAVRx separates ocean pixels from land and ice using the University of Maryland’s 8-km land and ice masks (available online at http://www-surf.larc.nasa.gov/surf/pages/sce_type.html). The flowchart of the CLAVRx cloud masking process is shown on Fig. 1. CLAVRx classifies all ocean pixels into four categories—clear, probably clear, probably cloudy, and cloudy—based on a series of cloud tests. The pixel is classified as cloudy if it fails at least one out of the 11 cloudy tests listed on Fig. 1. The pixels, which pass all cloudy tests and have at least one bordering cloudy pixel, are labeled probably cloudy. Other pixels are marked clear or probably clear based on the results of three “probably clear” tests. The four cloudy tests—the reflectance gross contrast test (RGCT), reflectance ratio contrast test (RRCT), channel 3a/3b albedo test (C3AT), and channel 3B emission test (EMS3bT)—exploit reflectance properties of clouds in the visible and near-infrared spectral ranges and are applicable only in the daytime and outside the sun glint area. Four other tests—the thermal gross contrast test (TGCT), uniform low stratus test (ULST), “four minus five” test (FMFT), and “three minus five” test (TMFT)—exploit radiative properties of clouds in the thermal IR spectral range. The channel 4 climatology test (CTGCT) cuts off cold pixels if the BT in the AVHRR Ch4 is lower than a local minimum clear-sky value taken from the precalculated static dataset. The CSST test checks for consistency between the SST estimated from AVHRR measurements, with a reference SST field [1° Reynolds weekly optimum interpolation SST (OISST; Reynolds et al. 2002)]. (Note that in the considered CLAVRx version the algorithm for SST estimation, used in CSST, is different from the one used in operational TS). The “probably clear” CLAVRx tests detect residual fractional and semitransparent cloudiness by elevated local spatial variability of AVHRR radiances.
The flowchart of the ACSM, used in ACSPO version 1.10, is shown in Fig. 2. This version of ACSPO uses the same land mask as CLAVRx and generates ice mask from the dataset of sea ice concentration available within the DSST dataset. The ocean pixel is used for SST if the ice concentration is less than 10%. Unlike CLAVRx, ACSM classifies ocean AVHRR pixels over ocean into three categories: clear, probably clear, and cloudy. The “probably cloudy” category, available in CLAVRx, was omitted in ACSPO, which is a clear-sky application and therefore only requires information on whether the pixel is usable for SST or not. The “probably clear” category is kept for users with less stringent quantitative requirements for SST. ACSM exploits the input of other ACSPO modules (not shown on Fig. 2), which calculate TS for all ocean pixels, simulate clear-sky BTs, and estimate biases between TS and TR and between measured and simulated BTs.
The pixel is classified as cloudy if it fails at least one of the seven cloudy tests. Three of the cloudy tests—the RTM test, static SST test, and adaptive SST test—contribute to the CRTM adequacy check. These tests are employed at all solar zenith angles during day and night. The RTM test verifies the accuracy of approximating the observed BTs with clear-sky BTs, simulated with CRTM using TS and NCEP GFS atmospheric data as input. The static SST test generates the first-guess cloudy and clear clusters by detecting unrealistically cold ΔTS and then the adaptive SST test refines their boundaries by analyzing statistics of clear and cloudy pixels within the neighborhood of the tested pixel. The group of cloudy tests also includes the tests, inherited from CLAVRx, that detect residual cloudiness, which might have not been captured by the CRTM adequacy check. Two of these tests, TMFT and ULST, use the AVHRR Ch3b and are only applicable in the nighttime. These tests were adopted from CLAVRx with modifications to reduce their false cloud detection rate (Petrenko et al. 2008). The other two cloudy tests, C3AT and RRCT, are based on reflectance channels and are used in the daytime only. The two latter tests were adopted from CLAVRx without changes.
The pixels that pass all cloudy tests are preliminarily classified as clear. They are subsequently tested with the SST uniformity test, which detects fractional subpixel cloudiness by elevated spatial variability in retrieved SST and reclassifies a part of the clear pixels as probably clear.
3. Regression SST and CRTM simulations
ACSPO version 1.10 estimates TS with regression algorithms (McClain et al. 1985). During the daytime, the split-window nonlinear SST (NLSST) algorithm is used:
During the nighttime, the multichannel SST (MCSST) algorithm is used:
Here, T3B, T4, and T5 are observed BTs in AVHRR Ch3B, Ch4, and Ch5; a0, a1, a2, a3, and b0, b1, b2, b3, b4, b5 are coefficients derived from the regression of observed BTs against in situ SST measurements. ACSPO version 1.10 adopts the regression coefficients from the MUT system without change.
Consequently, two estimates of SST are available within ACSPO—reference SST, TR, and regression SST, TS—and two clear-sky approximations of observed BTs can be produced using either TR or TS along with the same vector of NWP atmospheric variables X:
Here, TCS is a vector of simulated clear-sky BTs and F is a clear-sky CRTM function; D is a vector of BT derivatives with respect to SST, computed on the GFS grid and interpolated to AVHRR pixels; TCS includes three components (Ch3B, Ch4, and Ch5) in the nighttime and two components (Ch4 and Ch5) in the daytime; and TCS(TR) is obtained by simulation on the 1° grid and does not include components with spatial scales less than 1° and temporal variations on scales shorter than one day. In contrast, TCS(TS) contains a pixel-scale timely component, introduced by the second term on the right-hand side of (4). This makes TCS(TS) a better fit for observed BTs than TCS(TR). Tables 2 and 3 show for the nighttime and daytime respectively the biases and the standard deviations of differences TB − TCS(TR) and TB − TCS(TS) for each of four AVHRRs onboard different platforms, calculated from an ensemble of clear-sky pixels, detected by ACSM during 100 orbits over 1–7 August 2008. For all satellites and during both day and night, TCS(TS) compares with AVHRR BTs more favorably than TCS(TR). The improvement in STD is greatest in the more transparent nighttime Ch3B and smallest in Ch5, likely because this latter channel is most affected by inaccuracy in NWP water vapor. The nighttime regression also reduces the biases in all cases except for Ch3b of NOAA-16. The abnormal positive BT biases in NOAA-16 Ch3b in the nighttime are caused by essential negative bias in TS, retrieved with the NOAA-16 regression algorithm (see Table 4 in section 4), which, in turn, is likely due to long-term calibration trends in the NOAA-16 channels. In the daytime the biases of TCS(TR) and TCS(TS) are comparable.
Since TCS(TS) approximates clear-sky BTs more accurately than TCS(TR), TCS(TS) is used as ACSM cloud predictor. Although the accuracy of this approximation is also limited, cloud-induced variations in observed BTs often exceed the errors of approximation of TB with TCS(TS) under clear-sky conditions. As a result, the residual TB − TCS(TS) appears to be a useful predictor of clouds. It is expected that a 2D RTM inversion algorithm (Merchant et al. 2008, 2009b), which is currently under implementation within ACSPO (Shabanov et al. 2009), will improve the accuracy of clear-sky simulation and make the CRTM adequacy check more efficient than with the regression algorithms.
4. SST and BT bias estimation within ACSPO
As shown in Tables 2 and 3, the approximation vector TCS(TS) is biased with respect to the observation vector TB. This may be due to inaccurate NWP data or incomplete (i.e., missing aerosol attenuation) or not fully accurate CRTMs (e.g., Liang et al. 2009). Also, TS can be biased with respect to TR (which is anchored to in situ SST; Reynolds et al. 2007) because of inaccurate regression coefficients. The biases of deviations ΔTB = TB − TCS(TS) and ΔTS may vary in time because of sensor calibration trends and orbital drift. If these biases are not accounted for, they can affect the results of clear-sky identification. Therefore, the biases are estimated online within ACSPO and accounted for in the corresponding ACSM tests. The most common method of clear-sky bias estimation is to average the corresponding deviations over clear pixels, determined with a clear-sky mask (e.g., Merchant et al. 2006; Liang et al. 2009). However, this method may create undesirable crosstalk between the classification of pixels by ACSM and the bias estimates. Therefore, the biases are estimated within ACSPO independently from and prior to ACSM as positions of peaks of ΔTS and ΔTB histograms, accumulated over ocean, without separating pixels into clear and cloudy. Although the percentage of “clear-sky” ocean pixels is typically only about 15%, the corresponding clear-sky BT and SST anomalies are concentrated in a relatively narrow range and form the peaks of all-sea-pixels histograms. ACSPO processes AVHRR data files sequentially by segments, containing 1024 GAC (768 LAC or FRAC) lines each; TS and TCS(TS) are estimated for all ocean pixels within the segment. The all-sea-pixels histograms of ΔTS and ΔTB are updated recursively for each new segment:
Here Si (i = 1, 2, …) is the histogram, accumulated from the ith data segment only; Hi is a recursively updated histogram after processing the ith segment. The coefficient k is 0.99000 for GAC and 0.99575 for LAC data. With k < 1, the contribution of a given segment to Hi reduces in time; so that for one day-old segment this contribution is equal approximately to 0.08. Since daytime and nighttime TS are produced with different algorithms (NLSST and MCSST, respectively), and SST is subject to diurnal variability, the SST and BT histograms are accumulated separately for day and night.
Figure 3 shows time series of biases in ΔTS, retrieved with the nighttime algorithm and estimated by averaging ΔTS over clear-sky pixels for a given orbit and from the location of the peak of the all-sea-pixels histogram, accumulated according to (5a) and (5b). Tables 2 and 3 compare the biases in ΔTB, estimated in both ways for each of the four platforms. As shown in Table 2, in the nighttime the differences between the two BT bias estimates do not exceed 0.12 K. During the daytime, as shown in Table 3, this difference is within 0.21 K. The estimates of ΔTS bias, obtained in two ways, are shown in Tables 4 and 5 for nighttime and daytime, respectively. At night the two bias estimates are consistent to within 0.09 K, with the biases varying among the platforms from −1.02 K for NOAA 16 to −0.14 K for MetOp-A. In the daytime the difference between the two biases is the smallest (<0.14 K) for the morning platforms NOAA-17 and MetOp-A and the greatest (0.32 K) for NOAA-16. Both for BT and SST, the difference between the two bias estimates is much smaller than the STD of the bias, estimated by averaging over the clear pixels.
5. ACSM tests
a. RTM test
The RTM test verifies the accuracy of fitting observed BTs, TB, with TCS(TS). The test uses the following condition for the pixel being clear-sky:
If condition (6) is met, the pixel is set to clear; otherwise it is set to cloudy. Here, BBT is the vector of BT biases, estimated as described in section 4 and N is the number of channels used in SST retrieval: N = 2 in the daytime (Ch4 and Ch5) and N = 3 in the nighttime (Ch3b, Ch4, and Ch5). In ACSPO version 1.10 the threshold DBT is set to 1 K2.
b. Static SST test
The predictor for the static SST test is ΔTS, corrected for the bias BSST, estimated as described in section 4. The test cuts off obviously unrealistic negative SST anomalies with the following condition:
If yes, then the pixel is set to clear; otherwise, it is set to cloudy. The threshold DSST is location and time specific; DSST is defined using the estimated SST error standard deviation σSST, available from the DSST dataset, as follows:
The values of σSST typically vary from 0.1 to 0.7 K, depending on location; hence, DSST is close to −2 K for most of the world’s ocean. The liberal setting of the threshold reduces the chance of false cloud detections. On the other hand, it may allow misclassifications of cloudy pixels as clear, especially at the boundaries of cloudy systems, often surrounded with relatively warm ambient cloudiness.
c. Adaptive SST test
The adaptive SST test further refines the initial classification by the static SST test. It detects ambient cloudiness at the boundaries of cloudy systems, initially determined with condition (7). The test analyzes local statistics of ΔTS in clear and cloudy clusters within a sliding window, surrounding the tested pixel. For GAC, the size of the sliding window was empirically chosen to be 15 × 15 pixels (60 × 60 km2). For FRAC and LAC, it was set to 31 × 31 pixels (31 × 31 km2) because of computation time limitations. All clear pixels within the window are tested with the following condition:
If yes, then the pixel is set to clear, otherwise it is set to cloudy. The value ρCLD in (9) is the difference between ΔTS in a given pixel and mean of ΔTCLD averaged over cloudy pixels within the sliding window, normalized to STD σCLD of ΔTS over cloudy pixels within the same window:
and ρCLR is ΔTS normalized to σCLR = DSST/3:
Parameters ΔTCLD and σCLD are subject to change on each iteration if new pixels are classified as cloudy according to condition (9). The procedure repeats itself until either the classification of the pixels within the window stabilizes or the tested (central) pixel in the window becomes cloudy.
d. Nighttime TIR tests
ACSM preserves two CLAVRx nighttime TIR tests, TMFT and ULST (Heidinger 2004), which cover a small yet statistically significant amount of cloudy pixels in addition to those detected by RTM and SST tests. Both these tests are used at solar zenith angles θ > 85° because of contamination of Ch3b with solar reflected radiance in the daytime. In the original test formulations, the pixel was classified as clear if the following conditions were met:
Here, T3B, T4, and T5 are observed BTs in Ch3B, Ch4, and Ch5; DTMFT(T4, θ) is a threshold, taken from the precalculated lookup table as a function of T4 and satellite view angle θ (e.g., Heidinger 2004). It has been found that in the original formulation these tests produced an undesirably high rate of misclassification of clear pixels as cloudy. At the same time, they allowed occasional leakages of too cold SST pixels into the clear cluster, indicating that the problem was due to incomplete accounting for cloud emission properties by the test predictors rather than to inaccurate threshold selection (e.g., Petrenko et al. 2008).
e. Daytime reflectance tests
ACSM adopts two CLAVRx daytime tests, RRCT and C3AT, without changes. These tests are applied within the range of solar zenith angles θ0 < 85°. The test condition for RRCT is
Here, R1 and R2 are observed reflectances in Ch1 and Ch2. The threshold DRRCT(θ, θ0, ϕ) was precalculated by Heidinger (2004) using the 6S radiative transfer model (Vermote et al. 1997) for clear-sky atmosphere as a function of satellite view zenith angle θ, solar zenith angle θ0, and solar azimuth ϕ. The test conditions for the C3AT test are as follows:
If the corresponding test condition is met, the pixel is clear. Here R3A is the observed reflectance in Ch3a and R3B is calculated reflectance in Ch3b (Heidinger 2004); D3A and D3B are thresholds, precalculated the same way as for DRRCT.
f. The SST spatial uniformity test
Residual subpixel clouds, missed by other cloud tests, can be detected by elevated spatial variability in observed BT or reflectance. This concept is based on the texture, or spatial uniformity, tests used in many cloud masking algorithms. Usually, the predictor for the texture test is spatial RMS variation in BT or reflectance in a small neighborhood of a given pixel (e.g., Ackerman et al. 1998; Kriebel et al. 2003; Heidinger 2004; Derrien and Le Gleau 2005; Merchant et al. 2005). The potential risk of using this predictor is a possible false detection of clouds in clear-sky ocean areas with high thermal gradients.
In ACSM, the uniformity test has the following peculiarities (e.g., Petrenko et al. 2008). First, it analyzes the field of TS rather than observed BTs (i.e., allowing screening residual cloud contaminations directly in the SST product. Second, the predictor for the ACSM uniformity test is the STD of the difference TS − median(TS) rather than STD of TS. Median(TS) is the TS field, passed through the 2D median filter. The window size for GAC was set to 3 × 3 pixels to avoid excessive loss of clear pixels. For FRAC and LAC in ACSPO version 1.10 it was set to 11 × 11, which is consistent with GAC window size in kilometers. In the latest ACSPO versions, the window size for FRAC and GAC has been reduced to 5 × 5 pixels in order to preserve more clear pixels. The threshold for the uniformity test is selected to be somewhat above the RMS level of random noise in SST. Since the median filter preserves regular contrasts but suppresses random noise (e.g., Gonzalez and Woods 2003), the difference TS − median(TS) is more sensitive to random variations in TS, typical for subpixel cloud effects rather than for more regular surface contrasts caused by ocean thermal fronts. This reduces the risk of misclassification of ocean fronts as cloudy pixels. Following CLAVRx, the pixels that fail the uniformity test are classified as probably clear.
g. Example of ACSM performance
Figure 4 demonstrates the performance of the ACSM tests with composite maps of ΔTS from nighttime MetOp-A observations over the Gulf of Mexico on 1 August 2008. Figure 4a shows the distribution of ΔTS without the clear-sky mask imposed. While most negative ΔTS values are caused by cloud contaminations, positive ΔTS values are mainly due to inaccuracy in TR, which has a 1-day time resolution and a spatial resolution of 0.25°. In particular, TR inaccuracy shows itself in coastal areas and in dynamic areas of the ocean. The RTM test detects most cold pixels (Fig. 4b), but a large part of them survive this test. Some pixels with positive ΔTS, at which condition (6) is not met, are also rejected. Figure 4c shows a combined effect of RTM and static SST tests. The static SST test additionally masks out the pixels that pass condition (6) but have excessively cold ΔTS. The effect of ambient clouds is noticeable in Fig. 4c: the cloudy pixels are often surrounded by relatively cold ΔTS. The adaptive SST test (Fig. 4d) eliminates a large fraction of these colder pixels. The TMFT and ULST tests additionally detect a small amount of cloudy pixels and the SST uniformity test reclassifies a number of clear pixels into the probably clear category (Fig. 4e).
6. Statistics of clear-sky ΔTS
Tables 4 and 5 compare the statistics of ΔTS over clear pixels detected by ACSM and CLAVRx in the nighttime and daytime, respectively. Globally, ACSM produces 30% to 40% more clear pixels than CLAVRx, and ΔTS distributions produced by ACSM have warmer biases and smaller STD. The only exception is NOAA-16 nighttime observations. This is because the ACSM flexibly accounts for biases in ΔTB and ΔTS (note that the nighttime bias for NOAA-16 ΔTS, estimated from the position of the histogram peak, is −1.01 K). The estimates of skewness and kurtosis in Table 5 and especially in Table 4 show that the distributions of ACSPO ΔTS are closer to Gaussian shape. Figure 5 shows histograms of ΔTS over clear pixels in a logarithmic scale. Both for CLAVRx and ACSM the ΔTS values were calculated with respect to the same reference SST field, TR. While identifying more clear pixels than CLAVRx, the ACSM performs more conservative screenings of cold ΔTS. On the warm side of the histograms, the ACSM preserves more clear pixels, thus reducing a false cloud detection rate. The CLAVRx histograms have heavy cold tails, which is the reason of increased skewness and kurtosis of ΔTS statistics by CLAVRx, as shown in Tables 4 and 5. The comparison of ACSM and CLAVRx testifies that the product-oriented clear-sky mask provides a higher quality of the clear-sky product than the generic cloud mask.
Figure 6 demonstrates the evolution of the clear-sky ΔTS histogram as the new ACSM tests are sequentially added. The statistics of the corresponding ΔTS distributions are presented in Table 6. The BT test rejects approximately 55% of sea pixels, mainly on the left cold wing of the histogram. However, a part of unrealistically cold pixels, at which CRTM approximates observations with sufficient accuracy, passes the BT test. The static SST test sharply cuts off the left wings of the histograms, rejecting about 16% of sea pixels. The adaptive SST test additionally rejects more than 6% of sea pixels in the neighborhood of cloud boundaries, determined by the static SST test, and makes the shape of the histogram closer to Gaussian. The CLAVRx tests (nighttime: TMFT and ULST; daytime: RRCT and C3AT) only slightly affect the clear-sky statistics of ΔTS. Finally, the uniformity test screens out about 7% of sea pixels. This test mainly affects the ΔTS bias, warming it up by 0.05 K in the daytime and by 0.07 K in the nighttime.
ACSPO version 1.10 essentially exploits the DSST, interpolated to AVHRR pixels; TR is used in the daytime regression SST algorithm (1) and in the ACSM static and adaptive SST tests. When processing AVHRR data in real time, DSST may be unavailable for the exact date of observations. In this case, the graceful degradation is provided by using the latest available TR data. The following simulation was performed to estimate the effect of using outdated TR on the clear-sky statistics of ΔTS. Figure 7 shows that the increase in the TR delay results in widening of the clear-sky ΔTS histogram. Note that while an outdated TR was used as a reference SST field in the ACSM tests, ΔTS values were calculated with respect to TR for the exact date of observations. Table 7 demonstrates that the growing TR delay causes cooling down ΔTS bias and increase of STD. Increasing skewness and kurtosis show that TR delays cause deviations of ΔTS histogram from the Gaussian shape. It is important that ΔTS statistics degrade gradually with the increase in TR delay.
7. Conclusions and outlook
The ACSPO clear-sky mask was developed using CLAVRx as a first cut. It was optimized and fine-tuned in an attempt to improve the quality of the SST product. The major features of the ACSPO clear-sky masking process are as follows:
ACSPO employs online clear-sky CRTM simulations and real-time NWP information. This allows posing the clear-sky masking problem as testing CRTM for adequacy with observed brightness temperatures. According to this concept, ACSM includes the RTM test, which evaluates the accuracy of fitting observed BTs with CRTM and a combination of the static SST test, which performs initial screening using liberal restrictions on negative deviations of regression SST from the reference SST field, and the adaptive SST test, which refines the initial classification based on statistics of clear and cloudy ΔTS in the neighborhood of the tested pixel.
The ACSPO incorporates estimation of global biases in retrieved SST minus reference SST and in observed BTs minus simulated clear-sky BTs. The biases are estimated online, upstream and independently from ACSM. Accounting for these biases in the ACSM tests enhances temporal and cross-platform consistency of AVHRR pixel classification by ACSM.
The spatial uniformity test is applied directly to retrieved SSTs rather than to observed BTs or reflectances. This allows direct screening of cloud contaminations in the product. The test has been reformulated to minimize false cloud detections over ocean thermal fronts but still efficiently detect random SST variations caused by subpixel cloudiness.
ACSM version 1.10 also inherits four CLAVRx tests, nighttime TMFT and ULST and daytime RRCT and C3AT, which additionally screen out residual clouds. However, the relative amount of pixels identified with these tests is small. These tests will be revisited in future versions of ACSM.
The new ACSM features mentioned above have improved the quality of the SST product. The product-oriented ACSM produces an average 30% to 40% more “clear” pixels, warmer ΔTS biases, and smaller STD compared to CLAVRx version 4 (dated 12 May 2006). During the nighttime, ACSM prevents misclassifications of pixels with too cold retrieved SST as clear. At the same time, ACSM is fairly insensitive to the reference SST field. Using a one-to-two-week delayed reference field instead of the one for the exact date of observations causes a slight gradual degradation in statistics of clear-sky SST anomalies.
Future improvements to ACSM will include the following:
Implementation of an SST retrieval algorithm based on multivariate RTM inversion, which will allow more accurate fitting of observed BTs with CRTM and hence make the CRTM adequacy check more efficient.
Further optimization of existing ACSM tests and development of new tests, particularly for more efficient use of AVHRR reflectance channels.
Special attention will be paid to improving the consistency between the daytime and nighttime clear-sky masks. Currently, the daytime and nighttime clear-sky masks are still different because of using different daytime and nighttime SST algorithms and different sets of cloud tests.
ACSM improves the statistics of clear-sky SST over CLAVRx version 4, mainly due to the use of online CRTM and near-real-time NWP and SST information. However, a graceful degradation of the clear-sky masking process should be provided in case the information required by the ACSM is unavailable. It is planned therefore that in future ACSPO versions the ACSM will be separated into two modules, cloud mask and quality control (Petrenko et al. 2009). While the quality control will extensively use real-time NWP information, the CLAVRx-like cloud mask will emphasize the use of static reference fields and precalculated thresholds. Under normal conditions the cloud mask will perform initial liberal cloud filtering to reduce the amount of pixels to be processed with computationally more expensive CRTM, SST algorithms, and quality control.
We thank our colleagues John Supper (NESDIS/OSDPD), Xing-Ming Liang, Feng Xu, and Prasanjit Dash (STAR/CIRA), Nikolay Shabanov (STAR/IMSG), and Denise Frey (OSDPD/QSS) for valuable assistance, discussions, and feedback at the different stages of work. Our special thanks go to the reviewers, whose constructive comments greatly helped us to improve the manuscript. This work is conducted under the Algorithm Working Group funded by the GOES-R Program Office. The views, opinions, and findings contained in this report are those of the authors and should not be construed as an official NOAA or U.S. Government position, policy, or decision.
Corresponding author address: Boris Petrenko, Room 601-4, 5200 Auth Rd., Camp Springs, MD 20746. Email: email@example.com