1. Introduction

Meteorological Operation (MetOp)-A, the first in a series of three planned European Organization for the Exploitation of Meteorological Satellites (EUMETSAT) polar-orbiting satellites, was successfully launched in October 2006 and carries a wide array of instruments for measuring various atmospheric, oceanic, and surface parameters. Included in the instrument suite are several heritage instruments provided by the National Oceanic and Atmospheric Administration (NOAA), such as the Advanced Very High Resolution Radiometer (AVHRR) and the Advanced Microwave Sounding Unit (AMSU). In addition to these heritage instruments, MetOp-A carries a new generation of advanced instruments, which include the Infrared Atmospheric Sounding Interferometer (IASI).

IASI is a cross-track-scanning Michelson interferometer that measures 8461 channels at 0.25 cm⁻¹ spacing between 645 and 2760 cm⁻¹ (3.6–15.5 μm) in a 2 × 2 array of circular footprints with a nadir spatial resolution of roughly 50 km × 50 km (with a corresponding single footprint spatial resolution at nadir of roughly 12 km). Spectral measurements from the IASI contain information on the vertical temperature profile, surface parameters (e.g., temperature, emissivity, reflectivity), clouds, and the vertical distribution of tropospheric and stratospheric trace gases such as H₂O, CO, CH₄, CO₂, HNO₃, and O₃ (Cayla 1993; Maddy et al. 2009). In addition, comparisons with other high spectral resolution spaceborne sounders, such as the Atmospheric Infrared Sounder (AIRS) flying onboard the National Aeronautics and Space Administration’s (NASA’s) Earth Observing System (EOS) Aqua platform, have demonstrated the excellent in-orbit calibration and performance of IASI. While both AIRS and IASI have demonstrated improvement to forecast models, the accurate treatment of clouds has long been a limiting factor to maximizing the utility of IR sounder data (Le Marshall et al. 2006; Collard and McNally 2009). This is true because clouds have a considerable effect on observed IR radiances, and at 12-km spatial resolution less than 10% of IASI footprints are expected to be cloud free. Methods to handle clouds are therefore required to optimally utilize IR sounder data in numerical weather prediction (NWP) models and for various other operational and research purposes.

There are several approaches for handling the effect of clouds in the IR, the most common of which include the following: avoiding the clouds by screening for clear-sky footprints, directly modeling the radiative effect of the clouds using sophisticated radiative transfer and cloud microphysical models, and estimating the clear-sky portion of an IR scene by using a number of adjacent and variably cloudy footprints coupled with an estimate of the clear-sky radiance from a forecast model or collocated satellite instrument that is less likely to be affected by clouds. The last approach, termed cloud clearing, is currently used at NOAA/National Environmental Satellite, Data, and Information Service (NESDIS) for operational IASI processing and is briefly described in the following.

NOAA currently operationally processes 100% of IASI data from calibrated and apodized level 1C (L1C) spectral measurements to geophysical level 2 (L2) products and distributes these products to the NOAA/Comprehensive Large Array-Data Stewardship System (CLASS) (available online at http://www.class.ngdc.noaa.gov/saa/products/welcome). The current algorithm used to produce the L2 products from IASI is largely based on the AIRS science team (AST) algorithm (Aumann et al. 2003), including the fast radiative transfer algorithm (RTA) (Strow et al. 2003) and fast eigenvector regression (Goldberg et al. 2003; Zhou et al. 2008), as well as cloud-clearing and physical retrieval methodologies (Susskind et al. 2003), and is described in the IASI L2 Algorithm Theoretical Basis Document (ATBD).

The current NOAA operational cloud-clearing methodology uses the same fast eigenvector regression methodology that is described in Goldberg et al. (2003) to provide temperature and moisture geophysical profiles as well as surface parameters using MetOp-A cloudy-sky IASI spectral measurements and AMSU microwave sounder brightness temperatures as inputs. These regression output parameters are then matched with climatological trace gas abundances (e.g., O₃, N₂O, etc.) and used as inputs to an RTA (Strow et al. 2003) to produce a clear-sky radiance estimate. This clear-sky radiance estimate is then used to extrapolate cloud-cleared radiances (CCs) from a spatial interpolation of multiple cloudy infrared footprints in the IASI 2 × 2 array of footprints collocated to the microwave footprint. The 2 × 2 array of footprints is sometimes referred to as a field of regard (FOR).

As the surface-leaving radiance in the 2 × 2 array of IASI footprints becomes obscured because of increasing cloudiness, the regression operator relies more heavily on the microwave measurements to determine the atmospheric profiles and surface temperature. Unfortunately, broad vertical weighting functions and possible sidelobe contamination limit the information content of microwave sounders such as AMSU in the lower atmosphere. In addition, because the clear-sky estimate is produced via a radiative transfer model, accurate a priori assumptions about infrared surface characteristics, such as emissivity, are required to compute accurate radiances. Therefore, scenes with low-altitude clouds where the surface-leaving radiances are constrained entirely by the microwave measurements can produce errant CCs that, in turn, produce errant sounding products. Radiances computed from the corrupted products can agree with the measurements within the error budget, making detection and removal of the errant scenes impractical. These and other limitations in using AMSU for cloud clearing as applied to the AIRS cloud-clearing algorithm were discussed in Barnet et al. (2005) and form part of the impetus for the work described in this paper.

In this paper we will describe future upgrades to the operational cloud-clearing algorithm being used for IASI processing within NOAA/NESDIS. Specifically, our new cloud-clearing algorithm leverages off of the MetOp-A AVHRR Clouds from AVHRR (CLAVR-x) cloud mask (Heidinger 2010; Thomas et al. 2004) to provide high-quality, high spatial resolution IR window clear-sky scene radiance estimates required for cloud-clearing inputs and quality assurance. For instance, Wang and Cao (2008) showed that the mean difference between collocated AVHRR and IASI for AVHRR channels 4 and 5 is generally less than 0.4 K, with a standard deviation of 0.3 K. Therefore, the direct use of AVHRR clear-sky measurements decreases limitations of the current algorithm to provide high-quality clear-sky radiance estimates throughout the atmospheric column, and especially near the surface to a high degree of accuracy. In section 2 we describe the IASI–AVHRR collocation procedures and the AVHRR cloud mask products. In section 3 we fully describe our synergistic IASI–AVHRR cloud-clearing algorithm and provide an analysis of the performance of the new algorithm in section 4.

2. AVHRR–IASI collocation and AVHRR CLAVR-x cloud masking

AVHRR/3 is a six-channel imaging and scanning radiometer that measures three solar channels in the visible–near infrared region and three thermal infrared channels. AVHRR has an instantaneous field of view of 1.3 mrad, corresponding to a 1.1-km footprint at nadir. The cross-track scan swath of the instrument extends ±55.4° on either side of nadir, providing a swath that extends beyond the IASI cross-track swath width of ±48.3° on either side of nadir. Two-point (deep space and internal blackbody) calibration of the thermal IR channels is performed on a scan-line-by-scan-line basis, and a prelaunch nonlinearity correction has been performed on the data (Sullivan 1999).

a. Collocation between AVHRR and IASI measurements

A typical IASI spectrum and the spectral response functions (SRFs) of AVHRR Channels 4 and 5 are shown in Fig. 1. IASI’s spectral range fully overlaps AVHRR longwave thermal infrared channels 4 and 5, with nominal spectral centroids of 10.8 and 12 μm, respectively. The complete spectral overlap between IASI and AVHRR in the longwave IR window region provides a unique opportunity to characterize subpixel variability within the IASI footprints because these split window thermal infrared channels are generally used to derive sea surface temperature and other surface properties. High spatial resolution AVHRR measurements collocated within the IASI spatial footprints therefore ideally enable the detection and removal of the spectral fingerprint of clouds from IASI spectra.

IASI spectrum for Air Force Geophysics Laboratory (AFGL) U.S. Standard Tropical Atmosphere, 1976 (black) and overlaid AVHRR SRFs (red) for AVHRR channels 4 and 5.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

IASI spectrum for Air Force Geophysics Laboratory (AFGL) U.S. Standard Tropical Atmosphere, 1976 (black) and overlaid AVHRR SRFs (red) for AVHRR channels 4 and 5.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

IASI spectrum for Air Force Geophysics Laboratory (AFGL) U.S. Standard Tropical Atmosphere, 1976 (black) and overlaid AVHRR SRFs (red) for AVHRR channels 4 and 5.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

Collocation between IASI and AVHRR uses an algorithm developed for use with AIRS and Moderate Resolution Imaging Spectroradiometer (MODIS) data on NASA’s Aqua satellite (Sun et al. 2006) and is an extension of the algorithms described in Li et al. (2005). Explained briefly, this algorithm finds the closest AVHRR observation to the center of the IASI footprint and performs an outward search to find all of the AVHRR pixels falling within the IASI footprint. A weight, herein termed the integrated point spread function (IPSF), is assigned to each collocated AVHRR pixel, which depends on the angular difference between the AVHRR pixels and the center pixel. For instance, weights nearest to the center of the IASI footprint are given a value of 1, while weights on the edge of the IASI footprint are given a weighting of 0.

b. CLAVR-x cloud masking

The CLAVR-x product (Thomas et al. 2004; Heidinger 2010) provides high spatial resolution (≈1 km) cloud masking in one of four categories, with 0 corresponding to confidently clear, 1 corresponding to probably clear, 2 corresponding to probably cloudy, and 3 corresponding to cloudy. In our processing we integrate various surface parameters using the CLAVR-x mask to determine all-sky (mask = 0, 1, 2, 3), confidently clear-sky (mask = 0), and confidently and probably clear-sky (mask = 0, 1) AVHRR radiances as well as the average cloud-top temperature and pressure and the standard deviation of the cloud-top temperature from the CLAVR-x product. For instance, for pixels determined to be confidently clear sky by the CLAVR-x cloud mask, we calculate the clear AVHRR radiance in AVHRR spectral band i in each IASI footprint as follows:

View Expanded

In Eq. (1), is the radiance of the confidently clear-sky AVHRR pixel l, is the number of confidently clear-sky AVHRR pixels collocated to the IASI footprint, and IPSF_l is the integrated point spread function for pixel l. We have also assumed that the IPSF has been normalized to unity.

Example collocations for the single day of IASI and AVHRR data obtained on 3 October 2010 are shown in Fig. 2. IASI measurements R at wavenumber ν (R_ν) are spectrally averaged onto the AVHRR SRF for channel i, SRF_i,ν using

View Expanded

and plotted against the spatially collocated and averaged [using Eq. (1)] all-sky and confidently clear-sky AVHRR measurements. The number of successful collocations, that is, those corresponding to cases where both the IASI and AVHRR quality assurance (QA) flags indicate highest quality, is 1 268 749. The correlation between the all-sky AVHRR measurements and IASI measurements for this set of cases is very high, giving a value of 0.998 for both channels considered. A summary of statistics for the IASI–AVHRR collocations for both all-sky and clear-sky (92 347 cases) are provided in Table 1 for AVHRR channels 4 and 5.

Collocations of AVHRR BTs and IASI BTs for 3 Oct 2010. IASI data were spectrally convolved onto the AVHRR channel 4 SRF, while AVHRR was spatially convolved onto the IASI footprints. Collocations for all cases are shown (black dots), while collocation for cases determined by the CLAVR-X cloud mask to be clear are also shown (red). Results for AVHRR channel 5 are similar.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Collocations of AVHRR BTs and IASI BTs for 3 Oct 2010. IASI data were spectrally convolved onto the AVHRR channel 4 SRF, while AVHRR was spatially convolved onto the IASI footprints. Collocations for all cases are shown (black dots), while collocation for cases determined by the CLAVR-X cloud mask to be clear are also shown (red). Results for AVHRR channel 5 are similar.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

Collocations of AVHRR BTs and IASI BTs for 3 Oct 2010. IASI data were spectrally convolved onto the AVHRR channel 4 SRF, while AVHRR was spatially convolved onto the IASI footprints. Collocations for all cases are shown (black dots), while collocation for cases determined by the CLAVR-X cloud mask to be clear are also shown (red). Results for AVHRR channel 5 are similar.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

Table 1.

Bias, standard deviation (std dev) and correlation coefficient r between spatially convolved AVHRR measurements and spectrally convolved IASI measurements for AVHRR channels 4 and 5 for both all-sky and clear-sky cases.

View Table

Similar to the findings discussed in Wang and Cao (2008), the differences between AVHRR and IASI for our nonuniform and clear-sky scenes show temperature- and scan-dependent biases. These temperature-dependent biases suggest possible problems with nonlinearity in AVHRR calibration (Wang and Cao 2008). In what follows, we have performed a brightness temperature–dependent bias correction to the AVHRR measurements ,

View Expanded

to make them better agree with the IASI measurements. The correction coefficients are listed in Table 2. We have also found scan angle–dependent biases between IASI and AVHRR that are symmetric about nadir and are on the order of ≈0.2 K for both AVHRR channels 4 and 5. At this point, we have not attempted to correct these scan angle–dependent differences because they are much smaller than the sidelobe corrections required to use the AMSU. Wang and Cao (2008) discuss these scan angle–dependent differences and the possible causes for the scan angle dependence in more detail.

Table 2.

Slope and offset coefficients between AVHRR and IASI measurements.

View Table

3. A review of cloud-clearing methodology

The two-spot, adjacent footprint cloud-clearing methodology assumes that the spectral radiance in two adjacent footprints, denoted , differ only in the product of the cloud fraction and cloud emissivity according to

View Expanded

where is the measured radiance in footprint j, and and are the true clear-sky and true cloudy-sky column radiances, respectively, for footprint j = 1, 2. By defining a new parameter , and assuming the cloud emissivities are equal in footprints 1 and 2 (i.e., ), we can simultaneously solve both equations for to enable estimation of the cloud-cleared radiance , giving

View Expanded

With those substitutions, the problem of determining the cloud-cleared radiance in the two adjacent footprints then reduces to the determination of the parameter η. The authors note that is not guaranteed to be exactly equal to the true clear-sky scene radiance from Eq. (4) because measurements are susceptible to instrument noise and there is a possibility that our assumption that the two footprints differ only in their respective cloud fractions is not true (e.g., water vapor or surface variability between the two adjacent footprints).

Smith (1968), Chahine (1974), and McMillin and Dean (1982) showed that a single channel or small subset of channels can remove the radiative effect of clouds from entire spectrum provided that an independent estimate of the clear-sky radiance for the two footprints is given. In our notation, the solution for adjacent spot, single-channel cloud clearing is given by solving Eq. (5) for η, yielding

View Expanded

Chahine (1977) and Chahine et al. (1977) showed that the formulation of the cloud-clearing equations in the η notation enables determination of clear-sky IR spectra affected by J − 1 cloud formations in J footprints. Joiner and Rokke (2000) employed the η notation in a variational context to cloud clear Television and Infrared Observation Satellite (TIROS) Operational Vertical Sounder data with J = 3. Susskind et al. (2003) used the 3 × 3 array of AIRS pixels collocated to an AMSU footprint and Chahine’s η methodology to determine as many as 4ηs to cloud clear AIRS spectra. The AIRS approach provides the basis for the operational IASI–AMSU cloud-clearing algorithm currently employed by NOAA/NESDIS, and therefore shares many of the benefits and limitations of the AIRS algorithm.

As described in section 1, the coupled AIRS–AMSU or IASI–AMSU algorithm relies heavily on the AMSU measurements to provide inputs to a forward model, which in turn provides the clear-sky estimate . The IR forward model requires the complete atmospheric state, including temperature, moisture, and trace gas profiles as well as IR surface properties, such as surface emissivity and reflectivity in order to compute . AMSU measurements are generally only sensitive to temperature profiles, surface temperature, and moisture profiles requiring one to make assumptions regarding the IR surface properties and trace gas profiles. Therefore, any biases the AMSU geophysical profiles, either assumed surface parameters or trace gas profiles, and/or biases in the forward model itself directly affect the determination of η and the error characteristics of the inferred .

Clear-sky radiance estimates from collocated high spatial resolution imager measurements (e.g., from MODIS) can also be used to remove the effects of clouds from IR sounder measurements. For instance, Smith et al. (2004) and Li et al. (2005) showed that the collocated AIRS IR sounder and Aqua MODIS imager measurements enable direct calculation of high-quality cloud-cleared radiances without the use of a forward model to estimate . Their methods rely on the high spatial resolution MODIS measurements and cloud mask to estimate clear-sky measurements in MODIS IR spectral bands spatially collocated and averaged onto the AIRS footprints. The use of IR spectral bands covering the spectral domains sampled by the AIRS instrument enables direct comparison of the clear-sky MODIS measurements to AIRS, and therefore does not require a priori assumptions about the geophysical state (i.e., surface properties, trace gas concentrations, and/or water vapor abundances) to enable calculation of clear-sky radiances.

In a similar fashion, in the following we utilize the spatially averaged and collocated AVHRR clear-sky radiances aggregated onto the IASI footprints and denoted to produce η and . We expand on the methodology described in Li et al. (2005) by providing an error analysis of with respect to the composite input variables to the CC algorithm. Explicit treatment of the errors induced through cloud clearing leads to an improved quality control scheme and a more optimal selection of footprints enabling a reduction in the noise in the cloud-clearing algorithm. In addition, our use of the η notation has the advantage that multiple cloud formations (e.g., type, height, etc.) can be cleared and will be the subject of a future publication.

a. A description of the AVHRR/IASI cloud-clearing algorithm

Although there are various methods to determine η given and , the method of least squares enables a simple solution to our problem. The least squares problem for determining η from the clear-sky AVHRR pixels can be written as the minimization of the objective function given in Eq. (7):

View Expanded

In Eq. (7), N_chan corresponds to the number of AVHRR channels used, which is 2, and is the sum of the expected variance in the AVHRR clear-sky radiance in channel i and the expected variance in the spectrally averaged IASI radiance in channel i. For simplicity we assume that the radiance unit squared; however, we have found that modifying these weights does not affect the quality of the cloud-cleared radiances to a large degree. Taking the derivative of Eq. (7) with respect to η,

View Expanded

and setting the result equal to zero enables minimization of this objective function. Solving the previous equation for η yields the least squares solution for η and is given by

View Expanded

From Eq. (9), we can see that the magnitude of extrapolation parameter η linearly depends on the contrast and nonlinearly depends on the contrast between the adjacent footprints . In the next section we further develop the mathematics that enable the characterization of the response of Eqs. (5) and (9) to the contrast between various input variables to the algorithm and to the uncertainties in the input variables themselves.

b. Cloud-clearing error estimates

The potentially large corrections (e.g., many tens of kelvins) that are required by cloud clearing warrant quantification of the uncertainties in the cloud-clearing process. Calculating the differential of Eq. (5) as

View Expanded

where , enables estimation of the bias in the cloud-cleared radiance . The bias is composed of two terms—the first arising from fact that noise in is a linear combination of instrument noise in and , and denoted and respectively; and the second arising from biases in η. Generally speaking, the magnitude of δη, which is a function of the uncertainty in the composite variables used to determine η (e.g., and ), and the contrast between the footprints used to derive η modulates the magnitude of the spectral correlation of . We also note that self-apodization of the spectra resulting from instrument FOV geometry and instrument imperfections as well as user-selected apodization of the interferogram (e.g., either Gaussian or Blackmann apodization) to reduce sidelobes also introduces spectral correlation to the IASI random noise. Because the first two terms of Eq. (10) are linearly proportional to η, we can therefore expect that the magnitude of η itself will also effect the magnitude of the spectral correlation in . In addition, because η depends on the squared reciprocal of the contrast between footprints, δη will be a strongly nonlinear function of the contrast between footprints, multiplied by the uncertainties in the two footprints used for cloud clearing.

On a case-by-case basis it is not generally possible to estimate the bias in ; however, calculating the covariance of Eq. (10) and taking the expectation of the result enables us to statistically estimate some of the terms of the error covariance of . It can be shown that the error covariance of the cloud-cleared radiance takes the form

View Expanded

where E(·) denotes the statistical expectation. Although small differences in the calibration between the IASI footprints exist (see Collard and McNally 2009), statistically, the expectation of noise in the IASI footprints should be equal to the spectral error covariance of the IASI instrument, which we denote as S_e (i.e., ). We can therefore rewrite Eq. (11) as

View Expanded

It is clear from Eq. (12) that the cloud-clearing process amplifies the spectral error covariance of the IASI measurements S_e by a factor of

View Expanded

the square root of which we will term the amplification factor. From Eq. (13), we can also see that as η → 0 (i.e., the footprints are cloud free), then α(η) → 1.

The error analysis described by Eqs. (11) and (12) is complicated by the cross-correlation terms [e.g., ] that will be difficult to estimate. Nonetheless, from these equations it is clear that a well-designed cloud-clearing algorithm should minimize both the random noise amplification and the spectral error correlation introduced by the cloud-clearing process. This can be accomplished for each set of IASI footprints and collocated AVHRR clear-sky pixels aggregated onto the IASI footprint by selecting

1. cases where the contrast between footprints is large in order to minimize , and also the spectral correlation as given in Eq. (10); and

2. cases where the distance from the IASI radiance to the clear-sky estimate is small (i.e., ), in order to minimize .

A description of our cloud-clearing algorithm is given in the next section.

c. Cloud-clearing algorithm implementation

The use of AVHRR enables characterization of subpixel cloud variability within the IASI footprints and, more importantly, enables detection of cloud-free or clear-sky footprints. As described in the section 3b, cloud clearing amplifies the random components of noise in the IASI measurements; therefore, our approach to handle clouds is two pronged. If AVHRR determines that any of the IASI footprints are clear sky, then we average those clear-sky footprints and skip cloud clearing (this is commonly referred to as “hole hunting”); otherwise, we perform cloud clearing. The steps of our algorithm are presented in more detailed in the following and a schematic of the algorithm is shown in Fig. 3:

1. Aggregate the collocated clear-sky AVHRR radiance for each channel onto the IASI footprints. If fewer than 3% of all collocated AVHRR pixels are masked clear by the CLAVR-x mask, then reject the current case.

2. If any IASI footprint is determined by AVHRR to be clear-sky, then is set equal to the average IASI spectral radiances in those clear-sky footprints. In this case α(η) is equal to , where n_clr is the number of clear-sky IASI footprints within the IASI 2 × 2 array. Proceed to step 9.

3. If no clear-sky IASI footprints are found, then sort the j IASI footprints by .

4. For each sorted cloudy IASI footprint (j = 1, 2, … , 4) select a neighboring footprint (k = 2, 3, 4, j ≠ k order matters), giving a total of six possibilities. We order the FOVs such that the warmest FOV always corresponds to in order to minimize the noise amplification.

5. For each pair (j, k) calculate η(j, k) using Eq. (9).

6. Calculate R^cc(j, k) from Eq. (5).

7. Calculate and α[η(j, k)].

8. Define a figure of merit for each pair of footprints (j, k) with .

9. Select the pair of footprints (j′, k′) with fom(j′, k′) = min fom(j, k).

10. Apply quality control to the selected cloud-cleared radiance by requiring that χ²(j′, k′) ≤ 5.0 K and α[η(j′, k′)] ≤ 10.

The quality control thresholds for χ²(j, k) and α[η(j′, k′)] are very liberal and were selected such that we maximize the number of cases that get through our cloud-clearing algorithm.

Schematic of the cloud-clearing algorithm illustrating that the radiance in each footprint is assumed to be a linear combination of a clear-sky radiance and cloudy-sky radiance with the relative weighting described by the cloud fraction N^j in each footprint j. The collocated AVHRR measurements for channel A_i that are determined to be clear by the CLAVR-x cloud mask for pixel l and denoted in the figure are averaged and used as an estimate of the clear radiance . For clarity, these subpixel measurement locations are shown for only one footprint and the size of the subpixel footprints is exaggerated. To compare apples to apples, the IASI spectral measurements are also spectrally integrated onto the AVHRR bandpasses. The cloud-clearing algorithm cycles through various combinations of the IASI footprints (e.g., {j = 2, k = 1}, {j = 2, k = 3}, {j = 4, k = 2}, etc.), estimates η(j, k) using Eq. (9), and produces a cloud-cleared radiance via Eq. (5) for each combination. The algorithm then selects the optimal combination of footprints; i.e., the ones that minimize the figure of merit fom(j, k) described in section 3c. It is not possible to tell from this general example which footprints would be used in our algorithm; however, the algorithm would likely not choose footprints 1 and 3 to perform cloud clearing because the cloud fraction in these two footprints is very similar (i.e., N³ ≈ N¹), and hence .

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Schematic of the cloud-clearing algorithm illustrating that the radiance in each footprint is assumed to be a linear combination of a clear-sky radiance and cloudy-sky radiance with the relative weighting described by the cloud fraction N^j in each footprint j. The collocated AVHRR measurements for channel A_i that are determined to be clear by the CLAVR-x cloud mask for pixel l and denoted in the figure are averaged and used as an estimate of the clear radiance . For clarity, these subpixel measurement locations are shown for only one footprint and the size of the subpixel footprints is exaggerated. To compare apples to apples, the IASI spectral measurements are also spectrally integrated onto the AVHRR bandpasses. The cloud-clearing algorithm cycles through various combinations of the IASI footprints (e.g., {j = 2, k = 1}, {j = 2, k = 3}, {j = 4, k = 2}, etc.), estimates η(j, k) using Eq. (9), and produces a cloud-cleared radiance via Eq. (5) for each combination. The algorithm then selects the optimal combination of footprints; i.e., the ones that minimize the figure of merit fom(j, k) described in section 3c. It is not possible to tell from this general example which footprints would be used in our algorithm; however, the algorithm would likely not choose footprints 1 and 3 to perform cloud clearing because the cloud fraction in these two footprints is very similar (i.e., N³ ≈ N¹), and hence .

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

Schematic of the cloud-clearing algorithm illustrating that the radiance in each footprint is assumed to be a linear combination of a clear-sky radiance and cloudy-sky radiance with the relative weighting described by the cloud fraction N^j in each footprint j. The collocated AVHRR measurements for channel A_i that are determined to be clear by the CLAVR-x cloud mask for pixel l and denoted in the figure are averaged and used as an estimate of the clear radiance . For clarity, these subpixel measurement locations are shown for only one footprint and the size of the subpixel footprints is exaggerated. To compare apples to apples, the IASI spectral measurements are also spectrally integrated onto the AVHRR bandpasses. The cloud-clearing algorithm cycles through various combinations of the IASI footprints (e.g., {j = 2, k = 1}, {j = 2, k = 3}, {j = 4, k = 2}, etc.), estimates η(j, k) using Eq. (9), and produces a cloud-cleared radiance via Eq. (5) for each combination. The algorithm then selects the optimal combination of footprints; i.e., the ones that minimize the figure of merit fom(j, k) described in section 3c. It is not possible to tell from this general example which footprints would be used in our algorithm; however, the algorithm would likely not choose footprints 1 and 3 to perform cloud clearing because the cloud fraction in these two footprints is very similar (i.e., N³ ≈ N¹), and hence .

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

4. Performance of the cloud-clearing algorithm

In this section we perform an analysis of the results of the cloud-clearing algorithm by comparing the cloud-cleared radiances to the subpixel clear-sky AVHRR measurements . To test the accuracy of the cloud-cleared radiances over a large range of atmospheric conditions, we selected a subset of 66 night granules from five partial IASI orbits on 3 October 2010. For the analysis that follows, the results have been restricted to latitudes between 70°S and 75°N. The authors note that the polar orbit of MetOp-A samples the poles more than the middle or low latitudes. A restriction to investigate latitudes between 70°S and 75°N was made due to a lower availability of subpixel clear-sky estimates from AVHRR at higher latitudes and a lower acceptance rate of the IASI L2 retrievals.

Figure 4 shows the AVHRR CLAVR-X cloud mask for the several partial MetOp-A orbits that form our dataset. In creating the figure, we restricted viewing angles from AVHRR to be within those viewed by IASI. It is worthwhile to note that for this dataset the CLAVR-X cloud mask determined ≈10% of the single FOV IASI footprints to be clear sky, ≈2.5% of the 2 × 2 array of IASI footprints to be clear sky, and ≈39% of the 2 × 2 array of IASI footprints to be completely covered with clouds (i.e., they are overcast).

CLAVR-X cloud mask for several partial MetOp-A orbits on 3 Oct 2010. For this dataset, ≈10% of the single FOV IASI footprints are clear, ≈2.5% of the 2 × 2 IASI FORs are clear, and ≈39% of the 2 × 2 IASI FORs are completely overcast.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

CLAVR-X cloud mask for several partial MetOp-A orbits on 3 Oct 2010. For this dataset, ≈10% of the single FOV IASI footprints are clear, ≈2.5% of the 2 × 2 IASI FORs are clear, and ≈39% of the 2 × 2 IASI FORs are completely overcast.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

CLAVR-X cloud mask for several partial MetOp-A orbits on 3 Oct 2010. For this dataset, ≈10% of the single FOV IASI footprints are clear, ≈2.5% of the 2 × 2 IASI FORs are clear, and ≈39% of the 2 × 2 IASI FORs are completely overcast.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

Figure 5 illustrates the improvement in yield resulting from the use of a cloud-clearing algorithm and also the performance of the algorithm. The top left panel of Fig. 5 shows the AVHRR channel 4 brightness temperature for any IASI FOR where at least one footprint was determined to be clear sky. The top right panel shows the spectrally convolved cloud-cleared brightness temperature for the equivalent AVHRR channel 4 SRF where met the quality control thresholds described in section 3c. The bottom right panel illustrates the ability of the cloud-clearing algorithm to fit the subpixel clear-sky AVHRR measurement in AVHRR channel 4 by showing the difference between and , where each are converted to a brightness temperature. The bottom left panel shows the coldest brightness temperature in the IASI FOR spectrally convolved for AVHRR channel 4.

(top left) Map of AVHRR measurements averaged onto the IASI footprints where the any of 2 × 2 IASI footprints comprising the IASI FOR were determined to be clear sky. (top right) Map of the cloud-cleared IASI measurements spectrally averaged onto the AVHRR SRF for AVHRR channel 4. (bottom left) Map of the coldest IASI footprint (FOV) in the IASI 2 × 2 array. (bottom right) Map of the difference between the IASI cloud-cleared radiances and the clear estimate for AVHRR channel 4.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

(top left) Map of AVHRR measurements averaged onto the IASI footprints where the any of 2 × 2 IASI footprints comprising the IASI FOR were determined to be clear sky. (top right) Map of the cloud-cleared IASI measurements spectrally averaged onto the AVHRR SRF for AVHRR channel 4. (bottom left) Map of the coldest IASI footprint (FOV) in the IASI 2 × 2 array. (bottom right) Map of the difference between the IASI cloud-cleared radiances and the clear estimate for AVHRR channel 4.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

(top left) Map of AVHRR measurements averaged onto the IASI footprints where the any of 2 × 2 IASI footprints comprising the IASI FOR were determined to be clear sky. (top right) Map of the cloud-cleared IASI measurements spectrally averaged onto the AVHRR SRF for AVHRR channel 4. (bottom left) Map of the coldest IASI footprint (FOV) in the IASI 2 × 2 array. (bottom right) Map of the difference between the IASI cloud-cleared radiances and the clear estimate for AVHRR channel 4.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

Apparent from the bottom right panel, the cloud-clearing–hole-hunting algorithm fits the AVHRR subpixel clear-sky scenes extremely well. The probability distribution function and cumulative distribution function for the difference between the subpixel clear-sky AVHRR measurements for AVHRR channel 4 and IASI cloud-cleared radiances are also shown in Fig. 6. For this dataset of partial orbits, the root-mean-squared difference (RMSD) and bias between the and for AVHRR channel 4 is 0.2225 and −0.1429 K. Likewise, the root-mean-squared difference and bias between the and for AVHRR channel 5 is 0.2376 and −0.1648 K. One area for possible improvement of the algorithm is for land cases. Over land the bottom right panel shows larger departures especially over high surface terrain, such as the U.S. and Canadian Rockies and the Andes range in South America. These larger differences over land surfaces could be due to a number of factors, including differences in the subpixel AVHRR footprint versus the IASI footprint aggregate surface emissivity, surface temperature, water vapor amount, or other geophysical variability not properly accounted for by our algorithm. Nevertheless, if we only consider land cases in this partial set of orbits we find the RMSD and bias for AVHRR channel 4 is 0.3251 and −0.0347 K and the RMSD and bias for AVHRR channel 5 is 0.3285 and −0.03213 K.

PDF (solid) and CDF (dotted) of the difference between the IASI cloud-cleared radiances and the clear estimate for AVHRR channel 4 (928.15 cm⁻¹) for the five partial MetOp-A orbits on 3 Oct 2010.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

PDF (solid) and CDF (dotted) of the difference between the IASI cloud-cleared radiances and the clear estimate for AVHRR channel 4 (928.15 cm⁻¹) for the five partial MetOp-A orbits on 3 Oct 2010.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

PDF (solid) and CDF (dotted) of the difference between the IASI cloud-cleared radiances and the clear estimate for AVHRR channel 4 (928.15 cm⁻¹) for the five partial MetOp-A orbits on 3 Oct 2010.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

In the previous section we developed equations to describe the error characteristics of and argued that a successful algorithm should minimize both the difference between the clear estimate and cloud-cleared radiance and also the noise amplification resulting from cloud clearing. Figure 7 compares the amplification factor α[η(j, k)] resultant from two systems that use different figures of merit (foms) to select optimal footprints for cloud clearing. The black curves in Fig. 7 correspond to a system that uses a fom that minimizes the root-mean-squared agreement between the subpixel clear-sky radiances observed by AVHRR and IASI cloud-cleared radiances χ²(j, k) and the amplification factor α[η(j, k)], while the red curves correspond to a system that uses a fom that minimizes only the RMSD agreement χ²(j, k). The data plotted in this figure are for only those cases that passed our quality control described above. While the probability distribution functions (PDFs) and cumulative distribution functions (CDFs) of α[η(j, k)] for each fom agree well for α[η(j, k)] ≤ 1, for larger amplification factors we see that the black CDF curves (fom uses both quantities) approaches 1 much faster than the red CDF curve. This means that our algorithm will limit the random noise amplification much better than an algorithm that does not consider α[η(j, k)].

PDF (solid) and CDF (dashed) of the amplification factor as calculated from Eq. (12). The PDF and CDF when the figure of merit used in the algorithm to decide footprints j′ and k′ consists of only the χ² term (red curves), and the PDF and CDF when the figure of merit includes both χ² and α(η) (black curves) are shown.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

PDF (solid) and CDF (dashed) of the amplification factor as calculated from Eq. (12). The PDF and CDF when the figure of merit used in the algorithm to decide footprints j′ and k′ consists of only the χ² term (red curves), and the PDF and CDF when the figure of merit includes both χ² and α(η) (black curves) are shown.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

PDF (solid) and CDF (dashed) of the amplification factor as calculated from Eq. (12). The PDF and CDF when the figure of merit used in the algorithm to decide footprints j′ and k′ consists of only the χ² term (red curves), and the PDF and CDF when the figure of merit includes both χ² and α(η) (black curves) are shown.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

It is also worthwhile to note that for this ensemble, roughly 48% of our 2 × 2 IASI FORs include at least one clear-sky FOV; that is, α(η) ≤ 1. This means that over the scale of the IASI FOR, which is ≈50 km, if 3% of the 1-km AVHRR clear-sky pixels collocated to IASI footprints are found to be cloud free (i.e., the FOR is not overcast), then 18% of the atmosphere is cloud free over a scale of 50 km, 30% of the atmosphere is cloud free over a scale of × 50 km, 35% of the atmosphere is cloud free over a scale of 25 km (roughly half an IASI footprint), and 48% of that the scene is cloud free over a scale of the IASI footprint size, 12 km. Thus, 3% of the AVHRR pixels collocated to an IASI footprint is ≈six 1-km AVHRR pixels. This finding is extremely important to note in the context of the development of future IR sounders with high spatial resolution, and also for cloud model parameterizations and studies of cloud spatial scaling; however, the use of dataset with more days and geographical scenes would be required to provide robustness to these findings.

Assuring good accuracy of relative to does not guarantee good performance of the radiances themselves. Because IASI is a thermal sounder, the ability of our algorithm to remove the effect of clouds is directly dependent on the thermal contrast between clouds and the surface-leaving radiances. We would therefore expect that the largest degree of difficulty for the algorithm would be for undetected low clouds. To test the quality and usefulness of we also run the NOAA operational IASI retrievals using as inputs to produce estimates of the geophysical state observed by IASI. To determine the quality of our for near-surface properties, we compared the ECMWF model output ocean skin temperature to retrieved ocean surface skin temperature for two system configurations. The first configuration used the current operational AIRS-based IASI plus AMSU cloud-clearing algorithm and L2 processor described in section 1, while the second configuration used the combined AVHRR, IASI, and AMSU cloud-clearing algorithm described in this paper. In the combined AVHRR, IASI, and AMSU system configuration the L2 processor was told that the cloud-cleared radiances were clear with nominal IASI NEΔN. As discussed in section 3b, the process of cloud clearing amplifies the random components of IASI spectral noise. We would therefore expect that the performance of the retrieval algorithm that the AVHRR, IASI, and AMSU cloud-cleared radiances would be suboptimal due to the fact that noise amplification is not explicitly handled.

The benefits of adding AVHRR to the cloud-clearing algorithm are shown in Fig. 8. Here, the blue curves correspond to surface temperature retrievals from cloud-cleared radiances that utilized information from AVHRR, IASI, and AMSU, while red curves correspond to surface temperature retrievals from cloud-cleared radiances that utilized information from IASI and AMSU only. Quality control for the IASI retrievals is based on a series of threshold tests for various retrieval convergence criteria and coarse data quality checks. To ensure our comparison was fair, we used a common rejection for each system so that the same ensemble is considered in each PDF (section 3c). The percent of cases accepted by both systems was 42.7% and statistics are summarized in Table 3.

PDF of the difference between retrieved ocean skin temperatures and ECMWF analysis modeled skin temperatures for the five partial MetOp-A orbits on 3 Oct 2010. The surface temperature retrievals where the cloud-clearing algorithm utilized information from AVHRR, IASI, and AMSU (blue), and surface temperature retrievals where the cloud-clearing algorithm utilized information from IASI and AMSU only (red) are shown. Quality control for each system was common so that the same ensemble was used in each system configuration.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

PDF of the difference between retrieved ocean skin temperatures and ECMWF analysis modeled skin temperatures for the five partial MetOp-A orbits on 3 Oct 2010. The surface temperature retrievals where the cloud-clearing algorithm utilized information from AVHRR, IASI, and AMSU (blue), and surface temperature retrievals where the cloud-clearing algorithm utilized information from IASI and AMSU only (red) are shown. Quality control for each system was common so that the same ensemble was used in each system configuration.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

PDF of the difference between retrieved ocean skin temperatures and ECMWF analysis modeled skin temperatures for the five partial MetOp-A orbits on 3 Oct 2010. The surface temperature retrievals where the cloud-clearing algorithm utilized information from AVHRR, IASI, and AMSU (blue), and surface temperature retrievals where the cloud-clearing algorithm utilized information from IASI and AMSU only (red) are shown. Quality control for each system was common so that the same ensemble was used in each system configuration.

Citation: Journal of Atmospheric and Oceanic Technology 28, 9; 10.1175/JTECH-D-10-05045.1

• Download Figure
• Download figure as PowerPoint slide

Table 3.

Bias, standard deviation (std dev), correlation coefficient r, and % outliers between ECMWF model ocean surface skin temperatures and retrieved skin temperatures that utilized either AVHRR, IASI, and AMSU cloud clearing or the AIRS-based IASI plus AMSU cloud clearing. We define an outlier as the % of cases falling outside |3 K| about the mean difference.

View Table

Generally speaking the system that utilizes AVHRR to quality control and produce cloud-cleared radiances shows a much smaller, if nonexistent, cold tail in the differences between retrieved ocean surface skin temperatures and ECMWF model ocean surface skin temperatures. This cold tail, which is a tendency for the retrievals to be colder than the model, is a direct result of cloud contamination in the cloud-cleared radiance. Adding AVHRR to the cloud-clearing algorithm also tightens the PDF toward a more Gaussian shape with a smaller standard deviation, which again illustrates the high quality and low noise (correlated and random) of the cloud-cleared radiances produced using our approach.

5. Conclusions

In this paper we have developed a methodology that enables the determination of high-quality cloud-cleared radiances from high spectral resolution sounders using collocated high spatial resolution imager measurements. Building on the results of Smith et al. (2004) and Li et al. (2005) and examining the propagation of errors through the cloud-clearing algorithm lead us to an improved quality control scheme and more optimal selection of footprints (a combination cloud-clearing–hole-hunting approach) that demands low-magnitude noise amplification for the cloud-cleared radiances. In addition, by formulating the problem in the η notation, our approach has the advantage that multiple cloud formations can be cleared from the spectra simultaneously, which will be the subject of a future publication.

When faced with real data from MetOp-A, the combination AVHRR, IASI, and AMSU algorithm successfully removes the effects of clouds from the IASI radiances for ≈42% of cases attempted. Considering that ≈39% of cases were rejected because the 2 × 2 array of IASI footprints was determined by the CLAVR-x cloud mask to be completely covered with clouds, a 42% yield over the entire ensemble corresponds to a ≈ 70% = 100[42/(100 − 39)]% success rate for the cloud-clearing algorithm. In addition, after correcting for some calibration differences between AVHRR and IASI, these cloud-cleared radiances agree with subpixel clear-sky AVHRR radiances to better than ≈0.2-K RMSD, with almost no bias for either surface sensitive AVHRR window channel. Although the dataset used to test the algorithm was not global in extent, the dataset included a wide variety of atmospheric conditions, so it is expected that the algorithm performance should extend to global conditions.

To guarantee the performance of the cloud-cleared radiances, we ran our cloud-cleared radiances through the operational L2 processor for IASI assuming that the radiances were clear. For the partial set of MetOp-A orbits considered on 3 October 2010 between 70°S and 75°N latitude, surface temperature retrievals run using the combined AVHRR, IASI, and AMSU algorithm agree with the ECMWF model surface skin temperatures to better than 0.2 K in the mean, with a standard deviation of ≈1.2 K, and demonstrate the high accuracy and precision of these cloud-cleared radiances for channels spanning the atmospheric column and including the surface. Relative to the current operational system, these statistics represent a ≈2-K improvement in the bias and a ≈1-K improvement in the random component of error for the surface temperature retrieval. As noted in section 4, we did not handle the noise amplification of the cloud-cleared radiances for the AVHRR, IASI, and AMSU algorithm, and therefore we would expect that the algorithm performance is better than that reported here.

Another interesting finding that has implications for the design of future IR sounding instruments as well as the understanding of cloud size and spatial scaling follows from the spatial scaling of cloud-free pixels over the IASI 2 × 2 array of footprints. We found for this ensemble that in the 61% of cases that at least 3% of the 1-km AVHRR pixels collocated onto the IASI footprints were determined to be cloud free over the 50-km IASI 2 × 2 array, all four footprints were cloud free 18% of the time, three footprints were cloud free 25% of the time, two footprints were cloud free 35% of the time, and at least one of the 12-km footprints were cloud free 50% of the time. This indicates that there is no simple progression in the probability of finding clear-sky pixels in smaller FOVs and also that there is a potential to run the IASI retrievals at a higher spatial resolution than a 50-km AMSU footprint.

We anticipate that the next release of the IASI operational retrievals will incorporate the AVHRR data into the cloud-clearing algorithm.