• Ackerman, S. A., , K. I. Strabala, , W. P. Menzel, , R. A. Frey, , C. C. Moeller, , and L. E. Gumley. 1998. Discriminating clear sky from clouds with MODIS. J. Geophys. Res. 103:3214132157.

    • Search Google Scholar
    • Export Citation
  • Aumann, H. H., and R. J. Pagano. 1994. Atmospheric Infrared Sounder on the Earth Observing System. Opt. Eng. 33:776784.

  • English, S. J., , J. R. Eyre, , and J. A. Smith. 1999. A cloud-detection scheme for use with satellite sounding radiances in the context of data assimilation for numerical weather prediction. Quart. J. Roy. Meteor. Soc. 125:23592378.

    • Search Google Scholar
    • Export Citation
  • Gadd, A. J., , B. R. Barwell, , S. J. Cox, , and R. J. Renshaw. 1995. Global processing of satellite sounding radiances in a numerical weather prediction system. Quart. J. Roy. Meteor. Soc. 121:615630.

    • Search Google Scholar
    • Export Citation
  • Griffiths, P. R., and J. A. de Haseth. 1986. Fourier Transform Infrared Spectrometry. Vol. 83, Chemical Analysis, John Wiley and Sons, 671 pp.

    • Search Google Scholar
    • Export Citation
  • Huang, H. L., and P. Antonelli. 2001. Application of principal component analysis to high-resolution infrared measurement compression and retrieval. J. Appl. Meteor. 40:365388.

    • Search Google Scholar
    • Export Citation
  • Lee, A. C. L. 2003. Strategy for dissemination and use of advanced sounder radiances. Met Office (RSI) Branch Working Paper 205. [Available from National Meteorological Library, FitzRoy Road, Exeter, EX1 3PB, United Kingdom.].

    • Search Google Scholar
    • Export Citation
  • Masiello, G., , M. Matricardi, , R. Rizzi, , and C. Serio. 2002. Homomorphism between cloudy and clear spectral radiance in the 800–900-cm−1 atmospheric window region. Appl. Opt. 41:965973.

    • Search Google Scholar
    • Export Citation
  • Nicholls, S. 1978. Measurements of turbulence by an instrumented aircraft in a convective boundary layer over the sea. Quart. J. Roy. Meteor. Soc. 104:653676.

    • Search Google Scholar
    • Export Citation
  • Saunders, R. W., and K. T. Kriebel. 1988a. An improved method for detecting clear sky and cloudy radiances from AVHRR data. Int. J. Remote Sens. 9:123150.

    • Search Google Scholar
    • Export Citation
  • Saunders, R. W., and K. T. Kriebel. 1988b. Errata. Int. J. Remote Sens. 9:13931394.

  • Scott, N. A., and A. Chedin. 1981. A fast line-by-line method for atmospheric absorption computations—The automatized atmospheric absorption atlas. J. Appl. Meteor. 20:802812.

    • Search Google Scholar
    • Export Citation
  • Serio, C., , A. M. Lubrano, , F. Romano, , and H. Shimoda. 2000. Cloud detection over sea surface by use of autocorrelation functions of upwelling infrared spectra in the 800–900-cm−1 window region. Appl. Opt. 39:35653572.

    • Search Google Scholar
    • Export Citation
  • Simeoni, D., , C. Singer, , and G. Chalon. 1997. Infrared atmospheric sounding interferometer. Acta Astronautica 40:113118.

  • Stephenson, D. B. 2000. Use of “odds ratio” for diagnosing forecast skill. Wea. Forecasting 15:221232.

  • Strabala, K. I., , S. A. Ackerman, , and W. P. Menzel. 1994. Cloud properties inferred from 8–12-μm data. J. Appl. Meteor. 33:212229.

  • Taylor, J. P., , J. M. Edwards, , M. D. Glew, , P. Hignett, , and A. Slingo. 1996. Studies with a flexible new radiation code. II: Comparisons with aircraft short-wave observations. Quart. J. Roy. Meteor. Soc. 122:839861.

    • Search Google Scholar
    • Export Citation
  • Wilks, D. S. 1995. Statistical Methods in the Atmospheric Sciences: An Introduction. Vol. 59, International Geophysics Series, Academic Press, 467 pp.

    • Search Google Scholar
    • Export Citation
  • Wilson, S. H. S., , N. C. Atkinson, , and J. A. Smith. 1999. The development of an airborne infrared interferometer for meteorological sounding studies. J. Atmos. Oceanic Technol. 16:19121927.

    • Search Google Scholar
    • Export Citation
  • View in gallery

    (a) The noise used in normalization (dashed line) and the average measured ARIES spectra for flight A677 (solid line). (b) The same spectra after normalization by the noise

  • View in gallery

    Time series of cobs(89) for flight A678. Squares denote cloud seen in the DFC imagery. The solid line shows the results for the main tests over the full spectral range. The dashed line shows an alternative cobs(89) for which all calculations have been made on spectra truncated above 2000 cm−1 wavenumber. This approach creates a nondaylight case because the solar component is removed. The horizontal lines show limits used in detecting cloud: the dotted line is for the main tests, and the dash–dot line is for the alternative, nondaylight, case

  • View in gallery

    The first 10 EOFs plotted individually, numbered from 0 to 9, with eigenvalues w noted alongside. The spectra were noise normalized before SVD to EOFs. Values over 1.0 are above the ARIES noise level

  • View in gallery

    (a) The first EOF in 𝗨all that contains cloud information (solid line) and the running mean (dashed line). Both have been denormalized for comparison with the (b) radiance difference—the averages of all cloud spectra minus all clear spectra

  • View in gallery

    (a) The ranked eigenvalues w, illustrating how they quickly fall. (b) The rms of the residual R, computed as the eigenvectors within 𝗨all are used one after another to create each element of cobs. The lower line in (b) shows the result on compression of an ARIES spectra that was used within 𝗨all. The upper line in (b) shows the result when compressing an ARIES clear spectra from the same flight, but not used within 𝗨all

  • View in gallery

    Plots of cloudy spectra minus the spectra from a nearby clear view. The plots are in pairs for two cloudy views: (a), (b) missed cloud at spectrum 104 and (c), (d) cloud detected at spectrum 91. Here (a) and (c) show the difference at full ARIES spectral resolution and (b) and (d) have been boxcar smoothed by 25 cm−1 wavenumbers

  • View in gallery

    Apollo test results on one flight's (A678) data. (a) The difference at 12 μm with SST taken to flight altitude. The threshold is marked by a dotted line. (b) The second test of a running calculation of standard deviation over five consecutive spectra. The threshold is marked by a dashed line. True cloud in the FOV is marked by boxes over the plots. Cloud as analyzed by Apollo is marked by crosses independently for each test

  • View in gallery

    The trispectral band differences plotted against 11-μm BT for all three test flights: A674, A678, and A741. The 8- minus 11-μm data are squares or “+”, with a dotted horizontal line showing the threshold. The 11- minus 12-μm data are diamonds or “×” with a dashed line showing the threshold. The SST-based thresholds are the vertical dotted lines, one for each flight. Clear FOVs are marked by “+” or “×”; otherwise cloud was seen within the FOV (denoted by squares or diamonds)

  • View in gallery

    A subset of the trispectral data from Fig. 8: only the 11-μm band BT for one flight, A678. Cloud detected by the trispectral tests is marked with “×”; squares mark where true cloud was seen within the FOV

  • View in gallery

    The homogeneity results, plotting |hs| for one flight, A678. The threshold on |hs| is shown by the dotted line. Clouds detected by the tests are marked by “×”; true clouds seen in the FOV are marked by squares over the plot

  • View in gallery

    The sensitivity of |hs| to the reference spectra chosen, illustrated by two reference spectra at either end of the A677 flight track. The solid line has a reference at the beginning, spectrum 20. The dotted line has a reference at the end, spectrum 197. Overplotted squares denote cloud seen within the FOV

  • View in gallery

    Fig. A1. A time series of the summed residual, Σ Ra (longest wavelength window), during the A677 flight. The four points at which Σ Ra = 0 are the ARIES-measured clear views added to the modeled clear spectra. Dashed vertical lines mark calibration periods. Clouds were present in most views from spectra 130 to 160

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 6 5 1
PDF Downloads 2 2 1

Initial Cloud Detection Using the EOF Components of High-Spectral-Resolution Infrared Sounder Data

View More View Less
  • 1 Met Office, Bracknell, Berkshire, United Kingdom
© Get Permissions
Full access

Abstract

Measured cloud spectral signatures in high-resolution infrared interferometer data have been separated from the clear-air signatures using singular value decomposition. Sets of empirical orthogonal functions (EOFs) have then been created from these signatures to investigate the possibility of cloudy view discrimination without the use of any background data. The measured data have been taken by the Airborne Research Interferometer Evaluation System (ARIES), which is specifically designed to gather data from an aircraft that are representative of the forthcoming Infrared Atmospheric Sounding Interferometer (IASI). EOF sets were based on 78 diverse modeled clear-air spectra, supplemented by selected measured spectra. Video data gave independent verification of cloudy and cloud-free views. The development of a cloud-detection scheme is detailed, and several possible cloud-detection procedures were tested. The most promising procedure is presented. Comparative tests are made with cloud-detection algorithms developed for earlier satellite instruments. The results are encouraging; clouds were detected in the measured test data with similar success to other schemes but without requiring prior information or even the uncompressing of transmitted data. With the prospect of IASI (and similar) data being compressed for transmission using EOFs, the procedure here could be implemented in NWP centers as an initial very inexpensive but accurate method to create a cloud-filtering mask.

Corresponding author address: Dr. Jonathan A. Smith, ECMWF, Shinfield Park, Reading RG2 9AX, United Kingdom. jonathan.smith@ecmwf.int

Abstract

Measured cloud spectral signatures in high-resolution infrared interferometer data have been separated from the clear-air signatures using singular value decomposition. Sets of empirical orthogonal functions (EOFs) have then been created from these signatures to investigate the possibility of cloudy view discrimination without the use of any background data. The measured data have been taken by the Airborne Research Interferometer Evaluation System (ARIES), which is specifically designed to gather data from an aircraft that are representative of the forthcoming Infrared Atmospheric Sounding Interferometer (IASI). EOF sets were based on 78 diverse modeled clear-air spectra, supplemented by selected measured spectra. Video data gave independent verification of cloudy and cloud-free views. The development of a cloud-detection scheme is detailed, and several possible cloud-detection procedures were tested. The most promising procedure is presented. Comparative tests are made with cloud-detection algorithms developed for earlier satellite instruments. The results are encouraging; clouds were detected in the measured test data with similar success to other schemes but without requiring prior information or even the uncompressing of transmitted data. With the prospect of IASI (and similar) data being compressed for transmission using EOFs, the procedure here could be implemented in NWP centers as an initial very inexpensive but accurate method to create a cloud-filtering mask.

Corresponding author address: Dr. Jonathan A. Smith, ECMWF, Shinfield Park, Reading RG2 9AX, United Kingdom. jonathan.smith@ecmwf.int

Introduction

The next generation of satelliteborne infrared sounders, such as the Advanced Infrared Sounder (AIRS; Aumann and Pagano 1994) and Infrared Atmospheric Sounding Interferometer (IASI; Simeoni et al. 1997), will produce much greater volumes of data than previous sounders. The transfer, storage, and processing of these data will require considerable effort both in time and computing resources. Part of this increase will be accommodated through data-compression techniques. Huang and Antonelli (2001) detail a scheme with compression ratio of ∼15 for AIRS data; Lee (2003) presents a different scheme for IASI data that gives a compression ratio of ∼300. Presented here is an initial cloud-detection scheme that will further help to accommodate the increased data volumes. At an early stage in processing it would allow the selection of infrared data that are unlikely to finally be used in first, “day one,” assimilation systems because of large amounts of cloud. Computation time could be saved by filtering out the selected cloudy data from further processing. The scheme itself can run with minimal computation.

The current (English et al. 1999) and previous (Gadd et al. 1995) cloud-detection schemes in the Met Office NWP model (and similar schemes elsewhere) require background information. These background data (such as predicted temperature profile and surface temperature to calculate expected radiances) have to be gathered and matched with the sounder data. Accurate cloud masks derived from such schemes are obtained at the expense of computing effort. Such effort should be concentrated on the “best” infrared sounder views—those that contain no significant cloud. The cloud-detection scheme described here would be in addition to (more sophisticated) cloud detection further downstream within the data processing. Our initial scheme is designed to require no model or spectral background data and to operate in tandem with a probable method of data compression for IASI data transmission.

Proposed cloud-detection scheme

The algorithm presented below has been developed for IASI data. A specific and probable method of data compression (Lee 2003) is assumed, though the detail of data transmission for IASI still has to be fixed. If the assumed data compression is not used, the algorithm here would require an extra small expense to preprocess the data.

The algorithm is based around the encoding, and compression, of the IASI spectra of both clear and cloud views using empirical orthogonal functions (EOF), which is described in the next section. With IASI not yet in orbit and high-spectral-resolution models of cloud spectra still under development, to use cloud data that are as realistic as possible it was decided to use spectra measured by the Airborne Research Interferometer Evaluation System (ARIES; Wilson et al. 1999). The data were recorded during three transit flights (two over the Atlantic Ocean in May and one flight over the Baltic Sea in December) carried out by the Met Office C-130 research aircraft (Nicholls 1978; Taylor et al. 1996). Care was taken to ensure that data were chosen for times at which there were no cirrus clouds above the aircraft. Clear data came from both the measured data and modeled spectra calculated to aircraft altitude from a diverse set of 78 atmospheric profiles. Several methods of detecting cloud have been tested, and sensitivity to the main variables was quantified. Results from the tests are presented in section 3, showing the route that led to the chosen algorithm and demonstrating what effects will have to be accounted for as the IASI data compression scheme is defined. The chosen algorithm is a simple threshold test on a single number, which corresponds to the main “cloudy” EOF. In section 4 this chosen algorithm is a compared with previous and proposed cloud-detection schemes that were adapted to process the ARIES measured data. The final section is a summary of the development and findings.

The algorithm chosen could easily be implemented at little computational cost, if EOFs are used as here in the compression of the data. The algorithm should also be adaptable as the IASI data-compression scheme is refined, as would be expected when real IASI data become available after the launch of the Meteorological Operational Satellite (METOP).

Method

The requirement we aim to meet is the development of a fast and efficient first-stage cloud-detection scheme to filter out obviously cloudy data and so to reduce the volume of the dataset that will be passed into a more detailed cloud-detection scheme within the main NWP assimilation process. We also require this detection scheme to be stand alone—requiring no prior information.

The key stages of the proposed scheme are presented below, and in later sections this scheme is tested with ARIES data. The scheme is described in general terms that can be adapted to any high-resolution infrared sounder. Note that stages 1–6 below need only be conducted once to create the EOF. Stage 7 is the data compression that may be carried out before transmission to NWP centers, and stage 8 is the initial cloud-detection test. The finer details of the scheme are given later as the use of the scheme using ARIES data is discussed.

In stage 1, a database of representative clear-sky spectra of radiances needs to be generated. For an earth orbiter, this dataset would need to be representative of the entire globe. The radiance is view dependent and is also sensitive to the nature of the underlying surface. This database of clear spectra can be obtained from model calculations based on the representative profile set or measurement spectra from the sounder in known cloud-free conditions. It is imperative that cloud is not in the field of view (FOV) of the sounder if its data are to be used in this stage, and so preprocessing of these data with a more detailed cloud-detection scheme would be required. The radiances in this database must be at the same spectral resolution as the sounder and must represent the instrument's line shape. In the ideal case, this database will be created once for the sounder and will only be updated intermittently as experience dictates (e.g., if significant changes in spectroscopy are discovered). If each spectrum is a vector x(m) with m spectral elements, then our database is 𝗗(m, n) containing n spectra x. Here, the number of cloud-free spectra (for any one view angle) is given by n = Nclr.

In stage 2, singular value decomposition is performed on this database 𝗗 of clear-sky spectra to generate the set of EOFs 𝗨(m, n)clr by
T
where 𝗨 and 𝗩 are orthogonal matrices, superscript T denotes the transpose of the matrix, 𝗪 is an m by n matrix having zero nondiagonal elements and diagonal elements that are the eigenvalues w ordered from greatest downward, and the first n columns of 𝗨 are eigenvectors u(m) (more correctly the left-hand singular vectors) that give 𝗨(m, n)clr. The similarity with the standard eigenvalue problem can be more clearly seen by creating the m by m matrix 𝗗𝗗T, and then, for the eigenvalues w and associated eigenvectors u. defined by Eq. (1),
i1520-0450-43-1-196-e2
(for this equation, the matrix and vector dimensions are left in).

In stage 3, a database of radiance spectra from atmospheric columns that include cloud is then generated. As with the clear-sky spectra, these will be view dependent and will need to match the sounder's instrument line shape; this database of cloudy scenes can again be generated from modeled profiles or from carefully selected observations. The modeling of these spectra requires great caution to ensure that the spectra are truly representative of real clouds, so, and for instance, scattering and particle shape would need to be treated accurately. The optimum solution would probably require observations of clouds from the sounder. The number of cloudy spectra in the database (for any one view angle) is given by Ncld.

In stage 4, each cloudy spectrum x from the database Ncld is projected onto the EOF set generated from the cloud-free spectra database 𝗨clr to give the set of eigenvector scores c(n):
cTclrx
The eigenvector scores are then used to “rebuild” the spectra, and the residual difference between original observation and rebuilt spectra R(m) is computed:
Rxclrc
Because the eigenvector scores c were generated from cloudy spectra being projected onto an EOF set representative of cloud-free spectra, the residuals R will contain the information from the spectra associated with cloud.

In stage 5, singular value decomposition is now performed on the database of residuals R to generate the set of EOFs 𝗨cld, an m × Ncld matrix. Because the residual dataset is orthogonal to the clear-sky dataset, in stage 6 the two sets of EOFs 𝗨clr and 𝗨cld can be concatenated to give 𝗨all, an m × (Nclr + Ncld) matrix.

To use the cloud-detection scheme, in stage 7 each measurement spectrum x(m)obs, where the condition of the atmosphere (i.e., cloudy or clear) is not known, is projected onto the EOF set 𝗨all to find the eigenvector scores, c(n)obs, where
cobsTallxobs

To test for cloud in the measurement spectra, in stage 8 the eigenvector scores cobs are examined. For cobs, dimension n is equal to Nclr + Ncld, where the first Nclr elements contain information related to cloud-free conditions and the last Ncld elements contain only information about clouds. In the ideal case in which 𝗨clr was truly representative of all cloud-free conditions and 𝗨cld was truly representative of all cloudy conditions, the eigenvector scores cobs for an observed spectrum that was cloud free would only contain information in the first Nclr elements. In converse, an unknown observation that was cloudy would have high eigenvector scores within both the first Nclr elements and the subsequent Ncld elements. Therefore, to detect the presence of cloud in a measured spectrum the user need only to examine the magnitude of one (or more) of the eigenvector scores cobs. If we identify the position of each eigenvector score with a number N, then cloud information will appear in the eigenvector scores cobs(N), where NNclr + 1.

Construction and testing of the cloud-detection scheme using ARIES data

The cloud-detection scheme outlined above is now tested using data gathered by the ARIES instrument on the Met Office C-130 aircraft.

Noise normalization

For optimum performance of the cloud-detection scheme, all radiance spectra (observations, clear-sky dataset, and cloudy-sky dataset) require normalization by the expected instrument noise. The ARIES noise was evaluated during the Validation of IASI Radiative Transfer: Experiment and Modeling project (Wilson et al. 1999). Interferometer noise is reduced by averaging multiple interferograms—the noise is reduced by a factor inversely proportional to the square root of the number of interferograms used. A representation of the noise is shown in Fig. 1a by the dashed line. This quantity has been calculated for 40 coadded spectra to match the number of ARIES spectra coadded to generate each spectra used in this work [coaddition is the averaging of coherent interferometer spectra as defined in Griffiths and de Haseth (1986)]. On the same graph is shown the average of the ARIES test spectra (defined later) before noise normalization. As an example of the data used in the tests, Fig. 1b shows ARIES test spectra after normalization. In addition to noise normalization, an extreme noise value [103 W (m2 sr cm−1)−1] was set at specific wavenumbers to remove the impact of spurious “spikes.” For clarity these are not shown in Figs. 1a or 1b. These spikes, mainly within the center of the water vapor band at specific wavenumbers between 1200 and 2000 cm−1, are seen in ARIES data when the onboard calibration targets are obscured by strong absorption in the open path between them, the scan mirror, and the pressure-sealed instrument window.

Note that in this cloud-detection scheme radiance spectra are used, as opposed to brightness temperature spectra. Instrumental noise for interferometers, such as ARIES or IASI, varies across the spectral range. For any given wavenumber, this noise can be treated as a constant when expressed in terms of radiance, whereas, when using brightness temperature (BT), the noise, expressed as a noise equivalent temperature NEΔT, will vary with scene brightness temperature. Therefore, because the spectra are to be normalized by the expected noise, radiances must be used.

Derivation of the cloud-free dataset and its EOFs

The set of clear-sky spectra Nclr were either calculated by the automatized atmospheric absorption atlas (4A) code (Scott and Chedin 1981) by using a diverse, globally representative profile set or were ARIES clear spectra gathered from C-130 flight A677 (A677 being a unique identifier of the flight), a flight over the tropical Atlantic Ocean. The number of diverse profiles, totaling 78, was made up as follows: 43 radiosonde profiles of temperature and humidity, with 42 from the thermodynamic initial-guess retrieval (TIGR) database and the last being the mean of the 42, plus 35 profiles that contain a wider variation in ozone, taken from selected National Environmental Satellite, Data, and Information Service and Antarctic data.

The 4A spectra were calculated at 0.01 cm−1 wavenumber resolution and then were processed to match the ARIES self-apodized resolution of 0.048 cm−1 wavenumber. Calculations were performed over 28 atmospheric levels up to the same altitude as the ARIES measurements, 6400 m. All of the views, calculated and measured, are in the nadir.

These 78 spectra from the diverse profiles were supplemented by a set of four ARIES-measured clear spectra plus an additional six spectra for surface skin BT (T*) variation. The ARIES spectra were selected to represent better the tropical regions where the test datasets were gathered. The ARIES data were carefully selected using coincident downward-facing video imagery to ensure there was no cloud contamination. The six spectra, included to account for surface skin temperature variation, were created using a single profile from the 78, modeled six times, but with T* ranging from 274.5 to 309.5 K. The profile chosen was similar to the air that was flown over, with a surface-layer temperature of 294 K and a dewpoint depression of 4 K or greater throughout.

These 88 clear-sky spectra, Nclr, were then normalized by ARIES noise before creating 𝗨clr by singular value decomposition (stage 2 of the scheme). This process was carried out across the whole spectral range (taken as 600–2975 cm−1 wavenumber rather than the ARIES limit of 3000 cm−1 to facilitate artifact-free matching by FFT of the 4A modeled spectra to ARIES resolution.

Derivation of the cloudy dataset and its EOFs

Cloud views were taken from selected ARIES data gathered during flight A677 at 24°N, 18°W and around 1600 UTC (approximately 1500 LT). Spectra with clouds in the FOV were identified from video imagery from the C-130 downward-facing camera (DFC) and examination of the ARIES spectra. The ARIES spectra were examined for increases in the solar window and decreases across the 8–12-μm wavelength window in comparison with spectra from a nearby cloud-free view. For these studies it was only necessary to select four cloudy spectra in order to represent the cloud types observed in the tropical Atlantic. For a satelliteborne sounder, to be used in NWP, the number of cloudy spectra would need to be significantly increased. Indeed, the number of ARIES spectra was increased when the scheme was tested against observations of clouds in subarctic atmospheres, presented later. After the cloudy radiance spectra were selected, they were noise normalized and passed through stage 4 of the scheme, as detailed above, to generate a set of residuals Ri. It is worth remembering at this stage that this labor-intensive process of selecting representative cloudy spectra need only be conducted once for the lifetime of any particular sounder and can be done outside of any time-critical NWP process.

Once the set of residuals for the cloudy spectra has been computed, singular value decomposition of them is performed (stage 5 above) to generate a set of EOFs 𝗨cld. This orthogonal set of EOFs is then concatenated to the clear spectra set 𝗨clr to form 𝗨all (stage 6 above).

The preprocessing stage of the cloud-detection scheme is now complete, and any observed spectra gathered with unknown atmospheric conditions (i.e., cloud free or cloudy) can be projected on to 𝗨all to find the eigenvector scores cobs. Because in this test of the scheme we have used four cloudy spectra, N (defined in stage 8 above) totals 92, and we will examine cobs(89) to detect the presence of cloud (detailed in stages 7 and 8 above), cobs(89) being the first eigenvector score expected to contain cloudy information (88 clear spectra + 1) if cloud is present in the scene.

Testing the cloud-detection scheme

Because the cloudy spectra from flight A677 have been used previously, data used to test the cloud-detection scheme were taken from other flights. For the detailed tests reported in this section, ARIES spectra from the subsequent flight (A678) were used. Views were again over both clear air and cloud, as before identified from DFC images. Spectra were recorded from 30° to 45°N along 16°W between 1030 and 1330 UTC. Altitude was initially 300 m lower than before, at 6100 m, and then rose to 300 m higher, at 6700 m. As a sensitivity test, spectra were modeled at the two different altitudes above an altostratus layer and tested through the cloud scheme, but this test had no impact on the results. It is not expected that changes in the spectra due to these small altitude changes would significantly mask, or falsely create, the cloud signals that are within the first cloudy EOF. The data have, therefore, all been treated as if they were at 6400-m altitude. Altitude changes are clearly not a problem with a satellite-based sensor and only arise here from the use of an aircraft flying in the troposphere.

Figure 2 shows the value for cobs(89) during flight A678. Analysis is given for two measuring periods; the first, cloudy, period runs from spectra 66 to 87 along the x axis; the second, mainly clear, period runs from spectra 88 to 114. These latter data were taken at around 1300 UTC and are at the higher altitude of 6700 m.

Extensive cloud was seen during the first measuring period, with opaque cumulus and stratocumulus completely filling the FOV. The cloud clears at the beginning of the second period, followed by a patch of semitransparent cumulus passing through part of the FOV in the middle of this period. All of the values above the horizontal dotted line correspond to cloud in the ARIES FOV, as confirmed by the DFC. However, as for the single spectra in the middle of the second period (at number 104), a few cloud views have a value below this. The spectral differences between a cloud view with cobs(89) just above the line and one just below the line are discussed in the following section.

Toward the end of the flight A678 (for spectra numbered 160 and above) the last three clear views correspond to a change in heading and a tripling in aerosol content around the aircraft. The value of cobs(89) rose up to the level of the horizontal dotted line in Fig. 2. Either the change in heading or aerosol, or both, could be the reason behind the increase for these last three points. Both this rise and the variation in clear-view values suggest that more clear views, to account for different relative sun angles and aerosol, need to be added to the final version of 𝗨clr before creating cloudy EOF.

Monitoring the residuals

Because the set of residuals Ri contains information regarding detail in the spectra that cannot be represented by the cloud-free set of EOFs, it is reasonable to assume that simple monitoring of the magnitude of each residual spectrum may give an indication of whether an unknown spectrum contains clouds. Such an analysis was conducted and is presented in the appendix. In summary, this analysis showed that the monitoring of residuals was not robust enough for cloud-detection purposes. However, routine monitoring of the residual spectra may be of use to NWP centers for monitoring the state of the instrument because changes in the instrument's performance could result in structure within the residuals.

Properties of the eigenvectors and the eigenvalues

The first 10 EOFs in 𝗨clr are shown in Fig. 3. The eigenvalue w calculated for each EOF (eigenvector) during the singular value decomposition (SVD) is shown within each plot. Because the spectra used to create these EOFs are noise normalized, a value of 1.0 in these plots is equivalent to information in the spectra at the noise level of the ARIES instrument. Values above 1.0 are above the noise. The first cloud EOF (added to 𝗨clr to be EOF 89 in 𝗨all) is shown in Fig. 4a; the mean value is shown by the dashed line and is offset by −1 × 10−5. Unlike the EOFs in Fig. 3, the EOF here has been denormalized to contrast with the mean difference due to cloud in Fig. 4b. The first cloud EOF in Fig. 4a shows a solar signal at the shortwave end of the spectrum, more clearly seen in the mean. No such structure was evident in the other three cloudy EOFs in any part of the spectrum.

The eigenvalues w fall off quickly to be below 1.0 after 47 EOFs (see Fig. 5a, which shows the w associated with all the eigenvectors in 𝗨all). Figure 5b shows two lines, each the incremental residual sum difference to a measured spectrum x, because data are decoded from cobs one element at a time. If the x encoded using 𝗨all is one of the ARIES spectra used in creating 𝗨all, then the residual after each eigenvector falls of quickly as well (Fig. 5b, lower line). However, if x is one of the remaining ARIES-measured data (not used to create 𝗨all), the residual never falls below 1000 (Fig. 5b, upper line). This result suggests that just four ARIES clear spectra are not enough, and, to optimize further the scheme, additional information in the ARIES spectra could be added to 𝗨all.

Cloud-detection sensitivity (in brightness temperature)

The plots in Figs. 6a and 6c show the difference between two cloud-view spectra and a clear-view spectrum. The plots in Figs. 6b and 6d show the same data, boxcar smoothed by 25 cm−1 wavenumbers (and no data plotted for the first and last 25 cm−1 wavenumber). For the cloudy view spectrum, number 91, the value of cobs(89) is above the value for any clear view (dotted in Fig. 2). For the cloudy view spectrum, number 104, the cobs(89) value is just below the dotted line—less than the value for the final three clear views in Fig. 2, when the aircraft turned.

The summed differences in BT over two window regions are shown in Table 1, along with the values for maximum cobs(89) and the mean of detected clouds. For spectrum 91, there is relatively little change in the longer wavelength window, but there is a greater change in the shortest wavelength window with a solar component. For spectrum 104, the opposite is true. Both anomalies can be explained by low- and high-level-topped small cumulus only partially filling the FOV. With low cloud tops, there is little contrast between the radiance from the cloud top and sea at longer wavelengths. However, solar wavelengths are reflected more efficiently from the cloud top, leading to a large solar component only. Clouds with higher tops can cast shadows across the FOV, which reduces part of the reflected solar radiance; if the shadowed sides of the clouds are also within the FOV, then the solar component can be little changed (or even reduced) from the clear view, but the colder cloud top will reduce radiance at the longer wavelengths.

To investigate further the effect of solar reflection in cloud detection, it was removed from these data by truncating the spectrum at 2000 cm−1, leaving only the one window region (at longer wavelengths). The changes in cloud detection using cobs(89) are shown in Fig. 2 by the dashed line. In effect, this has created a “nondaylight” case (by removing the shortest wavelength window, it will not correspond directly to the full spectra seen at nighttime). The dashed plot in Fig. 2 still shows variations that correspond to the cloud signal detected using the full spectrum; however, cloudy views are below the highest clear-view values. The horizontal dash–dot line has, therefore, been set using the value at spectrum 104. To be the same as the full spectral case (dotted line), this cloudy view is just missed by the threshold. It is shown later that, though this cloud-detection threshold has a high probability of detecting cloud, the measured skill using these truncated spectra is the lowest of all (Table 4, below).

Comparison with other infrared cloud-detection schemes

The data used to test the EOF cloud detection have also been tested using three other schemes. The schemes were chosen to include one from the early processing of the Television and Infrared Observation Satellite (TIROS) Operational Vertical Sounder (TOVS), another for a current filter-based radiometer, and one for a future interferometric sounder. The first is the Apollo scheme (Saunders and Kriebel 1988a,b) used for the Advanced Very High Resolution Radiometer (AVHRR) data, but only using the tests for the infrared channels. The second was proposed as a Moderate-Resolution Imaging Spectroradiometer (MODIS) infrared cloud detection test in Strabala et al. (1994); there the scheme was tested with the High-Resolution Interferometer Sounder (HIS) and MODIS Airborne Simulator data. It is now included in the MODIS cloud-mask calculation. The final scheme was a proposal by Serio et al. (2000) that was developed further in Masiello et al. (2002) for IASI cloud detection using the spectral features found in high-resolution data, who tested their scheme with interferometric monitor of greenhouse gases (IMG) data.

Data from two further flights were also used to test all of the schemes over different cloud types. During flight A674 over the South Atlantic, in transit from Ascension Island to Recife in Brazil, small cumulus were flown over (as in flights A677 and A678), followed by an altostratus layer. The altostratus was seen to vary visually from semitransparent to opaque in the DFC imagery. Data were tested from two areas, both at about 8°S. The first, with small cumulus only, was at around 18°W, with data taken at around 0800 LT from an altitude of 6100 m. The second area was farther westward, around 34°W, where data were recorded above an altostratus layer at approximately local noon from 7300-m altitude. For part of the data there were also cumulus clouds below the altostratus. At both locations, atmospheric profile data came from aircraft measurements during descents.

Layers of cirrus above mixed and broken cloud over the Baltic Sea were flown over during flight A741. Data were taken at an altitude of 7600 m around 56°N between 17° and 20°E. Sensitivity tests showed the detection scheme not to be sensitive to the higher altitude flown during this flight. However, with the cloud tops only a few hundred meters below the aircraft, rather than several kilometers below (as was the case for the other flights), the cloud information in 𝗨all was extended. Adding four cloudy spectra from this flight when creating 𝗨cld was sufficient to represent this cloud, giving a total of 96 EOFs in 𝗨all. With the SVD performed on all eight cloudy residuals to create a 𝗨cld from both subtropical and arctic cloud, the cloud test was still a threshold on cobs (89).

The DFC imagery showed no clear spectra for A741—every recording period included some cloud in the FOV, and most were completely filled with cloud. To confirm that cloud detection was working, some clear spectra were required. Atmospheric profile data were taken from a Met Office research sonde launched from nearby Visby Island and from a dropsonde launched from the C-130 and were used to model four clear spectra.

The results for each scheme are first analyzed below individually in detail for the A678 test-case data. At the end of this section, the statistical results from all three test flights A674, A678, and A741 are given. Cloud limits for each scheme have been adapted using data from the separate flight A677 and from local SST data. For the A678 test case, the schemes have been compared using the probability of detection (PoD) of cloud, a false-alarm rate (FAR) for cloud that is analyzed but not actually present in the FOV, and the Heidke skill score (HSS) to assess the schemes when taking account of the expected success of chance alone (Wilks 1995). Table 2 shows the 2 × 2 contingency table for the cloud detection, with the analyzed cloud presence along the rows and the true cloud presence from the DFC imagery down the columns. Using a, b, c, and d as defined in Table 2, these statistics are derived following Wilks (1995). The ratio of analyzed cloud to all true cloud views is used for PoD:
aac
FAR has been taken as the ratio of falsely analyzed cloud to all analyzed cloud:
bab
HSS is defined as
i1520-0450-43-1-196-e8
PoD ranges from 0 to 1, the higher being the better in this case. FAR also ranges from 0 to 1, with lower being better.1 The lowest possible FAR is desired here so that clear FOVs are not lost to the following NWP system. The HSS ranges from −1 to 1. A perfect analysis of cloud would score 1, a random analysis would score 0, and an analysis that was worse than random would score below 0. To reflect the final use of the algorithm, removing obviously cloudy data from further processing, 1 − PoD (the probability of a true cloud being detected as clear) can be considered. However, this is less important for our scheme, because a more detailed cloud-detection scheme using background information is intended to follow this initial test. The yield of clears will be more important:
dbd

Yield ideally would be 1.0. This is the case when FAR = 0.0, which implies b = 0. All of the tests with FAR = 0.0 have the ideal yield to pass on to a later, more sophisticated, test.

Apollo for AVHRR

The Apollo tests for AVHRR include visible data, AVHRR channels 1 and 2, and, only at night, channel-3 data. Because the ARIES data were gathered in daylight hours, we cannot use the nighttime tests, because we have solar contamination. For the remaining AVHRR channels (4 and 5), wavelength limits and wavenumber equivalents are shown in Table 3. The Apollo scheme consists of a sequence of tests that are applied in order, with failure at any one test causing an AVHRR pixel to be flagged as cloudy (Saunders and Kriebel 1988a).

The first test compares the channel-5 brightness temperature with a reference brightness temperature. Over the sea this is 2 K less than the radiance expected from an analyzed SST, transmitted through a representative atmosphere to the top of the atmosphere (TOA) and converted to a brightness temperature. For the data here, the SST were taken at positions along the flight track from the National Oceanic and Atmospheric Administration (NOAA) Satellite Active Archive (SAA) 50-km SST analysis and the Baltic Sea analysis archives of the German Bundesamt für Seeschiffahrt und Hydrographie (BSH). For each flight's dataset, a profile taken at, or nearby, the experimental area was selected. The profile data were from dropsondes released from the C-130 research aircraft, descents of the aircraft from altitude, or a balloonsonde. The chosen atmospheric profile was then used with a range of SSTs as input to the 4A radiative transfer code to calculate the expected brightness temperatures at the aircraft altitude. These expected BTs were then compared with the ARIES-measured BTs over the AVHRR channel-5 pass band (for the radiative transfer calculation and the averaging of ARIES data to the channel pass band, a top-hat filter function was assumed between the wavenumber limits given in Table 3).

The results are shown in Fig. 7a, for the same spectra as in Fig. 2. The differences between the ARIES “AVHRR channel 5” BT minus the “SST” BT (derived from the nearest archived SST transmitted through the local atmosphere to the flight altitude) are plotted. Crosses mark where the difference is equivalent to a cloud flag in the Apollo test.

The boxes show where clouds were seen in, or adjacent to, the ARIES FOV in the DFC imagery and showed changes in the ARIES spectra—by a reduction in the window temperature and/or an increase in the solar region when compared with nearby clear spectra.

The second Apollo test measures the spatial coherence in a 9 × 9 box from AVHRR channel-4 images. To transfer the test to the one-dimensional data of the flight track, the test was carried out over five consecutive ARIES spectra “pixels”—the ARIES BT measured over the channel-4 pass band was treated as a pixel value. The standard deviation was calculated over this pixel and the two pixels before and after it. The central pixel was then flagged as cloudy if the standard deviation exceeded the Apollo threshold of 0.2 K. Pixels adjacent to a calibration and, therefore, a break in recording spectra were not included in the flagging. The results are shown in Fig. 7b. The horizontal dashed line is at 0.2 K. Crosses mark where this Apollo test would flag a pixel as cloudy. The limit of 0.2 K is consistent with the data; Fig. 7b shows that setting the limit to catch more cloud would lose more clears. The results in Table 4 show that the first test with SST has a zero FAR, the second test shows more skill but has a considerable FAR, and the combination of the two tests has the greater skill.

Trispectral tests for MODIS

A cloud scheme for MODIS, tested with HIS data, is described in Strabala et al. (1994). The technique is trispectral, using data from pass bands centered at 8.35, 11.1, and 12.0 μm (Table 3). The nearest MODIS channels are 29, 31, and 32—the channels used in creating the bit-18 flag in the MODIS cloud mask (Ackerman et al. 1998).

For each ARIES spectrum, three brightness temperatures were calculated over the three bands of Strabala et al. (1994). Again, a top-hat filter function was assumed in the calculations, using the wavenumber limits given in Table 3. From Table 2 in Strabala et al. (1994), three of the “clear over water” BT tests have been implemented: the 8-μm band minus 11-μm band is less than a threshold (−0.5 K here), the 11-μm band minus 12-μm band is less than another threshold (1.1 K here), and the 11-μm band is greater than a threshold based on local SST. The first two thresholds were set using data from flight A677. For comparison, the results in Fig. 8 are shown in a similar manner to Fig. 13 in Strabala et al. (1994). To give several cloud types, all three test flights' data (from A674, A678, and A741) are plotted. The colder, mainly ice cloud, data are all to the left, <280 K “11-μm BT.” The warmer tropical and subtropical data are to the right, >274 K 11-μm BT. The square or “+” is 8-μm band minus 11-μm band data. The diamonds and “×” are the 11-μm band minus 12-μm band data. Both the square and diamond represent cloud seen in the FOV. The horizontal dotted line shows the threshold for clear spectra for the 8-μm band minus 11-μm band difference—all of the clear “+” are below this line. In a similar way for the 11-μm band minus 12-μm band difference threshold, the horizontal dashed line has all of the clear “×” below it. The local SST for each flight are shown by the vertical solid lines; the dotted vertical lines to the right of each show the margin used for the threshold.

Figure 8 shows some of the crescent shapes seen in the data of Strabala et al. (1994, their Fig. 13), but here there is considerable clustering for the warmer data. When combined, the test results have a zero FAR, but the HSS is slightly less than the combined Apollo tests (Table 4).

Figure 9 shows the 11-μm wavelength BT calculated from the ARIES data for flight A678. The clouds, both detected by the trispectral tests and true, are marked as they were in Fig. 7 for the Apollo tests.

Spectral similarity for IMG and IASI

In Serio et al. (2000) a technique is presented for testing IASI spectra for cloud in the FOV over sea and then is tested with IMG spectra.

The test derives an index “hs” that measures the degree of homogeneity in BT between measured spectra and a closely matching clear reference spectrum over a window range of 900–800 cm−1 wavenumber (11.11–12.50 μm). Correlation and cross correlation are computed for several lags in wavenumber between the spectra; the results are summed and then are normalized to create |hs|. In Serio et al. (2000), reference spectra were taken from a database of clear-sky spectra; the spectra with the minimum difference to the observed spectra were chosen. This database was calculated using the Line-by-Line Radiative Transfer Model code. Here, a nearby ARIES clear spectrum (no cloud in the DFC image) was chosen where possible, rather than a calculated spectrum. The results are shown in Fig. 10. The x axis is as in Fig. 2, and against this axis is plotted the absolute value of the calculated index, |hs|, with the reference spectra taken early on in the flight A678, at spectrum 22. In Serio et al. (2000) the test for flagging the IMG TOA spectra as cloudy was |hs| > 0.3; here the limit was set in the same manner as in the trispectral tests, by only having clear spectra below the limit value from the separate A677 flight data. This limit was also |hs| > 0.3. The statistical summary in Table 4 shows a skill close to the Apollo tests but with zero FAR, and the PoD is less than 0.5.

The performance of |hs| on the A677 data used to set the limit was very sensitive to the reference spectra chosen. Results for two different clear reference spectra for this first flight are shown in Fig. 11. The solid line shows |hs| with the reference spectra taken early on in flight A677, spectrum 20. The second, dashed line shows |hs| with the reference taken later on, at spectrum 196. The results show how critically |hs| depends on the reference spectra. Both regions from which the reference spectra were chosen were devoid of any cloud, but dropsonde data showed the earlier region to be much drier above the marine boundary layer. The 0.3 limit for |hs| was set using clear reference spectra at several positions to match best the local atmosphere.

Results over varied cloud

The statistical analysis has been performed for all schemes over all three test flights' data, including the Arctic cirrus and layers of cloud from A741, the subtropical cumulus from A678, and the tropical cumulus and altostratus from A674. The limits not dependent on SST were identical to those used for the A678-only tests. Results for all three flights are given in Table 5. A bias figure has also been added. It is defined as the ratio of all analyzed cloud to all true cloud (Wilks 1995) and, in the notation of Table 2, is given by
abac

An unbiased analysis would give a bias figure of exactly 1. Analyzing too much cloud gives a bias greater than 1, whereas too little cloud analyzed gives a bias less than 1. For the tests here, a bias of less than 1 would be preferred so that clear FOV were not rejected.

The Apollo tests again show a higher skill when combined, this time with a lower FAR. The trispectral and homogeneity tests still have zero FAR and a skill very close to the Apollo tests. The EOF scheme testing cobs(89) produces the highest skill by a small margin, with zero FAR. Trials were carried out using two thresholds, that is, by examining the value of combinations of cobs(n) associated with different cloud types. However, this approach gave no improvement in cloud detection for the extra complication. The simplest approach, one test on cobs(89) for cloud, worked best.

For all schemes the bias is toward too little cloud, as would be desired in any initial cloud-clearing scheme that was preceding more accurate computation.

Discussion

Proposed encoding and cloud-detection algorithm

If EOFs are used to encode the calibrated data by the data receiver prior to dissemination to the NWP center, then the scheme presented here for cloud detection has the potential to be performed extremely quickly, depending on the nature of the encoding. Lee (2003) suggests a method of data compression that is entirely consistent with this cloud-detection scheme.

The optimum compression occurs if the data receiver [e.g. European Organization for the Exploitation of Meteorological Satellites (EUMETSAT) for IASI] constructs an EOF set 𝗨all as described in stages 1–6 above. This EOF set can then be sent once to each NWP center. Prior to dissemination of the data, the data receiver projects each observed spectrum xobs onto 𝗨all (stage 7 above) to generate cobs. The residual Robs can then be found from
Robsxobsallcobs

The data receiver then needs only to transmit cobs and Robs to the NWP center, which can rebuild the spectra using a rearrangement of the above equation. The data compression comes about because the dimension of cobs is an array of dimension N equal to the number of spectra used in the generation of 𝗨all, which will be significantly less than the number of spectral points in the radiance spectra (IASI has ∼8000 spectral points). Although Robs has the same number of points as the original radiance spectra (∼8000 for IASI), because it represents the residual information not contained within 𝗨all, it is the case that, provided 𝗨all is constructed with care, the number of bits used to represent each number in Robs can be greatly reduced without losing significant information. Indeed, if 𝗨all were a perfect representation of all cloud-free and cloudy atmospheres, then Robs would only contain instrument noise. Lee (2003) shows that an IASI radiance spectrum requires ∼203 000 bits when the necessary 24 bits per channel are adopted. He goes on to show that with the binary universal form for the representation of meteorological data (BUFR) encoding, this value reduces to ∼78 000 bits per spectra. If the data are encoded using the technique suggested by Lee (2003), based on EOFs, then Lee (2003) shows that the data volume reduces to ∼24 000 bits for Robs and less than 700 bits for cobs [for quantization at half-IASI noise-equivalent differential temperature (NEΔT) specification].

Such an encoding of the data clearly represents a reduction in cost to the data receiver, and, if data are transmitted in this manner, the NWP center can quickly and cheaply (in terms of numerical processing) detect cloud in the spectra by looking at cobs(N), where N represents the first element that contains cloudy information. A threshold on cobs(N) would be set from past data validated with any other available data from satellite instruments. This threshold may differ by day or night and by region. Once the threshold was exceeded, the encoded spectra would be deemed cloudy.

When data are available from further cloud types it may be possible to discriminate between types of cloud, or clouds of different radiative significance (where, e.g., low-level cloud might be passed as “clear” if techniques are available to retrieve atmospheric properties above that cloud layer in a useful way).

Cloud-detection algorithm for no encoding

If no encoding is used, then the algorithm can readily be applied to the full spectra by just using one EOF, the first cloudy one, 𝗨(m, 89)all. Stage 7 in section 2 is then modified, with Eq. (5) becoming
cobsm,TAllxmobs
This equation could be extended to subsequent cloudy EOFs if required, and stage 8 could be followed as before to detect cloud.

Conclusions

A simple cloud-detection scheme based on the use of EOFs has been presented and has been tested using a limited set of aircraft observations. The scheme reduces to a simple threshold technique that is efficient and simple, especially if the data are received at the NWP center using an encoding scheme like that suggested by Lee (2003). For the data presented here, only one value was required to be tested. With only eight instrument spectra in 𝗨all, the detection limit is around a 1-K change in BT across the 800–1200 cm−1 wavenumber window.

This scheme, when applied to the test data summarized in Table 5, is of comparable skill to other schemes. However, for data already transmitted as EOFs, the only external data it requires are the single threshold value for cobs(89), the element corresponding to the first cloudy EOF. No other prior information, such as SST, or reference atmospheric profiles are required. The first Apollo test, which also requires no background information, performs nearly as well but has lower PoD and HSS. The combined Apollo test, requiring background data, has a better bias but less skill (Table 5).

Without having to fetch or collocate prior data, the EOF scheme offers a very efficient method of removing obviously cloudy data at the initial stages of (or even before) processing. This quality will allow a concentration of resources in time and computation on the best data—the spectra from which parameters can be retrieved to greater accuracy than is given by the background field. With the large volumes of data due to be processed for AIRS and IASI, this optimizing of resources should be most beneficial to NWP centers. By compressing and then sending full spectral data, rather than just a subset of channels, NWP centers are given the freedom to use the data that best fit their particular purpose and resources. They can control how many channels, or which subsets, they use and when. It also allows the whole spectrum to be monitored against NWP models, which is particularly important for an interferometric radiometer for which problems in the measured interferogram may appear over the spectrum of derived radiances.

The scheme has been tested with a set of observations gathered by ARIES, utilizing only a limited number of clear-sky and cloudy spectra. If the scheme is adopted for a spaceborne sounder, then the number of spectra used in constructing 𝗨all would necessarily be larger, which would act to improve the performance of the cloud-detection scheme. In other words, the very fact that the scheme matches (and in some cases outperforms) the performance of other cloud-detection techniques, using only a limited set of ARIES data, is a good indication that it will perform well with an operational sounder. With collocated AIRS and MODIS data becoming available, we would hope next to test the scheme with these data. Not only would this test use data measured from space, but it would also allow performance over many more and varied cloud scenes to be investigated.

Acknowledgments

We are indebted to several colleagues for many useful discussions about this work, to the reviewers for many useful comments, and to one reviewer in particular for the idea in section 5b. We thank Marco Matricardi of ECMWF for providing the TIGR profiles, the NOAA SAA for the global 50-km-resolution SST analysis data, and the BSH for the Baltic Sea SST analysis. We must also thank the other staff in the Measurement of Tropospheric Humidity (MOTH) projects and the flight crew of the C-130 research aircraft for their part in gathering the data presented here.

REFERENCES

  • Ackerman, S. A., , K. I. Strabala, , W. P. Menzel, , R. A. Frey, , C. C. Moeller, , and L. E. Gumley. 1998. Discriminating clear sky from clouds with MODIS. J. Geophys. Res. 103:3214132157.

    • Search Google Scholar
    • Export Citation
  • Aumann, H. H., and R. J. Pagano. 1994. Atmospheric Infrared Sounder on the Earth Observing System. Opt. Eng. 33:776784.

  • English, S. J., , J. R. Eyre, , and J. A. Smith. 1999. A cloud-detection scheme for use with satellite sounding radiances in the context of data assimilation for numerical weather prediction. Quart. J. Roy. Meteor. Soc. 125:23592378.

    • Search Google Scholar
    • Export Citation
  • Gadd, A. J., , B. R. Barwell, , S. J. Cox, , and R. J. Renshaw. 1995. Global processing of satellite sounding radiances in a numerical weather prediction system. Quart. J. Roy. Meteor. Soc. 121:615630.

    • Search Google Scholar
    • Export Citation
  • Griffiths, P. R., and J. A. de Haseth. 1986. Fourier Transform Infrared Spectrometry. Vol. 83, Chemical Analysis, John Wiley and Sons, 671 pp.

    • Search Google Scholar
    • Export Citation
  • Huang, H. L., and P. Antonelli. 2001. Application of principal component analysis to high-resolution infrared measurement compression and retrieval. J. Appl. Meteor. 40:365388.

    • Search Google Scholar
    • Export Citation
  • Lee, A. C. L. 2003. Strategy for dissemination and use of advanced sounder radiances. Met Office (RSI) Branch Working Paper 205. [Available from National Meteorological Library, FitzRoy Road, Exeter, EX1 3PB, United Kingdom.].

    • Search Google Scholar
    • Export Citation
  • Masiello, G., , M. Matricardi, , R. Rizzi, , and C. Serio. 2002. Homomorphism between cloudy and clear spectral radiance in the 800–900-cm−1 atmospheric window region. Appl. Opt. 41:965973.

    • Search Google Scholar
    • Export Citation
  • Nicholls, S. 1978. Measurements of turbulence by an instrumented aircraft in a convective boundary layer over the sea. Quart. J. Roy. Meteor. Soc. 104:653676.

    • Search Google Scholar
    • Export Citation
  • Saunders, R. W., and K. T. Kriebel. 1988a. An improved method for detecting clear sky and cloudy radiances from AVHRR data. Int. J. Remote Sens. 9:123150.

    • Search Google Scholar
    • Export Citation
  • Saunders, R. W., and K. T. Kriebel. 1988b. Errata. Int. J. Remote Sens. 9:13931394.

  • Scott, N. A., and A. Chedin. 1981. A fast line-by-line method for atmospheric absorption computations—The automatized atmospheric absorption atlas. J. Appl. Meteor. 20:802812.

    • Search Google Scholar
    • Export Citation
  • Serio, C., , A. M. Lubrano, , F. Romano, , and H. Shimoda. 2000. Cloud detection over sea surface by use of autocorrelation functions of upwelling infrared spectra in the 800–900-cm−1 window region. Appl. Opt. 39:35653572.

    • Search Google Scholar
    • Export Citation
  • Simeoni, D., , C. Singer, , and G. Chalon. 1997. Infrared atmospheric sounding interferometer. Acta Astronautica 40:113118.

  • Stephenson, D. B. 2000. Use of “odds ratio” for diagnosing forecast skill. Wea. Forecasting 15:221232.

  • Strabala, K. I., , S. A. Ackerman, , and W. P. Menzel. 1994. Cloud properties inferred from 8–12-μm data. J. Appl. Meteor. 33:212229.

  • Taylor, J. P., , J. M. Edwards, , M. D. Glew, , P. Hignett, , and A. Slingo. 1996. Studies with a flexible new radiation code. II: Comparisons with aircraft short-wave observations. Quart. J. Roy. Meteor. Soc. 122:839861.

    • Search Google Scholar
    • Export Citation
  • Wilks, D. S. 1995. Statistical Methods in the Atmospheric Sciences: An Introduction. Vol. 59, International Geophysics Series, Academic Press, 467 pp.

    • Search Google Scholar
    • Export Citation
  • Wilson, S. H. S., , N. C. Atkinson, , and J. A. Smith. 1999. The development of an airborne infrared interferometer for meteorological sounding studies. J. Atmos. Oceanic Technol. 16:19121927.

    • Search Google Scholar
    • Export Citation

APPENDIX

Analysis of the Residuals

A series of tests were performed to look at the potential cloud information contained within the residuals R as defined in Eq. (4). Note that at this stage the EOF set being used consists only of cloud-free spectra (𝗨clr). To study the residuals, the sum of the residual over three atmospheric window regions was computed:

  1. Σ Ra 750–975 and 1075–1200 cm−1 (missing the ozone band around 1000 cm−1),
  2. Σ Rb 2000–2200 cm−1, and
  3. Σ Rc 2400–2975 cm−1 wavenumber.

The test data, for this test on residuals, came from a consecutive series of ARIES spectra from which the four clear views in 𝗨clr were extracted in flight A677, including clear air and cloud views. Cloudy or clear FOVs were identified using the technique described in section 3.

Increases in these residual sums would indicate the presence of significant information that is not contained in 𝗨clr and that may indicate that cloud was present. The residual sum was tested to see if a distinct threshold value between clear and cloudy spectra could be determined.

Clouds showed up in the Σ Ra (750–1200 cm−1; Fig. A1), and in the other residuals (not shown here for reasons of brevity). The dashed vertical lines show breaks in the time sequence during ARIES calibrations. All of the cloud occurred in the fifth measuring sequence—spectra numbered from 131 to 156. The four ARIES spectra incorporated into 𝗨clr appear as zero values of each Σ Ra. These four ARIES spectra were necessary to reduce jumps in clear-view Σ Ra across calibrations and a slope in the Σ Ra between calibrations.

Tests were also made in which three separate SVD operations over the three discrete window regions of the spectrum defined above were performed. The results showed that the cloud signatures are not distinct without the spectra outside each window region being included in the calculation. In effect, changes in each single window region can be explained by clear-air changes when not constrained by the rest of the spectra.

The residual threshold scheme was found not to be robust when tested against other observations because the level required for the threshold was not consistent. This result is due to the high sensitivity of this technique to the correct representation of all cloud-free conditions within the dataset used to construct 𝗨clr.

Fig. 1.
Fig. 1.

(a) The noise used in normalization (dashed line) and the average measured ARIES spectra for flight A677 (solid line). (b) The same spectra after normalization by the noise

Citation: Journal of Applied Meteorology 43, 1; 10.1175/1520-0450(2004)043<0196:ICDUTE>2.0.CO;2

Fig. 2.
Fig. 2.

Time series of cobs(89) for flight A678. Squares denote cloud seen in the DFC imagery. The solid line shows the results for the main tests over the full spectral range. The dashed line shows an alternative cobs(89) for which all calculations have been made on spectra truncated above 2000 cm−1 wavenumber. This approach creates a nondaylight case because the solar component is removed. The horizontal lines show limits used in detecting cloud: the dotted line is for the main tests, and the dash–dot line is for the alternative, nondaylight, case

Citation: Journal of Applied Meteorology 43, 1; 10.1175/1520-0450(2004)043<0196:ICDUTE>2.0.CO;2

Fig. 3.
Fig. 3.

The first 10 EOFs plotted individually, numbered from 0 to 9, with eigenvalues w noted alongside. The spectra were noise normalized before SVD to EOFs. Values over 1.0 are above the ARIES noise level

Citation: Journal of Applied Meteorology 43, 1; 10.1175/1520-0450(2004)043<0196:ICDUTE>2.0.CO;2

Fig. 4.
Fig. 4.

(a) The first EOF in 𝗨all that contains cloud information (solid line) and the running mean (dashed line). Both have been denormalized for comparison with the (b) radiance difference—the averages of all cloud spectra minus all clear spectra

Citation: Journal of Applied Meteorology 43, 1; 10.1175/1520-0450(2004)043<0196:ICDUTE>2.0.CO;2

Fig. 5.
Fig. 5.

(a) The ranked eigenvalues w, illustrating how they quickly fall. (b) The rms of the residual R, computed as the eigenvectors within 𝗨all are used one after another to create each element of cobs. The lower line in (b) shows the result on compression of an ARIES spectra that was used within 𝗨all. The upper line in (b) shows the result when compressing an ARIES clear spectra from the same flight, but not used within 𝗨all

Citation: Journal of Applied Meteorology 43, 1; 10.1175/1520-0450(2004)043<0196:ICDUTE>2.0.CO;2

Fig. 6.
Fig. 6.

Plots of cloudy spectra minus the spectra from a nearby clear view. The plots are in pairs for two cloudy views: (a), (b) missed cloud at spectrum 104 and (c), (d) cloud detected at spectrum 91. Here (a) and (c) show the difference at full ARIES spectral resolution and (b) and (d) have been boxcar smoothed by 25 cm−1 wavenumbers

Citation: Journal of Applied Meteorology 43, 1; 10.1175/1520-0450(2004)043<0196:ICDUTE>2.0.CO;2

Fig. 7.
Fig. 7.

Apollo test results on one flight's (A678) data. (a) The difference at 12 μm with SST taken to flight altitude. The threshold is marked by a dotted line. (b) The second test of a running calculation of standard deviation over five consecutive spectra. The threshold is marked by a dashed line. True cloud in the FOV is marked by boxes over the plots. Cloud as analyzed by Apollo is marked by crosses independently for each test

Citation: Journal of Applied Meteorology 43, 1; 10.1175/1520-0450(2004)043<0196:ICDUTE>2.0.CO;2

Fig. 8.
Fig. 8.

The trispectral band differences plotted against 11-μm BT for all three test flights: A674, A678, and A741. The 8- minus 11-μm data are squares or “+”, with a dotted horizontal line showing the threshold. The 11- minus 12-μm data are diamonds or “×” with a dashed line showing the threshold. The SST-based thresholds are the vertical dotted lines, one for each flight. Clear FOVs are marked by “+” or “×”; otherwise cloud was seen within the FOV (denoted by squares or diamonds)

Citation: Journal of Applied Meteorology 43, 1; 10.1175/1520-0450(2004)043<0196:ICDUTE>2.0.CO;2

Fig. 9.
Fig. 9.

A subset of the trispectral data from Fig. 8: only the 11-μm band BT for one flight, A678. Cloud detected by the trispectral tests is marked with “×”; squares mark where true cloud was seen within the FOV

Citation: Journal of Applied Meteorology 43, 1; 10.1175/1520-0450(2004)043<0196:ICDUTE>2.0.CO;2

Fig. 10.
Fig. 10.

The homogeneity results, plotting |hs| for one flight, A678. The threshold on |hs| is shown by the dotted line. Clouds detected by the tests are marked by “×”; true clouds seen in the FOV are marked by squares over the plot

Citation: Journal of Applied Meteorology 43, 1; 10.1175/1520-0450(2004)043<0196:ICDUTE>2.0.CO;2

Fig. 11.
Fig. 11.

The sensitivity of |hs| to the reference spectra chosen, illustrated by two reference spectra at either end of the A677 flight track. The solid line has a reference at the beginning, spectrum 20. The dotted line has a reference at the end, spectrum 197. Overplotted squares denote cloud seen within the FOV

Citation: Journal of Applied Meteorology 43, 1; 10.1175/1520-0450(2004)043<0196:ICDUTE>2.0.CO;2

i1520-0450-43-1-196-fa01

Fig. A1. A time series of the summed residual, Σ Ra (longest wavelength window), during the A677 flight. The four points at which Σ Ra = 0 are the ARIES-measured clear views added to the modeled clear spectra. Dashed vertical lines mark calibration periods. Clouds were present in most views from spectra 130 to 160

Citation: Journal of Applied Meteorology 43, 1; 10.1175/1520-0450(2004)043<0196:ICDUTE>2.0.CO;2

Table 1.

BT differences due to cloud from the A678 test data for selected cloudy spectra minus a clear-view spectrum

Table 1.
Table 2.

The 2 × 2 contingency table that defines the notation (ad ) used in comparing the analysis of cloud from ARIES spectra tested against the true cloud presence derived from the DFC imagery

Table 2.
Table 3.

ARIES wavenumber ranges and corresponding wavelengths used for AVHRR and trispectral tests

Table 3.
Table 4.

Results from testing cobs (89) with the other schemes when detecting cloud in the A678 ARIES data. Observations totaled 64 cloudy and 98 clear views. The minimum FAR and maximum PoD and HSS scores are italicized

Table 4.
Table 5.

Comparison of results when detecting cloud for all test case flights: A674 (tropical, with altostratus and cumulus), A678 (subtropical, small cumulus), and A741 (subarctic, cirrus). Observations totaled 199 cloudy and 109 clear views. As in Table 4, minimum FAR and maximum PoD and HSS scores are italicized, as is the bias closest to unity

Table 5.
1

FAR is sometimes defined with all true clears on the denominator rather than all analyzed cloud [see Eq. (3) in Stephenson 2000)], which would be expressed as b/(b + d). The difference between the two definitions obviously becomes less as the number of falsely analyzed clouds b approaches zero.

Save