A physical inversion scheme has been developed dealing with cloudy as well as cloud-free radiance observed with ultraspectral infrared sounders to simultaneously retrieve surface, atmospheric thermodynamic, and cloud microphysical parameters. A fast radiative transfer model, which applies to the clouded atmosphere, is used for atmospheric profile and cloud parameter retrieval. A one-dimensional (1D) variational multivariable inversion solution is used to improve an iterative background state defined by an eigenvector-regression retrieval. The solution is iterated in order to account for nonlinearity in the 1D variational solution. It is shown that relatively accurate temperature and moisture retrievals can be achieved below optically thin clouds. For optically thick clouds, accurate temperature and moisture profiles down to cloud-top level are obtained. For both optically thin and thick cloud situations, the cloud-top height can be retrieved with relatively high accuracy (i.e., error <1 km). National Polar-orbiting Operational Environmental Satellite System (NPOESS) Airborne Sounder Testbed Interferometer (NAST-I) retrievals from the The Observing-System Research and Predictability Experiment (THORPEX) Atlantic Regional Campaign are compared with coincident observations obtained from dropsondes and the nadir-pointing cloud physics lidar (CPL). This work was motivated by the need to obtain solutions for atmospheric soundings from infrared radiances observed for every individual field of view, regardless of cloud cover, from future ultraspectral geostationary satellite sounding instruments, such as the Geosynchronous Imaging Fourier Transform Spectrometer (GIFTS). However, this retrieval approach can also be applied to the ultraspectral sounding instruments to fly on polar satellites, such as the Infrared Atmospheric Sounding Interferometer (IASI) on the European MetOp satellite, the Cross-track Infrared Sounder (CrIS) on the NPOESS Preparatory Project, and the follow-on NPOESS series of satellites.
Nadir observations from a spacecraft- or an aircraft-flown infrared instrument can be used to infer the atmospheric temperature, moisture, and concentration of other chemical species using radiative transfer equation inversion techniques. The retrievals of atmospheric state, temperature, and moisture profiles obtained from infrared radiometric measurements will contain intolerable error near and below the cloud level if the cloud radiation and the attenuation of infrared radiation emitted from the earth’s surface and the atmosphere below the clouds are not properly accounted for in the retrieval process. Clouds greatly complicate the interpretation of infrared sounding data. The new ultraspectral resolution infrared spectral radiance data contain the information needed to alleviate much of the ambiguity between cloud, atmospheric temperature, and moisture contributions that exist in lower spectral resolution sounding radiance data. Since there are vast cloudy regions of the globe, a great deal of effort has gone into cloud detection and cloud-clearing processes (Smith et al. 2004). Nevertheless, the schemes dealing with cloud detection and cloud clearing (Smith 1968) remain a major source of error in the final retrieval products. Some schemes limit themselves to dealing with the observations unaffected by clouds (e.g., Chedin et al. 1985), while others make direct use of the cloudy radiances and attempt to retrieve temperature and moisture along with the cloud parameters (e.g., Susskind et al. 1984). Recently, fast molecular and cloud transmittance models (Huang et al. 2004) have been developed and used to retrieve cloud optical thickness from Atmospheric Infrared Sounder (AIRS) measurements (Wei et al. 2004). The retrieval methodology is reliant on other cloud and thermodynamic parameters, such as cloud-top pressure and atmospheric profiles of temperature, moisture, and ozone. Fast molecular and cloud transmittance models have also been used to enable the infrared radiances to be used under cloudy conditions with the accuracy required for sounding retrieval processing; the empirical orthogonal function (EOF) statistical regression retrieval algorithm (e.g., Smith and Woolf 1976; Zhou et al. 2002) has been expanded to include realistic cloud parameters (e.g., cloud-top height, effective particle diameter, and optical thickness) to deal with cloudy as well as cloud-free observations (Smith et al. 2004; Zhou et al. 2005a). With that recently developed EOF regression algorithm, cloud parameters as well as atmospheric profiles are retrieved simultaneously from infrared (IR) spectral radiance observations.
The National Polar-orbiting Operational Environmental Satellite System (NPOESS) Airborne Sounder Testbed Interferometer (NAST-I) has been successfully operating on high-altitude aircraft since 1998 (e.g., Cousins and Smith 1997; Smith et al. 2005). NAST-I is designed to support the development of future satellite temperature and moisture sounders such as the Infrared Atmospheric Sounding Interferometer (IASI) on the MetOp satellite, the Cross-track Infrared Sounder (CrIS) on the NPOESS Preparatory Project (NPP), and the follow-on NPOESS series of satellites, as well as the Geosynchronous Imaging Fourier Transform Spectrometer (GIFTS) designed to fly on geostationary satellites. For obtaining dynamics (i.e., moisture flux and winds) from geostationary ultraspectral infrared radiance data, atmospheric profiles must be retrieved for every field of view, regardless of cloud cover, in order to obtain motion measurements from a time sequence of the three-dimensional images of the atmospheric state variables (i.e., contiguous and continuous retrieval data are needed to construct the three-dimensional images). Here we report that a physical retrieval scheme is developed to further improve retrieval accuracy based on EOF regressions. The physical retrieval scheme is described alone with the retrieval results for demonstration. Retrieval results of cloud and atmospheric properties from NAST-I observations are compared with coincident observations obtained from the nadir-pointing cloud physics lidar (CPL) and dropsondes, respectively.
2. Physical retrieval scheme
a. Radiative transfer model and Jacobian matrix
The radiance measurements within the short-wavelength region, where the observed radiance may be affected by reflected solar radiation, are typically not used during daytime observing conditions. The cloud transmissive and reflective functions can be coupled with a fast molecular radiative transfer model. The NAST-I fast transmittance model used here is a combination of the Optimal Spectral Sampling (OSS) fast molecular radiative transfer model (Moncet et al. 2001; Liu et al. 2003) and the physically based cloud radiative transfer model based on the Discrete Ordinate Radiative Transfer (DISORT; Stamnes et al. 1988) calculations performed for a wide variety of cloud microphysical properties (e.g., Yang et al. 2001; Huang et al. 2004). In the OSS approach, an extension of the Exponential Sum Fitting Transmittance method (Wiscombe and Evans 1977), radiance for each instrument channel is represented as a linear combination of radiances computed at a few preselected monochromatic frequencies within the domain spanned by the instrument line shape function (Moncet et al. 2001). Since the OSS model calculates channel radiances and transmittances at a few representative monochromatic frequencies, the top-of-atmosphere radiance for an atmosphere containing a single cloud layer is expressed by the monochromatic radiative transfer equation at those frequencies,
where R is the upwelling spectral radiance at the top of atmosphere. Here, FT and FR are the cloud transmissive (including both direct and diffuse parts) and reflective (or albedo) functions, respectively. The terms R0, Rc, Rl, and R↓1 are upwelling emission below the cloud, emission from the cloud, upwelling emission above the cloud, and downwelling emission above the cloud, respectively. These terms are expressed below:
where ɛ refers to the earth’s surface emissivity, B is the Planck function, τ is the total transmittance from any given level to an upper boundary such as cloud level or the top of the atmosphere, and τ′ is the total transmittance from any given level to a lower boundary such as cloud level or the earth’s surface. The transmittance between the cloud level and the earth’s surface is τcs, while τtc is the transmittance between the top of the atmosphere and cloud level. The subscripts s, c, and t denote the surface, cloud, and top of atmosphere, respectively. Figure 1 illustrates the various terms described above. It is noted that the top of atmosphere is assumed to be at the aircraft altitude for applications with the NAST airborne data. The transmissive and reflective functions at a given wavelength are taken from the precalculated database. In this study, particle habits are assumed to be hexagonal columns. Examples of cloudy radiance spectra simulated by the NAST-I forward radiative transfer model against cloud-free conditions are illustrated in Fig. 2, showing that different cloud features are captured mostly in the long-wavelength window regions.
The cloud phase (i.e., clear, liquid, and ice cloud) is simply defined according to the retrieval. In other words, retrieved cloud-top temperature determines whether the cloud is liquid or ice. The relationship between the visible optical thickness (ξvis), ice water path (IWP), and particle size is simplified by the relationship between the IWP and cloud optical thickness based on the parameterization of the balloon and aircraft cloud microphysical database (Heymsfield et al. 2003). Here, using Eq. (6) of Heymsfield et al. (2003) and fitting the relationship between IWP and ξvis obtained from observations of Heymsfield et al. (2003, their Fig. 7a), a simplified relation is used to specify cloud effective particle diameter from the cloud optical thickness,
where a, b, and w are constants, De is the effective cloud particle diameter, and ξvis (later just ξ) is the visible optical thickness of the cloud. The definition for effective cloud particle size can be found in Yang et al. (2001). It should be mentioned that the assumptions and/or simplifications made herein for cloud microphysics (e.g., the particle habit, phase determination) are idealized. These assumptions will contribute to retrieval error but they are necessary in an ill-posed problem dealing with the inversion of infrared spectral radiance measurements. Two key cloud parameters, namely cloud-top pressure and cloud optical thickness, remain in the regression and the first part of the physical iteration inversion. The weighting functions (or Jacobian matrices) for cloud parameters are computed by a numerical perturbation method while others are computed by an analytical scheme. NAST-I analytical Jacobian matrices for temperature, water vapor, and skin temperature were computed as shown in Zhou et al. (2002). The numerical Jacobian matrix for cloud-top pressure and optical thickness is expressed as
where i is channel index, j is cloud parameter ψ index (ψ1 is cloud-top pressure and ψ2 is cloud optical thickness), and R is the simulated spectral radiance. An example of a cloud parameter Jacobian is shown in Fig. 3. The different spectral distributions of these weighting functions indicate the different channel radiance sensitivity and/or response to the cloud height and optical thickness.
b. Inversion scheme
An iterative eigenvector-regression retrieval scheme was initially developed and used for retrieval analyses on both thermodynamic profiles and cloud parameters. These regression retrievals were compared with CPL and dropsonde data indicating a favorable agreement. Once the first guess is generated from the regression technique described by Zhou et al. (2005a), a nonlinear iterative procedure is set up to produce a retrieval that is an improvement of the first guess (i.e., the EOF regression retrieval results). A one-dimensional (1D) variational solution, also known as the regularization algorithm or the minimum information method (e.g., Twomey 1963; Tikhonov 1963; Rodgers 1976; Hansen 1998), is chosen for the NAST-I physical retrieval methodology, which uses the regression solution as the initial guess. This solution has been applied to NAST-I radiances for retrieving both thermodynamic parameters and trace gases under cloud-free conditions (Zhou et al. 2002, 2005b). Here, we expand the same methodology to include cloud parameters dealing with retrieval problems under both cloudy and cloud-free conditions.
If the NAST-I observed radiance Rmν of each channel is known, then Rmν can be considered as a nonlinear function of the atmospheric temperature profile (T), water vapor mixing ratio profile (q), surface skin temperature (Ts), surface emissivity (ɛ), cloud-top pressure (Pc), effective cloud particle diameter (De), cloud visible optical thickness (ξ), cloud phase (ϕ), etc. Therefore, Rmν equals Rν(T, q, Ts, ɛ, Pc, De, ξ, ϕ, . . .) + σν, where σν is the instrument plus other sources of noise. Notice that De is initially a function of ξ as in Eq. (7). In general,
where the state vector X contains atmospheric temperatures, atmospheric moisture mixing ratios, surface skin temperature, cloud optical thickness, cloud-top height, etc. Here, Ym contains N (number of channels used) observed radiances. The linear form of Eq. (9) is
where Y′ is the linear tangent of the forward model Y (or R), the weighting function (or Jacobian) matrix. Here, the linear model Y′ uses an efficient analytical form (Li et al. 2000) for thermodynamic parameters and a numerical perturbation method for cloud parameters. A general form of the 1D minimum variance solution minimizes the following penalty function (Rodgers 1976):
where superscript T denotes the transpose. Using the Newtonian iteration,
the following quasi-nonlinear iterative form (Eyre 1989) is obtained:
where δXn = Xn − X0, δYn = Ym − Y(Xn), X is the thermodynamic and cloud parameters to be retrieved, X0 is the initial state of these parameters or the first guess, Ym is the vector of the observed radiances or brightness temperatures used in the retrieval process, 𝗘 is the observation error covariance matrix (which includes instrument noise and forward model error), and 𝗛 is the a priori matrix that constrains the solution. Here, 𝗛 can be the inverse of the a priori first-guess error covariance matrix or another type of matrix. If the statistics of both the measurement and a priori error covariance matrix are Gaussian, then the maximum likelihood solution is obtained. However, if the a priori error covariance matrix is not known or is estimated incorrectly, the solution will be suboptimal (Eyre 1989). Here, we apply 𝗛 = γ𝗜 in Eq. (13), where γ is a Lagrangian multiplier that serves as a smoothing factor. Equation (13) becomes
which is commonly referred to as the minimum information solution. It is noted that γ is dependent upon the observations, the observation error, and the first guess of the atmospheric profile; often it is chosen empirically (e.g., Susskind et al. 1984; Smith et al. 1985; Hayden 1988). In the NAST-I retrieval procedure, the Discrepancy Principle (e.g., Morozov 1984; Carfora et al. 1998; Li and Huang 1999) is applied to determine the appropriate smoothing factorγ. Thus,
where σ2 = ΣNk=1 e2k, ek is the square root of the diagonal of 𝗘 or the observation error of channel k, which includes instrument error and forward model error. Error is shown in the equation e2k = η2k + f 2k, where ηk is the instrument noise of channel k while fk is the forward model error, which is assumed to be 0.3 K for the same channel. Usually σ2 can be estimated from the instrument noise and the validation of the atmospheric transmittance model used in the retrieval. Since Eq. (15) has a unique solution for γ (Li and Huang 1999), the atmospheric parameters and the smoothing factor can be determined simultaneously. In NAST-I retrieval processing, a simple numerical approach is adopted for solving Eq. (15); γ is changed in each iteration according to γn+1 = qnγn (here q is a factor for γ increasing or decreasing). Based on Eq. (15), q is obtained in each iteration by satisfying the following conditions:
The q factor has been found from empirical experience to ensure that the solution is stable between iterations. Thus, γ keeps changing until the iteration stops.
In the retrieval processing, several checks are made for retrieval quality control. The root-mean-square (RMS) of quantity [Y(Xi) − Ym] from all selected channels χi is computed to check the convergence (or divergence). If χi+1 > χi within 2 iterations (i.e., iteration diverges), then the iteration is stopped and the retrieval is set to the first guess (or the previous atmospheric state); otherwise, iteration continues until χn < 1.0 K and |χn − χn−1| < 0.01 K or until a maximum of 10 iterations is reached. The degree of convergence of each iteration depends on the accuracy of the previous atmospheric and surface state. In addition, at each iteration, each level of the water vapor profile is checked for supersaturation. A unity magnitude of relative humidity is assumed at any supersaturated level.
The cloud particle size is derived using the relationship of Eq. (7) between the particle size and optical thickness. This relationship plays an influential role in both regression retrieval and physical matrix inversion. However, this relationship may not be true from case to case; on the other hand, the cloud signatures are captured in the long wavelength window region as shown in Fig. 2, illustrating that the radiance spectral slope is sensitive to particle size and that radiance magnitude is sensitive to optical thickness. Therefore, cloud microphysical parameters, namely effective particle diameter and visible optical thickness, are further refined with the radiances observed within the 10.4–12.5-μm window region, similar to what was described by Wei et al. (2004). The fitting of the magnitude and spectral slope of the radiance spectrum within the window region is important for refining the cloud microphysical parameters. If these cloud parameters vary by more than 10% during this stage, the matrix inversion is performed again to produce atmospheric temperature and moisture profiles. To summarize this hybrid inversion scheme, a simplified flowchart is shown in Fig. 4.
3. Case study and validation
NAST-I instrumentation, measurements, calibration, and radiance validation are documented elsewhere (e.g., Cousins and Smith 1997; Smith et al. 1999, 2005; Larar et al. 2002). NAST-I provides relatively high spectral resolution (0.25 cm−1) measurements in the spectral region of 645–2700 cm−1 with moderate spatial resolution (a linear resolution equal to 13% of the aircraft altitude at nadir) cross-track scanning. While a large amount of data has been collected since 1998 under a variety of meteorological conditions, results from only a very limited dataset are presented herein for the purpose of “cloudy” retrieval demonstration. Retrievals from the THORPEX Atlantic Regional Campaign (ATReC; e.g., Shapiro and Thorpe 2004) are used to demonstrate this inversion methodology. These data, together with the radiosondes and dropsondes released from the National Oceanic and Atmospheric Administration (NOAA) G-4 aircraft that flew below the National Aeronautics and Space Administration (NASA) ER-2 aircraft, provide a unique dataset for the detailed analysis of retrieval resolution and accuracy. During this field campaign, cloud properties were also provided by the nadir-pointing CPL on board the NASA ER-2 aircraft (McGill et al. 2002). All coincident observations obtained during this experiment are used to understand the atmospheric state and cloud microphysical properties for validating NAST-I retrievals.
The experiment of 5 December 2003 is chosen to test and demonstrate this inversion scheme with a realistic cloud radiative transfer model. The target scenes (latitude from 32° to 42°N, longitude from 68° to 76°W), shown in Fig. 5, covered a variety of conditions desired by the scientific objectives of the experiment. These included a variety of cloud conditions, such as midlevel altocumulus, as well as low-level cumulus, thunderstorm, and extensive high cirrus in the ATReC region covered by the ER-2 and G-4 aircraft. The EOF regression retrievals have already shown reasonable agreement with the dropsondes and CPL observations (Zhou et al. 2005a). Here, the improvements through the physical retrieval scheme described above are emphasized by the demonstration of both radiance fitting and retrieval parameter validation.
a. Physical retrieval results and validation
The physical retrieval results of cloud and thermodynamic parameters are shown in Fig. 6. NAST-I retrieved cloud-top height (Hc) from the nadir observations against CPL measured cloud-top heights of the top two layers, and Fig. 6b shows the cloud optical thickness inferred from NAST-I measurements against that of the CPL 1064-nm channel measurements. It is noted that NAST-I spatial resolution (at the cloud height and at nadir) is 13% of the distance between the aircraft altitude and the cloud height (i.e., 1.56 km when the cloud height is at 8 km and the ER-2 is at 20 km), while the CPL horizontal resolution is about 0.2 km; furthermore, the NAST-I vertical resolution is about 1 km while the CPL vertical resolution is 0.03 km. Despite the differences of the instruments and of their spatial resolutions, the cloud-top heights inferred from NAST-I compare very well with CPL measurements for the variety of cloud conditions observed. The measurement sensitivity and accuracy of cloud optical thickness inferred from the infrared measurement is expected to be much poorer than that measured by the CPL because of the spatial resolution differences between the two instruments. Even so, NAST-I cloud optical thickness retrievals compare favorably to CPL observations.
NAST-I retrieved temperature and relative humidity (RH) vertical cross sections are shown in Figs. 6d,e, respectively. The areas whited out are under the clouds where the cloud optical thickness is larger than one. The variation of atmospheric conditions is captured very well by NAST-I retrievals, not only for the clear regions above optically thick clouds but also for regions below optically thin clouds. These soundings are also validated by the dropsondes released from the G-4 aircraft. The dropsondes are used to reveal the retrieval sounding accuracy under cloudy conditions. Intercomparisons between each dropsonde and the NAST-I local mean (i.e., 5 × 5 single field of views) retrieval are presented in Fig. 7. In general, the retrievals show a good agreement above the clouds; the sounding comparison continues to show a good agreement under the optically thin clouds to the second-layer clouds, as is indicated by the CPL observations, and retrievals are relatively accurate under the optically thin clouds. The erroneous retrievals below opaque clouds are shown. These errors, which were due to the limitation of infrared observations, were expected and should be discarded. As shown in Figs. 6 and 7, retrievals of temperature and moisture above the clouds are not disturbed by the clouds below. Both temperature and relative humidity profiles show a continuity variation from time to time (i.e., from location to location), indicating that the atmospheric features are well captured by NAST-I observations and retrievals.
b. Improved with physical inversion
The regression retrievals with realistic cloudy radiance training have already shown the improvement over the clear and/or isothermal equivalent radiance training (Zhou et al. 2005a). However, the physical inversion scheme further improves the retrieval accuracy from the cloudy iteration regression. Figure 8 shows the deviation of retrieved parameters from the regression to the physical algorithms used. Several approaches are used herein to show that the retrievals are evidently improved through physical inversion. The cloud particle size is produced as a function of optical thickness in order to reduce a number of retrieval parameters in the regression process, which results in an artificial correlation between these parameters, as in Eq. (7). Plotted in Fig. 9 are the retrieval results from regression (gray dots) to physical retrievals (black dots); the physical outcomes still obey the prediction of Eq. (7) but with expected scattering.
The cloud parameters retrieved from NAST-I observations plotted in Fig. 6 indicate that a variety of cloud conditions were also observed by the Geostationary Operational Environmental Satellite 8 (GOES-8) visible and infrared imaging sensors (Fig. 5). The radiance fitting statistics over this large-diversity dataset give a clear reflection of the integrity of this retrieval algorithm. A few thousand NAST-I observations (all nadir of 5 December 2003 flight) are used to compare with their associated retrieval simulations in order to show retrieval accuracy. The standard deviation error (STDE) and mean bias in brightness temperature are plotted in Fig. 10; the retrieval accuracy is improved significantly through the physical inversion. The relatively large error in the short wavelength region (i.e., wavenumber greater than 2200 cm−1) is mainly due to NAST-I instrument noise, a solar component not accounted for in the simulation, and the cloud model was cut off at a wavenumber of 2500 cm−1. Despite the uncertainty of the radiative transfer model (especially in the short wavelength region), the STDE from the physical retrieval is more like the instrument noise estimated from instrument calibration. This improvement evidently indicates that retrieved parameters, in general, are accurately retrieved through this physical inversion scheme.
Finally, spectral radiance comparisons of individual scans have also been examined over each dropsonde location where the temperature and moisture profiles can be validated to ensure that the deviations shown in Fig. 8 are in favor of approaching the real atmospheric conditions through the physical inversion. Two samples, the first and seventh dropsondes indicated in Fig. 6d, are shown in Figs. 11 and 12, respectively. Again, the radiance and retrieved parameters were compared to show the integrity of the algorithm. Figures 11a,b, 12a,b illustrate the radiance convergence to NAST-I observation, while Figs. 11c–e, 12c–e show the temperature and moisture convergence to the dropsonde through the physical inversion. The retrieved cloud parameters are indicated in the figure captions.
4. Concluding remarks
Our previous regression results indicate some success in the ability to retrieve information below scattered and partially transparent cirrus clouds, or clouds with an effective optical thickness of less than one. The initial EOF regression has laid a first step in dealing with infrared sounding data under cloudy conditions, which is now significantly improved by the physical iteration inversion described in this study. Results achieved with airborne NAST-I observations show that accuracies close to those achieved in totally cloud-free conditions can be achieved down to cloud-top levels. The accuracy of the profile retrieved below cloud-top level is dependent upon the optical thickness and fractional coverage of the clouds. The retrieval accuracy of temperature and moisture profiles is greatly improved by the physical inversion as shown by dropsonde validation. Therefore, the radiances can be accurately simulated by using physically inverted results, which is important for direct assimilation into a forecast model. The thermodynamic profile information might be obtained through a combination of cloud clearing and direct retrieval from the clouded radiances using a realistic cloud radiative transfer model. This work has demonstrated a novel retrieval methodology. The assumption of the cloud habit has been idealized, which does introduce errors. More realistic cloud habit assumptions and their associated parameters are strongly desired for future retrieval applications as cloud research progresses and cloudy radiative transfer models improve. The correct implementation still requires a considerable research development effort; however, cloudy sky radiative transfer models should enable the extraction of profile information from cloud-contaminated radiances suitable for numerical weather prediction application.
The NAST-I program is supported by the NPOESS Integrated Program Office (IPO), NASA Headquarters, and the NASA Langley Research Center. The authors also acknowledge support from NASA Headquarters Research Division Director Dr. Jack Kaye and IPO Program Manager Dr. Karen St. Germain. The authors express sincere thanks to the NAST-I team members from various organizations. The CPL data were kindly provided by Dr. Matthew J. McGill of NASA Goddard Space Flight Center. The fast cloud radiative transfer model was provided by Dr. Ping Yang of Texas A&M University. The OSS molecular radiative transfer model was provided by Atmospheric and Environmental Research, Inc.
Corresponding author address: Daniel K. Zhou, Mail Stop 401A, NASA Langley Research Center, Hampton, VA 23681. Email: email@example.com