A Spectral Data Compression (SDCOMP) Radiative Transfer Model for High-Spectral-Resolution Radiation Simulations

Chao Liu Collaborative Innovation Center on Forecast and Evaluation of Meteorological Disasters, and School of Atmospheric Physics, Nanjing University of Information Science and Technology, Nanjing, China

Search for other papers by Chao Liu in
Current site
Google Scholar
PubMed
Close
,
Bin Yao Collaborative Innovation Center on Forecast and Evaluation of Meteorological Disasters, and School of Atmospheric Physics, Nanjing University of Information Science and Technology, Nanjing, China

Search for other papers by Bin Yao in
Current site
Google Scholar
PubMed
Close
,
Vijay Natraj Jet Propulsion Laboratory, California Institute of Technology, Pasadena, California

Search for other papers by Vijay Natraj in
Current site
Google Scholar
PubMed
Close
,
Fuzhong Weng Chinese Academy of Meteorological Sciences, Beijing, China

Search for other papers by Fuzhong Weng in
Current site
Google Scholar
PubMed
Close
,
Tianhao Le Division of Geological and Planetary Sciences, California Institute of Technology, Pasadena, California

Search for other papers by Tianhao Le in
Current site
Google Scholar
PubMed
Close
,
Run-Lie Shia Division of Geological and Planetary Sciences, California Institute of Technology, Pasadena, California

Search for other papers by Run-Lie Shia in
Current site
Google Scholar
PubMed
Close
, and
Yuk L. Yung Division of Geological and Planetary Sciences, California Institute of Technology, Pasadena, California

Search for other papers by Yuk L. Yung in
Current site
Google Scholar
PubMed
Close
Free access

Abstract

With the increasing use of satellite and ground-based high-spectral-resolution (HSR) measurements for weather and climate applications, accurate and efficient radiative transfer (RT) models have become essential for accurate atmospheric retrievals, for instrument calibration, and to provide benchmark RT solutions. This study develops a spectral data compression (SDCOMP) RT model to simulate HSR radiances in both solar and infrared spectral regions. The SDCOMP approach “compresses” the spectral data in the optical property and radiance domains, utilizing principal component analysis (PCA) twice to alleviate the computational burden. First, an optical-property-based PCA is performed for a given atmospheric scenario (atmospheric, trace gas, and aerosol profiles) to simulate relatively low-spectral-resolution radiances at a small number of representative wavelengths. Second, by using precalculated principal components from an accurate radiance dataset computed for a large number of atmospheric scenarios, a radiance-based PCA is carried out to extend the low-spectral-resolution results to desired HSR results at all wavelengths. This procedure ensures both that individual monochromatic RT calculations are efficiently performed and that the number of such computations is optimized. SDCOMP is approximately three orders of magnitude faster than numerically exact RT calculations. The resulting monochromatic radiance has relative errors less than 0.2% in the solar region and brightness temperature differences less than 0.1 K for over 95% of the cases in the infrared region. The efficiency and accuracy of SDCOMP not only make it useful for analysis of HSR measurements, but also hint at the potential for utilizing this model to perform RT simulations in mesoscale numerical weather and general circulation models.

© 2020 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Vijay Natraj, vijay.natraj@jpl.nasa.gov

Abstract

With the increasing use of satellite and ground-based high-spectral-resolution (HSR) measurements for weather and climate applications, accurate and efficient radiative transfer (RT) models have become essential for accurate atmospheric retrievals, for instrument calibration, and to provide benchmark RT solutions. This study develops a spectral data compression (SDCOMP) RT model to simulate HSR radiances in both solar and infrared spectral regions. The SDCOMP approach “compresses” the spectral data in the optical property and radiance domains, utilizing principal component analysis (PCA) twice to alleviate the computational burden. First, an optical-property-based PCA is performed for a given atmospheric scenario (atmospheric, trace gas, and aerosol profiles) to simulate relatively low-spectral-resolution radiances at a small number of representative wavelengths. Second, by using precalculated principal components from an accurate radiance dataset computed for a large number of atmospheric scenarios, a radiance-based PCA is carried out to extend the low-spectral-resolution results to desired HSR results at all wavelengths. This procedure ensures both that individual monochromatic RT calculations are efficiently performed and that the number of such computations is optimized. SDCOMP is approximately three orders of magnitude faster than numerically exact RT calculations. The resulting monochromatic radiance has relative errors less than 0.2% in the solar region and brightness temperature differences less than 0.1 K for over 95% of the cases in the infrared region. The efficiency and accuracy of SDCOMP not only make it useful for analysis of HSR measurements, but also hint at the potential for utilizing this model to perform RT simulations in mesoscale numerical weather and general circulation models.

© 2020 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Vijay Natraj, vijay.natraj@jpl.nasa.gov

1. Introduction

Radiative transfer (RT) processes not only impact the Earth–atmosphere energy balance but also provide key information relevant to atmospheric remote sensing (Goody and Yung 1989; Liou 2002). Therefore, the transfer of radiation in the atmosphere is an essential problem in atmospheric modeling and measurements (Chandrasekhar 1960; Goody and Yung 1989; Liou 2002), and significant effort has been devoted to solving the RT equation under different circumstances and with differing degrees of accuracy. Since molecular absorption of atmospheric gases has significant spectral variation (absorption coefficients can change by a few orders of magnitude within a spectral region of 0.01 cm−1 or less), high spectral resolution (HSR) is normally required for accurate RT calculations. This implies that tens of thousands, if not more, of accurate RT simulations have to be performed for even a narrow spectral region. The resolution of monochromatic RT calculations is basically controlled by the width of the absorption line—stratospheric Doppler broadened lines are typically much narrower than tropospheric pressure-broadened lines. However, even for relatively broad absorption lines, scattering due to molecules, aerosols, and clouds makes the simulations complex and time consuming. The above considerations make it impractical to employ HSR simulations for applications such as satellite retrievals or weather and climate modeling.

On the other hand, HSR measurements can provide unique information about trace gases and other atmospheric constituents (e.g., Crevoisier et al. 2004; O’Dell et al. 2012). Such HSR instruments are on both polar orbiting and geostationary satellites. Examples include the Atmospheric Infrared Sounder (Aumann et al. 2003), the Infrared Atmospheric Sounding Interferometer (Hilton et al. 2012), the Cross-track Infrared Sounder (Han et al. 2013), the Visible Infrared Imaging Radiometer Suite (Yang et al. 2017), Orbiting Carbon Observatory-2 (Eldering et al. 2017) and TANSAT (Liu et al. 2013). Ground-based HSR instruments have also been developed for gas and aerosol retrievals [e.g., Total Carbon Column Observing Network (Wunch et al. 2011) and California Laboratory for Atmospheric Remote Sensing (Fu et al. 2014)]. Analysis of these HSR observations and development of retrieval algorithms require accurate and computationally efficient forward RT simulations. A distinction should be made between HSR RT simulations and HSR measurements. The former is normally performed at monochromatic wavelengths with a spectral resolution much finer than that of the actual measurements; the latter can be visualized as the convolution of the former over the instrument spectral response function.

Several rigorous RT schemes, for example, discrete ordinates (Chandrasekhar 1960; Liou 1973), Monte Carlo (Plass and Kattawar 1968), and adding–doubling (Twomey et al. 1966) methods, have been developed to perform accurate RT simulations, particularly for scattering atmospheres. Meanwhile, a HSR line-by-line model has to be applied to compute the gaseous absorption (Clough et al. 1992). The approach of using a line-by-line model coupled with exact RT calculations is reasonable for sensitivity studies or model validation for limited atmospheric scenarios; however, it becomes impractical for applications (such as radiance simulations for satellite retrieval/assimilation or flux simulations for climate and weather forecasting) that involve a large number of simulations over vast spatiotemporal domains. As a result, several fast RT methods have been developed for a variety of applications. Generally, there are two approaches to improve the efficiency of RT simulations. One approach achieves computational efficiency by accelerating every RT simulation but keeping the number of independent RT simulations unchanged (Wang et al. 2013; Liu et al. 2015). The second approach alleviates the computational burden by decreasing the number of RT simulations needed to compute HSR radiances (Liu et al. 2006; Moncet et al. 2008). The accuracy and efficiency of several fast RT approaches have also been investigated (Clough et al. 2005; Oreopoulos et al. 2012; Randles et al. 2013; Aumann et al. 2018).

A variety of fast RT models have been employed for HSR simulations (Duan et al. 2005; Natraj et al. 2005; Liu et al. 2006; Moncet et al. 2008; O’Dell 2010; Spurr et al. 2013; Wang et al. 2015; Kopparla et al. 2016; Somkuti et al. 2017). These models include two varieties of principal component analysis (PCA)-based techniques. Natraj et al. (2005) perform PCA on spectrally binned sets of optical properties; costly multiple-scattering RT calculations are only done for a few PCA-derived optical states. This approach will henceforth be referred to as optical-property-based PCA (Opt-PCA). On the other hand, Liu et al. (2006) perform rigorous RT calculations at only a small number of representative wavelengths, and then compute radiances at higher spectral resolution using PCA in the radiance domain. This methodology employs PCA in radiance space, and will be referred to as radiance-based PCA (Rad-PCA).

Both Opt-PCA and Rad-PCA have been tested and validated for various applications. Opt-PCA was first applied to scalar radiance simulations (no polarization) at high resolution in and around the O2 A-band (Natraj et al. 2005), and later expanded to RT modeling with polarization (Natraj et al. 2010). Subsequently, the PCA procedure was analytically differentiated and Jacobians developed for the PCA-based radiation fields (Spurr et al. 2013). A slightly different method for calculating the Jacobians was demonstrated by Efremenko et al. (2014a). Further work was also done to extend the technique to broadband radiances and fluxes (Kopparla et al. 2016), to improve treatment of aerosol scattering (Kopparla et al. 2017), and to demonstrate application for retrievals (Somkuti et al. 2017). Efremenko et al. (2014b) discussed Opt-PCA within the general framework of dimensionality reduction in optical property space. Rad-PCA is designed to fully exploit a large amount of information from spectral radiances (Liu et al. 2006; Matricardi 2010; Havemann et al. 2018). The technique has been extended for solar spectrum simulations (Liu et al. 2016; Yang et al. 2016), and been applied to retrievals of atmospheric profiles, surface characteristics, and cloud properties under all-sky conditions, as well as in satellite data assimilation (Liu et al. 2009; Matricardi and McNally 2014; Wu et al. 2017).

In this study, we combine the advantages of Opt-PCA and Rad-PCA to further enhance the computational efficiency of HSR RT simulations with negligible loss of accuracy. The new technique compresses spectral information in both optical property and radiance domains, and will hence be referred to as spectral data compression (SDCOMP) model. To the best of the authors’ knowledge, this is the first attempt at unifying the optical and radiance-based PCA approaches to both reduce the number of monochromatic wavelengths at which RT calculations are performed and optimize the RT calculation efficiency at each wavelength. Widely utilized fast RT models, such as the ones developed by Liu et al. (2006), Matricardi (2010), Moncet et al. (2015), and Havemann et al. (2018), fail to do the latter. A similar methodology (that couples the PCA-based approaches in a different manner) was recently applied for radiance computations in the Hartley-Huggins band of ozone (del Águila et al. 2019). However, their work only included the effects of gas absorption and Rayleigh scattering (no aerosols or clouds). Further, its performance for wider spectral ranges is still unclear.

The paper is organized as follows. The new model is introduced in section 2. Section 3 discusses the performance (in terms of accuracy and computational efficiency) of the algorithm for both solar and infrared spectral regions. We provide concluding remarks in section 4.

2. The SDCOMP model

Rad-PCA saves computational time by performing monochromatic RT calculations on a reduced set of representative wavelengths and extending these results to the whole spectral range using PCA in the radiance domain. On the other hand, Opt-PCA accelerates the simulations by efficient computation at each wavelength through PCA in the optical property (e.g., optical thickness, single-scattering albedo) domain. Clearly, the two PCA techniques are complementary; we combine the two approaches and investigate the performance of the unified model for simulations of HSR top of the atmosphere radiance. This study considers a solar spectral region between 0.75 and 0.92 μm and an infrared spectral region between 2000 and 2250 cm−1 to illustrate the performance of the method for both solar and thermal radiation sources. Spectral resolutions of 0.05 and 0.005 cm−1 are used for the solar and infrared regions, respectively; both scenarios contain 50 000 monochromatic wavelengths. It should be noted that this study is intended to present the general methodology of SDCOMP; the spectral resolution is not chosen to meet any specific application. Clearly, finer spectral resolutions should be used for more accurate results; this will be done in future studies. Indeed, the numerical efficiency of our method will increase with finer spectral resolution since the computation time of SDCOMP is a very weak function of spectral resolution; on the other hand, the computation time of exact RT methods scales linearly with the number of wavelengths.

Figure 1 shows schematically the SDCOMP methodology. The basic idea is to extend Opt-PCA results (left) at representative wavelengths to all required HSR wavelengths using the Rad-PCA approach (right). Opt-PCA is executed on a much coarser wavelength grid than the desired HSR grid. Opt-PCA follows a three-step procedure. First, the spectral region is split into bins, where each bin is characterized by grouping optical properties (such as atmospheric-layer trace gas optical thickness values or single-scattering albedos) that are similar within the bin. Second, PCA is performed on the optical properties so as to obtain the eigenstates with the largest contributions to the variance. Finally, accurate RT calculations (using 32 computational quadrature angles in this work) are carried out only for a few PCA-projected states.

Fig. 1.
Fig. 1.

Flowchart to illustrate the basic concept of the SDCOMP RT model.

Citation: Journal of the Atmospheric Sciences 77, 6; 10.1175/JAS-D-19-0238.1

Opt-PCA partitions the radiation field into single and multiple scattering (MS) contributions. The former can be computed efficiently and is calculated at every wavelength. The latter is computationally expensive. Opt-PCA computes two-stream (2S) approximations of the MS contributions at every wavelength; numerically exact MS contributions (using a large number of quadrature angles) are evaluated only for the mean optical state within the bin and a small number (typically 2–4) of PCA-perturbed optical states. PCA-derived correction factors are then used to compute more accurate approximations to the exact MS field. The correction factor is a metric to quantify the difference between 2S and numerically exact values of the MS field. The radiance calculated by Opt-PCA at the n representative wavelengths, Xn, can be expressed as
Xn=InSS+CnIn2S,
where InSS is the single-scattering (SS) contribution, In2S is the 2S approximation to the MS contribution and Cn is the correction factor. Details about the procedure to calculate Cn can be found in Spurr et al. (2013).
In Rad-PCA, the first l principal component (PC) vectors Vl×m are calculated offline from a large radiance training dataset at m wavelengths. Then, the radiances computed by Opt-PCA are used to compute PC scores Wl using the formula
Wl×Vl×n=Xn.
Here, Vl×n is a subset of the full set of PC vectors corresponding to the n representative wavelengths. n is normally much smaller than m, but larger than l; that is, m > > n > l. Since Vl×n is not a square matrix, we use the pseudoinverse (Vl×n)+ to find the least squares solution of Eq. (2); the dimension of this pseudoinverse matrix is n × l. Then, we have
WlXn×(Vl×n)+.
Hence, the full spectral matrix Xm can be calculated as follows:
XmWl×Vl×m=Xn×(Vl×n)+×Vl×m.
Rad-PCA calculates the radiances at the reference wavelengths, Xn, using exact RT simulations; the computational speed up of Rad-PCA is, therefore, approximately m/n. On the other hand, SDCOMP uses Opt-PCA to calculate Xn, thereby speeding up the reference radiance calculations. Meanwhile, Opt-PCA and SDCOMP differ in the total number of wavelengths at which radiances are calculated; the former needs to perform calculations over the entire set of m wavelengths, whereas the latter only requires computations at the n reference wavelengths. It is clear that SDCOMP optimizes the advantages of both the optical and radiance-based PCA models. For further details on Opt-PCA and Rad-PCA, the readers are referred to prior publications on those models (Natraj et al. 2005, 2010; Liu et al. 2006, 2016; Spurr et al. 2013; Kopparla et al. 2016; Yang et al. 2016; Kopparla et al. 2017; Somkuti et al. 2017).

Radiances from exact RT simulations (including absorption, scattering and, in the thermal infrared, emission effects) for a large variety of atmospheric conditions are needed to calculate the PC vectors used by Rad-PCA. For this purpose, we use an atmospheric profile dataset developed by the Copernicus Atmospheric Monitoring Service (CAMS) of the European Centre for Medium-Range Weather Forecasts (ECMWF), which is a collection of representative model atmospheres (Eresmaa and McNally 2016). The dataset includes 40 000 atmospheric profiles on a 60-level vertical grid from the surface to 0.1 hPa covering realistic annual and diurnal variations in temperature, specific humidity, and the mixing ratio of ozone, carbon monoxide, nitrogen dioxide, sulfur dioxide, and formaldehyde.

For consideration of aerosol scattering, we employ simulations of the Modal Aerosol Dynamics Model for Europe (MADE) (Ackermann et al. 1998). MADE provides vertical profiles of aerosol optical thickness for five aerosol types (black carbon, organic carbon, accumulation mode sea salt, coarse mode sea salt, and sulfate). Corresponding spectrally dependent optical properties for each aerosol type are precalculated using Mie theory with size distribution parameters and refractive indices from the Optical Properties of Aerosols and Clouds (OPAC) database (Hess et al. 1998), except for black carbon, where they are taken from d’Almeida et al. (1991).

For the infrared spectral region, we use monthly mean emissivity from the University of Wisconsin–Madison Baseline Fit Emissivity Database (Seemann et al. 2008). For the visible to near-infrared spectral range, we use monthly mean surface reflectivity spectra based on the visible and near-infrared bidirectional reflectance distribution function (BRDF) atlas (Vidot et al. 2018), which was developed for the fast RT model RTTOV (Saunders et al. 2018).

The High-Resolution Transmission 2016 (HITRAN2016) molecular spectroscopic database (Gordon et al. 2017) is used to obtain gas absorption optical thickness profiles; all gases that absorb in the relevant spectral ranges are considered in the simulations. The RT calculations for the training dataset employ the linearized discrete ordinate radiative transfer (LIDORT) model (Spurr et al. 2001).

Figure 2 presents some examples of the atmospheric temperature, water vapor, and aerosol profiles and the surface albedo spectra; for each parameter, 100 different scenarios are shown in the figure. The figure clearly illustrates the large dynamic range of the parameters, which is required to obtain a representative HSR radiance dataset to be used by Rad-PCA for training. The top panel of Fig. 3 shows the resulting reflectance spectra, based on the above profiles, within the solar spectral region. We calculated spectra for 7000 randomly selected atmospheric scenarios: (i) 5000 scenarios are used as training spectra to calculate the PC vectors, (ii) 1000 scenarios are used to choose the representative wavelengths, and (iii) 1000 scenarios are used for model evaluation (SDCOMP accuracy). The bottom panel of Fig. 3 shows the first five PCs, as well as the average reflectances, in the solar spectral region. The average reflectance clearly shows the dominant gas absorption features, whereas the PC vectors capture spectral variations of the reflectances (due to absorption, scattering, and surface reflection) from one wavelength to another. In this study, we use the first 100 PCs since the results become stable and additional PCs do not further improve the accuracy. This can be modified based on the spectral information within the wavelength range of interest.

Fig. 2.
Fig. 2.

Examples of atmospheric temperature, water vapor, and aerosol optical depth (AOD) profiles, and surface albedo spectra used in the simulations.

Citation: Journal of the Atmospheric Sciences 77, 6; 10.1175/JAS-D-19-0238.1

Fig. 3.
Fig. 3.

(top) Examples of simulated reflectance spectra at the top of the atmosphere used for training Rad-PCA. (bottom) Corresponding reflectance mean and first five principal components (PC-1 to PC-5). To illustrate the curves on the same plot, PC-1 to PC-5 are shifted by adding 0.08, 0.12, 0.16, 0.20, and 0.24, respectively, to their values.

Citation: Journal of the Atmospheric Sciences 77, 6; 10.1175/JAS-D-19-0238.1

It should be noted that the accurate monochromatic RT simulations used here for the training and validation datasets are computationally expensive; however, since the training is performed only once during the model development stage, the large computational burden is acceptable. On the other hand, Opt-PCA simulations are accurate enough for the calculation of the training dataset. We have tested SDCOMP based on training with Opt-PCA results (not shown here), and have found errors due to this process to be negligible. This procedure will make future SDCOMP development for wider spectral regions much more efficient.

Liu et al. (2006) selected the representative wavelengths to predict PC scores by considering correlation coefficients between radiances at different wavelengths. This study uses an alternative, iterative method to pick the wavelengths. It should be noted that neither method might provide the “optimal” choice; more research is required to optimize the procedure. Figure 4 shows schematically the iterative method. We start with an equal-spaced grid in the spectral domain, and iteratively modify the grid based on the ensemble errors of calculated SDCOMP radiances. We follow two rules for the modification. First, wavelengths at which SDCOMP radiances have the largest errors (compared to an “exact” calculation using the LIDORT RT model) replace the closest grid point from the prior iteration, so that the “quasi uniform” distribution of the grid will not be significantly affected. Second, the total number of grid points will be kept the same, and only ~25% of the grid points are changed during each iteration. Typically, after approximately three to four iterations, convergence is achieved; that is, the overall errors cannot be improved any more by changing the grid (keeping the total number of points fixed). Note that, after convergence, if SDCOMP errors at particular wavelengths are still significant, we can add those monochromatic wavelengths to the grid to further reduce computational errors.

Fig. 4.
Fig. 4.

Schematic of the iterative method to choose the representative monochromatic wavelengths.

Citation: Journal of the Atmospheric Sciences 77, 6; 10.1175/JAS-D-19-0238.1

Figure 5 illustrates the key steps of the aforementioned iterative procedure. First, 400 monochromatic points are chosen uniformly within the given spectral region. Based on this initial grid, SDCOMP results at the full HSR set of 50 000 wavelengths are obtained for 1000 different atmospheric scenarios and evaluated against accurate radiances/reflectances at the top of the atmosphere from LIDORT calculations. The average relative errors for the HSR wavelength grid are illustrated in Fig. 5a. Figure 5b illustrates a particular spectral region (corresponding to the region with blue background in Fig. 5a) with relatively large relative errors; the blue circles (eight overall for this region) indicate the initial equally spaced grid points. Clearly, the choice of uniformly separated grid points introduces significant errors; ~100 (out of 50 000) relative errors are over ~2%, which we set as the threshold for grid replacement. The gray-shaded background of Fig. 5 indicates the threshold. We replace some of the initial points with wavelengths at which the relative errors are larger than the threshold; the arrows between Figs. 5b and 5c illustrate this procedure. The replacement is done such that the total number of representative wavelengths does not change between iterations. SDCOMP is then invoked again to perform calculations based on the updated grid, and the new errors are shown in Fig. 5d. It is evident that the average errors become much smaller, overall, although there are still some regions with large errors (yellow-shaded region in Fig. 5d). The procedure is repeated with a new but much smaller error criterion (~0.4%, i.e., gray-shaded region in Fig. 5d); again, most wavelengths that give large errors can be included in the grid for the next iteration. It should be noted that once a wavelength is picked to replace an initial one, it will not be replaced in the later iterations. The errors become stable after three to four iterations, and little improvement obtained with further modification.

Fig. 5.
Fig. 5.

Example to illustrate the iterative method for choosing the representative wavelengths. (a) Average relative errors at 50 000 wavelengths by using 400 equally spaced monochromatic representative wavelengths. (b) As in (a), but for a much narrower spectral range [blue region in (a)]; corresponding representative wavelength grid points are indicated by blue circles. (c) Modified grid points for next iteration (green circles) by replacing some of the prior ones with those that produce large (as defined by values in the gray-shaded area) relative errors. (d) Average relative errors after adjustment.

Citation: Journal of the Atmospheric Sciences 77, 6; 10.1175/JAS-D-19-0238.1

For the solar spectral region in Fig. 5, a grid with 400 representative wavelengths after the iteration is enough to give very accurate results. However, for infrared wavelengths, significant overlap between absorption by multiple gases, and the relatively dense nature of the absorption lines, make the relationship between radiances at different wavelengths much more complicated; therefore, the simple iterative scheme discussed above is not enough to obtain similar accuracy levels as that at solar wavelengths. To address this issue, wavelengths at which the errors are large are added as additional grid points. Note that additional wavelengths imply more Opt-PCA calculations. However, Opt-PCA computes only approximate (2S and single scatter) RT solutions at these points; these are computationally efficient and take negligible time compared to exact RT calculations. Once the representative grid is obtained (offline), it is not changed for any other arbitrary user-generated scenarios. Using 5000 (instead of 1000) training atmospheres did not result in any significant changes, either in the chosen reference wavelengths or in the radiances.

3. SDCOMP performance

Figure 6 compares the accuracy of Opt-PCA, Rad-PCA, and SDCOMP. The exact results illustrated in the top panel are computed using 32-stream LIDORT simulations at a spectral resolution of 0.05 cm−1; relative errors (REs) produced by Opt-PCA, Rad-PCA, and SDCOMP are shown in the bottom panel. RE is defined as
RE=RFREMax[RE]×100%,
where RF and RE are the reflectances obtained from the fast and exact models, respectively, and Max[RE] is the maximum reflectance within the spectral region. This definition is employed to avoid artificial amplification of errors due to division by small numbers (Kopparla et al. 2017). All three models give accurate results, with comparable biases with respect to the exact LIDORT model. The maximum REs are less than 0.2%, and the average absolute REs are lower than 0.05%. The errors show similar structures over the spectral region, other than slightly larger errors for SDCOMP at strongly absorbing wavelengths.
Fig. 6.
Fig. 6.

(top) Example of reflectance at the top of the atmosphere computed using exact radiative transfer simulations. (bottom) Corresponding relative errors produced by Opt-PCA (blue), Rad-PCA (red), and SDCOMP (green).

Citation: Journal of the Atmospheric Sciences 77, 6; 10.1175/JAS-D-19-0238.1

Figure 7 illustrates the probability and cumulative probability distributions of spectral mean REs for 1000 different atmospheric trace gas and aerosol scenarios. The 1000 profiles are different from those used for the Rad-PCA training and those employed to choose the representative wavelengths, thereby providing an unbiased representation of SDCOMP errors. Most mean REs are less than 0.1%. Over 90% of Rad-PCA monochromatic results have REs less than 0.05%. Opt-PCA and SDCOMP provide slightly larger errors, but most of the errors (over 70% of the monochromatic REs) are less than 0.05%. SDCOMP uses Opt-PCA calculations at the reference wavelengths, whereas Rad-PCA employs exact RT calculations at those wavelengths. Therefore, SDCOMP REs are, by design, worse than those from Rad-PCA. The larger errors for Opt-PCA are because PCA is performed on the optical properties (that have a nonlinear relationship with the radiance), whereas Rad-PCA employs PCA directly on the radiance field. Finally, SDCOMP accrues errors due to PCA in both optical property and radiance domains and is therefore the least accurate of the three models. Probability distribution results for the 5000 training atmospheres are almost indistinguishable from those obtained here, demonstrating that the testing profiles span a wide range of atmospheric, surface and viewing geometry conditions, and also corroborating the robustness of the training process.

Fig. 7.
Fig. 7.

(left) Probability distributions and (right) cumulative probability distributions of spectrally averaged Opt-PCA, Rad-PCA, and SDCOMP relative errors for 1000 atmospheric trace gas and aerosol scenarios.

Citation: Journal of the Atmospheric Sciences 77, 6; 10.1175/JAS-D-19-0238.1

Figure 8 is similar to Fig. 6 but shows brightness temperatures (BTs) given by accurate simulations and the corresponding BT differences (BTDs) due to the three models. Since there are many more absorbing gases and more finely spaced spectral lines in the infrared, 800 representative wavelengths and 200 PCs are required for the monochromatic calculations (compared to 400 wavelengths and 100 PCs in the near-infrared). The BTDs are generally under 0.1 K; less than 1% of all results have BTDs larger than 0.1 K. Compared to the solar region, the differences are less systematic in the infrared. The average BTDs for Opt-PCA, Rad-PCA, and SDCOMP are approximately 0.012, 0.012, and 0.017 K, respectively. For the IR simulations, since the computations of the monochromatic gas absorptions themselves are quite time consuming, the efficiency gained by combining the two PCA approaches is even more significant; this will be discussed in greater detail at the end of the section.

Fig. 8.
Fig. 8.

As in Fig. 6, but for results in the infrared expressed in terms of brightness temperature.

Citation: Journal of the Atmospheric Sciences 77, 6; 10.1175/JAS-D-19-0238.1

Figure 9 is similar to Fig. 7 but shows the probability and cumulative probability distributions of the mean BTDs of the three models. The average BTDs for Opt-PCA and Rad-PCA show largest probability less than 0.03 K; the probability also decreases sharply as BTD increases. Over 90% of the results show BTDs less than 0.05 K; 99% show BTDs less than 0.1 K. The BTDs for SDCOMP are slightly larger, but also mostly under 0.1 K (over 95%). It should be noted that all REs and BTDs reported in this study are at monochromatic wavelengths, which are expected to be much larger than those for convolved radiances, representative of typical instrument channel measurements, used for assimilation of hyperspectral sounder data.

Fig. 9.
Fig. 9.

As in Fig. 7, but for results in the infrared expressed in terms of brightness temperature differences.

Citation: Journal of the Atmospheric Sciences 77, 6; 10.1175/JAS-D-19-0238.1

Figures 69 show that the accuracy of SDCOMP is similar to that of Opt-PCA and Rad-PCA. On the other hand, SDCOMP is more computationally efficient than the other two PCA-based models, as demonstrated in Table 1. To avoid the speed up of the fast RT approaches to be dependent on the choice of the accurate RT model, we employ LIDORT with 32 streams as the numerically exact RT model for all four scenarios. An “exact” model requires 50 000 accurate monochromatic RT calculations. In comparison, Rad-PCA needs only 400 accurate RT simulations for the solar spectral region; radiance-based PCA is used to extend these results to the full set of 50 000 wavelengths. The PCA process takes negligible computational time. For Opt-PCA, a combination of 2S and SS calculations is performed at every wavelength (i.e., 50 000 total); these take negligible computational time as well. Following the treatment by Kopparla et al. (2017), the spectral region is split into 11 intervals (bins) based on the total gas absorption optical thickness; a maximum of (2k + 1) accurate RT simulations are needed for each bin, where k is the number of PCs that are retained. We employ four PCs in this study, implying that no more than 99 (2 × 4 + 1 = 9 calculations for each bin) accurate simulations are required. For SDCOMP, the 400 accurate Rad-PCA simulations are replaced by fast 2S/SS calculations. Further, for the bin RT calculations, it should be noted that bins containing less than nine wavelengths are merged into an adjacent bin. Since there are far fewer monochromatic wavelengths for SDCOMP (~400) compared to Opt-PCA (50 000), several bins are either empty or have very few wavelengths. In practice, only about 50 accurate calculations are necessary. Table 1 lists only the main RT components required for the three PCA-based models; computation times that are shared by all three models (e.g., modeling setup, input/output) are not considered here. Considering that the computational times for exact LIDORT simulations differ dramatically for different atmospheric scenarios, we will not provide absolute CPU times here; numbers in Table 1 simply demonstrate the speed up compared to exact calculations.

Table 1.

Comparison of relative computational efficiency of the three models for the spectral range between 0.75 and 0.92 μm.

Table 1.

The situation for the infrared region is slightly different. Table 2 compares the relative efficiencies of the three PCA approaches compared to the exact solution. There are about 800 representative wavelengths (compared to 400 for the solar case) for Rad-PCA; therefore, more accurate RT simulations are needed for all PCA models compared to the solar scenario. Meanwhile, the difference in the number of accurate RT simulations required for Opt-PCA and SDCOMP becomes smaller. However, in this spectral region, computational times for the modeling setup (mainly calculation of gas absorption optical thicknesses) and accurate RT simulations are comparable, with a ratio approximately 3:10. Thus, if we assume that each accurate RT simulation uses a computational unit of 1.0, the corresponding model setup will take an additional computational unit of ~0.3. For the exact RT (50 000 exact RT simulations and 50 000 setups) and SDCOMP (800 exact RT simulations and 800 setups) simulations, over 20% of the total computational time is spent in the modeling setup; the fraction becomes over 99% for Opt-PCA (50 000 setups and ~100 exact RT simulations). The advantage of SDCOMP compared with Opt-PCA is clearly due to the significant decrease in modeling setup time. SDCOMP is three orders of magnitude faster than the exact simulation.

Table 2.

Comparison of relative computational efficiency of the three models for the spectral range between 2000 and 2250 cm−1.

Table 2.

4. Conclusions

This study utilized the complementarity of two different PCA-based approaches to develop a spectral data compression model that optimizes the advantages of the two models. In this SDCOMP approach, PCA is used twice independently: first, to compute radiances at representative wavelengths utilizing correlations in optical properties, and second, to extend radiances at these wavelengths to those at a larger set of monochromatic wavelengths using radiance correlations. Our results indicate that SDCOMP further improves the computational efficiency of the two existing PCA-based approaches while providing similar accuracy as those techniques (Figs. 69). The accuracy and efficiency may be further improved by optimizing the choice of representative wavelengths and/or the number of PC vectors.

Recent work has shown that the accuracy of SDCOMP can be further improved by applying machine learning techniques (Le et al. 2020). Future efforts will investigate the applicability and performance of SDCOMP over wider spectral ranges. Only aerosol scattering is considered here; results for cloudy atmospheres will be analyzed in forthcoming endeavors. Furthermore, SDCOMP may have the potential to be applied to flux and atmospheric heating-/cooling-rate simulations in weather and climate models. In order for SDCOMP to be relevant for atmospheric retrieval systems and numerical weather models, calculation of radiances alone is not enough; the model needs to also have the capability to compute Jacobians. The current SDCOMP model does not include Jacobian calculations. On the other hand, there is no difficulty per se to perform such calculations. This has already been demonstrated for Opt-PCA by Spurr et al. (2013). The same methodology will be employed for SDCOMP in future work.

Acknowledgments

This research was supported in part by the National Natural Science Foundation of China (NSFC) (41975025) and the Young Elite Scientists Sponsorship Program of CAST (2017NRC001). A portion of this research was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration (80NM0018D0004). V. N. acknowledges support from the NASA Earth Science U.S. Participating Investigator program (Solicitation NNH16ZDA001N-ESUSPI).

REFERENCES

  • Ackermann, I. J., H. Hass, M. Memmesheimer, A. Ebel, F. S. Binkowski, and U. Shankar, 1998: Modal Aerosol Dynamics model for Europe: Development and first applications. Atmos. Environ., 32, 29812999, https://doi.org/10.1016/S1352-2310(98)00006-5.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Aumann, H. H., and Coauthors, 2003: AIRS/AMSU/HSB on the Aqua mission: Design, science objectives, data products, and processing system. IEEE Trans. Geosci. Remote Sens., 41, 253264, https://doi.org/10.1109/TGRS.2002.808356.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Aumann, H. H., and Coauthors, 2018: Evaluation of radiative transfer models with clouds. J. Geophys. Res. Atmos., 123, 61426157, https://doi.org/10.1029/2017JD028063.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Chandrasekhar, S., 1960: Radiative Transfer. Dover Publications, 393 pp.

  • Clough, S. A., M. J. Iacono, and J.-L. Moncet, 1992: Line-by-line calculation of atmospheric fluxes and cooling rates: Application to water vapor. J. Geophys. Res., 97, 15 76115 785, https://doi.org/10.1029/92JD01419.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Clough, S. A., M. W. Shephard, E. J. Mlawer, J. S. Delamere, M. J. Iacono, K. E. Cady-Pereira, S.-A. Boukabara, and P. D. Brown, 2005: Atmospheric radiative transfer modeling: A summary of the AER codes. J. Quant. Spectrosc. Radiat. Transfer, 91, 233244, https://doi.org/10.1016/j.jqsrt.2004.05.058.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Crevoisier, C., S. Heilliette, A. Chédin, S. Serrar, R. Armante, and N. A. Scott, 2004: Midtropospheric CO2 concentration retrieval from AIRS observations in the tropics. Geophys. Res. Lett., 31, L17106, https://doi.org/10.1029/2004GL020141.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • d’Almeida, G. A., P. Koepke, and E. P. Shettle, 1991: Atmospheric Aerosol: Global Climatology and Radiative Characteristics. A. Deepak Publishing, 561 pp.

  • del Águila, A., D. S. Efremenko, V. M. Garcia, and J. Xu, 2019: Analysis of two dimensionality reduction technique for fast simulation of the spectral radiances in the Hartley-Huggins band. Atmosphere, 10, 142, https://doi.org/10.3390/atmos10030142.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Duan, M., Q. Min, and J. Li, 2005: A fast radiative transfer model for simulating high-resolution absorption bands. J. Geophys. Res., 110, D15201, https://doi.org/10.1029/2004JD005590.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Efremenko, D. S., D. G. Loyola, R. Spurr, and A. Doicu, 2014a: Acceleration of radiative transfer model calculations for the retrieval of trace gases under cloudy conditions. J. Quant. Spectrosc. Radiat. Transfer, 135, 5865, https://doi.org/10.1016/j.jqsrt.2013.11.014.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Efremenko, D. S., A. Doicu, D. Loyola, and T. Trautmann, 2014b: Optical property dimensionality reduction techniques for accelerated radiative transfer performance: Application to remote sensing total ozone retrievals. J. Quant. Spectrosc. Radiat. Transfer, 133, 128135, https://doi.org/10.1016/j.jqsrt.2013.07.023.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Eldering, A., and Coauthors, 2017: The Orbiting Carbon Observatory-2: First 18 months of science data products. Atmos. Meas. Tech., 10, 549563, https://doi.org/10.5194/amt-10-549-2017.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Eresmaa, R., and A. P. McNally, 2016: Diverse profile datasets based on the CAMS atmospheric composition forecasting system. ECMWF Rep., 12 pp.

  • Fu, D., T. J. Pongetti, J.-F. Blavier, T. J. Crawford, K. S. Manatt, G. G. Toon, K. W. Wong, and S. P. Sander, 2014: Near-infrared remote sensing of Los Angeles trace gas distributions from a mountaintop site. Atmos. Meas. Tech., 7, 713729, https://doi.org/10.5194/amt-7-713-2014.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Goody, R. M., and Y. L. Yung, 1989: Atmospheric Radiation. Oxford University Press, 519 pp.

    • Crossref
    • Export Citation
  • Gordon, I. E., and Coauthors, 2017: The HITRAN2016 molecular spectroscopic database. J. Quant. Spectrosc. Radiat. Transfer, 203, 369, https://doi.org/10.1016/j.jqsrt.2017.06.038.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Han, Y., and Coauthors, 2013: Suomi NPP CrIS measurements, sensor data record algorithm, calibration and validation activities, and record data quality. J. Geophys. Res. Atmos., 118, 12 73412 748, https://doi.org/10.1002/2013JD020344.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Havemann, S., J.-C. Thelen, J. P. Taylor, and R. C. Harlow, 2018: The Havemann-Taylor fast radiative transfer code (HT-FRTC): A multipurpose code based on principal components. J. Quant. Spectrosc. Radiat. Transfer, 220, 180192, https://doi.org/10.1016/j.jqsrt.2018.09.008.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hess, M., P. Koepke, and I. Schult, 1998: Optical properties of aerosols and clouds: The software package OPAC. Bull. Amer. Meteor. Soc., 79, 831844, https://doi.org/10.1175/1520-0477(1998)079<0831:OPOAAC>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hilton, F., and Coauthors, 2012: Hyperspectral Earth observation from IASI: Five years of accomplishments. Bull. Amer. Meteor. Soc., 93, 347370, https://doi.org/10.1175/BAMS-D-11-00027.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kopparla, P., V. Natraj, R. Spurr, R.-L. Shia, D. Crisp, and Y. L. Yung, 2016: A fast and accurate PCA based radiative transfer model: Extension to the broadband shortwave region. J. Quant. Spectrosc. Radiat. Transfer, 173, 6571, https://doi.org/10.1016/j.jqsrt.2016.01.014.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kopparla, P., V. Natraj, D. Limpasuvan, R. Spurr, D. Crisp, R.-L. Shia, P. Somkuti, and Y. L. Yung, 2017: PCA-based radiative transfer: Improvements to aerosol scheme, vertical layering and spectral binning. J. Quant. Spectrosc. Radiat. Transfer, 198, 104111, https://doi.org/10.1016/j.jqsrt.2017.05.005.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Le, T., C. Liu, B. Yao, V. Natraj, and Y. L. Yung, 2020: Application of machine learning to hyperspectral radiative transfer simulations. J. Quant. Spectrosc. Radiat. Transfer, 246, 106928, https://doi.org/10.1016/j.jqsrt.2020.106928.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Liou, K. N., 1973: A numerical experiment on Chandrasekhar’s discrete-ordinate method for radiative transfer: Application to cloudy and hazy atmospheres. J. Atmos. Sci., 30, 13031326, https://doi.org/10.1175/1520-0469(1973)030<1303:ANEOCD>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Liou, K. N., 2002: An Introduction to Atmospheric Radiation. Academic Press, 583 pp.

  • Liu, C., P. Yang, S. L. Nasiri, S. Platnick, K. G. Meyer, C. Wang, and S. Ding, 2015: A fast visible infrared imaging radiometer suite simulator for cloudy atmospheres. J. Geophys. Res. Atmos., 120, 240255, https://doi.org/10.1002/2014JD022443.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Liu, X., W. L. Smith, D. K. Zhou, and A. Larar, 2006: Principal component-based radiative transfer model for hyperspectral sensors: Theoretical concept. Appl. Opt., 45, 201209, https://doi.org/10.1364/AO.45.000201.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Liu, X., D. K. Zhou, A. M. Larar, W. L. Smith, P. Schluessel, S. M. Newman, J. P. Taylor, and W. Wu, 2009: Retrieval of atmospheric profiles and cloud properties from IASI spectra using super-channels. Atmos. Chem. Phys., 9, 91219142, https://doi.org/10.5194/acp-9-9121-2009.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Liu, X., Q. Yang, H. Li, Z. Jin, W. Wu, S. Kizer, D. K. Zhou, and P. Yang, 2016: Development of a fast and accurate PCRTM radiative transfer model in the solar spectral region. Appl. Opt., 55, 82368247, https://doi.org/10.1364/AO.55.008236.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Liu, Y., Z. Cai, D. Yang, M. Duan, and D. , 2013: Optimization of the instrument configuration for TANSAT CO2 spectrometer. Chin. Sci. Bull., 58, 27872789, https://doi.org/10.1360/972013-518.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Matricardi, M., 2010: A principal component based version of the RTTOV fast radiative transfer model. Quart. J. Roy. Meteor. Soc., 136, 18231835, https://doi.org/10.1002/qj.680.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Matricardi, M., and A. McNally, 2014: The direct assimilation of principal components of IASI spectra in the ECMWF 4D-Var. Quart. J. Roy. Meteor. Soc., 140, 573582, https://doi.org/10.1002/qj.2156.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Moncet, J.-L., G. Uymin, A. E. Lipton, and H. E. Snell, 2008: Infrared radiance modeling by optimal spectral sampling. J. Atmos. Sci., 65, 39173934, https://doi.org/10.1175/2008JAS2711.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Moncet, J.-L., G. Uymin, P. Liang, and A. E. Lipton, 2015: Fast and accurate radiative transfer in the thermal regime by simultaneous optimal spectral sampling over all channels. J. Atmos. Sci., 72, 26222641, https://doi.org/10.1175/JAS-D-14-0190.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Natraj, V., X. Jiang, R.-L. Shia, X. Huang, J. S. Margolis, and Y. L. Yung, 2005: Application of principal component analysis to high spectral resolution radiative transfer: A case study of the O2 A band. J. Quant. Spectrosc. Radiat. Transfer, 95, 539556, https://doi.org/10.1016/j.jqsrt.2004.12.024.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Natraj, V., R.-L. Shia, and Y. L. Yung, 2010: On the use of principal component analysis to speed up radiative transfer calculations. J. Quant. Spectrosc. Radiat. Transfer, 111, 810816, https://doi.org/10.1016/j.jqsrt.2009.11.004.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Oreopoulos, L., and Coauthors, 2012: The continual intercomparison of radiation codes: Results from phase I. J. Geophys. Res., 117, D06118, https://doi.org/10.1029/2011JD016821.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • O’Dell, C. W., 2010: Acceleration of multiple-scattering hyperspectral radiative transfer calculations via low-stream interpolation. J. Geophys. Res., 115, D10206, https://doi.org/10.1029/2009JD012803.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • O’Dell, C. W., and Coauthors, 2012: The ACOS CO2 retrieval algorithm—Part I: Description and validation against synthetic observations. Atmos. Meas. Tech., 5, 99121, https://doi.org/10.5194/amt-5-99-2012.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Plass, G. N., and G. W. Kattawar, 1968: Monte Carlo calculations of light scattering from clouds. Appl. Opt., 7, 415419, https://doi.org/10.1364/AO.7.000415.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Randles, C., and Coauthors, 2013: Intercomparison of shortwave radiative transfer schemes in global aerosol modeling: Results from the AeroCom Radiative Transfer Experiment. Atmos. Chem. Phys., 13, 23472379, https://doi.org/10.5194/acp-13-2347-2013.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Saunders, R., and Coauthors, 2018: An update on the RTTOV fast radiative transfer model (currently at version 12). Geosci. Model Dev., 11, 27172737, https://doi.org/10.5194/gmd-11-2717-2018.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Seeman, S. W., E. Borbas, R. O. Knuteson, G. R. Stephenson, and H.-L. Huang, 2008: Development of a global infrared land surface emissivity database for application to clear sky sounding retrievals from multi-spectral satellite radiances measurements. J. Appl. Meteor. Climatol., 47, 108123, https://doi.org/10.1175/2007JAMC1590.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Somkuti, P., H. Boesch, V. Natraj, and P. Kopparla, 2017: Application of a PCA-based fast radiative transfer model to XCO2 retrievals in the shortwave infrared. J. Geophys. Res. Atmos., 122, 10 47710 496, https://doi.org/10.1002/2017JD027013.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Spurr, R. J. D., T. P. Kurosu, and K. V. Chance, 2001: A linearized discrete ordinate radiative transfer model for atmospheric remote sensing retrieval. J. Quant. Spectrosc. Radiat. Transfer, 68, 689735, https://doi.org/10.1016/S0022-4073(00)00055-8.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Spurr, R. J. D., V. Natraj, C. Lerot, M. Van Roozendael, and D. Loyola, 2013: Linearization of the principal component analysis method for radiative transfer acceleration: Application to retrieval algorithms and sensitivity studies. J. Quant. Spectrosc. Radiat. Transfer, 125, 117, https://doi.org/10.1016/j.jqsrt.2013.04.002.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Twomey, S., H. Jacobowitz, and H. B. Howell, 1966: Matrix methods for multiple-scattering problems. J. Atmos. Sci., 23, 289298, https://doi.org/10.1175/1520-0469(1966)023<0289:MMFMSP>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Vidot, J., P. Brunel, M. Dumont, C. Carmagnola, and J. Hocking, 2018: The VIS/NIR land and snow BRDF atlas for RTTOV: Comparison between MODIS MCD43C1 C5 and C6. Remote Sens., 10, 21, https://doi.org/10.3390/rs10010021.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wang, C., P. Yang, S. Platnick, A. K. Heidinger, B. A. Baum, T. Greenwald, Z. Zhang, 2013: Retrieval of ice cloud properties from AIRS and MODIS observations based on a fast high-spectral-resolution radiative transfer model. J. Appl. Meteor. Climatol., 52, 710726, https://doi.org/10.1175/JAMC-D-12-020.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wang, C., P. Yang, and X. Liu, 2015: A high-spectral-resolution radiative transfer model for simulating multilayered clouds and aerosols in the infrared spectral region. J. Atmos. Sci., 72, 926942, https://doi.org/10.1175/JAS-D-14-0046.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wu, W., and Coauthors, 2017: The application of PCRTM physical retrieval methodology for IASI cloudy scene analysis. IEEE Trans. Geosci. Remote Sens., 55, 50425056, https://doi.org/10.1109/TGRS.2017.2702006.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wunch, D., and Coauthors, 2011: The Total Carbon Column Observing Network. Philos. Trans. Roy. Soc., 369A, 20872112, https://doi.org/10.1098/rsta.2010.0240.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Yang, J., Z. Zhang, C. Wei, F. Lu, and Q. Guo, 2017: Introducing the new generation of Chinese geostationary weather satellites, Fengyun-4. Bull. Amer. Meteor. Soc., 98, 16371658, https://doi.org/10.1175/BAMS-D-16-0065.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Yang, Q., X. Liu, W. Wu, S. Kizer, and R. R. Baize, 2016: Fast and accurate hybrid stream PCRTM-SOLAR radiative transfer model for reflected solar spectrum simulation in the cloudy atmosphere. Opt. Express, 24, A1514A1527, https://doi.org/10.1364/OE.24.0A1514.

    • Crossref
    • Search Google Scholar
    • Export Citation
Save
  • Ackermann, I. J., H. Hass, M. Memmesheimer, A. Ebel, F. S. Binkowski, and U. Shankar, 1998: Modal Aerosol Dynamics model for Europe: Development and first applications. Atmos. Environ., 32, 29812999, https://doi.org/10.1016/S1352-2310(98)00006-5.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Aumann, H. H., and Coauthors, 2003: AIRS/AMSU/HSB on the Aqua mission: Design, science objectives, data products, and processing system. IEEE Trans. Geosci. Remote Sens., 41, 253264, https://doi.org/10.1109/TGRS.2002.808356.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Aumann, H. H., and Coauthors, 2018: Evaluation of radiative transfer models with clouds. J. Geophys. Res. Atmos., 123, 61426157, https://doi.org/10.1029/2017JD028063.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Chandrasekhar, S., 1960: Radiative Transfer. Dover Publications, 393 pp.

  • Clough, S. A., M. J. Iacono, and J.-L. Moncet, 1992: Line-by-line calculation of atmospheric fluxes and cooling rates: Application to water vapor. J. Geophys. Res., 97, 15 76115 785, https://doi.org/10.1029/92JD01419.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Clough, S. A., M. W. Shephard, E. J. Mlawer, J. S. Delamere, M. J. Iacono, K. E. Cady-Pereira, S.-A. Boukabara, and P. D. Brown, 2005: Atmospheric radiative transfer modeling: A summary of the AER codes. J. Quant. Spectrosc. Radiat. Transfer, 91, 233244, https://doi.org/10.1016/j.jqsrt.2004.05.058.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Crevoisier, C., S. Heilliette, A. Chédin, S. Serrar, R. Armante, and N. A. Scott, 2004: Midtropospheric CO2 concentration retrieval from AIRS observations in the tropics. Geophys. Res. Lett., 31, L17106, https://doi.org/10.1029/2004GL020141.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • d’Almeida, G. A., P. Koepke, and E. P. Shettle, 1991: Atmospheric Aerosol: Global Climatology and Radiative Characteristics. A. Deepak Publishing, 561 pp.

  • del Águila, A., D. S. Efremenko, V. M. Garcia, and J. Xu, 2019: Analysis of two dimensionality reduction technique for fast simulation of the spectral radiances in the Hartley-Huggins band. Atmosphere, 10, 142, https://doi.org/10.3390/atmos10030142.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Duan, M., Q. Min, and J. Li, 2005: A fast radiative transfer model for simulating high-resolution absorption bands. J. Geophys. Res., 110, D15201, https://doi.org/10.1029/2004JD005590.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Efremenko, D. S., D. G. Loyola, R. Spurr, and A. Doicu, 2014a: Acceleration of radiative transfer model calculations for the retrieval of trace gases under cloudy conditions. J. Quant. Spectrosc. Radiat. Transfer, 135, 5865, https://doi.org/10.1016/j.jqsrt.2013.11.014.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Efremenko, D. S., A. Doicu, D. Loyola, and T. Trautmann, 2014b: Optical property dimensionality reduction techniques for accelerated radiative transfer performance: Application to remote sensing total ozone retrievals. J. Quant. Spectrosc. Radiat. Transfer, 133, 128135, https://doi.org/10.1016/j.jqsrt.2013.07.023.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Eldering, A., and Coauthors, 2017: The Orbiting Carbon Observatory-2: First 18 months of science data products. Atmos. Meas. Tech., 10, 549563, https://doi.org/10.5194/amt-10-549-2017.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Eresmaa, R., and A. P. McNally, 2016: Diverse profile datasets based on the CAMS atmospheric composition forecasting system. ECMWF Rep., 12 pp.

  • Fu, D., T. J. Pongetti, J.-F. Blavier, T. J. Crawford, K. S. Manatt, G. G. Toon, K. W. Wong, and S. P. Sander, 2014: Near-infrared remote sensing of Los Angeles trace gas distributions from a mountaintop site. Atmos. Meas. Tech., 7, 713729, https://doi.org/10.5194/amt-7-713-2014.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Goody, R. M., and Y. L. Yung, 1989: Atmospheric Radiation. Oxford University Press, 519 pp.

    • Crossref
    • Export Citation
  • Gordon, I. E., and Coauthors, 2017: The HITRAN2016 molecular spectroscopic database. J. Quant. Spectrosc. Radiat. Transfer, 203, 369, https://doi.org/10.1016/j.jqsrt.2017.06.038.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Han, Y., and Coauthors, 2013: Suomi NPP CrIS measurements, sensor data record algorithm, calibration and validation activities, and record data quality. J. Geophys. Res. Atmos., 118, 12 73412 748, https://doi.org/10.1002/2013JD020344.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Havemann, S., J.-C. Thelen, J. P. Taylor, and R. C. Harlow, 2018: The Havemann-Taylor fast radiative transfer code (HT-FRTC): A multipurpose code based on principal components. J. Quant. Spectrosc. Radiat. Transfer, 220, 180192, https://doi.org/10.1016/j.jqsrt.2018.09.008.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hess, M., P. Koepke, and I. Schult, 1998: Optical properties of aerosols and clouds: The software package OPAC. Bull. Amer. Meteor. Soc., 79, 831844, https://doi.org/10.1175/1520-0477(1998)079<0831:OPOAAC>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hilton, F., and Coauthors, 2012: Hyperspectral Earth observation from IASI: Five years of accomplishments. Bull. Amer. Meteor. Soc., 93, 347370, https://doi.org/10.1175/BAMS-D-11-00027.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kopparla, P., V. Natraj, R. Spurr, R.-L. Shia, D. Crisp, and Y. L. Yung, 2016: A fast and accurate PCA based radiative transfer model: Extension to the broadband shortwave region. J. Quant. Spectrosc. Radiat. Transfer, 173, 6571, https://doi.org/10.1016/j.jqsrt.2016.01.014.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kopparla, P., V. Natraj, D. Limpasuvan, R. Spurr, D. Crisp, R.-L. Shia, P. Somkuti, and Y. L. Yung, 2017: PCA-based radiative transfer: Improvements to aerosol scheme, vertical layering and spectral binning. J. Quant. Spectrosc. Radiat. Transfer, 198, 104111, https://doi.org/10.1016/j.jqsrt.2017.05.005.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Le, T., C. Liu, B. Yao, V. Natraj, and Y. L. Yung, 2020: Application of machine learning to hyperspectral radiative transfer simulations. J. Quant. Spectrosc. Radiat. Transfer, 246, 106928, https://doi.org/10.1016/j.jqsrt.2020.106928.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Liou, K. N., 1973: A numerical experiment on Chandrasekhar’s discrete-ordinate method for radiative transfer: Application to cloudy and hazy atmospheres. J. Atmos. Sci., 30, 13031326, https://doi.org/10.1175/1520-0469(1973)030<1303:ANEOCD>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Liou, K. N., 2002: An Introduction to Atmospheric Radiation. Academic Press, 583 pp.

  • Liu, C., P. Yang, S. L. Nasiri, S. Platnick, K. G. Meyer, C. Wang, and S. Ding, 2015: A fast visible infrared imaging radiometer suite simulator for cloudy atmospheres. J. Geophys. Res. Atmos., 120, 240255, https://doi.org/10.1002/2014JD022443.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Liu, X., W. L. Smith, D. K. Zhou, and A. Larar, 2006: Principal component-based radiative transfer model for hyperspectral sensors: Theoretical concept. Appl. Opt., 45, 201209, https://doi.org/10.1364/AO.45.000201.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Liu, X., D. K. Zhou, A. M. Larar, W. L. Smith, P. Schluessel, S. M. Newman, J. P. Taylor, and W. Wu, 2009: Retrieval of atmospheric profiles and cloud properties from IASI spectra using super-channels. Atmos. Chem. Phys., 9, 91219142, https://doi.org/10.5194/acp-9-9121-2009.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Liu, X., Q. Yang, H. Li, Z. Jin, W. Wu, S. Kizer, D. K. Zhou, and P. Yang, 2016: Development of a fast and accurate PCRTM radiative transfer model in the solar spectral region. Appl. Opt., 55, 82368247, https://doi.org/10.1364/AO.55.008236.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Liu, Y., Z. Cai, D. Yang, M. Duan, and D. , 2013: Optimization of the instrument configuration for TANSAT CO2 spectrometer. Chin. Sci. Bull., 58, 27872789, https://doi.org/10.1360/972013-518.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Matricardi, M., 2010: A principal component based version of the RTTOV fast radiative transfer model. Quart. J. Roy. Meteor. Soc., 136, 18231835, https://doi.org/10.1002/qj.680.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Matricardi, M., and A. McNally, 2014: The direct assimilation of principal components of IASI spectra in the ECMWF 4D-Var. Quart. J. Roy. Meteor. Soc., 140, 573582, https://doi.org/10.1002/qj.2156.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Moncet, J.-L., G. Uymin, A. E. Lipton, and H. E. Snell, 2008: Infrared radiance modeling by optimal spectral sampling. J. Atmos. Sci., 65, 39173934, https://doi.org/10.1175/2008JAS2711.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Moncet, J.-L., G. Uymin, P. Liang, and A. E. Lipton, 2015: Fast and accurate radiative transfer in the thermal regime by simultaneous optimal spectral sampling over all channels. J. Atmos. Sci., 72, 26222641, https://doi.org/10.1175/JAS-D-14-0190.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Natraj, V., X. Jiang, R.-L. Shia, X. Huang, J. S. Margolis, and Y. L. Yung, 2005: Application of principal component analysis to high spectral resolution radiative transfer: A case study of the O2 A band. J. Quant. Spectrosc. Radiat. Transfer, 95, 539556, https://doi.org/10.1016/j.jqsrt.2004.12.024.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Natraj, V., R.-L. Shia, and Y. L. Yung, 2010: On the use of principal component analysis to speed up radiative transfer calculations. J. Quant. Spectrosc. Radiat. Transfer, 111, 810816, https://doi.org/10.1016/j.jqsrt.2009.11.004.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Oreopoulos, L., and Coauthors, 2012: The continual intercomparison of radiation codes: Results from phase I. J. Geophys. Res., 117, D06118, https://doi.org/10.1029/2011JD016821.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • O’Dell, C. W., 2010: Acceleration of multiple-scattering hyperspectral radiative transfer calculations via low-stream interpolation. J. Geophys. Res., 115, D10206, https://doi.org/10.1029/2009JD012803.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • O’Dell, C. W., and Coauthors, 2012: The ACOS CO2 retrieval algorithm—Part I: Description and validation against synthetic observations. Atmos. Meas. Tech., 5, 99121, https://doi.org/10.5194/amt-5-99-2012.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Plass, G. N., and G. W. Kattawar, 1968: Monte Carlo calculations of light scattering from clouds. Appl. Opt., 7, 415419, https://doi.org/10.1364/AO.7.000415.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Randles, C., and Coauthors, 2013: Intercomparison of shortwave radiative transfer schemes in global aerosol modeling: Results from the AeroCom Radiative Transfer Experiment. Atmos. Chem. Phys., 13, 23472379, https://doi.org/10.5194/acp-13-2347-2013.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Saunders, R., and Coauthors, 2018: An update on the RTTOV fast radiative transfer model (currently at version 12). Geosci. Model Dev., 11, 27172737, https://doi.org/10.5194/gmd-11-2717-2018.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Seeman, S. W., E. Borbas, R. O. Knuteson, G. R. Stephenson, and H.-L. Huang, 2008: Development of a global infrared land surface emissivity database for application to clear sky sounding retrievals from multi-spectral satellite radiances measurements. J. Appl. Meteor. Climatol., 47, 108123, https://doi.org/10.1175/2007JAMC1590.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Somkuti, P., H. Boesch, V. Natraj, and P. Kopparla, 2017: Application of a PCA-based fast radiative transfer model to XCO2 retrievals in the shortwave infrared. J. Geophys. Res. Atmos., 122, 10 47710 496, https://doi.org/10.1002/2017JD027013.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Spurr, R. J. D., T. P. Kurosu, and K. V. Chance, 2001: A linearized discrete ordinate radiative transfer model for atmospheric remote sensing retrieval. J. Quant. Spectrosc. Radiat. Transfer, 68, 689735, https://doi.org/10.1016/S0022-4073(00)00055-8.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Spurr, R. J. D., V. Natraj, C. Lerot, M. Van Roozendael, and D. Loyola, 2013: Linearization of the principal component analysis method for radiative transfer acceleration: Application to retrieval algorithms and sensitivity studies. J. Quant. Spectrosc. Radiat. Transfer, 125, 117, https://doi.org/10.1016/j.jqsrt.2013.04.002.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Twomey, S., H. Jacobowitz, and H. B. Howell, 1966: Matrix methods for multiple-scattering problems. J. Atmos. Sci., 23, 289298, https://doi.org/10.1175/1520-0469(1966)023<0289:MMFMSP>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Vidot, J., P. Brunel, M. Dumont, C. Carmagnola, and J. Hocking, 2018: The VIS/NIR land and snow BRDF atlas for RTTOV: Comparison between MODIS MCD43C1 C5 and C6. Remote Sens., 10, 21, https://doi.org/10.3390/rs10010021.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wang, C., P. Yang, S. Platnick, A. K. Heidinger, B. A. Baum, T. Greenwald, Z. Zhang, 2013: Retrieval of ice cloud properties from AIRS and MODIS observations based on a fast high-spectral-resolution radiative transfer model. J. Appl. Meteor. Climatol., 52, 710726, https://doi.org/10.1175/JAMC-D-12-020.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wang, C., P. Yang, and X. Liu, 2015: A high-spectral-resolution radiative transfer model for simulating multilayered clouds and aerosols in the infrared spectral region. J. Atmos. Sci., 72, 926942, https://doi.org/10.1175/JAS-D-14-0046.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wu, W., and Coauthors, 2017: The application of PCRTM physical retrieval methodology for IASI cloudy scene analysis. IEEE Trans. Geosci. Remote Sens., 55, 50425056, https://doi.org/10.1109/TGRS.2017.2702006.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wunch, D., and Coauthors, 2011: The Total Carbon Column Observing Network. Philos. Trans. Roy. Soc., 369A, 20872112, https://doi.org/10.1098/rsta.2010.0240.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Yang, J., Z. Zhang, C. Wei, F. Lu, and Q. Guo, 2017: Introducing the new generation of Chinese geostationary weather satellites, Fengyun-4. Bull. Amer. Meteor. Soc., 98, 16371658, https://doi.org/10.1175/BAMS-D-16-0065.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Yang, Q., X. Liu, W. Wu, S. Kizer, and R. R. Baize, 2016: Fast and accurate hybrid stream PCRTM-SOLAR radiative transfer model for reflected solar spectrum simulation in the cloudy atmosphere. Opt. Express, 24, A1514A1527, https://doi.org/10.1364/OE.24.0A1514.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fig. 1.

    Flowchart to illustrate the basic concept of the SDCOMP RT model.

  • Fig. 2.

    Examples of atmospheric temperature, water vapor, and aerosol optical depth (AOD) profiles, and surface albedo spectra used in the simulations.

  • Fig. 3.

    (top) Examples of simulated reflectance spectra at the top of the atmosphere used for training Rad-PCA. (bottom) Corresponding reflectance mean and first five principal components (PC-1 to PC-5). To illustrate the curves on the same plot, PC-1 to PC-5 are shifted by adding 0.08, 0.12, 0.16, 0.20, and 0.24, respectively, to their values.

  • Fig. 4.

    Schematic of the iterative method to choose the representative monochromatic wavelengths.

  • Fig. 5.

    Example to illustrate the iterative method for choosing the representative wavelengths. (a) Average relative errors at 50 000 wavelengths by using 400 equally spaced monochromatic representative wavelengths. (b) As in (a), but for a much narrower spectral range [blue region in (a)]; corresponding representative wavelength grid points are indicated by blue circles. (c) Modified grid points for next iteration (green circles) by replacing some of the prior ones with those that produce large (as defined by values in the gray-shaded area) relative errors. (d) Average relative errors after adjustment.

  • Fig. 6.

    (top) Example of reflectance at the top of the atmosphere computed using exact radiative transfer simulations. (bottom) Corresponding relative errors produced by Opt-PCA (blue), Rad-PCA (red), and SDCOMP (green).

  • Fig. 7.

    (left) Probability distributions and (right) cumulative probability distributions of spectrally averaged Opt-PCA, Rad-PCA, and SDCOMP relative errors for 1000 atmospheric trace gas and aerosol scenarios.

  • Fig. 8.

    As in Fig. 6, but for results in the infrared expressed in terms of brightness temperature.

  • Fig. 9.

    As in Fig. 7, but for results in the infrared expressed in terms of brightness temperature differences.