1. Introduction
Many different operational and research applications in the atmospheric sciences require water vapor and temperature profiles at high temporal resolution. Recently, the National Research Council (NRC) issued two reports that strongly recommended the development of a network of ground-based thermodynamic profilers (Committee on Developing Mesoscale Meteorological Observational Capabilities to Meet Multiple National Needs 2009; Committee on Progress and Priorities of U.S. Weather Research and Research-to-Operations Activities 2010). There are several different technologies that could be used to address this measurement need (Hoff and Hardesty 2012). Active laser-based profilers such as differential absorption lidars (e.g., Nehrir et al. 2011; Wulfmeyer and Bösenberg 1998) and Raman lidars (e.g., Goldsmith et al. 1998; Ferrare et al. 2006; Whiteman et al. 2006) are able to measure water vapor profiles with high vertical and temporal resolution. Raman lidars are also able to profile temperature in the boundary layer and free troposphere (e.g., DiGirolamo et al. 2004; Newsom et al. 2013). Passive techniques that take advantage of the spectral nature of the microwave (e.g., Hewison 2007; Löhnert et al. 2004; Cimini et al. 2011) and infrared (e.g., Smith et al. 1999; Feltz et al. 2003) emission by the atmosphere are also capable of providing high-temporal-resolution profiles, albeit at lower vertical resolution than is capable with lidar-based techniques.
The Atmospheric Emitted Radiance Interferometer (AERI) is a passive infrared spectrometer that is one of the possible technologies that could be deployed as part of the ground-based networked envisioned by the NRC reports (Hoff and Hardesty 2012). Thermodynamic profiles retrieved from AERI data have been used for many applications, including the investigation of cold fronts and drylines (Turner et al. 2000; Feltz et al. 1998), monitoring the evolution of convective indices during a severe weather event (Feltz and Mecikalski 2002), characterizing the differences between convective indices in tornadic and nontornadic storms (Wagner et al. 2008), and as input into a cumulus entrainment rate retrieval scheme (Wagner et al. 2013).
The AERI studies above used the physical retrieval method, henceforth called AERIprof, to derive thermodynamic profiles from the AERI-observed radiances. The AERIprof algorithm is based upon an onion-peeling technique (Smith et al. 1999). This method is typically faster than other physical retrieval methods such as optimal estimation because the method only requires that the diagonal of the Jacobian matrix be computed (Rodgers 2000). However, a significant drawback of the onion-peeling method, especially for nonlinear applications such as infrared radiative transfer inversions, is that a good first guess is required. The original AERIprof algorithm used a statistical first guess (Feltz et al. 1998; Smith et al. 1999), but often inadequacies of the first-guess profile in the midtroposphere would result in the algorithm either not converging or producing poor quality retrievals. The first guess then evolved to include temperature and humidity profiles retrieved from the Geostationary Operational Environmental Satellite series that were blended with the statistical retrieval (Feltz and Mecikalski 2002). The current version of AERIprof now uses a first guess taken from a numerical model such as the Rapid Update Cycle or the Global Forecast System (Feltz et al. 2003).
Another limitation of the AERIprof algorithm is the inability of the algorithm to retrieve thermodynamic profiles under low- and midlevel clouds. The downwelling infrared radiance is very sensitive to changes in the liquid water path, especially when the liquid water path is less than 100 g m−2 (Turner 2007); this makes the incorporation of the cloud into the first guess of the AERIprof algorithm extremely problematic.
A third limitation of the AERIprof algorithm is that the uncertainty of the retrieved profiles is not produced by the algorithm. Rodgers (2000) points out that uncertainties can be computed for the onion-peeling technique; to do this requires significantly more computation time as the entire Jacobian matrix must be computed. Since the weighting functions become significantly broader with altitude away from the instrument, the uncertainties in the retrieved solution will be correlated but the magnitude of these correlations for an AERI-retrieved profile is unknown.
In this study, a new physical retrieval algorithm is developed that addresses some of the limitations of the AERIprof algorithm. This new algorithm, henceforth called AERIoe, uses a Gauss–Newton iterative scheme in an optimal-estimation approach (Rodgers 2000). A simple modification to the traditional optimal-estimation equations, which was suggested by Masiello et al. (2012), is used that allows the algorithm to be relatively insensitive to the first-guess profile, and even a very poor first-guess profile will almost always lead to a solution. Furthermore, the new AERIoe algorithm is able to retrieve thermodynamic profiles in both clear and cloudy scenes, and liquid water cloud properties are retrieved simultaneously in the latter. We note that this has been already done for airborne- and satellite-based infrared spectrometers (e.g., Zhou et al. 2007), but not for ground-based instruments like the AERI. In this study, only cases that are cloud free or have a single-layer liquid-only cloud will be considered (more complex cloud situations will be handled in a future effort), and the cloud will be assumed to be emissive only (i.e., nonscattering). Last, and most important, a full error characterization of the algorithm is performed, and the information content of the AERI observations is presented.
2. Instrument
The AERI is an operational ground-based infrared spectrometer that measures downwelling radiance at approximately 1 cm−1 resolution from 520 to 3000 cm−1 (i.e., from 19.2 to 3.3 μm) (Knuteson et al. 2004a). The AERI was designed by the University of Wisconsin—Madison for the U.S. Department of Energy Atmospheric Radiation Measurement (ARM) Program (Stokes and Schwartz 1994), and is currently deployed by ARM and other entities around the world. Two well-characterized blackbodies (one at ambient air temperature; the other held at 60°C) and the application of a nonlinearity correction for the detectors result in the radiometric accuracy of the radiance measurements being better than 1% of the ambient radiance (Knuteson et al. 2004b). The initial temporal resolution of the AERI radiance observations was approximately 10 min; this was chosen to achieve good signal-to-noise ratios for thermodynamic profiling and spectroscopic analyses. In the mid-2000s, however, most AERI instruments were modified to acquire data at approximately 20-s temporal resolution to improve cloud property retrievals that utilize AERI data (e.g., Turner 2005, 2007; Mace et al. 1998). However, as the signal-to-noise ratio in these “rapid sample” AERI observations is considerably smaller than that in the “slow sample” AERI observations, these data are processed with a principal component–based noise filter (Antonelli et al. 2005; Turner et al. 2006) to improve the signal-to-noise ratio before they are utilized in the AERIoe algorithm.
The optimal-estimation technique requires that the covariance matrix of the observations be known. The calibration of each of the over 5000 spectral elements in the AERI data is determined independently from each other (Knuteson et al. 2004b), and thus it has often been assumed that the AERI covariance matrix is diagonal with no significant off-diagonal elements. To test this, an AERI system was configured to view a third blackbody, which had a fixed temperature of 40°C, in its zenith port for over 24 h. Since the expected radiance was known, the radiometric error in each spectral channel could be computed. The derived correlation matrix (not shown) demonstrated that the distribution of the channel-to-channel correlation values was Gaussian distributed with a mean of zero and that the absolute value of the channel-to-channel correlation was well below 0.3 for all channel pairs and, thus, supports the use of a diagonal observation covariance matrix 
The AERI’s 1-σ radiance uncertainty spectrum can be derived several ways. Operationally, this is computed from the spectral standard deviation of the calibrated imaginary radiance spectrum (Knuteson et al. 2004b). However, this can also be computed as the square root of the diagonal of the covariance matrix 
3. Algorithm
The optimal-estimation algorithm is an iterative Gauss–Newton retrieval technique that utilizes a forward model F (see the appendix for information on the forward model) to compute an estimate of the observation vector Y from the current state vector X; the algorithm then adjusts the state vector and continues to iterate until the change in the computed radiance F(X) between successive iterations is small. In our application, the observation vector is the AERI-observed radiance, where the radiance observations have first been subjected to the noise filter. The state vector used in AERIoe consists of four components: the temperature profile, the water vapor mixing ratio (WVMR) profile, the cloud liquid water path (LWP), and the cloud droplet effective radius (Reff). The AERIoe retrievals are performed on a fixed vertical grid, which has a resolution of 25 m at the surface that decreases in an exponential-like manner to 800 m at 3 km and 2000 m at 6 km.
Many retrieval problems are ill determined, and thus additional information is needed in order to stabilize the retrieval and yield a solution. In the optimal-estimation approach, this additional information is given in the form of an a priori probability density function (PDF). This PDF is assumed to be Gaussian and thus described by a mean state vector Xa of the atmosphere and the covariance 
The AERI is very sensitive to LWP when the LWP is less than 60 g m−2; this sensitivity to LWP is much higher than the sensitivity of the commonly used microwave radiometers (MWR) that have channels between 20 and 35 GHz (Turner 2007). Thus, it would be difficult to create two disjoint thermodynamic profile datasets from the ARM radiosonde data without using the AERI in the classification process. Hence, we have elected to combine all of the radiosonde profiles into a single prior dataset and to not make any correlations between the thermodynamic profiles and clouds. This assumption provides the least constraint on the retrieval algorithm, allowing it the most flexibility to find a solution that matches the observed radiance spectra.
A lidar was used to determine the altitude of the base of the lowest cloud overhead, and this height was used as a fixed parameter in the retrieval algorithm for that time sample. If no cloud was detected, then lidar data for several hours around the AERI sample were used to determine the likely cloud-base altitude. If still no clouds were found, then a default cloud-base altitude of 2 km was used in the retrieval. The specification of a likely cloud-base height, even if the lidar does not report a cloud, is done to allow the retrieval to overcome any differences in sensitivity or spatial or temporal mismatch between the AERI and the lidar and, thus, allows the algorithm to retrieve LWP in all conditions. After a cloud-base height is determined, the algorithm assumes that the cloud has the geometrical thickness of one of the fixed layers used in the retrieval (i.e., 25 m thick if the cloud base is at the surface, 800 m thick if it is at 3 km, etc.), and the cloud properties (e.g., liquid water content and effective radius) are assumed to be constant over the depth of the cloud. If the sky is truly cloud free, then we expect that the retrieved LWP should be 0 g m−2 within its uncertainties.
An additional source of uncertainty is the accuracy of the forward model F, which includes uncertainties in the spectroscopy, the error due to the application of the radiative transfer model on a discrete height grid, and other factors (D. D. Turner et al. 2013, unpublished manuscript). One way to capture these uncertainties is to inflate the observational covariance matrix (Masiello et al. 2012); however, objectively determining the inflation factor needed to account for forward model uncertainty is difficult. Thus, even though we are applying the retrieval to noise-filtered data, we will use the 1-σ noise estimate derived from the unfiltered data in the retrieval. Our inflation factor (i.e., the difference in the 1-σ noise levels used in the observational covariance matrix) is about a factor of 3, which is within the range used by Masiello et al. (2012). Inflating the observational covariance matrix has a smoothing effect on the retrieval and prevents overfitting to the data. As a result, the inflated observational covariance matrix will result in lower values of DFS, SIC, and vertical resolution of the solution than would result if the original observational covariance matrix was used.
Figure 1 shows several iterations from an example AERIoe retrieval. This example uses a simulated AERI observation (with random error) in a cloudy atmosphere; this allows us to know the true atmospheric state profile. The true profile was chosen to be significantly different than the prior state for this example; the entire temperature profile is about 15°C colder than the prior and the WVMR is drier by nearly a factor of 3. Note that the prior was used as the first guess. The information content in the AERI is highest near the surface (to be demonstrated later), and thus it is the lowest portions of the profile (as well as the LWP, not shown) that are adjusted the most in the early iterations. As the 
An example showing the profiles in the state vector at various iterations during a retrieval. The prior mean profile (green) was used as the first guess, and the gray-shaded area illustrates the 1-σ uncertainties in the prior. The AERI observation used in this example was simulated using the true profile shown with the blue lines. A 200-m-thick cloud with an LWP of 32 g m−2 has its base located at the dashed horizontal line. The black dotted line shows the first iteration that would result if γ = 1; in this case, the retrieval did not converge with additional iterations. However, the retrieval did converge when γ > 1 was used to constrain the solution during the early iterations (solid purple, brown, and red lines), where the γ factors were 1000, 300, and 1 for iterations 1, 2, and 7, respectively.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
An example showing the profiles in the state vector at various iterations during a retrieval. The prior mean profile (green) was used as the first guess, and the gray-shaded area illustrates the 1-σ uncertainties in the prior. The AERI observation used in this example was simulated using the true profile shown with the blue lines. A 200-m-thick cloud with an LWP of 32 g m−2 has its base located at the dashed horizontal line. The black dotted line shows the first iteration that would result if γ = 1; in this case, the retrieval did not converge with additional iterations. However, the retrieval did converge when γ > 1 was used to constrain the solution during the early iterations (solid purple, brown, and red lines), where the γ factors were 1000, 300, and 1 for iterations 1, 2, and 7, respectively.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
An example showing the profiles in the state vector at various iterations during a retrieval. The prior mean profile (green) was used as the first guess, and the gray-shaded area illustrates the 1-σ uncertainties in the prior. The AERI observation used in this example was simulated using the true profile shown with the blue lines. A 200-m-thick cloud with an LWP of 32 g m−2 has its base located at the dashed horizontal line. The black dotted line shows the first iteration that would result if γ = 1; in this case, the retrieval did not converge with additional iterations. However, the retrieval did converge when γ > 1 was used to constrain the solution during the early iterations (solid purple, brown, and red lines), where the γ factors were 1000, 300, and 1 for iterations 1, 2, and 7, respectively.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
The AERIoe algorithm is able to use any spectral region observed by the AERI. Here, we elected to use the same spectral bands used in the AERIprof algorithm (Feltz et al. 1998; Smith et al. 1999) so that the results shown here would provide illumination regarding the information content of the AERIprof-retrieved profiles (since the AERIprof algorithm does not directly supply uncertainty or information content estimates). Because we are simultaneously retrieving cloud properties, we have added five additional spectral regions in the atmospheric window (Table 1) to provide additional sensitivity to the cloud; however, these additional windows are chosen in microwindows between absorption lines and thus have little sensitivity to the vertical profile of temperature or humidity. An optimal selection of spectral elements to be employed in the AERIoe retrievals, following a method such as that of Merrelli and Turner (2012), will be used in the future.
Spectral regions used in the AERIoe retrieval, and the primary sensitivity of each spectral region.
4. Results
Turner et al. (2007a) demonstrated that clouds with an LWP of less than 100 g m−2 are difficult to observe accurately with many ground-based remote sensors. The AERI has very good sensitivity to liquid water clouds when the LWP is less than approximately 60 g m−2 (Turner 2007), however, but the infrared spectrum starts to become opaque for LWP values above this threshold. In this section, we used both simulated and real AERI observations to test the ability to retrieve liquid water cloud properties and thermodynamic profiles simultaneously using the AERIoe algorithm.
a. Simulated observations
One of the strengths of the AERIoe algorithm is that it is able to simultaneously retrieve thermodynamic profiles and liquid water cloud properties for a single-layer cloud where the cloud base is known. To evaluate the ability of the retrieval algorithm, AERI radiance observations were simulated in a variety of clear- and cloudy-sky conditions, covering a wide range of LWP (0–130 g m−2) and Reff (6–12 μm). For these tests, we used a mean of 0 g m−2 with a 1-σ uncertainty of 50 g m−2 for the assumed prior LWP; these values were guided by 40 months of MWR-retrieval (MWRRET) data (Turner et al. 2007b) at the SGP (where the MWR-retrieved LWP had a median of ~3 g m−2 and the 25th and 75th quartiles were −3 and 25 g m−2, respectively). Note that if during one of the iterations of the AERIoe algorithm the LWP was negative it was reset to 0 for that iteration (because the forward model requires nonnegative values of LWP). We assumed the a priori mean and uncertainty for the Reff to be 8 ± 4 μm, which is roughly in the range of values for warm stratiform clouds (Miles et al. 2000).
An example showing several of the AERIoe retrievals is shown in Fig. 2. In this example, two downwelling radiance spectra were simulated from the same thermodynamic profile; one simulation was for clear sky, and the other included a liquid water cloud. The cloud was 200 m thick with a base at 2 km, an LWP of 32.1 g m−2, and a Reff of 8.8 μm. The computed spectra, before random spectral noise was added, are shown in Fig. 2a.
(a) Simulated AERI radiance observations for the mean prior state (green), clear-sky conditions (brown), and cloudy-sky conditions (red). The cloud was 200 m thick with a base at 2 km, an LWP of 32.1 g m−2, and a Reff of 8.8 μm. Retrieval of the (left) temperature and (right) WVMR for three different retrieval configurations: (b),(c) clear-sky retrieval applied to the clear-sky radiance, (d),(e) cloudy-sky retrieval applied to the clear-sky radiance, and (f),(g) cloudy-sky retrieval applied to the cloudy radiance. In (b)–(g), the green line and shaded region are the mean prior and its 1-σ uncertainty, respectively, and the blue and red lines are the truth profile and the retrieved profile, respectively, where the red lines include 1-σ error bars. The retrieved cloud properties, with the uncertainties in parentheses, for the clear-sky case in (d) and (e) and the cloudy-sky case in (f) and (g) are shown in the inset boxes in (e) and (g), respectively. In both cases, the retrieved LWP agreed with the true LWP, within uncertainty. The horizontal dashed lines in (d)–(g) illustrate the assumed cloud-base height.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
(a) Simulated AERI radiance observations for the mean prior state (green), clear-sky conditions (brown), and cloudy-sky conditions (red). The cloud was 200 m thick with a base at 2 km, an LWP of 32.1 g m−2, and a Reff of 8.8 μm. Retrieval of the (left) temperature and (right) WVMR for three different retrieval configurations: (b),(c) clear-sky retrieval applied to the clear-sky radiance, (d),(e) cloudy-sky retrieval applied to the clear-sky radiance, and (f),(g) cloudy-sky retrieval applied to the cloudy radiance. In (b)–(g), the green line and shaded region are the mean prior and its 1-σ uncertainty, respectively, and the blue and red lines are the truth profile and the retrieved profile, respectively, where the red lines include 1-σ error bars. The retrieved cloud properties, with the uncertainties in parentheses, for the clear-sky case in (d) and (e) and the cloudy-sky case in (f) and (g) are shown in the inset boxes in (e) and (g), respectively. In both cases, the retrieved LWP agreed with the true LWP, within uncertainty. The horizontal dashed lines in (d)–(g) illustrate the assumed cloud-base height.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
(a) Simulated AERI radiance observations for the mean prior state (green), clear-sky conditions (brown), and cloudy-sky conditions (red). The cloud was 200 m thick with a base at 2 km, an LWP of 32.1 g m−2, and a Reff of 8.8 μm. Retrieval of the (left) temperature and (right) WVMR for three different retrieval configurations: (b),(c) clear-sky retrieval applied to the clear-sky radiance, (d),(e) cloudy-sky retrieval applied to the clear-sky radiance, and (f),(g) cloudy-sky retrieval applied to the cloudy radiance. In (b)–(g), the green line and shaded region are the mean prior and its 1-σ uncertainty, respectively, and the blue and red lines are the truth profile and the retrieved profile, respectively, where the red lines include 1-σ error bars. The retrieved cloud properties, with the uncertainties in parentheses, for the clear-sky case in (d) and (e) and the cloudy-sky case in (f) and (g) are shown in the inset boxes in (e) and (g), respectively. In both cases, the retrieved LWP agreed with the true LWP, within uncertainty. The horizontal dashed lines in (d)–(g) illustrate the assumed cloud-base height.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
The clear-sky spectra were then used in two separate retrievals: the first retrieval had the LWP and its covariance in the prior set to zero, which effectively put the AERIoe algorithm in clear-sky mode by preventing the retrieval of LWP (Figs. 2b,c), whereas the second retrieval retrieved the profiles and cloud properties simultaneously (Figs. 2d,e). The first guess used in these retrievals was the prior. The quality and accuracy of the retrieved temperature and WVMR profiles are very similar in these two retrieval examples, and in the case in which the algorithm could retrieve the cloud properties the LWP was correctly determined by the algorithm to be zero. Note that if the LWP is zero then a cloud does not exist and thus there is no information in the data on Reff; this was seen in the DFS for Reff, which was zero in this second retrieval.
A third retrieval was performed using the simulated cloudy radiance spectrum as input (Figs. 2f,g). Again, the agreement between the retrieved and true profiles is very good. Above the cloud-base height, however, the accuracy of the retrieved profiles is degraded as the retrieved temperature profile no longer agrees with the truth within the uncertainty of the retrieval (shown as error bars on the retrieval; Fig. 2f). Furthermore, there is an increase in the magnitude of the 1-σ error bars for the altitudes above the cloud; this is most easily observed in the temperature profile by comparing the cloudy case (Fig. 2f) with the case in which there was no cloud (Fig. 2d). The retrieved LWP in the cloudy case is almost exactly the true value of 32.1 g m−2, and the retrieved Reff is within 2 σ of the true value of 8.8 μm.
The retrieval results from all of the simulated AERI cases are shown in Fig. 3, where Figs. 3a and 3b show the bias error and the 1-σ uncertainties in the retrieved LWP and Reff values, respectively, as a function of the true LWP. The bias in the retrieved LWP is better than 2 g m−2 for LWP < 60 g m−2, and the bias slowly increases for LWP > 60 g m−2. For the cases with larger LWP, the retrieval is negatively biased; this bias is due both to the saturation of the AERI radiance signal and to the influence of the prior on the retrieval that pulls the result toward the mean prior when the information content in the observations begins to decrease. This is also observed in the 1-σ error bars in the retrieved LWP, which become much more pronounced in size in Fig. 3a when the LWP > 60 g m−2. In a similar way, the bias in the retrieved Reff is less than 0.5 μm when the true LWP < 50 g m−2 (Fig. 3b), but the bias quickly increases when the LWP is larger. The DFS for the LWP and Reff are also presented in Fig. 3d; like the bias results, the DFS starts decreasing when the LWP becomes greater than 50 g m−2, especially for Reff. The strong sensitivity of the Reff to LWP is due to the lack of spectral information needed to determine Reff as the cloud optical depth increases, resulting in large portions of the infrared spectrum becoming opaque. For similar reasons, the DFS for Reff is zero (but the DFS for LWP is essentially 1) when the LWP is zero.
The bias between the truth minus the retrieved (a) LWP and (b) Reff, the (c) DFS for temperature and water vapor profiles, and the (d) DFS for LWP and Reff, all as a function of the cloud LWP. These retrievals used simulated AERI radiance observations, and the cloud was assumed to be 200 m thick with a base at 2 km.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
The bias between the truth minus the retrieved (a) LWP and (b) Reff, the (c) DFS for temperature and water vapor profiles, and the (d) DFS for LWP and Reff, all as a function of the cloud LWP. These retrievals used simulated AERI radiance observations, and the cloud was assumed to be 200 m thick with a base at 2 km.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
The bias between the truth minus the retrieved (a) LWP and (b) Reff, the (c) DFS for temperature and water vapor profiles, and the (d) DFS for LWP and Reff, all as a function of the cloud LWP. These retrievals used simulated AERI radiance observations, and the cloud was assumed to be 200 m thick with a base at 2 km.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
The results in Fig. 2 suggested that there was an impact on the accuracy of the retrieved profile above clouds. Using this set of simulated radiances, we computed the DFS for both the temperature and WVMR profiles (Fig. 3c). In these simulations, the cloud base was at 2 km. The DFS for the water vapor slowly decreases as the LWP, and hence the optical depth of the cloud, increase. The DFS for temperature is relatively independent of LWP for the cases shown in Fig. 3c, however. This result suggests that there is little information on the atmospheric temperature in the simulated AERI radiance observations above 2 km but that there is some information about water vapor above this altitude that is attenuated by the cloud. The vertical distribution of the information content in the AERI observations for temperature and water vapor profiles, and the differences between the two variables, will be investigated further in the next section with retrievals applied to real observations.
b. Real observations—clear sky
The AERIoe algorithm was applied to AERI observations collected at the SGP site. The algorithm was run in cloudy-sky mode; that is, it was configured to retrieve thermodynamic profiles and cloud properties simultaneously. As indicated earlier, the ARM SGP site launched nominally four radiosondes per day; thus, AERI retrievals that were coincident with radiosonde launches were compared.
Mean bias (blue) and RMS difference (red) profiles between the AERIoe retrievals and smoothed radiosonde data for (left) temperature and (right) WVMR, where the retrievals used all of the spectral bands listed in Table 1. RMS difference profiles that used the unsmoothed radiosonde data [i.e., that did not use Eq. (7)] are shown by the thin green lines. For the red and blue curves, the statistics (bias and RMS difference) profiles for the clear-sky and cloudy-sky cases are denoted with solid and dotted lines, respectively. There are 130 clear-sky comparisons between September and November 2008 included in this analysis. There are 18 cloudy cases in this period, where only cases for which the cloud base height was above 2 km and the cloud temperature was warmer than −15°C were retained. The dashed black profile shows the 1-σ uncertainties in the prior.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
Mean bias (blue) and RMS difference (red) profiles between the AERIoe retrievals and smoothed radiosonde data for (left) temperature and (right) WVMR, where the retrievals used all of the spectral bands listed in Table 1. RMS difference profiles that used the unsmoothed radiosonde data [i.e., that did not use Eq. (7)] are shown by the thin green lines. For the red and blue curves, the statistics (bias and RMS difference) profiles for the clear-sky and cloudy-sky cases are denoted with solid and dotted lines, respectively. There are 130 clear-sky comparisons between September and November 2008 included in this analysis. There are 18 cloudy cases in this period, where only cases for which the cloud base height was above 2 km and the cloud temperature was warmer than −15°C were retained. The dashed black profile shows the 1-σ uncertainties in the prior.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
Mean bias (blue) and RMS difference (red) profiles between the AERIoe retrievals and smoothed radiosonde data for (left) temperature and (right) WVMR, where the retrievals used all of the spectral bands listed in Table 1. RMS difference profiles that used the unsmoothed radiosonde data [i.e., that did not use Eq. (7)] are shown by the thin green lines. For the red and blue curves, the statistics (bias and RMS difference) profiles for the clear-sky and cloudy-sky cases are denoted with solid and dotted lines, respectively. There are 130 clear-sky comparisons between September and November 2008 included in this analysis. There are 18 cloudy cases in this period, where only cases for which the cloud base height was above 2 km and the cloud temperature was warmer than −15°C were retained. The dashed black profile shows the 1-σ uncertainties in the prior.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
Data from September to November 2008 were used for this comparison. During this period, the retrieval converged 90% of the time. Cases in which the AERIoe-retrieved LWP was less than 3 g m−2 were considered to be clear-sky scenes. The bias (RMS) profile between the smoothed radiosonde and retrieved profiles in clear-sky scenes is less than 0.5°C (1.5°C) and 0.2 (0.8) g kg−1 for all heights below 5 km for temperature and water vapor, respectively. These clear-sky RMS results are very similar to the comparison results of AERIprof versus radiosonde that were shown by Feltz et al. (2003). Note that applying the smoothing [Eq. (7)], which reduces the sampling error, only slightly reduces the RMS difference in both the clear-sky temperature and humidity profiles (the bias in both is essentially unchanged and thus is not shown in Fig. 4 for the unsmoothed data). Both the bias and the RMS between the radiosonde and the AERIoe retrievals increase dramatically for the cloudy cases, however. We have chosen to analyze only cloudy cases for which the cloud base was above 2 km (so that there was some reasonable vertical range to evaluate between the radiosonde and the AERIoe retrieval) and the cloud temperature was warmer than −15°C (to eliminate cases in which the cloud is likely not composed primarily of liquid water; Naud et al. 2010). Again, accounting for the vertical sampling [Eq. (7)] in the cloudy cases results in only a small impact on the RMS difference in the temperature profile, but the impact on the water vapor RMS difference is much larger, especially above 2 km.
Modified Taylor diagrams showing the uncertainty in the (a) retrieved temperature and (b) WVMR profiles derived from a Monte Carlo sampling of the posterior covariance for the clear-sky retrievals shown in Fig. 4. The dotted circles were subjectively determined to encompass the central regions of the distributions.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
Modified Taylor diagrams showing the uncertainty in the (a) retrieved temperature and (b) WVMR profiles derived from a Monte Carlo sampling of the posterior covariance for the clear-sky retrievals shown in Fig. 4. The dotted circles were subjectively determined to encompass the central regions of the distributions.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
Modified Taylor diagrams showing the uncertainty in the (a) retrieved temperature and (b) WVMR profiles derived from a Monte Carlo sampling of the posterior covariance for the clear-sky retrievals shown in Fig. 4. The dotted circles were subjectively determined to encompass the central regions of the distributions.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
Modified Taylor diagrams showing the correlation coefficient and standard deviation ratio between the smoothed radiosonde [Eq. (7)] and AERIoe-retrieved profiles, using only data between the surface and 4 km, for the (a) temperature and (b) WVMR profiles. The same data used to generate Fig. 4 were used in this figure. The dotted ovals denote the uncertainty range in the posterior (Fig. 5).
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
Modified Taylor diagrams showing the correlation coefficient and standard deviation ratio between the smoothed radiosonde [Eq. (7)] and AERIoe-retrieved profiles, using only data between the surface and 4 km, for the (a) temperature and (b) WVMR profiles. The same data used to generate Fig. 4 were used in this figure. The dotted ovals denote the uncertainty range in the posterior (Fig. 5).
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
Modified Taylor diagrams showing the correlation coefficient and standard deviation ratio between the smoothed radiosonde [Eq. (7)] and AERIoe-retrieved profiles, using only data between the surface and 4 km, for the (a) temperature and (b) WVMR profiles. The same data used to generate Fig. 4 were used in this figure. The dotted ovals denote the uncertainty range in the posterior (Fig. 5).
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
The modified Taylor diagrams that compare the vertically smoothed radiosonde profiles [using Eq. (7)] and AERIoe retrievals are shown in Fig. 6. The temperature comparison (Fig. 6a) shows an extremely good comparison between the radiosonde and the retrievals in clear sky (gray plus signs); the correlation coefficients are typically larger than 0.98, the ratio of the standard deviations is between 0.9 and 1.2, and the majority of the comparisons are within the black dotted oval that denotes the posterior uncertainty of the retrieval. The clear-sky WVMR comparison (Fig. 6b) is also reasonably good, with most of the profiles having r > 0.9 and 0.7 < SDR < 1.3 and landing in or near the posterior uncertainty oval. The cloudy cases (black square symbols) have poorer agreement with the radiosondes relative to the clear-sky cases in both temperature and water vapor. We will return in section 4c to the discussion about the accuracy in cloudy situations.
The retrieval algorithm is trying to find a solution that lies within the uncertainty specified by the prior yet that also agrees with the observations within their uncertainty. Since the prior provides potentially a very significant constraint to the solution, it is important to understand how much information content in the solution is from the observation (vs the prior) and how this information is distributed vertically. The DFS is one objective way to determine the information content in the observations. The DFS provides an estimate of the number of independent levels of temperature and humidity that can be derived from the observation, given the constraint provided by the prior (Rodgers 2000). The SIC [Eq. (6)] provides another objective way to quantify the information in the AERI observation. The SIC provides a measure for the number of unique profiles that the retrieval can resolve given the prior’s constraints. Both the DFS and SIC were computed for each individual retrieval, as the information content in the observation depends on the relative location of the solution in the prior’s state space.
The DFS and SIC of the AERIoe retrievals are given as a function of precipitable water vapor (PWV) in Figs. 7a and 7b, because as the PWV increases some spectral regions used in the retrieval become opaque, thereby reducing the information content in the observations (Löhnert et al. 2009). While the DFS for temperature shows little dependence on the PWV, the DFS for WVMR decreases with increasing PWV; the total SIC also decreases with increasing PWV, but this result is primarily due to the information content in the WVMR retrievals decreasing. The magnitude of the DFS for temperature and WVMR, and the dependence on PWV, are quantitatively similar to the simulation study by Löhnert et al. (2009). Figures 7c and 7e show the vertical distribution of the fractional DFS for temperature and WVMR, respectively. The fractional DFS for temperature grows very quickly with altitude, with approximately 75% of the temperature-profile signal in the AERI observations occurring in the lowest 1 km of the atmosphere and 90% in the lowest 3 km. The fractional DFS profile for WVMR changes less quickly with altitude, with only 50% of the information in the AERI observations being in the lowest 1 km and 90% of the information not being reached until approximately 4 km.
(a) DFS for temperature and WVMR profiles and (b) total SIC for combined temperature and WVMR retrieval as a function of PWV, using the same clear-sky data as in Fig. 4. The vertical distribution of the cumulative DFS for (c) temperature and (e) WVMR, and the vertical resolution derived from the averaging kernel for (d) temperature and (f) WVMR. The shaded gray regions in (c)–(f) denote the central 80% range of values (i.e., from the 10th to the 90th percentiles), and the solid black line is the median. The dashed line shows the median value that would result if 
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
(a) DFS for temperature and WVMR profiles and (b) total SIC for combined temperature and WVMR retrieval as a function of PWV, using the same clear-sky data as in Fig. 4. The vertical distribution of the cumulative DFS for (c) temperature and (e) WVMR, and the vertical resolution derived from the averaging kernel for (d) temperature and (f) WVMR. The shaded gray regions in (c)–(f) denote the central 80% range of values (i.e., from the 10th to the 90th percentiles), and the solid black line is the median. The dashed line shows the median value that would result if
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
(a) DFS for temperature and WVMR profiles and (b) total SIC for combined temperature and WVMR retrieval as a function of PWV, using the same clear-sky data as in Fig. 4. The vertical distribution of the cumulative DFS for (c) temperature and (e) WVMR, and the vertical resolution derived from the averaging kernel for (d) temperature and (f) WVMR. The shaded gray regions in (c)–(f) denote the central 80% range of values (i.e., from the 10th to the 90th percentiles), and the solid black line is the median. The dashed line shows the median value that would result if
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
The rows of the averaging kernel also provide information on the vertical resolution of the retrievals. The median vertical resolution and spread between the 10th and 90th percentiles are shown in Figs. 7d and 7f. As suggested by the fractional DFS profiles, the vertical resolution of the retrieved temperature profile decreases very quickly above 1 km, achieving a vertical resolution of about 7 km at 3-km altitude. The WVMR vertical resolution of the retrieval is much higher than that for temperature, especially above 1 km. These results are explained by two phenomena. The first is that the infrared-radiance weighting functions become more diffuse (i.e., less peaked) with distance away from the sensor; weighting functions closer to the sensor are much sharper and better defined. The second reason is that the level-to-level correlation for temperature–temperature (Fig. 8a) is larger than the level-to-level correlation in WVMR–WVMR (Fig. 8b) in the prior with altitude. In general, this level-to-level correlation in the prior reduces the ability of the retrieval to adjust the temperature in the mid- and upper troposphere (because of the stronger constraint), which contributes to the poorer vertical resolution in temperature.
The level-to-level correlation of the prior (below the diagonal) and posterior (above the diagonal) for (a) temperature–temperature, (b) WVMR–WVMR, and (c) temperature–WVMR for an AERIoe retrieval at 2350 UTC 1 Oct 2008. Also shown are the 1-σ uncertainties in the prior (blue) and posterior (red) for (d) temperature and (e) WVMR.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
The level-to-level correlation of the prior (below the diagonal) and posterior (above the diagonal) for (a) temperature–temperature, (b) WVMR–WVMR, and (c) temperature–WVMR for an AERIoe retrieval at 2350 UTC 1 Oct 2008. Also shown are the 1-σ uncertainties in the prior (blue) and posterior (red) for (d) temperature and (e) WVMR.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
The level-to-level correlation of the prior (below the diagonal) and posterior (above the diagonal) for (a) temperature–temperature, (b) WVMR–WVMR, and (c) temperature–WVMR for an AERIoe retrieval at 2350 UTC 1 Oct 2008. Also shown are the 1-σ uncertainties in the prior (blue) and posterior (red) for (d) temperature and (e) WVMR.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
One question that could be asked is, What would the impact be on the information content in the solution, quantified through the vertical distribution of DFS and vertical resolution, if there were no correlations between different levels in temperature and WVMR in the prior (i.e., if the prior covariance matrix 
An advantage of the optimal-estimation approach is that the error covariance of the solution [
c. Real observations—cloudy sky
The strength of the algorithm, as evidenced in Fig. 2, is in its ability to retrieve temperature and WVMR profiles when there are clouds overhead. The results in Fig. 2 show an idealized situation, however, in which the radiative transfer model used to simulate the AERI observations is the same as the forward model that is used in the retrieval. The real test of the ability of AERIoe to retrieve profiles under clouds is to use real radiance observations collected during cloudy periods.
The retrieval was applied to the rapid-sample AERI dataset collected on 7 October 2008 at the SGP site when there were cumulus clouds overhead (Fig. 9). The cloud heights needed by the algorithm were specified using the collocated ceilometer, as indicated above. If the algorithm was run in clear-sky mode (i.e., was not allowed to retrieve cloud properties by using an overly constrained prior in the cloud variables), the retrieved profiles exhibited large temporal discontinuities in both temperature and WVMR (Figs. 9a,c). When clouds were overhead, such as during the four periods at the vertical white dashed lines that correspond with collocated Total Sky Imager (TSI) data (Figs. 9e–h), the retrieved temperature profiles are often too cold around the cloud height and much too warm above the cloud (Fig. 9a); similar artifacts are seen in the WVMR (Fig. 9c). When the algorithm is enabled to simultaneously retrieve cloud LWP and Reff, the artifacts in the temperature are markedly reduced (Fig. 9b), but there are still large artifacts in the retrieved WVMR (Fig. 9d). The mean retrieved LWP for the cloudy cases during this period is 8.8 g m−2 with a mean 1-σ uncertainty of 0.2 g m−2. For comparison, the mean retrieved LWP in these cloudy cases from the MWRRET algorithm (Turner et al. 2007b) applied to the collocated 23.8- and 31.4-GHz microwave radiometer data is 11.8 g m−2, but the mean 1-σ uncertainty derived from the optimal estimation–based microwave radiometer retrieval algorithm is 10 g m−2 (i.e., 50 times as large).
AERIoe retrievals of (a),(b) temperature and (c),(d) WVMR during a period with cumulus overhead, as evidenced by TSI images at (e) 1724, (f) 1730, (g) 1748, and (h) 1750 UTC, and white vertical dotted lines in the time–height cross sections). The results for when AERIoe was run in clear-sky mode are shown in (a) and (c), and the results for when the AERIoe algorithm included clouds are shown in (a) and (d). The retrieval used all spectral bands included in Table 1. Cloud locations are marked with black circles with white centers.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
AERIoe retrievals of (a),(b) temperature and (c),(d) WVMR during a period with cumulus overhead, as evidenced by TSI images at (e) 1724, (f) 1730, (g) 1748, and (h) 1750 UTC, and white vertical dotted lines in the time–height cross sections). The results for when AERIoe was run in clear-sky mode are shown in (a) and (c), and the results for when the AERIoe algorithm included clouds are shown in (a) and (d). The retrieval used all spectral bands included in Table 1. Cloud locations are marked with black circles with white centers.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
AERIoe retrievals of (a),(b) temperature and (c),(d) WVMR during a period with cumulus overhead, as evidenced by TSI images at (e) 1724, (f) 1730, (g) 1748, and (h) 1750 UTC, and white vertical dotted lines in the time–height cross sections). The results for when AERIoe was run in clear-sky mode are shown in (a) and (c), and the results for when the AERIoe algorithm included clouds are shown in (a) and (d). The retrieval used all spectral bands included in Table 1. Cloud locations are marked with black circles with white centers.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
Why do the retrievals that include clouds still have significant artifacts in the temperature and humidity profiles? In the thermal infrared, the contribution of scattering is often assumed to be negligible. This is done for two reasons: 1) it is typically considered to be small and makes relatively little contribution to the upwelling or downwelling flux in most situations and 2) including scattering in the radiative transfer results in a significant computational expense. Indeed, it was because of the latter that we originally utilized an absorption-only forward model (see the appendix). The scattered radiation contribution has a strong wavelength dependence (due to the change in the refractive index of the cloud particles) in radiance, however, and is modulated by both the height of the cloud and the gaseous absorption between the cloud and the instrument.
Figure 10 shows the contribution of the scattering, as a fraction of the total downwelling radiance, for different spectral elements as computed by the LBLDIS radiative transfer model (Turner et al. 2003). [The LBLDIS model is a combined model that uses the discrete ordinates radiative transfer model (DISORT) with the high-spectral-resolution gaseous transmission data computed by the line-by-line radiative transfer model (LBLRTM) to treat scattering in a 1D plane-parallel sense while computing radiance and flux.] One spectral element that is sensitive to both scattering and water vapor (i.e., is in one of the spectral regions used by the retrieval for water vapor profiling) is at 1255 cm−1. This wavenumber element shows a scattering contribution up to 6% of the total downwelling radiance signal for the U.S. Standard Atmosphere, 1976 (Fig. 10, top). The scattering contribution in the 1352 cm−1 spectral element, which is also very close to the spectral region used for water vapor profiling (Table 1), increases as the atmosphere becomes drier than the U.S. Standard Atmosphere, 1976 (i.e., as PWV decreases below 1.4 cm), as illustrated by the difference between the U.S. Standard Atmosphere, 1976 and the sub-Arctic winter profile in Fig. 10 (bottom). Note, however, that the spectral elements in the other water vapor–sensitive spectral band from 538 to 588 cm−1 have a smaller amount of scattering signal, relative to the 1250–1350 cm−1 spectral region.
(top) The fraction of the downwelling radiance at the surface that is a result of scattering, computed as 100 × (with scattering − no scattering)/(with scattering), at different wavenumbers for a range of optical depth (LWP) values. The temperature and humidity structure of the atmosphere is the U.S. Standard Atmosphere, 1976, and the liquid water cloud is 100 m thick with its base at 500 m and an effective radius of 8 μm. (bottom) The difference in the scattering contribution relative to the U.S. Standard Atmosphere, 1976 result that results when the background atmosphere is changed to either a warmer and moister profile (midlatitude summer; plus signs) or a colder and drier profile (sub-Arctic winter; circles).
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
(top) The fraction of the downwelling radiance at the surface that is a result of scattering, computed as 100 × (with scattering − no scattering)/(with scattering), at different wavenumbers for a range of optical depth (LWP) values. The temperature and humidity structure of the atmosphere is the U.S. Standard Atmosphere, 1976, and the liquid water cloud is 100 m thick with its base at 500 m and an effective radius of 8 μm. (bottom) The difference in the scattering contribution relative to the U.S. Standard Atmosphere, 1976 result that results when the background atmosphere is changed to either a warmer and moister profile (midlatitude summer; plus signs) or a colder and drier profile (sub-Arctic winter; circles).
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
(top) The fraction of the downwelling radiance at the surface that is a result of scattering, computed as 100 × (with scattering − no scattering)/(with scattering), at different wavenumbers for a range of optical depth (LWP) values. The temperature and humidity structure of the atmosphere is the U.S. Standard Atmosphere, 1976, and the liquid water cloud is 100 m thick with its base at 500 m and an effective radius of 8 μm. (bottom) The difference in the scattering contribution relative to the U.S. Standard Atmosphere, 1976 result that results when the background atmosphere is changed to either a warmer and moister profile (midlatitude summer; plus signs) or a colder and drier profile (sub-Arctic winter; circles).
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
Thus, the retrieval could be modified in two ways: 1) to include scattering by using the LBLDIS as the forward model or 2) to reduce the impact of scattering by not using the 1250–1350 cm−1 spectral band in the retrieval, since this band shows the largest sensitivity to scattering. Since the LBLDIS model is nearly two orders of magnitude more computationally expensive than our absorption-only model, we have elected to evaluate the latter. We will demonstrate that, by using spectral regions that are less sensitive to scattering by cloud particles, accurate thermodynamic profiles can be retrieved in scenes with single-layer liquid water clouds.
Figure 11 shows the results of reapplying the AERIoe retrievals in the cumulus scene discussed above when radiance observations in the 1250–1350 cm−1 band were not included. The artifacts on the retrieved temperature and WVMR profiles when this spectral band was not used in the retrieval are notably different both when clouds are not included in the retrieval (Figs. 11a,c vs Figs. 9a,c) and when clouds are included in the retrieval (Figs. 11b,d vs Figs. 9b,d). The results that do not use the 1250–1350 cm−1 band and include clouds (i.e., Figs. 11b,d) are much more uniform over time, which would be expected in atmospheric conditions on this day, and thus are considered to be a better retrieval.
As in Fig. 9, except that the retrievals did not use the 1250–1350 cm−1 band.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
As in Fig. 9, except that the retrievals did not use the 1250–1350 cm−1 band.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
As in Fig. 9, except that the retrievals did not use the 1250–1350 cm−1 band.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
A natural question is, How much information is lost when the 1250–1350 cm−1 band is not included in the retrieval? Since less spectral information is used, we expect a decrease in the information content, which should result in a decrease in SIC and DFS and a decrease in the vertical resolution. In a mean sense, the DFS for WVMR and temperature decreases by approximately 0.8 and 0.5, respectively, and the SIC decreases by ~3.5. The decrease in the temperature DFS is expected because while the 1250–1350 cm−1 band is primarily sensitive to the water vapor profile it also has sensitivity to changes in the temperature profile. This decrease in information results in a slight degradation in the vertical resolution (Fig. 12), although the change in the vertical resolution is minimal in the lowest 500 m.
The median vertical resolution in clear-sky scenes for retrievals that use all of the spectral bands in Table 1 (solid lines) vs retrievals that do not use the 1250–1350 cm−1 band (dashed lines).
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
The median vertical resolution in clear-sky scenes for retrievals that use all of the spectral bands in Table 1 (solid lines) vs retrievals that do not use the 1250–1350 cm−1 band (dashed lines).
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
The median vertical resolution in clear-sky scenes for retrievals that use all of the spectral bands in Table 1 (solid lines) vs retrievals that do not use the 1250–1350 cm−1 band (dashed lines).
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
The retrieval algorithm, when data from the 1250–1350 cm−1 band were not included, was reapplied to the AERI observations collected at the SGP site from September to November 2008, and the bias and RMS difference profiles between the retrievals and the smoothed radiosondes were computed. The clear-sky bias and RMS for temperature are qualitatively unchanged when this spectral band is (Fig. 4) or is not (Fig. 13) used, but there is a slight increase in the absolute value of the clear-sky WVMR bias profile in the lowest 3 km (although the WVMR RMS remains qualitatively similar between Figs. 13 and 4). The bias and RMS difference profiles for the temperature are considerably improved when the 1250–1350 cm−1 band is not used in cloudy situations (Fig. 13) relative to the baseline (Fig. 4); the WVMR statistics in cloudy cases are qualitatively unchanged below 2 km but the RMS is reduced above this level when the 1250–1350 cm−1 band is not included.
As in Fig. 4, except that the retrievals did not use the 1250–1350 cm−1 band.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
As in Fig. 4, except that the retrievals did not use the 1250–1350 cm−1 band.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
As in Fig. 4, except that the retrievals did not use the 1250–1350 cm−1 band.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
Comparing the modified Taylor plots for when this spectral region is (Fig. 6) and is not (Fig. 14) used in the retrievals demonstrates better agreement with the radiosondes for the latter. The temperature intercomparisons are more tightly clustered around r = 1 and SDR = 1 in Fig. 14a than in Fig. 6a, for both the clear-sky (gray plus signs) and cloudy cases (black squares). The humidity intercomparisons are also improved, with an improved clustering of the results at r > 0.95 and SDR values between 0.7 and 1.3; this is especially true for the cloudy cases. Thus, by not using the 1250–1350 cm−1 spectral band, which is more sensitive to scattered radiation than are other spectral bands, in the retrieval there is an improvement in the accuracy of the retrieved profiles in both clear and, especially, cloudy situations, with a slight decrease in the information content in the observations used resulting in a decrease in the DFS, SIC, and vertical resolution.
As in Fig. 6, except that the retrievals did not use the 1250–1350 cm−1 band.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
As in Fig. 6, except that the retrievals did not use the 1250–1350 cm−1 band.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
As in Fig. 6, except that the retrievals did not use the 1250–1350 cm−1 band.
Citation: Journal of Applied Meteorology and Climatology 53, 3; 10.1175/JAMC-D-13-0126.1
5. Conclusions
High-temporal-resolution profiles of temperature and water vapor in the boundary layer are needed by many operational and research applications because of the potential rapid evolution of the boundary layer with time. The spectral downwelling infrared radiance is sensitive to the vertical thermodynamic structure of the atmosphere, and thus observations of downwelling radiance by the AERI can be used to provide quantitative estimates of the temperature and humidity profiles overhead. However, since the AERI is a passive radiometer, a priori information is needed by the retrieval algorithm to stabilize the inversion and provide a solution.
We have developed, for the first time, an optimal estimation–based physical retrieval algorithm to derive thermodynamic profiles and liquid water cloud properties from the AERI radiance observations in clear and cloudy scenes. The AERIoe algorithm has the advantage that the uncertainty of the AERI observation and the sensitivity of the forward model are propagated to provide a full characterization of the uncertainty of the solution. A unique aspect of this algorithm is the incorporation of a parameter that allows the relative weight between the prior and the observational covariance matrices to be systematically adjusted on each iteration. The use of this γ parameter was required to overcome the limitation of a poor first guess for the iterative retrieval algorithm and is absolutely essential to perform retrievals in cloudy situations because the downwelling infrared radiance is very sensitive to small changes in cloud properties if there is a cloud overhead. Using a large γ parameter for early iterations places more weight on the prior relative to the observations, and thus the updated profiles retrieved by the algorithm remain stable because of the constraints imposed by the prior. The γ factor is reduced to 1 as the algorithm iterates, allowing the retrieval to use more information from the observations with each step.
Simulated AERI observations were used to evaluate the algorithm, demonstrating that the AERIoe algorithm is able to retrieve thermodynamic profiles and cloud properties simultaneously. When the algorithm was applied to real AERI observations, the retrieved temperature and humidity profiles showed good agreement with radiosondes in cloud-free conditions, but in cloudy cases there was a considerable bias in the retrieved temperature profiles above 1 km. This was demonstrated to be due, at least in part, to inadequacy of the forward radiative transfer model used in the retrieval, which did not include any scattering contribution to the downwelling radiance. Scattering in cloudy situations contributes fractionally the most radiation in the 1250–1350 cm−1 spectral band of all of the spectral regions used in the AERIoe retrieval (Table 1). When the retrievals were performed without this spectral band, the accuracy of the temperature and water vapor profiles in cloudy scenes was improved, and for the water vapor profiles the improvement was substantial (cf. Figs. 9d and 11d), with a slight loss of information content (measured by DFS, SIC, and the vertical resolution) in both the temperature and humidity profiles.
An important conclusion of this work is that the DFS from the AERIoe algorithm applied to real AERI data are quantitatively the same as those shown by Löhnert et al. (2009), who used simulated AERI radiance data. These results also provide insight into the information content and vertical resolution of the retrievals that were derived by using the onion-peeling AERIprof method used by Feltz et al. (1998, 2003), Feltz and Mecikalski (2002), and others.
These results (as well as those by Löhnert et al.) used the same spectral bands that were initially selected by Smith et al. (1999) and Feltz et al.; future work will use an optimized spectral-element selection method outlined by Rodgers (2000) and utilized by others (e.g., Merrelli and Turner 2012), which may provide additional information content, especially aloft, by incorporating other spectral elements observed by the AERI into the retrieval. High-spectral-resolution radiance observations, such as from the AERI, also contain information on cloud height, which can be determined using methods such as CO2 slicing (Holz et al. 2006), and cloud phase (Turner 2005). Thus, there is promise that the AERIoe algorithm could be extended to retrieve cloud height, cloud phase, and the properties of ice clouds in addition to the properties that it is currently retrieving. In addition, future work will focus on an optimized integration of AERI with MWR measurements, which should prove beneficial when retrieving thermodynamic profiles as well as cloud properties in cases with LWP > 50 g m−2.
Acknowledgments
NOAA’s National Severe Storms Laboratory and the U.S. Department of Energy (DOE) Atmospheric System Research (ASR) program supported this work. We thank the AERI team at the Space Science and Engineering Center at the University of Wisconsin—Madison for helpful discussions about the calibration and noise characterization of the AERI as well as for serving as an initial sounding board for some of the retrieval ideas presented here. We also thank Drs. Qin Xu, Jonathan Gero, and Aronne Merrelli for providing very constructive comments on an earlier version of this manuscript and the two anonymous reviewers for their comments that also improved the final version of this paper. The German Research Foundation (DFG) funded the TransRegio32 project, which provided travel support for D. Turner’s stay at the University of Cologne. The data used in this paper were collected by the DOE Atmospheric Radiation Measurement (ARM) Program and are available from its data archive at http://www.archive.arm.gov.
APPENDIX
Description of the Forward Model
The forward radiative transfer model F used in this work is an absorption-only model. The gaseous monochromatic optical depths for each atmospheric layer are computed with version 12.1 of the LBLRTM (Clough and Iacono 1995). This model includes the gaseous absorption of water vapor, carbon dioxide (CO2), ozone, methane (CH4), nitrous oxide (N2O), chlorofluorocarbons, and other gases. We have used data from the Total Carbon Column Observing Network (TCCON) (Wunch et al. 2011), which makes observations at the SGP site, to provide mean values for CO2, CH4, and N2O for our retrievals since there is overlap between these three gases and the spectral bands used in Table 1. This work uses height-independent values of CO2, CH4, and N2O set to 384, 1.793, and 0.310 ppm, respectively. The uncertainties in these trace gases, in both their amounts and their vertical distributions, are not explicitly accounted for in the retrieval, although the inflation of the observation covariance matrix (section 3) does account for some of these forward-model uncertainties in a general sense.
The clouds used in this scheme are assumed to be purely emissive; scattering is not treated. Only liquid-water clouds are treated in this paper, where the optical properties of the clouds are computed from Mie theory using an assumed modified gamma size distribution following Turner (2007). The spectral absorption optical depth, which is output directly from the Mie calculation, is used instead of the extinction optical depth in the radiative transfer calculation. The radiative transfer is performed monochromatically, and then convolved with the AERI’s instrument function to reduce the resolution of the calculation to match that of AERI. Finite differences are used to compute the Jacobian.
REFERENCES
Antonelli, P., and Coauthors, 2005: A principal component noise filter for high spectral resolution infrared measurements. J. Geophys. Res., 109, D23102, doi:10.1029/2004JD004862.
Carissimo, A., I. De Feis, and C. Serio, 2005: The physical retrieval methodology for IASI: The δ-IASI code. Environ. Model. Software, 20, 1111–1126.
Cimini, D., and Coauthors, 2011: Thermodynamic atmospheric profiling during the 2010 Winter Olympics using ground-based microwave radiometry. IEEE Trans. Geosci. Remote Sens., 49, 4959–4969.
Clough, S. A., and M. J. Iacono, 1995: Line-by-line calculations of atmospheric fluxes and cooling rates. 2: Application to carbon dioxide, ozone, methane, nitrous oxide, and halocarbons. J. Geophys. Res., 100, 16 519–16 535.
Committee on Developing Mesoscale Meteorological Observational Capabilities to Meet Multiple National Needs, 2009: Observing Weather and Climate from the Ground Up: A Nationwide Network of Networks. National Academies Press, 234 pp.
Committee on Progress and Priorities of U.S. Weather Research and Research-to-Operations Activities, 2010: When Weather Matters: Science and Service to Meet Critical Societal Needs, National Academies Press, 182 pp.
Di Girolamo, P., R. Marchese, D. N. Whiteman, and B. Demoz, 2004: Rotational Raman lidar measurements of atmospheric temperature in the UV. Geophys. Res. Lett., 31, L01106, doi:10.1029/2003GL018342.
Feltz, W. F., and J. R. Mecikalski, 2002: Monitoring high-temporal-resolution convective stability indices using the ground-based Atmospheric Emitted Radiance Interferometer (AERI) during the 3 May 1999 Oklahoma–Kansas tornado outbreak. Wea. Forecasting, 17, 445–455.
Feltz, W. F., W. L. Smith, R. O. Knuteson, H. E. Revercomb, H. M. Woolf, and H. B. Howell, 1998: Meteorological applications of temperature and water vapor retrievals from the ground-based Atmospheric Emitted Radiance Interferometer (AERI). J. Appl. Meteor., 37, 857–875.
Feltz, W. F., W. L. Smith, H. B. Howell, R. O. Knuteson, H. Woolf, and H. E. Revercomb, 2003: Near-continuous profiling of temperature, moisture, and atmospheric stability using the Atmospheric Emitted Radiance Interferometer (AERI). J. Appl. Meteor., 42, 584–597.
Ferrare, R. A., and Coauthors, 2006: Evaluation of daytime measurements of aerosols and water vapor made by an operational Raman lidar over the southern Great Plains. J. Geophys. Res., 111, D05S08, doi:10.1029/2005JD005836.
Goldsmith, J. E. M., F. H. Blair, S. E. Bisson, and D. D. Turner, 1998: Turn-key Raman lidar for profiling atmospheric water vapor, clouds, and aerosols. Appl. Opt., 37, 4979–4990.
Hansen, C., 1992: Analysis of discrete ill-posed problems by means of the L-curve. SIAM Rev., 34, 561–580.
Hewison, T. J., 2007: 1D-VAR retrieval of temperature and humidity profiles from a ground-based microwave radiometer. IEEE Trans. Geosci. Remote Sens., 45, 2163–2168.
Hoff, R. M., and R. M. Hardesty, Eds., 2012: Thermodynamic Profiling Technologies Workshop Report to the National Science Foundation and the National Weather Service. NCAR Tech. Note NCAR/TN-488+STR, 80 pp.
Holz, R. E., S. A. Ackerman, P. Antonelli, F. Nagle, R. O. Knuteson, M. McGill, D. L. Hlavka, and W. D. Hart, 2006: An improvement to the high-spectral-resolution CO2-slicing cloud-top altitude retrieval. J. Atmos. Oceanic Technol., 23, 653–670.
Knuteson, R. O., and Coauthors, 2004a: The Atmospheric Emitted Radiance Interferometer. Part I: Instrument design. J. Atmos. Oceanic Technol., 21, 1763–1776.
Knuteson, R. O., and Coauthors, 2004b: The Atmospheric Emitted Radiance Interferometer. Part II: Instrument performance. J. Atmos. Oceanic Technol., 21, 1777–1789.
Löhnert, U., S. Crewell, and C. Simmer, 2004: An integrated approach toward retrieving physically consistent profiles of temperature, humidity, and cloud liquid water. J. Appl. Meteor., 43, 1295–1307.
Löhnert, U., D. D. Turner, and S. Crewell, 2009: Ground-based temperature and humidity profiling using spectral infrared and microwave observations. Part I: Simulated retrieval performance in clear sky conditions. J. Appl. Meteor. Climatol., 48, 1017–1032.
Mace, G. G., T. P. Ackerman, P. Minnis, and D. F. Young, 1998: Cirrus layer microphysical properties derived from surface-based millimeter radar and infrared interferometer data. J. Geophys. Res., 103, 23 207–23 216.
Masiello, G., C. Serio, and P. Antonelli, 2012: Inversion for atmospheric thermodynamical parameters of IASI data in the principal components space. Quart. J. Roy. Meteor. Soc., 138, 103–117.
Merrelli, A. M., and D. D. Turner, 2012: Comparing information content of upwelling far infrared and midinfrared radiance spectra for clear atmosphere profiling. J. Atmos. Oceanic Technol., 29, 510–526.
Miles, N. L., J. Verlinde, and E. E. Clothiaux, 2000: Cloud droplet distributions in low-level stratiform clouds. J. Atmos. Sci., 57, 295–311.
Naud, C. M., and Coauthors, 2010: Thermodynamic phase profiles of optically thin midlatitude clouds and their relation to temperature. J. Geophys. Res., 115, D11202, doi:10.1029/2009JD012889.
Nehrir, A. R., K. S. Repasky, and J. L. Carlsten, 2011: Eye-safe diode-laser-based micropulse differential absorption lidar (DIAL) for water vapor profiling in the lower troposphere. J. Atmos. Oceanic Technol., 28, 131–147.
Newsom, R. K., D. D. Turner, and J. E. M. Goldsmith, 2013: Long-term evaluation of temperature profiles measured by an operational Raman lidar. J. Atmos. Oceanic Technol., 30, 1616–1634.
Rodgers, C. D., 2000: Inverse Methods for Atmospheric Sounding: Theory and Practice. Series on Atmospheric, Oceanic and Planetary Physics, Vol. 2, World Scientific, 238 pp.
Smith, W. L., W. F. Feltz, R. O. Knuteson, H. E. Revercomb, H. M. Woolf, and H. B. Howell, 1999: The retrieval of planetary boundary layer structure using ground-based infrared spectral radiance measurements. J. Atmos. Oceanic Technol., 16, 323–333.
Stokes, G. M., and S. E. Schwartz, 1994: The Atmospheric Radiation Measurement (ARM) Program: Programmatic background and design of the Cloud and Radiation Testbed. Bull. Amer. Meteor. Soc., 75, 1201–1221.
Taylor, K. E., 2001: Summarizing multiple aspects of model performance in a single diagram. J. Geophys. Res., 106, 7183–7192.
Turner, D. D., 2005: Arctic mixed-phase cloud properties from AERI lidar observations: Algorithm and results from SHEBA. J. Appl. Meteor., 44, 427–444.
Turner, D. D., 2007: Improved ground-based liquid water path retrievals using a combined infrared and microwave approach. J. Geophys. Res., 112, D15204, doi:10.1029/2007JD008530.
Turner, D. D., W. F. Feltz, and R. A. Ferrare, 2000: Continuous water profiles from operational ground-based active and passive remote sensors. Bull. Amer. Meteor. Soc., 81, 1301–1317.
Turner, D. D., S. A. Ackerman, B. A. Baum, H. E. Revercomb, and P. Yang, 2003: Cloud phase determination using ground-based AERI observations at SHEBA. J. Appl. Meteor., 42, 701–715.
Turner, D. D., R. O. Knuteson, H. E. Revercomb, C. Lo, and R. G. Dedecker, 2006: Noise reduction of Atmospheric Emitted Radiance Interferometer (AERI) observations using principal component analysis. J. Atmos. Oceanic Technol., 23, 1223–1238.
Turner, D. D., and Coauthors, 2007a: Thin liquid water clouds: Their importance and our challenge. Bull. Amer. Meteor. Soc., 88, 177–190.
Turner, D. D., S. A. Clough, J. C. Liljegren, E. E. Clothiaux, K. Cady-Pereira, and K. L. Gaustad, 2007b: Retrieving liquid water path and precipitable water vapor from Atmospheric Radiation Measurement (ARM) microwave radiometers. IEEE Trans. Geosci. Remote Sens., 45, 3680–3690.
Wagner, T. J., W. F. Feltz, and S. A. Ackerman, 2008: The temporal evolution of convective indices in storm-producing environments. Wea. Forecasting, 23, 786–794.
Wagner, T. J., D. D. Turner, L. K. Berg, and S. K. Kruger, 2013: Ground-based remote retrievals of cumulus entrainment rates. J. Atmos. Oceanic Technol., 30, 1460–1471.
Whiteman, D. N., and Coauthors, 2006: Raman lidar measurements during the International H2O Project. Part I: Instrumentation and analysis techniques. J. Atmos. Oceanic Technol., 23, 157–169.
Wulfmeyer, V., and J. Bösenberg, 1998: Ground-based differential absorption lidar for water-vapor profiling: Assessment of accuracy, resolution, and meteorological applications. Appl. Opt., 37, 3825–3844.
Wunch, D., and Coauthors, 2011: The Total Carbon Column Observing Network. Philos. Trans. Roy. Soc., 369A, 2087–2112.
Zhou, D. K., W. L. Smith Sr., X. Liu, A. M. Larar, S. A. Mango, and H.-L. Huang, 2007: Physically retrieving cloud and thermodynamic parameters from ultraspectral IR measurements. J. Atmos. Sci., 64, 969–982.
