1. Introduction
The Cloud-Aerosol Lidar Infrared Pathfinder Satellite Observations (CALIPSO) mission (Winker et al. 2003) joined the A-Train (Stephens et al. 2002) constellation of satellites in late April 2006 and began acquiring scientific data in mid-June of that year. CALIPSO carries three, coaligned, nadir-viewing instruments: a dual-wavelength, dual-polarization lidar (Winker et al. 2007), an imaging infrared radiometer (Chomette et al. 2003) and a wide-field camera (Pitts et al. 2007). The main aims of the CALIPSO mission are to acquire global profile data on the distribution and properties of clouds and aerosols to, ultimately, improve the performance of weather and climate models. The achievement of these goals is aided by flying in formation with the other satellites, thereby allowing the analysis of clouds and aerosols to be enhanced by the synergistic combination of data from other instruments viewing the same atmospheric targets almost simultaneously.
CALIPSO’s lidar (the Cloud-Aerosol Lidar with Orthogonal Polarization, CALIOP) is a dual-wavelength, dual-polarization, elastic backscatter lidar that transmits linearly polarized pulses of laser light at wavelengths of 1064 and 532 nm. Energy backscattered from the atmosphere is received in a 1-m-diameter telescope and separated into one channel where the 1064-nm signal is detected using an avalanche photodiode, and one channel for each of the orthogonal polarizations at 532 nm where photomultipliers are used. (Hunt et al. 2009) CALIPSO data products are available in various “levels” that, according to National Aeronautics and Space Administration (NASA) Earth Observing System (EOS) standards, reflect the degree of processing involved (King et al. 2004). Level 1 data products include the calibrated, attenuated backscatter profiles at the two wavelengths along with various ancillary atmospheric and navigational data. These level 1 data are used to create the higher-level data products. Primary level 2 data products from the lidar are the locations of atmospheric regions containing particulate matter (clouds and aerosols), the identification of these particles according to type, and profiles and layer integrals of particulate backscatter and extinction in these regions. This paper focuses on the fully automated retrieval of profiles of particulate backscatter and extinction. Note that the level 2 algorithms covered here are applied to measurements made by a single instrument (CALIOP). Multisensor algorithms [e.g., CALIOP plus Aqua’s Moderate Resolution Imaging Spectroradiometer (MODIS)] are not presently being utilized in the production of the CALIOP level 2 lidar data products and, hence, are not presented in this work.
The analysis of space-based lidar data must contend with numerous complexities not typically encountered in ground-based measurements. Many atmospheric features (clouds and aerosol layers) are tenuous, and the large distance of the satellite from these features combined with the limits placed on the energy of the laser transmitter by the satellite power budget and eye-safety requirements can lead to low signal-to-noise ratios (SNRs) in the recorded data. This problem is exacerbated during daylight operations, when the weakly backscattering features are viewed against brightly illuminated backgrounds like dense water clouds or snow-covered or sandy surfaces. The usual method of increasing the SNR is to average many profiles. However, along-track inhomogeneities in the atmospheric features, combined with the high speed (typically 7.5 km s−1) of the satellite across these features and the relatively low firing rate of the laser (∼20 s−1), lead to situations where it is simply not possible to acquire a sufficient number of profiles before the subsatellite atmosphere changes significantly.
An added complication, not limited to the analysis of space lidar data, is the presence of both strongly scattering and weakly scattering features in the same atmospheric region. As the lidar equation is nonlinear, any attempt to retrieve profiles of particulate extinction from the averages of profiles containing features of grossly different intrinsic or extrinsic optical properties (and hence lidar ratios—ratios of the particulate volume extinction and backscatter coefficients) could produce results that would be unrepresentative of the real atmosphere.
A unique feature of the CALIPSO lidar analysis is that it attempts to average signals in atmospheric regions where the optical properties are uniform and the signal strengths are comparable (Vaughan et al. 2009). The way in which the CALIPSO analysis can identify and process regions of different horizontal and vertical extents in the same atmospheric region is outlined in section 2 below. The CALIPSO lidar analysis is a multistage process that includes the correction of signals for instrumental effects; the detection, boundary location, and classification of atmospheric features; and, finally, the retrieval of the backscatter and extinction of these features. In section 2 we explain the analysis architecture and how, in order to improve SNRs, the analysis is performed simultaneously at multiple horizontal resolutions using processes that work either on individual profiles or on whole “scenes” and the information on the features contained therein. The extinction retrieval process is intimately linked to the preceding stages of feature detection, boundary location, and feature classification. Because a clear understanding of this interaction is required before the extinction retrieval is discussed, a brief summary of the preceding processes and the information provided to the extinction algorithm is also presented.
In section 3 the rationale for the selection of the algorithm used in the CALIPSO analysis for the retrieval of profiles of particulate backscatter and extinction is outlined, along with details of its implementation, including the use of constrained and unconstrained solutions and retrieval stability issues. The methods by which the CALIPSO extinction profile algorithm are used to retrieve the extinction in a scene containing features of differing horizontal and vertical extents and features of different types that are vertically adjacent or are embedded in other, larger features are described in section 4 with the aid of a representative complex atmospheric scene simulated using measurements acquired by LITE (Winker et al. 1996). The performance of the CALIPSO Hybrid Extinction Retrieval Algorithm (HERA) is discussed in section 5 and an example of the analysis of an actual scene measured by CALIPSO is presented.
For reasons of available space, an analysis of the sensitivity of the extinction algorithm to errors and uncertainties in the input data and their propagation as the solution proceeds down through the atmosphere will be published separately. Similarly, a detailed description of the treatment of multiple scattering in the CALIPSO extinction retrieval algorithm, which takes account of the different features types and their locations in an atmospheric scene, will be presented in a future publication.
2. CALIPSO lidar analysis architecture
a. Definition of scene and profile processes in the CALIPSO lidar analysis
In our description of the CALIPSO lidar analysis, we refer frequently to features and scenes. We define a CALIPSO scene as a consecutive set of profiles of attenuated backscatter, extending 80 km along the CALIPSO track and to an altitude of 30 km, and containing a number of atmospheric features. Scenes are sections of what are commonly referred to as CALIPSO curtain files, which are two-dimensional, height versus along-track distance, sections through the atmosphere (e.g., see Fig. 3). For the purposes of our detection scheme, we define a feature as any extended and contiguous region of enhanced backscatter signal that rises significantly above the expected “clear air” value. The CALIPSO extinction analysis uses horizontal resolutions of 5, 20, and 80 km, which correspond, respectively, to averages of 15, 60, and 240 consecutive, single-shot profiles, and are the same spatial scales at which the feature finder detects features. The CALIPSO lidar analysis proceeds one 80-km scene at a time.
In the CALIPSO extinction analysis, features are analyzed or “solved” (their particulate backscatter and extinction profiles retrieved) using lidar ratios appropriate to the type of cloud or aerosol layer of which they are composed. As a scene, or any profile for that matter, can contain features of different types, different lidar ratios are used in different regions of the scene or profile. First, however, features need to be located and classified.
Although it is recognized that the lidar ratio may vary continuously as a function of range within a layer (e.g., Pappalardo et al. 2004), the approximation used here is the standard method used in the analyses of data from elastic backscatter lidars (e.g., Campbell et al. 2003; Comstock and Sassen 2001; Fernald 1984; Hlavka et al. 2005; Mitrescu et al. 2005; Platt 1979; Stephens et al. 2001; Welton et al. 2000). Within HERA, the lidar ratio is considered to be constant over certain intervals within each backscatter profile, as determined by the layer detection and scene classification algorithms. Unlike more sophisticated instruments, such as high spectral resolution lidar (HSRL) and Raman lidar, the measurements made by elastic backscatter lidars like CALIOP simply do not contain the additional information required for deriving estimates of range-resolved lidar ratios. Furthermore, it must also be recognized that while Raman and HSRL can derive extinction and backscatter directly, and thus derive range-varying lidar ratios, accurate retrieval of these data requires profile SNRs much higher than are readily available in the current generation of operational space-based lidars. Given the very high speed at which the CALIPSO satellite traverses its ground track, the extensive spatial averaging that would be required to generate SNR levels for Raman–HSRL retrievals (especially during daylight operations) would likely require averaging over dissimilar layer types, thus rendering the lidar ratio retrievals highly suspect at best. Thus, while CALIPSO cannot produce fine-resolution profiles of the lidar ratio, the analysis scheme employed by HERA improves on standard elastic backscatter lidar analyses by constraining the extinction retrievals with measurements of layer transmittance wherever possible, or by employing different lidar ratios in different regions of a scene according to the classification of the detected features. These lidar ratios have been derived from an extensive analysis of Aerosol Robotic Network (AERONET) sun-photometer data (Omar et al. 2009, hereafter OJTECH).
b. Analysis module dependencies
The CALIOP automated analysis consists of three primary modules: a feature-finding routine that detects layer boundaries, a suite of scene classification algorithms (SCAs) that differentiates between the various types of clouds and aerosols, and HERA, which is responsible for retrieving the optical properties of all layers detected. The operation of all three components relies on a nested, multigrid averaging scheme initially incorporated into the feature-finder module. As the SNRs of all but the strongest features are usually too low to permit their detection in single profiles, varying numbers of consecutive profiles are averaged in order to improve the SNR. Note that we average first and then analyze the averaged profile rather than the converse. The non-Gaussian nature of the noise in the CALIPSO profiles of weak features and the nonlinear nature of the lidar retrieval can lead to biases if single profiles are analyzed first and their retrieved extinction profiles averaged afterward. The process of averaging profiles and locating features is quite complex, and is fully covered in VJTECH; so, only sufficient detail to permit the understanding of the extinction algorithm will be provided here.
Prior to the execution of HERA, a selective, iterated boundary location (SIBYL) algorithm generates a map of all features detected within each 80-km scene. The spatial resolutions of these features vary according to the amount of horizontal averaging required to provide reliable differentiation of clouds and/or aerosol layers from the weaker molecular background signal. More tenuous layers require more averaging; robust features, on the other hand, require little or no averaging. The features in regions of heterogeneous backscatter intensity—for example, fair weather cumulus embedded in boundary layer aerosols, or the fall streaks in cirrus clouds—are frequently detected not at one single resolution, but instead at several different averaging resolutions, according to the magnitude of the localized signal strength. Conceptually, the result of this sort of analysis resembles a coarse contour plot of backscatter intensity within each 80-km scene. As a consequence of this behavior, features found at different spatial resolutions may lie vertically adjacent to one another, and features found at high spatial resolution may be wholly embedded within some other feature detected at a coarser spatial resolution.
After locating the layers within a scene, the CALIPSO feature finder identifies those elevated layers for which a reliable estimate of the 532-nm-layer two-way transmittance can be computed from the clear-air regions immediately above and below the feature. Using these estimates, together with the measured integrated attenuated backscatters of the elevated layers, SIBYL derives an initial estimate of the 532-nm lidar ratio (Young 1995). Where transmittance measurements are not possible, or are of low quality, the initial lidar ratios are assigned according to feature type. For clouds, the lidar ratios at both 532 and 1064 nm are assigned according to ice-water phase (Hu et al. 2009, manuscript submitted to J. Atmos. Oceanic. Technol., hereafter HJTECH). For aerosols, an aerosol subtyping algorithm selects lidar ratios at 532 and 1064 nm for each feature in the scene by using the scattering characteristics provided by SIBYL for each layer, along with the ancillary surface type from the International Geosphere–Biosphere Programme (IGBP) and the National Snow and Ice Data Center (NSIDC) (OJTECH).
Approximations used in the correction of the retrieved extinction profiles for the effects of multiple scattering are also selected at this time. These functions depend on the nature of the particles in the feature and on the proximity of the feature to any overlying features. Although the algorithm has been written to accommodate a range-dependent multiple-scattering function, for the current (version 2.01) data release a constant value of unity is used in all situations with the exception of cirrus layers, where a constant value of 0.6 is used.
Once all the features in a scene have been located and classified, the relevant information on the vertical locations, horizontal extents, and optical properties of the features is passed to HERA along with the 16 profiles of attenuated backscatter that have each been averaged to a horizontal resolution of 5 km. The task of HERA is to retrieve particulate extinction coefficients from the various regions of the CALIPSO scene, which can be considered as an array 16 columns wide, 1 for each of the 5-km resolution profiles, by 550 rows deep, each row corresponding to a range step or altitude. HERA combines an algorithm that selects which regions of the scene to process, and in which order, with an algorithm that retrieves profiles of particulate extinction from profiles of attenuated backscatter (the profile solver). These profiles of attenuated backscatter are created for each feature by averaging those sections of all the 5-km resolution profiles that lie within the boundaries of the features detected by the feature boundary locater (SIBYL).
The extinction retrieval process works from the top of the atmosphere downward to the surface, correcting the signals from lower regions for the attenuation caused by higher features as this is retrieved. However, in complex scenes, where dissimilar features can be vertically adjacent to, or embedded in, larger features, the solution pathway needs significant modification. The process by which regions of a complex scene are selected and analyzed is described in section 4.
Signals from features detected lower in the atmosphere suffer wavelength-dependent attenuation by higher regions. As this attenuation is unknown until the extinction profiles for the upper regions have been retrieved, the initial classification of the lower features, based on attenuated data, may be in error. Use of incorrect lidar ratios based on these incorrect classifications would result in incorrect retrievals of the attenuation of these lower features and subsequent propagation of these errors to still lower features in the scene. Therefore, once the attenuation of a region of the atmosphere above a feature to be solved has been retrieved, the SCA is invoked to use the just-retrieved overlying attenuation to provide an updated classification of the feature and its associated lidar ratios. The way in which the various analysis modules interact is depicted in Fig. 1.
3. Retrieval of profiles of particulate backscatter and extinction
a. The elastic backscatter lidar equation
P(r) is the backscattered signal power detected from range r from the lidar;
ξ is the lidar system parameter such that ξ = GAC, where GA is the amplifier gain and C is the lidar calibration coefficient;
E0 is the (average) laser energy for a single or averaged profile; and
- βM(r) is the molecular volume backscatter coefficient, which is proportional to the molecular number density profile;
σM(r) = SMβM(r) is the molecular volume extinction coefficient, and
SM is the molecular extinction-to-backscatter (or lidar) ratio;
αO3(r′) is the ozone volume absorption coefficient;
- βP(r) is the particulate volume backscatter coefficient;is the particulate two-way transmittance, where
- η(r) is a parameterization describing multiple scattering by particles;(r0 = 0, rn = r) is the particulate optical depth summed over the various layers of the atmosphere between the lidar and range r, and
SP is the particulate extinction-to-backscatter (lidar) ratio, which is assumed to be constant within identified layers (features).
The term P0 here represents the combined effects of signals from background illumination and from electrical or digital offsets applied during the electronic amplification or digitization of the detected signals. For CALIPSO, the background signal is determined from a measurement in the near field over the altitude range from 112 to 97 km where the molecular backscatter signal is effectively zero. The amplifier and digitizer zero offsets are measured and removed from the signal on board the satellite. At 532 nm, the lidar calibration coefficient is determined by comparing the measured signal, averaged over 55-km along-track distances and between altitudes of 30 and 34.2 km, with the modeled molecular signal in the same region (Powell et al. 2009). In the analysis of CALIPSO data, the profiles of the molecular number density and the ozone absorption coefficient are obtained from the NASA Global Modeling and Assimilation Office (GMAO) (Bloom et al. 2005). The initial estimate of the particulate lidar ratio, SP, and the multiple-scattering factor profile, η(r), are provided by the SCA (HJTECH; OJTECH).
b. Choice of the CALIPSO extinction retrieval algorithm
Various algorithms are available for retrieving profiles of particulate backscatter and extinction from signals measured using elastic backscatter lidars like that used by CALIPSO. The algorithms fall broadly into two classes. Closed-form analytical solutions, based on the solution to the Bernoulli equation and developed for the analysis of rainfall radar measurements (e.g., Hitschfeld and Bordan 1954), were adapted for the analysis of lidar data by early lidar researchers (e.g., Barrett and Ben Dov 1967; Viezee et al. 1969; Davis 1969). These solutions considered only one atmospheric scattering component and are not applicable to conditions where molecular and particulate scattering may be comparable in magnitude, as is the case in CALIPSO lidar signals. Two-component analytical solutions were developed by Fernald et al. (1972), Fernald (1984), and Klett (1985), among others, for the analysis of data recorded under these conditions. Iterative, numerical solutions (e.g., Gambling et al. 1971; Gambling and Bartusek 1972 a,b; Platt 1973) were also developed following those used by Elterman (1966) for the analysis of searchlight studies of aerosols.
The solutions described so far, with the exception of those of Davis (1969) and Platt (1973), assume that the signals detected by the lidar result from single interactions of the transmitted photons with the atmosphere. Under many conditions, determined by the lidar geometry and the nature of the scattering medium, this single-scattering approximation is not valid. Solutions using Platt’s (1973, 1979) parameterization of multiple scattering, in which the particulate extinction is modified by an efficiency factor, η(r), have been developed (e.g., Sassen and Cho 1992) for use under these conditions. For analysis of CALIPSO signals, however, η(r) modifies the optical depth as in Eq. (3). Also, unlike the common practice of replacing η(r) with a constant value, range-dependent functions are under development to describe the multiple scattering for each of the cloud and aerosol classes considered by CALIPSO (Winker 2003). For this implementation of the multiple-scattering function, the above algorithms are not suitable, so the CALIPSO analysis uses an iterative algorithm as described below. As the iteration usually converges in two to three steps, the increased processing time is not a significant concern.
c. Constrained and unconstrained retrievals and stability considerations
Solutions to the lidar equation essentially involve the calculation of the two-way particulate transmittance losses in Eq. (6) in order to solve for the particulate extinction or backscatter profile. The transmittance corrections are calculated using the retrieved backscatter and extinction from the preceding steps and, consequently, are sensitive to the boundary conditions set at the first point, and to the assumed values of the lidar ratio and the multiple-scattering function. For solutions that are initialized in the near field (forward solutions), the two-way transmittance decreases as the solution proceeds away from the near point. As both analytical and iterative algorithms working in the forward direction involve the correction of the attenuated backscatter profile by this decreasing transmittance, which appears in the denominator in both analytical and iterative expressions, solutions can diverge rapidly from the correct values, particularly in situations of moderate to high optical depth.
Klett (1981) explained the instability of the forward analytical solutions and showed that stable solutions could be ensured by initializing in the far field, beyond the feature being analyzed, and then working backward toward the lidar (backward solutions). Backward iterative solutions are also stable because the transmittance correction factor approaches unity as the solution approaches the near point, and it appears in the numerator. However, it is important to realize that stability does not imply accuracy. Some early researchers used backward-iterative solutions, primarily because far-point boundary values of backscatter could be assumed to be close to molecular values above 30 km for their aerosol studies in the stratosphere (Elterman 1966) or near the tropopause for their tropospheric (Gambling and Bartusek 1972a) aerosol studies, thereby ensuring that their retrievals were both stable and accurate. For lidar systems operating from space, however, backward solutions are often problematical because it is usually difficult to determine accurate, far-field boundary conditions with any confidence, especially under the conditions of reduced SNR encountered during daytime measurements. Whereas, for zenith-pointing observations from the surface, regions of clear air are more likely to be encountered in the far field, for space-based lidar observations, suitable clear regions are generally less likely to be found the farther the signal penetrates into the atmosphere.
The discussion so far refers to unconstrained retrievals, for which the layer optical depth is not known in advance of the solution. Fernald et al. (1972) constrained their analytical retrievals using an independently measured value of particulate transmittance, thus ensuring stable retrievals while, at the same time, deriving values of the particulate lidar ratio. By using forward solutions and iteratively adjusting lidar ratios so that the retrieved particulate backscatter on the far side of cirrus clouds returned to zero, various researchers (e.g., Davis 1969; Sassen and Cho 1992) have effectively constrained their retrievals while obtaining values of the lidar ratio for cirrus.
d. The HERA profile solver
To permit the tuning of various analysis parameters based on the experience gained from processing actual atmospheric data, and to permit the adjustment of these parameters in order to compensate for the expected reduction in performance (basically in SNR) over time or for significant changes in the atmosphere that would occur following the injection into the stratosphere of material from a major volcanic eruption, many parameters used to control the behavior of the software modules that create the CALIOP data products are read from a configuration file. These parameters include various thresholds and convergence tolerances, limiting values of retrieved optical depth or backscatter, and various default values of the lidar ratio. The parameters are kept constant during any data release version number and the values used are listed in the metadata provided with the various data products.
As explained earlier, the retrieved profile of βP(r) may diverge from the correct values if incorrect estimates of the lidar ratio, multiple scattering function, or correction for the attenuation of overlying features are used. Although any of these quantities could be adjusted to prevent divergence, as the uncertainties in the lidar ratio usually tend to be the largest, and the retrieval usually most sensitive to adjustments in this parameter, we choose to adjust the lidar ratio to prevent divergence in features. The profile solver detects if the retrieval is diverging in the positive direction (growing too large) by testing if successive estimates of βP,k(r) in Eq. (13) are not converging and by testing the values of f[βP,k(r)] at inflection points in its derivative, if any exist. Divergence in the negative direction is indicated by the existence of a specified number of negative values in the profile of βP(r), where the corresponding attenuated backscatter β′N(r) is positive. Upon detecting divergence, the profile solver algorithm is terminated, and then started anew using a modified value of the lidar ratio. For solutions diverging in the positive direction, the lidar ratio is reduced, and for solutions diverging in the negative direction, the lidar ratio is increased. As an adjustment of the lidar ratio in one direction may subsequently cause divergence in the opposite direction, the adjustments are linked. If fewer than five previous adjustments have been made in either direction, then the lidar ratio is adjusted by 1% in the required direction. Further reductions are by 5%, while further increases are determined by the average of the two previous values that were respectively too small and too large.
Whenever possible, extinction solutions are constrained by a determination of the two-way transmittance provided by SIBYL. This is done by adjusting the particulate lidar ratio iteratively using a variable secant algorithm (e.g., Fröberg 1966, section 2.2) until the retrieved particulate two-way transmittance differs from the supplied constraint by less than some predefined tolerance parameter. We choose to solve for SP iteratively rather than using an analytical equation as in Young (1995) because we use a range-dependent multiple-scattering function, which we may implement as an in-line calculation in the future.
For the analysis of aerosol layers in contact with the surface, no such constraint is available from the lidar measurements alone. In these situations, the retrieval of the correct extinction profile obviously depends on the correct identification of features and the selection of lidar ratios by the SCA.
e. Rescaling below solved regions of a CALIPSO scene
After the retrieval of the extinction profile through a feature, the attenuated backscatter profiles in all underlying regions are rescaled by dividing the attenuated backscatter data by the retrieved two-way transmittance ascribed to the feature. As features may not occupy all 16 of the 5-km columns, only the attenuated backscatter signals in those columns directly below the feature just analyzed are rescaled. This is identical to renormalizing the attenuated backscatter profile in each column to the molecular backscatter signal expected at an altitude just below the base of the feature just analyzed. However, the method we use is preferable to renormalization because the SNR in clear regions is often poor, especially below attenuating features in the 5-km resolution profiles, and renormalization in clear regions is not an option for the 1064-nm analysis, as there is little molecular signal at that wavelength. The first step in the analysis of those features detected below the features just analyzed is to create an average profile of attenuated backscatter from those columns and rows occupied by the feature as specified by SIBYL. The process just described ensures that attenuated backscatter profiles in the different columns contributing to the average will be correctly rescaled for the attenuation caused by overlying features before they are combined in the average profile. For features occupying several columns, this attenuation correction may be different in each column. This novel procedure used in the CALIPSO analysis ensures that we can now average “like with like” and thereby avoid nonlinear effects in the processing of the underlying features. (By “nonlinear,” we mean that the feature transmittance in the averaged profile would not be linearly related to the integrated attenuated backscatter in the averaged profile except for very low values of particulate backscatter and optical depth.) By correcting for the varying attenuation of overlying features before creating the averaged attenuated backscatter profile of the lower feature, we hope to retrieve physically meaningful data on the optical and radiative properties of the lower features.
4. The extinction retrieval process in a representative CALIPSO scene
Features to be analyzed by HERA are detected by SIBYL at horizontal resolutions of 5, 20, or 80 km; that is, they are only detectable after averaging over these horizontal resolutions. Therefore, as there is no information on how the material in a feature is distributed horizontally within the limits specified by SIBYL, the CALIPSO analysis algorithms assume that the material is distributed evenly across these horizontal dimensions, that is, that they occupy 1, 4, or 16 columns of the analysis arrays describing a CALIPSO scene.
Features in a scene can be either simple or complex. Simple features are vertically and horizontally contiguous at some horizontal resolution, are assumed to contain one type (or mixture) of particulates, and, consequently, are described by one set of optical parameters [SP and η(r)]. Simple features are not vertically adjacent to other features; that is, the tops and bases of simple features are bounded everywhere above and below by clear air and, thus, do not touch any part of any other feature. Complex features contain a number of otherwise simple features that share an upper or lower boundary with (i.e., are vertically adjacent to) another feature, or are wholly embedded in other features. We define embedded features as those that are totally enclosed within the boundaries of another feature of coarser horizontal resolution and do not share a common top or base (although embedded features may share a common side boundary with the enclosing feature). The main task of HERA is to recognize the different situations and to use the appropriate analysis pathways.
a. A representative complex scene
Figure 2 is a simplified representation of a scene containing both simple and complex features. For clarity, the number of rows (altitudes) in the array has been reduced from 550 to 36, and the vertical resolution set to 0.5 km for all altitude bins. All features are numbered, and any one feature may span several rows and/or several columns. Features F2–F7 and F11 are simple features, where there is only one unknown lidar ratio for each feature. These can all be solved with one call to the profile solver, and constrained retrievals are possible if a measurement of two-way transmittance can be determined from the reduction of the clear-air signal below each feature. In complex features there are more unknowns (lidar ratios) than possible measurements (two-way transmittances), so a unique, constrained solution does not exist. Here, we seek a solution (essentially a set of lidar ratios) that gives a retrieved optical depth, averaged across the whole complex feature, that is consistent (within a specified tolerance) with the value, measured over the same columns, of the reduction in the signal from the clear air below the complex feature. Provided that the optical thicknesses and their uncertainties are satisfactory, the consistency is achieved by adjusting the lidar ratios, which are initially set to model values, one at a time within acceptable limits, starting with the feature with the strongest integrated attenuated backscatter. (Note that the lower SNRs and the consequent, higher uncertainties usually preclude the use of this routine with daytime measurements.) For features in contact with the surface, like the complex surface layer shown here, a forward, unconstrained solution is used. In Fig. 2 there are three complex features. The first contains F1 and F8–F10, the second F12–F20, and the third F21–F38. Note that F11 is not considered to be part of the second complex feature because it is not vertically adjacent to any other feature and it can be solved simply in the same manner as F2–F7. Embedded features (like F20) pose an extra degree of difficulty, because the solution of F16 cannot be achieved without knowing the attenuation of its lower regions by F20, and the embedded feature (F20) cannot be solved until the attenuation caused by the top part of F16 is known.
It can be seen that the backscatter from clear-air regions is used in the calculation of the layer transmittances required for constrained and consistent solutions. Although SIBYL has detected no features in these regions, there may be features present that escape detection because of low SNR or low backscatter contrast against the molecular signal. Weakly backscattering features may still have appreciable particulate optical depth if their lidar ratios are high and if they are distributed diffusely over a wide height range. However, for the reasons just given, the particulate backscatter from the regions will be low, so any errors in measured layer transmittances resulting from the presence of aerosols in the clear regions are expected to be small.
b. Retrieval pathways
While the general philosophy of HERA is to solve the regions of a scene from the top of the atmosphere down to the surface, working across all horizontal resolutions and progressively rescaling profiles in underlying regions for the retrieved attenuation, the retrieval pathways for simple, complex, and embedded features are of increasing complexity. Note that “clear” regions can be solved in a similar manner to that for simple features, but this capability is only available at 532 nm, as there is generally insufficient signal at 1064 nm. If performed, retrievals of extinction profiles in these regions use default lidar ratios for the free troposphere or stratosphere as listed in the data product metadata.
The analysis procedures used to create the version 2.01 data products were similar for both 532 and 1064 nm. In the analysis, it was assumed that clear regions were completely clear of particles and thus, by definition, the particulate transmittance in these regions was unity. Therefore, only regions identified as containing features were processed. Also, the adjustment of lidar ratios in complex features in order to achieve “consistent” solutions as described in the previous section was not implemented in the analysis software in time for the version 2.01 data release. All features were processed using the lidar ratios provided by the SCA and only adjusted to prevent divergence.
We now describe the retrieval pathway for the 532-nm data with reference to Fig. 2. Note that, although in the current CALIPSO data release, the analysis routines are configured not to solve in clear regions, the description that follows is of the complete algorithm in which the whole scene is solved. Here, we assume that the clear regions are also to be solved. The first step is to create an average profile for clear-air region 1 from the 16 attenuated backscatter profiles. The number of columns contributing to the average will vary from 16 in rows 1 to 3 down to 3 in rows 5 and 6. The average profile is then solved and the attenuated backscatter profiles in all columns rescaled by the retrieved attenuation of that part of the clear region that overlies them. (Remember that we are assuming that the material in the features and clear regions is distributed evenly across their horizontal extents.) The attenuation correction will be greatest in columns F–H, where the clear region is deepest, and smallest in columns N–P, where it is shallowest. Because the highest unsolved feature is a member of a complex feature, the average optical depth of the complex feature to which it belongs is determined by measuring the reduction in the clear-air signal in rows 7–21 of columns M–P. An average attenuated backscatter profile is created for F1 from columns M–P and the resulting profile is solved initially using lidar ratios supplied by the SCA. The attenuated backscatter profile for F8 is then rescaled, temporarily, by the attenuation caused by F1 in row 4, column N, and its extinction profile retrieved, again using values of SP from the SCA. F9 and F10 are solved similarly. The total optical depth in each column of the complex feature is then calculated from the retrieved extinction profiles, and the average is compared with the previously “measured” value. If the values differ by more than some threshold value, the lidar ratio of the strongest of the four features is adjusted and the complex feature solved again in an attempt to achieve agreement. If adjustments within acceptable limits do not produce agreement, the lidar ratio of the next strongest feature is adjusted, and so on, until agreement is achieved or the maximum number of iterations is reached and the retrieval quality flag set to indicate this. After the complex feature is solved, the profiles underlying each of the constituent features are rescaled by the corresponding attenuations.
Next, each of the simple features F2–F7 is solved, using a transmittance constraint if one is available, and the underlying regions rescaled. The second clear-air region is analyzed next, using an average of the rescaled profiles in columns A–P. F11, a simple feature, is solved in the same manner as F2–F7. The second complex feature is solved in a similar manner to the first, with the exception of the embedded features F17, F18, and F20. The solution of the combination of F16 and F20 is performed in three steps. In the first step, the creation of the average profile constituting F16 initially assumes the attenuation caused by F20 is zero, and the extinction profile of F16 is then retrieved. Then, F20 can be rescaled correctly for the attenuation caused by that part of F16 that overlies it and is solved. The lower regions of F16 are rescaled for the attenuation caused by F20 and solved correctly. Once a satisfactory solution has been achieved for the second complex feature and the underlying regions rescaled, the third clear region is solved followed by the complex surface layer.
5. The algorithm in action
The correct performance of the extinction retrieval algorithm has been verified using a wide variety of simulated data (Powell 2005). Some tests have been of HERA operating in isolation with “perfect” feature boundaries and optical parameters supplied by SIBYL and the SCA, while others have been of all the algorithms operating together with “realistic” inputs supplied to HERA. The sensitivity of HERA to errors in the input parameters will be reported upon in another publication. Simulations using “noise free” data provide a very stringent test of the accuracy of the algorithm, as the effects of some of the weaker features in a complex scene like that discussed above can be subtle and hidden in any noise. In several tests, noise-free attenuated backscatter profiles were simulated based on variations of the arrangement of the features in Fig. 2. The results of the analysis with HERA showed that, when supplied with the correct input parameters, the retrieved extinction in all cells of the arrays differed from the input, modeled values by values that were comparable with the numerical precision of the computer.
Now we examine in Fig. 3 the performance of the algorithms using actual CALIPSO data recorded along the west coast of southern Africa on 1 January 2007. Variables are plotted as height versus latitude and longitude using color scales, as indicated at the base of each panel. In Fig. 3a, we present the 532-nm attenuated backscatter. The main features are a strong aerosol layer, elevated in places, extending from the left of the figure to 8.5°S. Between about 2° and 6°S this layer is obscured by a cloud layer that attenuates the signal completely. In the middle latitudes a strong signal from a cirriform cloud near 14 km can be seen, with a higher, more tenuous cloud farther to the south. A low, dense water cloud that, in places, attenuates the signal completely can be seen in the southern half of Fig. 3a. Figure 3b shows the horizontal distance over which profiles had to be averaged in order to detect the features shown. The strong water cloud was detected at single-profile resolution, while the very tenuous cloud in the upper right of Fig. 3b required averaging over 80 km and even then some sections of the cloud were below the detection threshold. In Fig. 3c, we plot the cloud–aerosol discrimination feature mask in which orange indicates aerosol and midblue cloud. It can be seen that some sections of the aerosol layer on the left (especially between the approximate latitudes −1.0° to −1.5°) have been misclassified as a result of the SNR or other characteristics of the signals (Liu et al. 2009). This can have significant consequences for the extinction products. In Figs. 3d and 3e, respectively, we show the classification subtypes of the aerosol and cloud features. These subtypes determine the initial lidar ratios supplied to HERA. The aerosol subtyping algorithm has shown the aerosol layer to be predominantly polluted dust overlying dust, with some embedded smoke. As expected, the high clouds are identified as being predominantly ice and the low clouds water, although there are occasional apparent anomalies.
Finally, in Fig. 4, we present the outputs of the extinction retrieval process. Figures 4a and 4b, respectively, show the 532-nm particulate backscatter and extinction. These are the primary retrieval outputs and show the backscatter and extinction of all particulates (cloud and aerosol) at the native 5-, 20-, and 80-km resolutions of HERA. (Note that HERA does not distinguish clouds from aerosols but simply produces a total particulate retrieval.) The remaining panels in Fig. 4 show the results of the postextinction processing (the boxes on the right in Fig. 1): panels c and d, respectively, show the 532-nm cloud and aerosol and extinction, and panel e presents the 1064-nm aerosol extinction retrievals after they have been reduced to the horizontal resolutions of 5 km (clouds) and 40 km (aerosols) and vertical resolutions of 60 m (clouds) and 120 m (aerosols) used in the CALIPSO level 2 extinction data products available from the Langley Atmospheric Sciences Data Center. While the effects of the occasional misclassification of aerosols as water clouds in the layer on the left can be can be seen in Fig. 4b and 4c, the overall performance of the chain of algorithms is very good.
6. Conclusions
A Hybrid Extinction Retrieval Algorithm (HERA) has been developed that successfully retrieves particulate backscatter and extinction profiles from attenuated backscatter profiles measured by the CALIPSO lidar in atmospheric scenes of varying complexity, where features (cloud or aerosol layers) of differing horizontal scales may be vertically adjacent or embedded in larger features. The accuracy of these retrievals depends to a large degree on the accuracy of the calibration of the data and of the lidar ratios supplied to HERA by the preceding algorithms in the processing chain. The sensitivity of the analysis to errors in these inputs will be treated in detail in a later publication. HERA is composed of two main elements: a top-level algorithm that selects the analysis pathway, order of processing, and analysis parameters that depend on the nature and spatial extent of the atmospheric features to be processed; and a profile solver or “extinction engine” that retrieves profiles of particulate extinction and backscatter coefficients from specified sections of an atmospheric scene defined by the top-level algorithm.
Where possible, retrievals of extinction profiles from simple, vertically isolated features are constrained by measurements of two-way transmittance determined by the reduction in the clear-air backscatter through the feature. For complex features, where there are more unknown lidar ratios than possible transmittance measurements, the algorithm attempts to find a solution in which the retrieved optical depth, averaged across the horizontal extent of the feature, is consistent with the average value determined from the clear-air signal reduction across the complex feature. (As noted above, however, this capability has not been implemented in the version 2.01 data release. Also, because of the virtual lack of 1064-nm backscatter from “clear” regions, enabling this capability at that wavelength is contraindicated.)
Acknowledgments
The authors are pleased to acknowledge helpful comments on the manuscript from the authors of the companion algorithms in the extinction retrieval process. The CALIPSO data used in this paper were obtained from the NASA Langley Atmospheric Sciences Data Center.
REFERENCES
Barrett, E. W., and Ben-Dov O. , 1967: Application of the lidar to air pollution measurements. J. Appl. Meteor., 6 , 500–515.
Bloom, S., and Coauthors, 2005: Documentation and validation of the Goddard Earth Observing System (GEOS) data assimilation system—Version 4. Tech. Rep. Series on Global Modeling and Data Assimilation, NASA/TM-2005-104606, Vol. 26, 165 pp. [Available online at http://gmao.gsfc.nasa.gov/pubs/tm/archive/tm_2005.php].
Campbell, J. R., Welton E. J. , Spinhirne J. D. , Ji Q. , Tsay S-C. , Piketh S. J. , Barenbrug M. , and Holben B. N. , 2003: Micropulse lidar observations of tropospheric aerosols over northeastern South Africa during the ARREX and SAFARI 2000 dry season experiments. J. Geophys. Res., 108 , 8497. doi:10.1029/2002JD002563.
Chomette, O., and Coauthors, 2003: Retrieval of cloud emissivity and particle size frame of the CALIPSO mission. Proc. IGARSS 2003: Int. Geoscience and Remote Sensing Symp., Vol. 3, Toulouse, France, Institute of Electrical and Electronics Engineers International, 1520–1522.
Comstock, J. M., and Sassen K. , 2001: Retrieval of cirrus cloud radiative and backscattering properties using combined Lidar and Infrared Radiometer (LIRAD) Measurements. J. Atmos. Oceanic Technol., 18 , 1658–1673.
Davis, P. A., 1969: The analysis of lidar signatures of cirrus clouds. Appl. Opt., 10 , 2099–2102.
Elterman, L., 1966: Aerosol measurements in the troposphere and stratosphere. Appl. Opt., 5 , 1769–1776.
Fernald, F. G., 1984: Analysis of atmospheric lidar observations: Some comments. Appl. Opt., 23 , 652–653.
Fernald, F. G., Herman B. M. , and Reagan J. A. , 1972: Determination of aerosol height distributions with lidar. J. Appl. Meteor., 11 , 482–489.
Fröberg, C-E., 1966: Introduction to Numerical Analysis. Addison-Wesley, 340 pp.
Gambling, D., and Bartusek K. , 1972a: Lidar observations of tropospheric aerosols. Atmos. Environ., 6 , 181–190.
Gambling, D., and Bartusek K. , 1972b: Lidar observations of tropospheric aerosols. Atmos. Environ., 6 , 869–870.
Gambling, D., Bartusek K. , and Elford W. G. , 1971: A 12-month study of aerosols below 60 km. J. Atmos. Terr. Phys., 33 , 1403–1413.
Hitschfeld, W., and Bordan J. , 1954: Errors inherent in the radar measurements of rainfall at attenuating wavelengths. J. Meteor., 11 , 58–67.
Hlavka, D. L., Palm S. P. , Hart W. D. , Spinhirne J. D. , McGill M. J. , and Welton E. J. , 2005: Aerosol and cloud optical depth from GLAS: Results and verification for an October 2003 California fire smoke case. Geophys. Res. Lett., 32 , L22S07. doi:10.1029/2005GL023413.
Hunt, W. H., Winker D. M. , Vaughan M. A. , Powell K. A. , Lucker P. L. , and Weimer C. , 2009: CALIPSO lidar description and performance assessment. J. Atmos. Oceanic Technol., in press.
King, M., Closs J. , Spangler S. , Greenstone R. , Wharton S. , and Myers M. , 2004: EOS Data Products Handbook. Vol. 1. EOS Project Science Office, NASA Goddard Space Flight Center, Greenbelt, MD, 260 pp. [Available from EOS Project Science Office, Code 900, NASA Goddard Space Flight Center, Greenbelt, MD 20771; or online at http://eospso.gsfc.nasa.gov/ftp_docs/data_products_1.pdf].
Klett, J. D., 1981: Stable analytical inversion for processing lidar returns. Appl. Opt., 20 , 211–220.
Klett, J. D., 1985: Lidar inversion with variable backscatter/extinction ratios. Appl. Opt., 24 , 1638–1643.
Liu, Z., and Coauthors, 2009: The CALIPSO lidar cloud and aerosol discrimination: Version 2 algorithm and initial assessment of performance. J. Atmos. Oceanic Technol., in press.
Mitrescu, C., Haynes J. M. , Stephens G. L. , Miller S. D. , Heymsfield G. M. , and McGill M. J. , 2005: Cirrus cloud optical, microphysical, and radiative properties observed during the CRYSTAL-FACE experiment: A lidar–radar retrieval system. J. Geophys. Res., 110 , D09208. doi:10.1029/2004JD005605.
Omar, A. H., and Coauthors, 2009: The CALIPSO automated aerosol classification and lidar ratio selection algorithm. J. Atmos. Oceanic Technol., in press.
Pappalardo, G., and Coauthors, 2004: Aerosol lidar intercomparison in the framework of the EARLINET project. 3. Raman lidar algorithm for aerosol extinction, backscatter, and lidar ratio. Appl. Opt., 43 , 5370–5385.
Pitts, M. C., Thomason L. W. , Hu Y. , and Winker D. M. , 2007: An assessment of the on-orbit performance of the CALIPSO wide field camera. Remote Sensing of Clouds and the Atmosphere XII, A. Comerón et al., Eds., International Society for Optical Engineering (SPIE Proceedings, Vol. 6745), doi:10.1117/12.737377.
Platt, C. M. R., 1973: Lidar and radiometer observations of cirrus clouds. J. Atmos. Sci., 30 , 1191–1204.
Platt, C. M. R., 1979: Remote sounding of high clouds: I. Calculation of visible and infrared optical properties for lidar and radiometer measurements. J. Appl. Meteor., 18 , 1130–1143.
Powell, K. A., 2005: Development of the CALIPSO lidar simulator. M.S. thesis, Dept. of Applied Science, College of William and Mary, 228 pp. [Available online at http://www-calipso.larc.nasa.gov/resources/publications.php].
Powell, K. A., and Coauthors, 2009: CALIPSO lidar calibration algorithms. Part I: Nighttime 532-nm parallel channel and 532-nm perpendicular channel. J. Atmos. Oceanic Technol., in press.
Sassen, K., and Cho B. S. , 1992: Subvisual thin cirrus lidar dataset for satellite verification and climatological research. J. Appl. Meteor., 31 , 1275–1285.
Stephens, G. L., Engelen R. J. , Vaughan M. , and Anderson T. L. , 2001: Toward retrieving properties of the tenuous atmosphere using space-based lidar measurements. J. Geophys. Res., 106 , 28143–28157.
Stephens, G. L., and Coauthors, 2002: The CloudSat mission and the A-Train: A new dimension of space-based observations of clouds and precipitation. Bull. Amer. Meteor. Soc., 83 , 1771–1790.
Vaughan, M., and Coauthors, 2009: Fully automated detection of cloud and aerosol layers in the CALIPSO lidar measurements. J. Atmos. Oceanic Technol., in press.
Viezee, W., Uthe E. E. , and Collis R. T. H. , 1969: Lidar observations of airfield approach conditions: An exploratory study. J. Appl. Meteor., 8 , 274–283.
Welton, E. J., and Coauthors, 2000: Ground-based lidar measurements of aerosols during ACE-2: Instrument description, results, and comparisons with other ground-based and airborne measurements. Tellus, 52B , 636–651.
Winker, D., 2003: Accounting for multiple scattering in retrievals from space lidar. 12th International Workshop on Lidar Multiple Scattering Experiments, C. Werner, U. G. Oppel, and T. Rother, Eds., International Society for Optical Engineering (SPIE Proceedings, Vol. 5059), 128–139.
Winker, D. M., Couch R. H. , and McCormick M. P. , 1996: An overview of LITE: NASA’s Lidar In Space Technology Experiment. Proc. IEEE, 84 , 164–180.
Winker, D. M., Pelon J. , and McCormick M. P. , 2003: The CALIPSO mission: Spaceborne lidar for observations of aerosols and clouds. Lidar Remote Sensing for Industry and Environment Monitoring III, U. N. Singh, T. Itabe, and Z. Liu, Eds., International Society for Optical Engineering (SPIE Proceedings, Vol. 4893), doi:10.1117/12.466539.
Winker, D. M., Hunt W. H. , and McGill M. J. , 2007: Initial performance Assessment of CALIOP. Geophys. Res. Lett., 34 , L19803. doi:10.1029/2007GL030135.
Young, S. A., 1995: Lidar analysis of lidar backscatter profiles in optically thin clouds. Appl. Opt., 34 , 7019–7031.
The relationship of HERA to other level 2 data product algorithms.
Citation: Journal of Atmospheric and Oceanic Technology 26, 6; 10.1175/2008JTECHA1221.1
The relationship of HERA to other level 2 data product algorithms.
Citation: Journal of Atmospheric and Oceanic Technology 26, 6; 10.1175/2008JTECHA1221.1
The relationship of HERA to other level 2 data product algorithms.
Citation: Journal of Atmospheric and Oceanic Technology 26, 6; 10.1175/2008JTECHA1221.1
A simplified version of a typical map of the location and extent atmospheric features as described by the feature finder (SIBYL). The 80-km scene contains 16 columns of data with horizontal resolutions of 5 km. For clarity the number of vertical levels has been reduced to 36.
Citation: Journal of Atmospheric and Oceanic Technology 26, 6; 10.1175/2008JTECHA1221.1
A simplified version of a typical map of the location and extent atmospheric features as described by the feature finder (SIBYL). The 80-km scene contains 16 columns of data with horizontal resolutions of 5 km. For clarity the number of vertical levels has been reduced to 36.
Citation: Journal of Atmospheric and Oceanic Technology 26, 6; 10.1175/2008JTECHA1221.1
A simplified version of a typical map of the location and extent atmospheric features as described by the feature finder (SIBYL). The 80-km scene contains 16 columns of data with horizontal resolutions of 5 km. For clarity the number of vertical levels has been reduced to 36.
Citation: Journal of Atmospheric and Oceanic Technology 26, 6; 10.1175/2008JTECHA1221.1
The creation of input data for the extinction retrieval algorithm (a) 532-nm attenuated backscatter (km sr)−1, (b) horizontal averaging required for the detection of the cloud and aerosol features, (c) cloud-aerosol discrimination, (d) aerosol subtyping algorithm, and (e) ice–water cloud classification.
Citation: Journal of Atmospheric and Oceanic Technology 26, 6; 10.1175/2008JTECHA1221.1
The creation of input data for the extinction retrieval algorithm (a) 532-nm attenuated backscatter (km sr)−1, (b) horizontal averaging required for the detection of the cloud and aerosol features, (c) cloud-aerosol discrimination, (d) aerosol subtyping algorithm, and (e) ice–water cloud classification.
Citation: Journal of Atmospheric and Oceanic Technology 26, 6; 10.1175/2008JTECHA1221.1
The creation of input data for the extinction retrieval algorithm (a) 532-nm attenuated backscatter (km sr)−1, (b) horizontal averaging required for the detection of the cloud and aerosol features, (c) cloud-aerosol discrimination, (d) aerosol subtyping algorithm, and (e) ice–water cloud classification.
Citation: Journal of Atmospheric and Oceanic Technology 26, 6; 10.1175/2008JTECHA1221.1
Outputs from HERA (km−1): (a) 532-nm particulate backscatter [(km sr)−1], (b) 532-nm particulate extinction at native resolution, (c) 532-nm cloud extinction product, (d) 532-nm 40-km aerosol extinction product, and (e) 1064-nm 40-km aerosol extinction product.
Citation: Journal of Atmospheric and Oceanic Technology 26, 6; 10.1175/2008JTECHA1221.1
Outputs from HERA (km−1): (a) 532-nm particulate backscatter [(km sr)−1], (b) 532-nm particulate extinction at native resolution, (c) 532-nm cloud extinction product, (d) 532-nm 40-km aerosol extinction product, and (e) 1064-nm 40-km aerosol extinction product.
Citation: Journal of Atmospheric and Oceanic Technology 26, 6; 10.1175/2008JTECHA1221.1
Outputs from HERA (km−1): (a) 532-nm particulate backscatter [(km sr)−1], (b) 532-nm particulate extinction at native resolution, (c) 532-nm cloud extinction product, (d) 532-nm 40-km aerosol extinction product, and (e) 1064-nm 40-km aerosol extinction product.
Citation: Journal of Atmospheric and Oceanic Technology 26, 6; 10.1175/2008JTECHA1221.1
