Testing IWC Retrieval Methods Using Radar and Ancillary Measurements with In Situ Data

1. Introduction

Clouds cover approximately 60% of the earth’s surface, strongly influencing its energy budget by controlling the amount of solar radiation reaching the earth’s surface and by controlling the loss of thermal energy to space. Because of their height in the atmosphere, ice clouds have a dominant effect on longwave forcing and on the earth’s net radiation budget (Hartmann et al. 1992). Details of the ice microphysics—including the ice water content (IWC), ice water path (IWP), extinction coefficient in visible wavelengths (σ), and ice particle shape—significantly affect ice cloud radiative properties.

Cloud satellite radar (CloudSat), with an onboard millimeter-wavelength (94.05 GHz) radar, and the Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observation (CALIPSO) satellite, with a dual wavelength (0.532 and 1.064 μm) and dual polarization lidar system, present new opportunities to characterize the microphysical properties of ice clouds on a global scale. CloudSat provides data enabling investigators to quantitatively evaluate the relationship between vertical profiles of cloud ice water content and cloud radiative properties and to utilize these results to improve the representation of ice clouds in climate models (Stephens et al. 2002). Because radar provides a measurement of the equivalent radar reflectivity Z_e, it is necessary to develop methods to convert Z_e to IWC. Early methods for this conversion used relationships between IWC and Z_e from ice particle size spectra measurements collected in situ (e.g., Heymsfield 1977) and at the surface (Sassen 1987). As pointed out by Atlas et al. (1995), there is no universal IWC–Z_e relationship because of large scatter and systematic shifts in particle size from day to day and cloud to cloud. For that reason, recently developed techniques have retrieved the IWC using more than radar reflectivity from single radar alone. These techniques include the use of radar combined with collocated lidar data (Intrieri et al. 1993; Wang and Sassen 2002a), Z_e and cloud optical depth derived from an IR radiometer (Matrosov et al. 1998), Z_e and cloud visible optical depth (Benedetti et al. 2003), Z_e measured at two frequencies (Hogan and Illingworth 1999), Z_e and cloud radiance derived from atmospheric emitted radiance measurements to derive layer-average IWC for thin cirrus (Mace et al. 1998), Z_e and Doppler fall speed (Matrosov et al. 2002; Mace et al. 2002; Delanoë et al. 2007; Sato and Okamoto 2006), and Z_e and temperature (Liu and Illingworth 2000; Hogan et al. 2006a; Protat et al. 2007).

If the goal of spaceborne radar is to provide vertical profiles of IWC and IWP for use in evaluating and improving the representation of clouds in climate models, it is necessary to assess the accuracy and limitations of the retrievals. Ground-based remote sensing measurements have been used in conjunction with in situ observations to evaluate retrieved IWC (Matrosov et al. 1995; Wang and Sassen 2002a). The evaluations relied on in situ measurements of particle size distributions (PSD) and estimates of ice particle mass and were based on samples from one or two midlatitude cirrus clouds. The IWCs derived in this way are accurate only to a factor of 2, so that the evaluations are not conclusive. Sassen et al. (2002) used a cloud model with explicit microphysics to test algorithms for retrieving cirrus cloud IWC from millimeter-wavelength radar reflectivity measurements. They found that radar Z_e-only approaches suffer from significant problems related to basic temperature-dependent cirrus cloud processes; however, excellent results were obtained when used with ancillary lidar or radiometric measurements. Mace et al. (2005) used a statistical approach to compare Moderate Resolution Imaging Spectroradiometer (MODIS) overpasses of cirrus with ground-based remote sensing observations. Using retrievals of cloud properties, it was found that there was a positive correlation in the effective particle size, the optical thickness, and the IWP between the satellite- and ground-based observations, although there were sometimes significant biases. Hogan et al. (2006b) utilized realistic 95-GHz radar and 355-nm lidar backscatter profiles simulated from aircraft-measured size spectra to evaluate the correction of lidar signals for extinction by ice cloud, a potential source of error in the retrieval of IWC from the lidar–radar approach.

Although there are in situ validation activities planned for CloudSat and CALIPSO, there are a number of issues beyond the retrievals that can lead to errors in the retrieved IWC. The lidar–radar approach, combining coincident measurements synergistically, may provide IWC better than could be derived from radar alone. However, there are issues related to converting lidar backscatter to extinction, although recently developed algorithms to derive visible extinction profiles are relatively insensitive to the details of ice microphysics, lidar backscatter-to-extinction ratio, and lidar calibration (Hogan et al. 2006b). Multiple scattering is an additional problem, as are attenuation of CloudSat’s radar beam when the radar reflectivities exceed ∼5 dBZ_e, spatial averaging scales from spaceborne radar, the difficulty in collocating an aircraft, radar, and an appropriate cloud, and the large differences in radar beam volume and the sample volume of an IWC measurement probe.

In this study, we perform a detailed evaluation of IWC retrieval methods, using test datasets derived from in situ microphysical measurements. The data provided to eight teams included all of the information needed for their retrieval methods: vertical profiles of Z_e, σ, temperature, and Doppler fall speed. However, the IWCs were not provided to the teams. Their retrieved IWCs were then compared with the measured values. This approach is not subject to the lidar and radar issues raised above, which obviously would add to the error. In section 2, we describe the test dataset and the methodology. In section 3, we evaluate the results, and in section 4, the principal findings are summarized.

2. Data overview and products provided to contributors

This section provides an overview of the temperatures and microphysical properties encountered during the cloud penetrations on 19 days used for this study and describes the methodology used to derive IWC, radar reflectivity, and extinction estimates provided to the participants.

Nineteen cloud penetrations from four field campaigns in low and midlatitudes constituted the microphysical dataset used in this study. Fifteen penetrations were from aircraft Lagrangian spiral descents where the aircraft descended at ∼1 m s⁻¹ while drifting in a spiral with the wind.¹

There was one aircraft spiral up through the cloud layer. There were also three balloon-borne ascents. The microphysics during the penetrations are treated as though they represent the vertical distribution of cloud microphysics through the layer, while recognizing that the aircraft samples portions of different ice source regions during each loop of a spiral descent or ascent and the balloons drift horizontally as they ascend. This potential problem should, however, in principle not impact the results of our comparisons since the instrument observables are simulated from the observations.

Figure 1a summarizes the temperatures encountered during the cloud penetrations. Each point represents a 5-s aircraft or a 25-m balloon-borne average and values are arranged chronologically unless altered for clarity where noted. The penetrations to the left of the star-shaped symbol are from cold temperature, midlatitude, synoptically generated ice clouds, evidently formed primarily through synoptic forcing at cirrus altitudes. The first eight cases are aircraft Lagrangian-type spiral descents from the First International Satellite Cloud Climatology Project (ISCCP) Regional Experiment (FIRE) I, 1986 (F1–8) and three additional ones are from the Atmospheric Radiation Measurement Program (ARM) 2000 intensive observing period (IOP) (A1–3). Three are from balloonborne ice crystal replicator ascents through cloud layers from FIRE II, 1991 (FR1–3). The constituent ice particles of the synoptically generated cirrus are predominantly bullet rosettes in various stages of development, and aggregates thereof [see Heymsfield and Iaquinta (2000), their Fig. 15 (case FR1, here); Heymsfield and Miloshevich (2003), their Fig. 4 (case A2, here); Heymsfield et al. (2004), their Fig. 4 (case A1, here); Heymsfield et al. (2007a, hereinafter H07a), their Fig. 3 (case A4, here)]. One case to the right of the star-shaped symbol is from the ARM 2000 IOP (A4). Case A4 took place at warmer temperatures than the other midlatitude cases and had more complex particle habits, largely composed of spatial-type ice crystals including bullet rosettes.

Four Lagrangian spirals were conducted in anvils/convective outflow regions during the Cirrus Regional Study of Tropical Anvils and Cirrus Layers Florida-Area Cirrus Experiment (CRYSTAL–FACE, hereinafter CF). The first three cases (CF1–3) covered the region from cloud top to base (CF1 was a spiral ascent). A fourth case (CF4) began near −10°C and is considered down to the melting layer. Constituent ice particle habits for the four spirals were complex, often aggregated and in various stages of riming (Heymsfield et al. 2004, their Fig. 4).

Figure 1b shows estimated radar reflectivities at three radar frequencies—13, 35, and 95 GHz—as derived from calculations using the particle size distributions. Condensed water content measurements were used to constrain the ice particle mass estimates associated with the PSD. The methodology used to calculate these reflectivities is elaborated upon below. The synoptically generated clouds have radar reflectivities primarily below 0 dBZ_e whereas the convectively generated ones are almost all above 0 dBZ_e. For the cold, synoptically generated cloud cases, the reflectivities are about the same at the lower and higher frequencies, signifying negligible Mie (non-Rayleigh) scattering. Conversely, there are numerous Mie scattering periods for the convectively generated clouds sampled.

Figure 2 shows the visible optical depths (τ) estimated for the various clouds sampled. The τ are found by integrating the extinction derived from the PSD and cross-sectional areas estimated from the particle probes [forward scattering spectrometer probes (FSSP) + 2D], as in H07a, from the cloud top to base (except for case CF4, which is included but did not sample to cloud top). The effects of ice shattering on the inlet of the 2D-C probe (Field et al. 2006) have been taken into account. The IWP are shown as a function of τ. More details of how the IWC were derived are given below. The cases studied include all cold cloud types given in the ISCCP classification as demarcated in the figure.

Figures 3 –5 provide an overview of the vertical distribution of IWC as a function of temperature for 18 of the 19 cloud profiles. (Spiral A4 is not plotted so as to reduce the number of figures.) Note the fluctuations in the IWC during the aircraft descents (e.g., Fig. 3b): this is due to the aircraft spiraling in and out of generating cells and is an unavoidable issue with aircraft spirals through cloud layers. The IWC fall in the range 0.001–0.3 g m⁻³. The exceptions are cases CF2 and CF4, with IWC ∼ 1 g m⁻³.

The vertical bars on the sides of each panel in Figs. 3 –5 provide information on the penetration depth of vertically pointing lidar (right) and radar (left) for the cloud layers (see legend, Fig. 4i). Given that a lidar beam is occulted at an optical depth of approximately 3 (Kinne et al. 1992), the layers in Figs. 3a–c can be fully penetrated by upward (ground based) and downward (e.g., spaceborne) lidar. Similarly, the CloudSat 94-GHz radar, with a detection threshold of −28 dBZ_e, would see through the depth of these cloud layers. (The method used to derive Z_e is discussed at the end of this section and in appendix A.) Few points had reflectivities above 0 dBZ_e, indicated by dark boxes on the left vertical bars.

The method to calculate the IWC and dBZ_e is best shown by first using examples from ARM IOP spiral descents A1–A3. Figures 3a–c plot the IWCs measured directly by a counterflow virtual impactor (CVI) probe (Twohy et al. 1997). Also shown in the panels (red symbols) are points representing the IWCs derived from the PSD (FSSP + 2D probes), with mass–dimensional relationships scaled to agree with the measured IWC when the IWC is above the CVI detection threshold of about 0.01 g m⁻³ (see H07a). The mass–dimensional relationships (m = aD^b) derived from the scaling process are used directly to produce reliable IWCs only when the measured values fall below the CVI’s detection threshold, as evidenced by application of these relationships to more recent datasets where the CVI threshold was ∼0.001 g m⁻³. H07a parameterizes the results of the scaling process. The coefficient a in the m(D) relationship has been represented in terms of temperature a(T) and the exponent b is evaluated using vertically pointing Doppler radar observations for cases A1, A4, and many other ARM observations (Heymsfield et al. 2007b, hereinafter H07b). Temperatures in these three cases spanned the range from −50° to −20°C (and for a case not shown, 12 March, to 0°C). The IWCs measured for the three cases are relatively constant with temperature and fall primarily in the range from 0.001 to several tenths of a gram per meter cubed.

Table 1 summarizes the application of the temperature-dependent mass–dimension relationships to the ARM 2000 spirals themselves. The median and mean values of the ratio r = IWC (PSD)/IWC (measured) are case dependent and are generally accurate to within 10%. The greater difference noted for the 9 March case is largely due to CVI measurement error. Sampling during this case was in and out of generating cells and the associated trails. The CVI exhibits hysteresis: water vapor remains inside the instrument’s housing, leading to small underestimates within the IWC region and greater overestimates when the IWC decreases rapidly, as was the case on 9 March. Note also that for a small minority of the periods during the spirals except for 5 March, the IWC fell below the CVI detection threshold (number of instances, Table 1) and were subsequently derived from the m(D) relationships together with the PSD.

Also note from Table 1 that the IWC in FSSP (small particle) sizes amounted to 12%–17% of the total IWC. The IWC in small particle sizes was considered in the derivation of the m(D) relationships, and although the precise value of the IWC in small particles is not well known, the error resulting from the addition of the FSSP IWC for those cases falling below the CVI detection threshold and for the FIRE I cases is at most 15%. Note that two cases A1 and A4 did not have supporting FSSP data. This would lead to an underestimate of the IWC for those periods when the IWC fell below the CVI’s detection threshold.

Particle habits were predominantly bullet rosettes and rosette aggregates for ARM cases A1–3. For that reason, we can use the same values for a(T) and b found for the ARM cases to estimate the IWC for the remaining midlatitude, synoptically generated ice cloud layers (cases F1–F8 and FR1–3). From Table 1, given that the particle habits from the FIRE cases are the same as for the ARM cases, we can expect that the application of the m(D) relationships to the FIRE I and II datasets will produce a mean error of ±10%.

As shown in Figs. 3d–f and 4a–h, the profiles of IWC for the synoptically generated cirrus exhibit considerable structure. The warmer temperature ice cloud layers were optically thick enough that a lidar beam was unable to penetrate the depth of several of these layers (see Sassen et al. 1994, who reported lidar observations for the FIRE cases). The reflectivities also reach 0 dBZ_e and above, highlighted by the dark vertical bars on the left side of the panels, in some instances. For the warmer case A4 (not plotted), where spatial-type ice crystals including bullet rosettes dominated, an a(T) relationship appropriate for the range of temperatures considered was developed and evaluated on the basis of the four ARM cases.

The IWCs from the four CF cloud layers sampled at temperatures from −45° to 0°C were measured directly rather than inferred from an m(D) relationship because the CVI detection threshold was usually exceeded (Table 1). We derived a(T) and b(T) in the m(D) relationship and compared the IWC derived from the PSD with those measured. The ratios are nearly unity except for the 18 July case where there were no FSSP PSD to include in the PSD estimate (Table 1).

The ice clouds sampled during CF had large IWCs and changed little in the vertical, indicating that they were primarily regions of fallout. In general, these clouds were optically thick, inhibiting lidar penetration into the cloud layer. The radar reflectivities above 0 dBZ_e, as indicated by the bars on the left side of each panel in Fig. 5, extend throughout the layers.

The participants received estimates of the radar reflectivities, derived from the PSD and m(D) relationships for frequencies of 9.6, 35, and 95 GHz (see Fig. 1b and the discussion in appendix A). (A frequency of 95 rather than 94 GHz is used in the remaining part of this paper to conform to retrievals reported in the literature, although differences for frequencies of 94 and 95 GHz are negligible.)

For midlatitude clouds, we used the Mie scattering estimates for spherical particles from Bohren and Huffman (1983). This method was chosen because the particles sampled were quasi-spherical: bullet rosettes with some aggregates of rosettes. Using a discrete dipole approximation for different crystal shapes, including quasi-spherical ice particles, Okamoto (2002) found that shape influences on Z_e are less than 2 dBZ_e for ensemble (PSD) volume-equivalent effective radii of less than 100 μm. This criterion was satisfied for all of the midlatitude PSD. The dual-wavelength ratio, dWR = dBZ_e (35 GHz)/dBZ_e (95 GHz), calculated from the midlatitude PSD, is generally less than a few Z_e decibels (Figs. 6a and 1b), indicating that errors resulting from the Mie scattering effects are generally negligible. The largest dWR values are for 12 March 2000 (spiral A4). In Fig. 6b, the measured dWR sampled by a ground-based radar on 12 March 2000, including the period of the spiral (see H07b), follow the same distribution with Z_e and has approximately the same magnitudes as in the calculations.

An estimate of the magnitude of the error of the derived midlatitude radar reflectivities given for the evaluations can be estimated by using the PSD from the four ARM spirals. The masses of all ice particles were increased by 10%, thereby increasing the IWC by an equal amount, expressing a reasonable degree of uncertainty in the CVI measurements and in the FSSP-sized particles. The net increase in radar reflectivity was only 0.65 ± 0.34 dBZ_e. On the basis of this sensitivity study and 1) because the midlatitude particles are quasi-spherical with negligible Mie effects, 2) the mass–dimension relationships produce accurate IWC (Table 1), and 3) there is reasonably good agreement between the calculated and measured reflectivities (H07b), we conclude that the Z_e derived from the forward model are probably accurate to 1 dBZ_e.

Non-Rayleigh effects are significant for the CF clouds (Fig. 1b) that were dominated by aggregates of complex crystal shapes. It is reasonable to assume that the CF particles that dominate the radar reflectivity are horizontally aligned aggregates of aspect ratio (height to diameter) equal to 0.6 based on the observations of Magono and Nakamura (1965) and Hanesch (1999). Matrosov et al. (2005) have developed a 𝗧-matrix scattering model that adequately describes the radar polarization backscattering properties of most nonspherical (oblate) atmospheric hydrometeors, including ice cloud particles, pristine snowflakes, and raindrops. Our forward model uses the Matrosov et al. (2005) model with an assumed particle aspect ratio of 0.6.

The few large particles sampled by the particle probes dominate the reflectivity. In H07a, we report on an evaluation of the influence of the statistics of the PSD on Z_e. Statistical variation of the ice particle concentrations in the largest sizes is used to evaluate the effects on Z_e. There is essentially no net bias in the calculated Z_e resulting from the statistics of the small sample, although in the retrievals this could lead to increases in the statistical uncertainty in IWC derived from spaceborne platforms.

The ensemble, reflectivity-weighted fall speeds V_Z at the three radar wavelengths and for Rayleigh scatterers were calculated from the masses and terminal velocities of the ice particles integrated across the PSD. H07b describe the methodology and compare the estimated distribution of V_Z with reflectivity with that measured by vertically pointing Doppler radar on two of the days used in this algorithm evaluation (A2 and A4). In appendix A, we discuss the impact of errors in our estimates of V_Z on the retrieved IWC.

The volume extinction coefficient in visible wavelengths σ was supplied for those participants employing a lidar–radar approach. The σ were estimated from twice the total particle area per unit volume in sizes from FSSP (assuming the particles were spherical) through to the largest particles measured by the imaging probes (based on the particle cross-sectional areas). Heymsfield et al. (2006) compare estimates of σ from the particle probes with those measured by the cloud integrating nephelometer (CIN) probe (Gerber et al. 2000) from CF. The σ from the CIN are, on average, about 2 times those derived from the particle probes. Heymsfield et al. (2006) provide reasons why the particle probe estimates are likely to be more reliable than those from the CIN, although we are not confident about the contribution from the FSSP. The potential impact of errors in σ on the retrieved IWC can be estimated by assuming, for example, that σ are over- or underestimated by 20% but that the radar reflectivity supplied to the investigators is exactly correct. Taking the FIRE cases (F1–F8) as an example, the Donovan and van Lammeren (2001) radar–lidar method predicts an increase in the IWC of 1.178 ± 0.014 g m⁻³ (for σ = +20%) and a decrease of 0.85 ± −0.01 g m⁻³ (σ = −20%). [Potential systematic overestimates in the σ provided to the investigators resulting from shattering on the inlet of the FSSP probe (Field et al. 2003) are evaluated in appendix B and quantified in Table B1.] Note that FSSP data were unavailable for cases A1, A4, and CF2.

3. Results

Eight teams, some with more than one IWC retrieval method, participated in this study. They were provided with vertical profiles of temperature (T), Z_e at frequencies of 9.6, 35, and 95 GHz, reflectivity-weighted fall speeds (V_Z) at these wavelengths σ and an optical depth τ as derived from σ integrated downward from cloud top and upward from cloud base. The vertical resolution of the profiles was ∼5 m.

Table 2 summarizes the IWC retrieval methods and the publications that describe their methodology, grouped between dark horizontal lines according to the methods used. Most of the algorithms use gamma-type particle size distributions but several use normalized PSD represented in terms of the melted equivalent diameter. The algorithms include the use of Z_e alone [1(Z95)], Z_e and τ [2(Z95_OD)], Z_e and T [3(ZT)], Z_e and σ [4(LiRad)], lidar–radar approach), and Z_e and V_Z [5(Z95V_Z)]; the “Doppler radar” approach). Note that some of the lidar–radar retrieval algorithms require direct estimates of extinction (e.g., Wang and Sassen 2002a, b) whereas others have the desirable feature that they use lidar backscatter directly (e.g., Donovan and van Lammeren 2001; Okamoto et al. 2003). The latter methods have been modified to use our extinction estimates.

The accuracy and applicability of the various approaches to the range of conditions sampled in this study can be evaluated by comparing the “retrieved” IWC (IWC_retr) with those “measured” (IWC_meas, shown in Figs. 3 –5). The latter were either measured by the CVI, or, when below its detection threshold or otherwise unavailable (FIRE I, FIRE II replicator), estimated from the PSD and m(D) relationships.

Figures 7 and 8 show the ratio r = IWC_retr/IWC_meas for the five approaches encompassing all teams, plotted as a series of 5-s data points along the abscissa, as in Fig. 1. The results differ widely amongst approaches. Comparable mean values of r averaged for all data points (listed in each panel) are obtained for approaches 1a(Z95) and 2a(Z95-OD) used by team 1 (Figs. 7a,b), producing good estimates of the IWC in a mean sense but with a large standard deviation. Their radar-only approach 1a(Z95) produces IWCs that are significantly underestimated for the low temperature midlatitude cases A1 and FR1–FR3, and are much improved with the addition of optical depth 2a(Z95_OD). It is also noteworthy that approach 2a produces IWC that are nearly equal to the measured values (r ∼ 1) for the CF cases, unlike the results for the other methods. The results for method 3(ZT) are either biased high (Figs. 7c,f) or have a large standard deviation (Fig. 7e). The lidar–radar methods 4(LiRad) show mean values of r of nearly unity and with relatively low standard deviations. The results for 4a(LiRad95) are close to unity throughout (Fig. 8a) with the exception of the CF cases with appreciable Mie effects (not considered in this algorithm), where the IWCs are significantly overestimated. Underestimates are noted for the low temperature, FIRE II replicator cases. The results for 4b(LiRad35) are nearly unity throughout (Fig. 8b), except for significant underestimates for the low temperature spiral CF2, with significant numbers of large particles. The results for 4c(LiRad95) agree well with the measured values (Fig. 8c), with a mean value near unity and with a low standard deviation. The results for 4d(LiRad95) are biased low throughout (Fig. 8d) except for the ARM cases. Note that the extinction values provided to investigators 2a and 4a–d may be overestimated because of shattering of large particles on the inlet of the FSSP probe (an issue addressed in appendix B and Table B1).

Use of Z_e and V_Z by 5a and 5b (Z95V_Z) produces values of r of nearly unity throughout (Figs. 8e,f); however, the IWC is significantly underestimated (especially for 5b) for the CF cases exhibiting appreciable Mie effects, which are treated differently by 5a and 5b.

The ratio r for each method and team has been examined in different ways to identify weaknesses and limitations in the retrieval algorithms. Some variables add insight (e.g., temperature); other variables, which may not be independent (e.g., Z_e and IWC), place geophysical limits on accuracy. The evaluations are presented in Figs. 9 –15 and the results are summarized in Fig. 16, according to method. We arbitrarily choose the range 0.75 < r < 1.25 to represent “good” agreement between the retrievals and measurements, given the uncertainties in the parameters, especially σ, supplied to the investigators. Outside of this range, there are biases suggesting weaknesses in a given approach. Note that a lidar beam is occulted at an optical depth of about 3 and CloudSat, for example, cannot detect cloud below −28 dBZ_e. (Figure 16 considers these detection thresholds.) To reduce the number of figures, the more sophisticated method for each team is chosen for these evaluations.

No strong biases of r with Z_e are noted for 2a(Z95_OD) (except for reflectivities below the CloudSat threshold, Fig. 9a), 4c(LiRad95) (Fig. 9c), 5a and 5b (Z95V_Z) (Figs. 9g,h), except for 5b at large Z_e (Fig. 9h). The greater bias for large Z_e for method 5b with respect to method 5a is consistent with the findings shown in Fig. 8 and may be related to a different treatment of the Mie effect (discrete dipole approximation in method 5b versus homogeneous spheres with corrected ice density in method 5a). Results from methods 4b(LiRad35) and 4d(LiRad95) (Figs. 9d–f) show an increasing low bias of IWC_retr with increasing Z_e, for reasons related to using Rayleigh scattering and/or the mass–dimension relationship chosen for their algorithm (from Brown and Francis 1995, hereinafter BF95). At higher reflectivities, 10 dBZ_e and above, the results from methods 3c(Z95T) and 4a(LiRad95) are overestimated (Figs. 9b,c). The treatment of nonspherical particles and Mie effects or the assumed breadth of the PSD may be the possible cause of these positive biases.

Biases in r as a function of the IWC are less pronounced than with Z_e (Fig. 10). At low IWC, the results for 2a(Z95_OD) are appreciably overestimated and those of 3c(Z95T) are biased high, especially for the higher IWC; these trends are consistent with the trends noted for Z_e. The results for 4d(LiRad95) are generally underestimated. Particularly low standard deviations are noted in the results for 4a, 4c, 5a, and 5b. Because a given IWC can be found over a wide range of temperatures, these low standard deviations suggest that these methods account properly for temperature.

There are few biases noted when the data are partitioned according to temperature (Fig. 11). The results for approach 2a(Z95_OD) for temperatures of −45°C and below are biased low, whereas those of 3c(Z95T), where temperature is a primary input variable, show a tendency to underestimate the IWC at low temperatures and overestimate it at warm temperatures. The above results are clearly consistent with those found in Figs. 9 and 10. The same result for this method was found at temperatures below −35°C by Hogan et al. (2006a). Method 4a(LiRad95) overestimates the IWC at temperatures above −10°C, not because of incorrect treatment of temperature but primarily because the Rayleigh approximation and the BF95 mass–dimension relationship were used. Methods 4c and 5a produce values of r of nearly unity for all temperatures, whereas those for 4d are biased low, mirroring earlier findings. Relatively low standard deviations are shown for groups 4c, 5a, and 5b, likely indicating that for a given temperature the PSD parameterizations and the treatment of the scattering signature in the Mie region are good.

As a lidar beam is occulted at an optical depth of about 3, an examination of how the lidar–radar approach performs at optical depths integrated from cloud top downward into cloud until τ = 3 is useful for assessing the accuracy of the lidar–radar approach under real conditions. All of the lidar–radar methods show little bias in the optical depth range 0.1–3 (Fig. 12), with the exception of those for 4d. It is noteworthy that if more penetrating lidars could be developed, lidar–radar methods presently available would still have the same level of accuracy at higher optical depths.

Figures 13 –15 examine factors that might point to errors in the way the various methods treat the particle size distributions. Most of the retrieval algorithms are based on gamma-type PSD,

View Expanded

where N₀ is the intercept parameter, D is the ice particle maximum diameter, μ is the dispersion, and λ is the slope. The lidar–radar approach implicitly eliminates the need for N₀ because Z_e and σ are each proportional to it, indicating why this method should be inherently more accurate than the other methods.

Distance into the cloud layer alters the PSD through aggregation, broadening it and reducing λ and N₀. In Fig. 13, the ratio r is examined as a function of distance below cloud top. (For case CF4, cloud top was not reached; the data, almost all above −10°C, are used in Fig. 13 for completeness.) The greatest biases are found for method 3c(Z95T) and to a lesser extent for 4d(LiRad95). The results for 2a, 4a, 4b, 4c, 5a, and 5b show little bias with depth below cloud top, implying that representations of the PSD are accurate. The low standard deviations for the lidar–radar approach demonstrate the merits of this approach, which removes errors by eliminating assumptions of N₀. The same is not true for method 2a(Z95_OD), which exhibits large standard deviations. In layers deeper than about 4 km, mostly from the CF cases, the results for methods 3c(Z95T) and 4a(LiRad95) are overestimated and those of 4b and 4c are slightly underestimated, largely as a result of how Mie effects in the radar part of the retrieval algorithm were considered. The results for 4d are negatively biased throughout.

We now evaluate whether the assumed values of the PSD parameters [Eq. (1)] used by the various retrieval algorithms lead to biases. Most of the retrieval algorithms use μ = 2.0. The dispersion of the PSD fitted to our data using the first, second, and sixth moments exhibit μ ranging from about −2 to +4, with a mean of 0.16 ± −1.6 and a median of −0.08 (essentially an exponential PSD). In Fig. 14, the ratio r is examined as a function of the μ derived from the measured PSD. In this comparison, we are not separating out other effects, such as the slope of the PSD; we do that below. Although there is sensitivity noted for the results for 3c9 (Z95T) and 4d, it is modest. Because the lowest values of μ are associated with the warmest temperatures in which Mie effects are significant (case CF4), and because method 4a did not consider Mie effects, the increase in error for method 4a where μ < 1 is not related to μ directly. We conclude from this comparison that the choice of μ is not negatively affecting the retrievals, although refinements could reduce the standard deviation of the estimates.

We can also infer whether the λ values used by the various groups have produced some of the errors noted in the retrieved IWC. From Heymsfield et al. (2005),

View Expanded

where ρ₀ is the density of liquid water, the term in parentheses with K_i and K_w converts the radar reflectivity with respect to solid ice to the equivalent radar reflectivity, a and b are the coefficient and exponent in the mass–dimension relationship, and r_mie is the ratio of the radar reflectivity for a given wavelength to the radar reflectivity for Rayleigh scatterers. In Eq. (2), we use the BF95 m(D) relationship, μ = 2, and a radar wavelength of 95 GHz to correspond to the input parameters used in 3c(Z95T), 4a, and 4b. For r_mie, we use our results for 95 GHz.

A systematically increasing positive bias in r with decreasing IWC/Z_e is noted for approach 3c(Z95T), and to a lesser extent 4a(LiRad95) (Fig. 15). The biases are a function of the Mie effect (where IWC/Z_e < 0.07; see horizontal bars on the right side of each panel in Fig. 15). If we use r_mie = 1 (Rayleigh scatterers), in Eq. (2) the biases largely disappear. Because the trend noted is not due directly to errors in the treatment of λ, the primary discrepancies noted in Fig. 15 are therefore due to the treatment of Mie effects and not λ.

The biases found in Figs. 9 –15 are shown graphically in Fig. 16. The biases are most prominent in the evaluations partitioned according to the radar reflectivity and IWC (Figs. 16a,b). At high reflectivities, we attribute differences in the retrieved and measured IWC to be primarily due to whether or how Mie and nonspherical particle scattering is treated. At low reflectivities, almost all approaches overestimate the IWC. This almost certainly has its roots in overestimated masses ascribed to small particles. Method 4d(LiRad95) is biased low for all metrics. The intercept parameter N₀ of the PSD is not at fault because r does not trend with IWC/Z_e. The mass–dimension relationship is not at fault because the BF95 relationship used is the same as the other LiRad approaches. We therefore conclude that the slopes of the PSD are biased high. High biases in r are noted for approach 3c(Z95T).

Figure 16h summarizes the results of the evaluation for all retrieved IWC, not partitioned by any variables but with radar and lidar thresholds considered. The approaches that use the lidar–radar combination, contained within the shaded regions, and the Doppler radar approaches (upper part of the figure) have a ratio of retrieved-to-measured IWC of nearly unity, with low standard deviations. These results demonstrate the utility of the LiRad approach. Methods 2a(Z95_OD) and 3(ZT) [directly applicable to the Aqua satellite constellation (A Train) datasets] produce good results overall, but the standard deviations are much larger (about a factor of 2 for IWC) than for the LiRad approach.

Values of the visible extinction coefficient provided to the investigators included contributions from particles sampled by the FSSP probe, although there were exceptions noted earlier. There may have been significant contributions to σ from large ice particles that shattered on the inlet of the FSSP—that is, artifacts. As shown in appendix B (and quantified in the second and fourth columns of Table B1), this might have resulted in up to a 25% uncertainty in the ratio of the retrieved-to-measured IWC. Overall, the results are still excellent.

Accurate retrievals of the IWC and IWP are central to improving the representation of ice clouds in climate models (Stephens et al. 2002). In Fig. 17, the IWP obtained using the results from the various groups are compared with the measured values for the 19 cases. Two sets of results are shown in the figure that pertain specifically to CloudSat–CALIPSO: 1) those for those portions of the cloud layer where the “measured” reflectivity exceeds −28 dBZ_e, to simulate what CloudSat would measure (CloudSat only), and 2) those for portions of the cloud layer where 1) is satisfied and where the cloud optical depth is 3 or less. The dropoff of IWP_retr/IWP_meas with increasing IWP_meas occurs because the IWP_retr remains constant when an optical depth of 3 is reached and the IWP_meas can continue to increase beyond that point. Also listed and plotted in each panel are the mean ratio IWP_retr/IWP_meas and its standard deviation, reflecting how accurately each retrieval algorithm estimated the IWP through cloud depth. In considering IWP_retr/IWP_meas, the results for 2a(Z95_OD) (the standard CloudSat algorithm) are good overall. The mean ratio of predicted-to-measured IWP is nearly unity and the standard deviation is relatively small. The results for 3c(Z95T) follow the earlier patterns: underestimates at the low IWPs and overestimates at the high ones, which translate into an overestimation by 20% of IWP. LiRad retrievals yield good results overall, except for errors induced where temperatures are warm and Mie effects are large. For example, method 4d produces underestimates at low IWP. Results for 5a([Z95V_Z) are good overall, with a slight positive bias of IWP but a relatively small standard deviation. The results from 5b(Z95V_Z) produce the best estimates of IWP among all methods.

Figure 17 also shows that a lidar–radar approach, if used alone, would drastically underestimate the IWP for the clouds with large IWP. The figure also suggests that accurate approaches are needed to derive the IWC from lidar at low IWP (when radar begins to fail to detect cloud) and from radar at high IWP (when the lidar beam is occulted). These points are illustrated graphically in Fig. 18a, which considers fictitious “perfect” retrievals, making the results independent of the approaches used by the study participants. The methods used in this figure include lidar alone (measuring from above cloud with the beam occulted at an optical depth of 3), 95-GHz cloud radar alone (with a minimum detectable reflectivity of −28 dBZ_e), and a combination of the two. We assume that the IWC is retrieved perfectly from the measurements meeting the detectability limitations and refer to this as IWC_det (detected). An additional scenario assumes that the cloud radar has the same minimum detectable reflectivity, but at Z_e of 6 dBZ_e and above there is a ±50% error in the retrieved IWC, resulting from uncertainties in how to treat nonspherical ice particles and non-Rayleigh scattering based on the dWR estimations in Fig. 6 and the findings shown in Fig. 9. When the IWP reaches about 100 g m⁻², the lidar beam begins to occult, a diminishing portion of the IWP would be measured, and cloud radar would underestimate the IWP below about 40 g m⁻².

Figure 18b shows that, when considered together, the LiRad approach yields accurate IWP (to within 20%) from the CALIPSO–CloudSat spaceborne remote sensors only in the relatively narrow range of 40–100 g m⁻² (Fig. 18b). The uncertainty in the treatment of the effects of nonspherical ice particles and Mie scattering for the higher reflectivities measured by cloud radar leads to the result that cloud radar is unable to retrieve IWP > 500 g m⁻². Note that there are obvious errors from nonspherical particle effects for the higher IWPs and attenuation (not considered), which is also significant in the high IWC/IWP layers.

4. Summary and conclusions

This study presents a comprehensive examination of the accuracy and limitations of algorithms used to retrieve the IWC from radar reflectivity alone and together with estimates of optical depth in visible wavelengths, lidar extinction, temperature, and mean reflectivity-weighted ice particle fall speed. It includes almost every category of methodology used to retrieve the IWC from radar-based algorithms and includes most but not every investigator working on this problem. The 19 cloud profiles used in the study derive from mid- and low-latitude ice clouds. The 3389 data points in the vertical, separated by ∼5 m, cover a wide range of conditions, spanning a temperature range from −65°C to 0°C, cloud depths ranging from 1.2 to 4.6 km, optical depths from thin to deep cirrus according to the ISCCP definition (τ ∼ 0.5–50), IWC from less than 0.001 to above 1 g m⁻³, IWP from 10 to 3000 g m⁻², and estimated radar reflectivities in the range from −50 to 30 dBZ_e at 9.6 GHz and from −50 to 15 dBZ_e at 95 GHz.

The various methods collectively and individually produced accurate IWC. The mean (median) ratio of all of the user-provided IWCs to the measured values was 1.15 (1.03) ± 0.66. The radar-only and radar–temperature retrievals (methods 1a and 3c) were less accurate, with a mean (median) ratio of 1.29 (1.20) ± 0.75. Lidar–radar approaches (methods 4) produced the best results overall, with a mean (median) ratio of 1.08 (1.00) ± 0.61 if there was no restriction on the optical depth. If only those periods are considered when the optical depth is less than 3 and the radar reflectivity is −28 dBZ_e or above to consider CloudSat–CALIPSO thresholds, this ratio is 1.08 (1.00) ± 0.53, demonstrating the utility of the lidar–radar approach. The results were almost as good for the standard CloudSat radar–visible optical depth approach although with a larger standard deviation. To evaluate the impact of potential errors in the measurement of small (<50 μm) ice crystals on the retrievals from the radar–optical depth and lidar–radar approaches, the contributions of the small ice crystals to the total extinction were removed completely except when they were obviously real. In this obviously extreme sensitivity study, the results for these approaches were still excellent: the mean (median) ratios of the retrieved-to-measured IWCs were 0.81 (0.75) ± 0.44 for the lidar–radar approach and 0.99 (0.84) ± 0.75 for the radar–visible optical depth algorithm. The Doppler radar retrievals (method 5) as a group also produced almost the same level of accuracy as the radar–lidar method, although these methods are not yet applicable to spaceborne instruments, with a mean (median) ratio of 1.06 (0.98) ± 0.56, and 1.14 (1.03) ± 0.60, with the above restriction on radar detectability. For actual clouds, the accuracy of the results reported above would be degraded because of attenuation of the 95-GHz radar beam, attenuation and multiple scattering of the lidar beam, errors involved in the conversion of lidar backscatter to extinction, spatial averaging scales of the radar and lidar beams, and the contribution of vertical air motions to the Doppler velocities (for the radar–Doppler fall speed approach).

Researchers participating in this investigation were provided with vertical profiles of radar reflectivity derived based on mass–dimension relationships that were constrained by direct measurements of the IWC and evaluated based on coincident radar–Doppler fall speed measurements (H07b). The associated vertical profiles of the extinction coefficient in visible wavelengths and Doppler fall speeds were derived from particle size distributions. Because Z_e, V_Z, and σ were not measured directly, there are potential errors or uncertainties in the values provided to the investigators. In sensitivity studies that evaluated approximate uncertainties, varying the IWC by 10% resulted in changes in the Z_e by <1dBZ_e, changing the V_Z by ±10% yielded IWC that were uncertain by ±25%, and varying the extinction by ±20% but assuming that the Z_e were correct resulted in an uncertainty of ±15%. Removing these uncertainties would reduce the standard deviation of the evaluations by ∼15%–20% but would not change the mean values unless there are biases uncovered in the instrumentation used to derive Z_e and V_Z.

New methods are needed to improve the range of utility of the lidar–radar approach and to derive the IWC from cloud radar and lidar alone. Although the lidar–radar approach was found to be more accurate than the other approaches, the range of usefulness of the approach is limited. It is shown from our empirically derived results that this approach can be accurate only within the IWP range from about 40 to 100 g m⁻², assuming the CALIPSO–CloudSat detection thresholds; below that, cloud radar detection threshold becomes important and above it, a lidar beam is occulted. For IWP above 500 g m⁻², non-Rayleigh effects become so important and are so uncertain that cloud radar alone cannot now be used to reliably retrieve the IWP.

Based on a number of tests designed to uncover weaknesses in the retrieval algorithms (Fig. 16), there are several areas where improvements can be made. A reflectivity–temperature method is potentially useful because it might only require a variable that could, but need not be, measured from a satellite-borne platform; for example, temperature could be derived from European Centre for Medium-Range Weather Forecasts (ECMWF) model forecasts. Improvements can be made to the representation of the mass–dimension relationship (H07b) by incorporating temperature dependence for the a and b coefficients in the m(D) relationship. The parameterization for the slope and dispersion of the PSD can be refined using currently available observations that cover a wide range of temperatures. Improvements might also be made through consideration of alternate approaches to account for nonspherical particle scattering and Mie effects (e.g., Okamoto 2002; Matrosov et al. 2005). The lidar–radar retrieval methods are accurate to within 10% in a mean sense and with a lower standard deviation and are not in need of major improvement, although the approach is limited. The results for the reflectivity–Doppler fall speed method are accurate and have low standard deviations. This method obviously holds much promise, although it cannot currently be used from satellite-based measurements [but there are plans for a European Space Agency Earth Clouds, Aerosols and Radiation Explorer (EarthCARE) mission, provided that Doppler velocity is accurately measured]. The radar–optical depth method, now currently in use for CloudSat retrievals, produce adequate results, but improvements are needed at the temperatures below −40°C and to refine the assumptions to reduce the standard deviation.

A major potential source of error identified here and in earlier studies is the treatment of nonspherical (Mie) effects, most significantly for 94 GHz (which is the frequency used for CloudSat and upcoming EarthCARE cloud radars). The differences between the retrieved and “measured” IWC for Z_e > 5 dBZ_e were large, for some methods overestimating and others underestimating the IWC. This situation occurs primarily at the warmer temperatures. It is not clear whether the method used to estimate nonspherical effects was considered properly here, although indications are that the results are reasonable agreement with observations. An effort needs to be made to establish proper treatment of nonspherical effects for Z_e > 5 dBZ_e and algorithms to correctly account for attenuation at 94 GHz are also required.