a. G-band DAR: VIPR

VIPR is a proof-of-concept DAR being developed at the Jet Propulsion Laboratory (JPL) for remote sensing of water vapor in cloudy volumes, specifically targeting the PBL. In addition to being the first water vapor DAR, VIPR is the first all-solid-state G-band cloud radar, and is operated in frequency-modulated, continuous-wave (FMCW) mode to increase the transceiver sensitivity relative to a pulsed system. An early version of the VIPR system, discussion of FMCW radar detection, and demonstration of the DAR measurement principle were previously described in detail in Cooper et al. (2018) and Roy et al. (2018). Since this early work, a higher-power transmitter and higher-gain optics have been installed, greatly increasing the sensitivity of the system, and the hardware has been modified for aircraft deployment. Technical details of the aircraft-compatible VIPR system can be found in Cooper et al. (2020), while relevant details of the instrument hardware for this work are discussed below and summarized in Table 1. Figure 1 shows VIPR during the ARM SGP deployment in both calibration and cloud-measurement configurations.

Table 1.

VIPR hardware and radar signal acquisition parameters.

View Table

View Full Size

VIPR deployment at the ARM SGP site in (a) calibration sphere measurement configuration and (b) zenith-pointing cloud measurement configuration. The radiosonde launches, Raman lidar, and calibration sphere are located at the central facility, with approximate distances of 900, 610, and 460 m from VIPR, respectively.

Citation: Journal of Atmospheric and Oceanic Technology 37, 6; 10.1175/JTECH-D-19-0122.1

• Download Figure
• Download figure as PowerPoint slide

The radar features a state-of-the-art, all-solid-state transmitter that is tunable between 167 and 174.8 GHz with 200 mW of CW output power, leveraging significant research and development in high-power millimeter-wave generation at JPL using GaAs Schottky diode frequency multiplication technology (Siles et al. 2018). The chosen radar band is a result of international regulations [National Telecommunications and Information Administration (NTIA); NTIA 2015] that prohibit G-band transmission in spectral regions reserved for passive satellite sensors, including the 174.8–191.8 GHz band used for humidity sounding. As a result, VIPR is positioned on the low-frequency flank of the 183 GHz water vapor absorption line, which makes the system primarily sensitive to PBL water vapor and allows the radar beam to penetrate through the whole atmosphere, whereas frequencies closer to the line center would experience too much absorption to feasibly probe all levels. As compared with high-peak-power, pulsed cloud radars with transmitter duty cycles on the order of 1%, the use of FMCW detection makes VIPR’s time-averaged output power equivalent to a 20 W pulsed system. Furthermore, the short wavelength of λ = 1.8 mm, large primary aperture gain, and low system noise figure result in a radar signal-to-noise ratio (SNR) of unity for a cloud reflectivity of −40 dBZ at 1 km range using nominal radar acquisition parameters (see Table 1).

Data acquisition and signal processing are performed on a rack-mounted PC, after the FMCW radar signal has undergone analog pulse compression in the front-end G-band mixer and subsequent I/Q down-conversion to baseband. A single computer program controls the necessary components for radar signal generation (e.g., chirp initiation, transmit frequency switching), digitizes the complex I/Q baseband signal, performs digital signal processing in parallel with radar signal acquisition, and provides a real-time display of the FMCW baseband time-domain signal, resulting power spectra, and 2D plots of cloud reflectivity. Digital signal processing steps include the application of a time-domain window to a buffered set of 400 radar pulses, computation of individual power spectra using fast Fourier transforms (FFTs), and averaging of the resulting power spectra. For memory considerations, spectral averages are computed for sets of 2000 pulses at each frequency and then stored in permanent memory. For the typical acquisition parameters in Table 1, a single logged measurement for two transmit frequencies takes 5 s, which sets the smallest time resolution for humidity profiling, and during which we rapidly switch between transmitter frequencies at a rate of 80 Hz.

Differential absorption measurements using cloud radars feature a few important distinctions from DIAL systems. First, the large width of molecular absorption lines at millimeter and submillimeter wavelengths in the troposphere requires that the “online” and “offline” transmitter positions span a fractional bandwidth of 1%–10%, while for DIALs this number is typically around 0.01%. Second, the scattering targets of interest for DAR are the hydrometeors that make up clouds and precipitation, while for DIAL, the dominant lidar signal originates from atmospheric aerosols. Because characteristic diameters of hydrometeor drop size distributions (DSDs) can vary by large amounts over small spatial scales, these two factors make DAR susceptible to systematic measurement biases stemming from frequency-dependent scattering. Furthermore, because the observed cloud and precipitation scenes can change quickly in time (e.g., from advection and sedimentation), it is critical to switch between DAR frequencies rapidly, as otherwise temporal dependence of reflectivity would be indistinguishable from frequency dependence and therefore humidity.

To mitigate the potential biases from frequency-dependent hydrometeor scattering, we implement a new DAR retrieval method that is based on a regularized least squares approach and is discussed in section 3b. Furthermore, to limit the deleterious effects from cloud scene dynamics we switch between transmit frequencies after 10 pulses of 1 ms duration at each frequency. With a 2 ms switching time for the Ka-band synthesizers that generate the radar signal, this results in a transmitter duty cycle of 80%. We note that in recent instrument simulation work, Battaglia and Kollias (2019) employed a retrieval that includes an extra fitting parameter to independently extract the spectral dependence of hydrometeor scattering and thus retrieve an unbiased humidity. However, this requires a large transmitter bandwidth to distinguish the linear frequency dependence of hydrometeor scattering from the higher-order dependence of the water vapor absorption coefficient. For VIPR, the transmit band from 167 to 174.8 GHz features little curvature of the water vapor absorption coefficient, and is fixed by a combination of international regulations and hardware constraints. For these validation measurements, we employ a two-frequency differential absorption measurement approach, with offline frequency f₁ = 167 GHz and online frequency f₂ = 174.8 GHz.

The primary quantity measured by VIPR is the radar echo power as a function of range and frequency,

Here Z(r, f) and Z_obs(r, f) are the unattenuated and observed reflectivity factors,

is the one-way optical depth, β_g and β_h are the gaseous and hydrometeor absorption coefficients, and C(f) is a radar hardware calibration factor. The procedure for determining C(f) and thus calibrating reflectivity profiles is outlined in the appendix. For DAR measurements of water vapor profiles inside of clouds, the value of C(f) is not needed because the retrieval involves the ratio of P_e(r, f) at two different ranges. Thus, the relevant sources of uncertainty in the retrieval of humidity are the random error in the measurement of P_e and the systematic uncertainty related to the frequency-dependent scattering parameters and stemming from our lack of knowledge about the true hydrometeor DSD. The systematic uncertainty terms will be discussed in more detail in section 3b.

In previous work (Roy et al. 2018), we demonstrated that the distribution of measured radar echo powers follows the well-known radar speckle model for randomly distributed scatterers, with resulting variance

where S_n = P_e(r)/P_n(r) is the signal-to-noise ratio, P_n(r) is the receiver noise power in range bin r, and N is the number of independent pulses averaged to obtain the echo power estimate. In contrast to pulsed radar systems, we write the noise power in FMCW radar explicitly as a function of range because of the mapping of range to frequency and the potential for nonuniform gain and noise power as a function of frequency in the baseband electronics. Additionally, the application of a time-domain window to the baseband radar signal prior to computing the power spectrum introduces covariance between neighboring range bins. The specific form of this covariance term can be derived by invoking the convolution theorem, which allows one to express the complex Fourier amplitudes of the windowed time-domain signal as a linear combination of those from the unwindowed signal, which themselves are statistically independent. The mean values and covariance matrix for the radar echo power spectrum are then computed using the squared moduli of the windowed amplitudes. For the Hann window used in this work, nearest neighbors have covariance

where P_e,± = [P_e(r) + P_e(r ± Δr)]/2, S_n,± = 2P_e,±/[P_n(r) + P_n(r ± Δr)], and Δr is the radar range resolution. In FMCW radar, the range resolution is given by Δr = c/2ΔF, where c is the speed of light and ΔF is the frequency chirp bandwidth. In this work, the nominal chirp bandwidth of ΔF = 10 MHz results in a range resolution of Δr = 15 m. Therefore, the measurement covariance matrix, which we will need for the regularized least squares retrieval in section 3b, is tridiagonal.

b. ARM instruments

The primary measurement used for DAR humidity validation comes from the radiosondes (Vaisala RS41-SG) launched from the ARM SGP central facility, located about 900 m NW of the radar’s location. In addition to the nominal ARM radiosonde (RS) launches at 0000, 0600, 1200, and 1800 UTC, there was an increased frequency of RS launches as high as twice per hour when thick clouds were present. The typical ascent rates were between 5 and 7 m s⁻¹, and temperature, relative humidity (RH), and GPS height were recorded every second. In this work, we report all height coordinates referenced to mean sea level. The radiosondes quickly advected away from the ARM site after launch with an average horizontal displacement of 30 km at 10 km height. While this introduces the potential for collocation errors between in situ and remotely sensed measurements, we find that humidity profiles from RS launches spaced 30 min apart show little variation on the scale of DAR humidity errors. Therefore, in our analysis we treat the entire profile measurement as having occurred at the time of launch. From the RH and temperature accuracy values reported on the Vaisala RS41-SG datasheet, we calculate an absolute humidity RS accuracy of 0.7, 0.4, 0.2, and 0.1 g m⁻³ at 20°, 10°, 0°, and −10°C, respectively.

In fact, for water vapor profiling inside of clouds, the RS in situ measurement is the only reliable one out of all the ARM instruments, since existing passive and active remote sensors are incapable of making this measurement accurately. However, for the measurements of IWV between the surface and cloud base, we utilize water vapor profiles retrieved using the ARM Raman lidar (RL) in addition to the RS values (Turner et al. 2016). The RL transmits a near-UV (355 nm), high-peak-power pulse, and detects the inelastically scattered photons from particular H₂O and N₂ vibrational transitions. The range-resolved ratio of detected inelastically scattered photons from H₂O and N₂ provides a profile that is proportional to the water vapor mixing ratio, with a multiplicative constant provided by an ancillary sensor for absolute calibration (Whiteman et al. 1992; Turner and Goldsmith 1999). The spatial and temporal resolution of the water vapor product used here are 60 m and 10 s (ARM 2015). In between radiosonde launches the RL provides valuable independent water vapor profiles with high temporal resolution that we utilize in the IWV measurement comparisons.

a. Standard differential absorption retrieval

The standard DAR approach (Roy et al. 2018; Battaglia and Kollias 2019) begins by combining the observed reflectivities to form the observed absorption coefficient

where r₂ − r₁ = R. Next, we write β_g(r, f) = ρ_υ(r)κ_υ(r, f) + β_g,d(r, f), where ρ_υ is the water vapor density, κ_υ is the mass extinction cross section including contributions from all H₂O absorption lines (i.e., the continuum contribution), and β_g,d is the dry-air absorption coefficient. In this work, we calculate gaseous absorption parameters using the millimeter-wave propagation model from the EOS Microwave Limb Sounder (Read et al. 2004). Making the assumption that the spatial variation of κ_υ and β_g,d is small on the scale R, we replace their values in the integrand in Eq. (5) with those at the midpoint $\overline{r}=\left(r_1+r_2\right)/2$ and evaluate the integral to yield

where α = (2R)⁻¹ ln[Z(r₁, f)/Z(r₂, f)] and ${\overline{\rho }}_{\upsilon }$ and ${\overline{\beta }}_h$ are the average values of the respective quantities between r₁ and r₂. Next, we perform a Taylor expansion of hydrometeor scattering quantities Z and β_h, which involve integrations of single particle scattering parameters over a DSD, and retain only the first-order frequency dependence because of the small fractional bandwidths used for DAR systems (1%–10%). Note that the equivalent parameters for DIAL systems can be taken to be independent of frequency because of the extremely small fractional bandwidths involved.

Thus, the standard DAR retrieval amounts to performing a least squares fit of the function

to the data, with the fitting parameters having the physical interpretations $a_1=\alpha \left(r_1,r_2,f_1\right)+{\overline{\beta }}_h\left(f_1\right)$, $a_2={\left(\partial /\partial f\right)\left(\alpha +{\overline{\beta }}_h\right)|}_{f_1}$, and $a_3={\overline{\rho }}_{\upsilon }$, where f₁ is a reference frequency within the transmitted band. Seemingly, then, measurements of Z_obs at three different frequencies should suffice to fully determine these coefficients and retrieve unbiased humidity estimates. However, if κ_υ is nearly linear across the range of frequencies used, then a₂ and a₃ become degenerate fitting parameters, and the choice of including a linear term leads to an inflation of the estimated humidity variance. Clearly, the measurement in this case cannot distinguish between frequency-dependence due to humidity and hydrometeor scattering. Hence, DAR retrievals will be most accurate when κ_υ exhibits strong curvature with respect to frequency. Unfortunately, however, κ_υ has very little curvature across the transmission band used in this work, and so utilizing the standard DAR retrieval would require setting a₂ = 0. The frequency-dependent hydrometeor terms would then enter directly as a bias. As shown in the next section, we attempt to limit these biases by retrieving the entire humidity profile in a single regularized least squares minimization routine.

b. Regularized least squares approach

In this section, we present a new retrieval algorithm for the case of two transmitted frequencies, f₁ and f₂, in which the entire humidity profile and corresponding covariance matrix are estimated in a single least squares minimization procedure, in contrast to previous approaches that iteratively retrieve the humidity at each height (Roy et al. 2018; Battaglia and Kollias 2019). This approach has the additional advantage of being amenable to regularization methods that constrain the form of the retrieved profile. The extension of this method to arbitrarily many frequencies is straightforward.

We begin the formulation of the least squares retrieval by defining the measurement vector y = [y₁, y₂], where [y_j]_i = ln[Z_obs(r_i, f_j)/Z₀] and $\mathbf{r}=[r_1,r_2,\dots ,r_{N_r}]$ is the collection of radar ranges for which the reflectivity measurements at all frequencies have SNR greater than some threshold value. For the in-cloud humidity profile retrievals presented in this work, we set this threshold to 1. The measurement vector therefore has length 2N_r. The choice of using the logarithm of reflectivity, and not linear units, in the measurement vector ensures that the forward model is linear in the humidity, and is justified so long as the relative error of the Z_obs measurement is small. For the measurements presented in this work, this condition is satisfied due to the large number of pulses N = 2000 averaged at each frequency, with a relative error of 0.25% for an SNR of 1. The resulting measurement covariance matrix is block diagonal, since the measurement errors at different frequencies are uncorrelated, with ${\mathsf{S}}_y=\mathrm{diag}\left({\mathsf{S}}_{{\mathbf{y}}_1},{\mathsf{S}}_{{\mathbf{y}}_2}\right)$. To determine the form of each individual block, we first calculate the covariance matrix for the reflectivity vector z = [Z_obs(f₁), Z_obs(f₂)], and then transform to the measurement space according to ${\mathsf{S}}_y={\mathsf{J}}_z{\mathsf{S}}_z{\mathsf{J}}_z^{\text{T}}$, where $\mathsf{S}$_z = diag[S_Z(f₁), S_Z(f₂)] is the covariance matrix for z and $\mathsf{J}$_z is the Jacobian of y with respect to z. The individual covariance matrices for observed reflectivity measurements are themselves tridiagonal, with

where the power variance and covariance terms are defined in Eqs. (3) and (4), and δ is the Kronecker delta. Note that if adjacent elements of the measurement vector do not correspond to adjacent radar range bins (i.e., if |r_j − r_k| ≠ Δr), then the off-diagonal term is zero.

Next, we must define the state vector x and forward model F(x, b), where b is a vector of parameters that are input to the forward model, but not retrieved. A full radiative model for the beam propagation would need to account for spatial distribution of the cloud microphysical properties (e.g., DSD parameters) as well as atmospheric parameters of temperature, pressure, and humidity. However, the DAR approach is powerful because it is largely insensitive to all of these details except for the humidity. Thus, we want a forward model that amounts to a fully determined linear system of equations, as is the case for the standard differential absorption retrieval [Eq. (7)]. We accomplish this by choosing the state vector elements to be x = [x₁, x₂], where [x₁]_i = ln{[Z_r(f₁)]_i/Z₀} and Z_r(f₁) = [Z_r(r₁, f₁), Z_r(r₂, f₁), …, Z_r(r_N, f₁)] consists of the retrieved reflectivities at one of the transmitted frequencies and includes attenuation from hydrometeors, but not gaseous attenuation, and x₂ = ρ_υ consists of the humidity values to be retrieved at ranges r_ρ = [r_ρ,1, r_ρ,2, …, r_ρ,M].

Because of the potential sparsity of the observed reflectivity profile, the definition of the retrieved humidity profile ρ_υ and its corresponding range axis requires some care. To maximally utilize the information in the measurements, we want the retrieval to both profile inside of continuous cloud volumes with resolution R and return IWV values in between cloud boundaries as well as between the radar and first cloud signal. Previous DAR retrieval methods based on fixed differencing in space of measurements separated by R lose the ability to extract information between cloud layers with large separations. As we will see, this problem is naturally avoided when using the least squares solution approach, since the entire humidity profile is retrieved from a minimization problem in which the forward model interpolates between all humidity points, and not just those separated by R. Anticipating that we will want to build up time series of humidity profiles with regular gridding in space, we enforce that the humidity positions be of the form r_ρ,i = n_iR for some integer n_i. Then, given a measurement range vector r with maximum value r_max, we form r_ρ by retaining all values n_i for which there is at least one element of r in the interval [(n_i − 1/2)R, (n_i + 1/2)R). Additionally, we always include the surface point n_i = 0 as the first entry in r_ρ.

The first step in constructing the forward model is to interpolate the lower-resolution humidity profile to the radar range resolution Δr, giving a new humidity vector $\tilde{\mathit{\rho }}$ with range axis $\tilde{\mathbf{r}}=\left[0,\mathrm{\Delta }r,2\mathrm{\Delta }r,\dots \right]$. Next, we use profiles of temperature and pressure at the same Δr resolution to calculate the resulting gas-absorption optical depth vector ${\tilde{\mathit{\tau }}}_g=\left[{\tilde{\mathit{\tau }}}_g\left(f_1\right),{\tilde{\mathit{\tau }}}_g\left(f_2\right)\right]$ according to

with ${\tilde{\mathit{\tau }}}_{g,0}=0$. Note that the gas absorption dependence on temperature and pressure are implicit in the $\tilde{r}$ dependence. For the retrievals presented in section 4, we use the radiosonde surface measurements and assume hydrostatic balance with a 6.5°C km⁻¹ lapse rate to calculate the temperature and pressure profiles. Last, the calculated optical depth vector is then masked to only include values at the measured range positions r, yielding the vector τ_g of length 2N_r. The recursive relationship in Eq. (9) can be cast as a matrix equation of the form

which will be necessary for the matrix inversion solution to the regularized least squares problem. Here τ_g,d is a constant vector of dry air optical depths, and $\mathsf{T}$ is a 2N_r × M matrix.

The forward model is then defined as F(x, b) = [F(x, b, f₁), F(x, b, f₂)], where

and d_i is a fixed scaling factor that accounts for differential backscattering coefficients between the two frequencies. Using Eq. (10) and the definition of x, we can write the forward model as

where

is defined in block form, ${\mathsf{I}}_{N_r}$ is the N_r × N_r identity matrix, and b includes both the d_i and τ_g,d terms. For this work, we choose the same value d for all d_i, informed by Lorenz–Mie calculations of liquid hydrometeor scattering (Bohren and Huffman 2004). This is equivalent to assuming that the DSD characteristic diameter is constant in space. The range of d values for different DSDs is shown in Fig. 2, where we assume a modified gamma distribution as in Roy et al. (2018) and vary the shape parameter ν from 1 to 4 to explore the sensitivity to the assumed DSD shape. Focusing on the ν = 4 case, which is a typical value used for nonprecipitating clouds, we see that the range of characteristic diameters over which b varies the most is 10 − 100 μm. To be conservative, we pick for our value of b the average of the two limits in Fig. 2, or d = 0.9, and assign a systematic uncertainty of σ_d = 0.1.

View Full Size

Ratio of reflectivity at f₂ = 174.8 GHz and f₁ = 167 GHz for liquid water spheres at 280 K, integrated over a modified gamma DSD for four different shape parameters ν.

Citation: Journal of Atmospheric and Oceanic Technology 37, 6; 10.1175/JTECH-D-19-0122.1

• Download Figure
• Download figure as PowerPoint slide

It is important to note that for DAR humidity profiling, systematic biases stemming from frequency-dependent backscattering would only arise if d_i varies in range. In the context of this retrieval, then, it is clear that our choice of constant d will not correct for these potential biases. However, this formulation gives us a way to propagate our uncertainty about these effects into our eventual estimates of humidity. Specifically, imperfect knowledge of the forward model parameters b can be represented by a systematic error covariance matrix in the measurement space according to (Rodgers 2000) ${\mathsf{S}}_e={\mathsf{J}}_{\mathbf{b}}{\mathsf{S}}_{\mathbf{b}}{\mathsf{J}}_{\mathbf{b}}^{\text{T}}$, where ${\mathsf{S}}_{\mathbf{b}}={\sigma }_d^2{\mathsf{I}}_{N_r}$ and $\mathsf{J}$_b is the 2N_r × N_r Jacobian of the forward model with respect to b. It is easy to show that the resulting $\mathsf{S}$_e is a 2N_r × 2N_r diagonal matrix with [$\mathsf{S}$_e]_ii − (σ_d/d)² for i > N_r, and [$\mathsf{S}$_e]_ii = 0 for i ≤ N_r. It may appear that, because Z_obs(r, f) ∞ C(f), the measurement error ought to include uncertainty in the determination of C(f). However, a miscalibrated value of C(f) will only affect the water vapor measurement between the radar and the first echo range, with subsequent retrieved humidity values being completely insensitive to this factor. Therefore, we do not include calibration uncertainty in the measurement covariance analysis, and will discuss the related column-integrated water vapor issues in the next section.

Finally, in order to mitigate the humidity biases that arise from the linear term in Eq. (7), we implement a regularization term. We begin by noting that these biases often occur over short spatial scales, for instance due to the differential backscattering at a boundary between intense and light precipitation, or between precipitation and a cloud base, and thus tend to produce large, unphysical gradients of water vapor in the retrieved profile. A natural regularization term to utilize, then, is one that penalizes sharp gradients in the retrieved state. For this term, we choose the sum of squared discrete derivatives of the humidity vector, which can be written as a quadratic form

for some matrix $\mathsf{A}$, where δ_ρ is some chosen scale for penalized humidity gradients, and is set to 10 g m⁻³ km⁻¹ in this work. The inverse algorithm is then the solution to a least squares minimization problem,

where the cost function is given by

and λ_reg ≥ 0 is a dimensionless parameter that determines how severely to penalize humidity gradients. To implement the gradient regularization scheme, we need to find the symmetric matrix $\mathsf{A}$ that satisfies Eq. (14). One can show that $\mathsf{A}$ is block diagonal and of the form $\mathsf{A}=\mathrm{diag}\left({\mathbf{0}}_{N_r},{\delta }_{\rho }^{-2}{\mathsf{D}}^{\text{T}}{\mathbf{\Delta }}_r^{-2}\mathsf{D}\right)$, where the first block is a N_r × N_r matrix of zeros, [$\mathsf{D}$]_ij = δ_i,j−1 − δ_i,j is a (M − 1) × M finite differencing matrix, and [Δ_r]_ij = δ_i,j(r_ρ,i+1 − r_ρ,i) is a (M − 1) × (M − 1) diagonal matrix. We set the regularization parameter λ_reg = 1, which amounts to a conservative penalty of gradients on the scale δ_ρ. This gradient regularization method is similar in nature to applying a spatial low-pass filter to the retrieved humidity profile, but performs the smoothing as part of a single least squares optimization routine, works for nonregularly spaced points, and naturally allows for uncertainty quantification in the retrieved, smoothed quantities. [An example scenario where this profile-smoothing approach is particularly effective is shown in Fig. 4 (second row, left panel).] Here the red curve represents the standard two-frequency DAR retrieval employed in Roy et al. (2018), which exhibits significant biases due to spatially varying hydrometeor properties, while the regularized least squares approach successfully mitigates these effects. We note that the value of δ_ρ is chosen as a compromise between limiting biases that appear as sharp humidity gradients and avoiding an overly constraining low-pass filter, and that using 10 g m⁻³ km⁻¹ results in good agreement between the median retrieval error and median bias with respect to radiosonde measurements (Fig. 5b). Because of this choice, the retrieval will smooth out real humidity variations that are on the scale of δ_ρ or larger.

Because the forward model is linear in the state vector, the minimization problem [Eq. (15)] can be solved by means of a matrix inversion,

Furthermore, the retrieved state covariance matrix is given by the Hessian of $\mathcal{C}\left(\mathbf{x}\right)$,

We note here that the regularization term is not introduced because of ill-posedness of the unregularized problem, as is the case with the introduction of the prior distribution in optimal estimation (Rodgers 2000), but rather to constrain the form of the retrieved profile. From the form of ${\widehat{\mathsf{S}}}_x$ in Eq. (18), it becomes apparent that inclusion of the systematic error covariance matrix $\mathsf{S}$_e can be viewed as effectively inflating the measurement variance. To get a sense of the scale of this variance inflation, we utilize the fact that the diagonal elements of $\mathsf{S}$_y are equal to the squared relative error of the corresponding echo power measurement, $\mathrm{var}\left[P_e\left(r,f\right)\right]/P_e^2\left(r,f\right)$, and those for $\mathsf{S}$_e are either zero or equal to (σ_d/d)². Thus, using Eq. (3) for the measurement relative error with N = 2000 pulses, we see that this approach amounts to an increase in variance by a factor of about 20 for high-SNR measurements, and by a factor of about 4 for measurements with SNR = 1.

c. Column-integrated water vapor

The IWV measurement between the surface and an elevated cloud base is an analogous measurement to future column measurements made from an airborne platform between the radar and the surface. These measurements are of interest for any DAR because in the absence of clouds, the surface return is the only radar measurement acquired. For the case of a single-range measurement at two frequencies, one can solve for the IWV between the radar and range r:

where $\text{IWV}={{\displaystyle \int }}_0^{r}d{r}^{\prime }{\rho }_{\upsilon }\left(r^{\prime }\right)$,

Δκ_υ(r) ≡ κ_υ(r, f₂) − κ_υ( r, f₁), and Δβ_h(r) ≡ β_h(r, f₂) − β_h(r, f₁). The first term in Eq. (19) will have an associated random error, while the remaining three may introduce systematic error. In this work, we only utilize measurements with negligible hydrometeor content between the surface and r and therefore can neglect the last term in this equation dealing with differential absorption from hydrometeors. Using ⟨Δκ_υ⟩ ≈ 0.25 dB km⁻¹ g⁻¹ m³, which is representative of lower-tropospheric conditions, we see from Eq. (19) that errors in the individual bias terms of 1 dB lead to a IWV bias of 2 mm.

To assess VIPR’s ability to accurately measure column-integrated water vapor, we perform a set of retrievals in which the surface-to-cloud-base IWV is the only humidity quantity retrieved. We assume that the water vapor density decays exponentially in the vertical with a scale height of 2 km, which is consistent with previous measurements at the ARM SGP site (Turner et al. 2001). For these retrievals, we lower the nominal SNR threshold to 0.5, and in some cases to 0.25 if there are no returns with SNR > 0.5, allowing for IWV measurements using cloud layers with weak reflectivities, and utilize reflectivity measurements within a distance R of the bottom cloud edge. The main reason for performing a separate set of IWV retrievals, as opposed to using the corresponding value from the profiling retrieval detailed in the previous section, is that the latter occurs on a fixed, equally spaced vertical grid. Therefore, depending on where the cloud base falls within a vertical grid cell of size R, the profiling retrieval will utilize a different number of reflectivity measurements to estimate the surface-to-cloud-base IWV, and thus have variable sensitivity. Furthermore, we change the scaling factor for differential backscattering between f₁ and f₂ from d = 0.9 to d = 1, since we expect scattering to be of Rayleigh character near the cloud edge where reflectivities are very weak. Unlike the case of in-cloud profiling, where taking the ratio of reflectivities at two different ranges makes the measurement insensitive to the value of d, incorrect specification of this parameter biases the IWV estimate [see Eq. (19)]. We take a practical approach to determining the calibration ratio C(f₁)/C(f₂), and choose the value that gives zero average IWV bias with respect to the ARM instrument ground truth. Therefore, average biases resulting from our choice of d and from hydrometeor absorption are implicitly included in the determined value of C(f₁)/C(f₂).

a. In-cloud humidity profiles

For validation of VIPR’s in-cloud profiling capabilities, we utilize cloud reflectivity measurements acquired on multiple days within the 2–14 April 2019 window. Three example datasets with significant spatiotemporal cloud coverage are presented in Fig. 3. We present the observed reflectivity measurements at f₁ = 167 GHz for which SNR > 1. These 2D plots have the nominal radar time resolution of about 5 s, and range resolution of 15 m. For these and all subsequent plots, the height is referenced to mean sea level, with the ground level at ARM being 0.3 km. While there is no near-range dead zone in FMCW radar, we do not use reflectivity measurements within the first 100 m, because these correspond to the antenna near field and the calibration measurements were performed in the far field. Depending on the level of precipitation at the surface, VIPR is either positioned outside pointing at the zenith, or just inside the facility pointing between 20° and 30° off zenith. The measurements presented in Fig. 3 account for these adjustments in pointing angle.

View Full Size

Observed cloud reflectivities at 167 GHz and corresponding retrieved humidity profiles for three different days. Dashed vertical lines correspond to radiosonde launch times. Time periods with no reflectivity measurements are either cloud-free scenes or periods when the radar was temporarily not acquiring data. Arrows indicate the melting-layer position.

Citation: Journal of Atmospheric and Oceanic Technology 37, 6; 10.1175/JTECH-D-19-0122.1

• Download Figure
• Download figure as PowerPoint slide

There are two features in the reflectivity datasets to highlight. First, there is a signature of the melting layer via an abrupt reflectivity change at a persistent height on each of the three days once precipitation is initiated, as we would expect for the cold rain process within the stratiform anvils associated with deep convective clouds. Second, there is a clear impact on the maximum detectable range in the presence of precipitation, due to the greatly increased hydrometeor absorption experienced by the radar beam. Additionally, note that because the PBL humidity on 13 April is less than half that on the previous days, the upper-tropospheric clouds appear to have a larger reflectivity as the beam absorption from water vapor is greatly reduced. In fact, for both of these reasons it is clear why DAR measurements from an airborne or spaceborne platform will offer a substantial increase in sampling precipitating clouds, as the hydrometeors and low humidity levels aloft will not extinguish the beam before it reaches the strongly reflective rain drops in the lower levels.

The lower panels in Fig. 3 are the retrieved, in-cloud humidity fields from the reflectivity measurements at 167 and 174.8 GHz (the latter are not shown). We utilize a humidity profile resolution of R = 180 m, which is a factor of 12 larger than the reflectivity resolution. The launch times of radiosondes from the ARM central facility are depicted with black dashed lines. Note that in-cloud humidity can only be retrieved where there are measurements at both frequencies, and since f₂ experiences more atmospheric absorption than f₁, the spatiotemporal coverage of humidity is slightly less than that for reflectivity at f₂.

A close scrutiny of the corresponding reflectivity and humidity plots in Fig. 3 reveals that some regions exhibit a correlation between spatial boundaries of dBZ and changes in ρ_υ. For instance, observe the marked depression in humidity correlated with and just below the melting layer position in periods with precipitation in Fig. 3c, or the temporal variations of magnitude >30% that are correlated with distinct bands of precipitation in the latter part of Fig. 3b. Comparisons of retrieved humidity profiles at the nominal 5 s radar time resolution with radiosonde measurements show that these variations are not physical. In fact, such features are a manifestation of the systematic bias from differential hydrometeor reflectivity, and are the reason for implementing the gradient-penalty regularization scheme. Specifically, if the characteristic diameter of the DSD changes quickly in space, we expect a large value of the second term in Eq. (7), which can significantly bias the retrieved value of ρ_υ. For instance, using the Lorenz–Mie calculations in Fig. 2 for ν = 4, if the characteristic diameter varies from 20 to 60 µm over 180 m, there would be a resulting DAR humidity bias of 5 g m⁻³.

While these particular variations in the retrieved humidity are not random in nature, we can reduce their effect on the accuracy of the measurement further by averaging vertical ρ_υ profiles in time. This will help to smooth out variations of the type seen in Fig. 3b, but not that associated with the melting layer in Fig. 3c. While this time-averaging procedure decreases the effective DAR ρ_υ time resolution, the situation will be much different when VIPR is deployed from an airborne platform. In this case, the fast aircraft motion allows VIPR to sample the spatial heterogeneities of a cloud scene much faster and therefore should permit a humidity measurement time resolution near the nominal value of 5 s. This is clear from low level of random noise evident in the humidity fields in Fig. 3, and from the fact that the error bars in the DAR-retrieved humidity are dominated by the introduction of the systematic error term $\mathsf{S}$_e, and have little contribution from the random measurement noise except where the SNR is low.

Because the time-varying systematic biases in ρ_υ are not random, we must account for covariance between adjacent time values in determining the resulting uncertainty. The recipe we use to compute time-averaged ρ_υ profiles involves first partitioning the 2D humidity fields into segments of duration T_avg = 10 min. Because of the sparsity of DAR datasets, it is often the case that there are many missing data points in each segment. Therefore, we require that each segment have a minimum of 10 measurements in order to record a final averaged humidity value. Given a set of humidity measurements at a given height z_i and between time t and t + T_avg, {ρ_υ(z_i, t₁), ρ_υ(z_i, t₂), …, ρ_υ(z_i, t_M)}, we compute the time-average mean and variance from ${\overline{\rho }}_{\upsilon }\left(z_i\right)=M^{-1}{{\displaystyle \sum }}_{j=1}^M{\rho }_{\upsilon }\left(z_i,t_j\right)$ and

where the ρ_υ time covariance matrix has elements ${\left[{\mathsf{S}}_{{\rho }_{\upsilon },i}\right]}_{jk}={\sigma }_{{\rho }_{\upsilon }}\left(z_i,t_j\right){\sigma }_{{\rho }_{\upsilon }}\left(z_i,t_k\right)\hspace{0.17em}\exp \left(-\left|t_j-t_k\right|/{\tau }_{\text{corr}}\right)$, ${\sigma }_{{\rho }_{\upsilon }}\left(z_i,t_k\right)$ is the uncertainty returned by the retrieval algorithm, and we set the exponential time-correlation constant to τ_corr = 1 min. Note that because of the imposed correlations in the time-averaging procedure, the improvement in humidity precision with averaging time is less than that for uncorrelated measurement averaging $\sqrt{T_{\text{avg}}/\mathrm{\Delta }t}$, where Δt = 5 s is the nominal temporal resolution.

The resulting comparisons of coincident DAR and RS humidity observations are presented in Fig. 4. The only profiles excluded from this figure are those in which the DAR profile spans less than 2 km. In these plots, the error bars correspond to the square root of the time-averaged humidity variance in Eq. (21) (i.e., 1σ error). These are the first-ever validated measurements of water vapor inside of clouds using an active remote sensing technique, and show the capability of VIPR to accurately profile in-cloud humidity not only in the PBL, but throughout the troposphere.

View Full Size

Comparisons of coincident radiosonde and DAR humidity profiles. The DAR profiles, which are initially retrieved with 5 s temporal resolution, are further averaged in time to 10 min resolution for comparison with radiosonde profiles. Error bars for the DAR measurements correspond to the square root of the variance defined in Eq. (21). For comparison, the left panel in the second row also shows the 10-min-averaged humidity profile retrieved using the simple two-frequency DAR method from Roy et al. (2018) (red line).

Citation: Journal of Atmospheric and Oceanic Technology 37, 6; 10.1175/JTECH-D-19-0122.1

• Download Figure
• Download figure as PowerPoint slide

While the coincident DAR/RS measurements make up a useful dataset for evaluating VIPR’s humidity accuracy, we would like to be able to use the remaining 10-min-averaged profiles as well. Therefore, we linearly interpolate the RS profiles in time at each height to the DAR temporal resolution T_avg, and note the observation that RS humidity profiles do not change appreciably in the intervals between launches (usually 0.5–2 h). Furthermore, we smooth the RS profiles in height to make the effective resolution equal to that of the DAR profiles by performing a convolution with a box of length R. The resulting comparison between remotely sensed and in situ humidity is presented in Fig. 5. The scatterplot in Fig. 5a has an associated correlation coefficient r = 0.96 and RMSE of 0.8 g m⁻³. The corresponding cumulative distribution of the absolute value of DAR humidity bias relative to the interpolated RS profiles is shown in Fig. 5b. Note that we describe bias in absolute terms (i.e., in units of g m⁻³) and not in terms of relative error, because the sensitivity of a differential absorption measurement is independent of the magnitude of ρ_υ (Roy et al. 2018). Therefore, this approach is a more appropriate assessment of the DAR accuracy. We find that biases are below 1 g m⁻³ 84% of the time, and below 2 g m⁻³ 98% of the time.

View Full Size

Summary statistics of in-cloud humidity profiling measurements performed in the ARM deployment. (a) 10 min averaged DAR humidity profiles (180 m resolution) vs temporally interpolated RS profiles. Measurements from different days are shown in different colors. Fitting a line to the data gives a slope of 1.01 and an intercept of 0.13 g m⁻³, highlighting the close agreement between remotely sensed and in situ measurements. (b) Cumulative probability distribution of absolute value of DAR biases relative to RS profiles (black line) and retrieval precision (blue line) for 10 min averaged humidity profiles. (c) Tukey box plot of height-binned biases.

Citation: Journal of Atmospheric and Oceanic Technology 37, 6; 10.1175/JTECH-D-19-0122.1

• Download Figure
• Download figure as PowerPoint slide

In Fig. 5c we bin the measurements by height to assess the bias dependence in various levels of the atmosphere, and present the resulting binned distributions as boxplots. It is interesting to note that the variance of the bias distribution decreases with height, which at first seems counterintuitive given that measurement SNR is largest in the lowest levels of the atmosphere. However, recall that the strongest biases in the DAR humidity measurement arise from sharp spatial gradients of the characteristic DSD diameter, which are likely to occur in the lower atmosphere (e.g., melting layer, localized precipitation populations). Nevertheless, the near-zero median bias in each vertical level reveals that the temporal-averaging approach successfully mitigates systematic DAR biases. In future DAR measurement systems, it will be interesting to explore the addition of a third transmission frequency outside of the 167–174.8 GHz band to independently measure the frequency-dependent hydrometeor scattering contributions to the differential absorption signal.

b. Surface-to-cloud-base column retrievals

We utilize all of the cloud datasets that do not feature strong precipitation to perform a measurement of IWV between the radar and cloud base. An example cloud scene and resulting IWV time series is presented in Fig. 6. In all of the DAR IWV measurements described in this section, we do not perform any additional averaging in time, and so the temporal resolution is 5 s. For the RS time series (blue line in Fig. 6b), we again interpolate the profiles linearly in time between launches at each height, and integrate the resulting profiles up to the cloud-base height determined by radar measurement. For this upper-tropospheric ice-cloud case, the remotely sensed and in situ IWV time series agree very well, even with the assumption of pure Rayleigh backscattering from spherical liquid hydrometeors. We note here that these DAR retrievals utilize the value for the radar calibration ratio C(f₁)/C(f₂) determined from the aggregate analysis described below.

View Full Size

(a) Observed reflectivity at f₁ = 167 GHz and (b) corresponding retrieved surface-to-cloud-base IWV with RS-integrated value (blue line) on 5–6 Apr 2019.

Citation: Journal of Atmospheric and Oceanic Technology 37, 6; 10.1175/JTECH-D-19-0122.1

• Download Figure
• Download figure as PowerPoint slide

For the case in Fig. 6, a radiosonde was launched at 2330 UTC, making the potential error from linear interpolation in time small. However, many of the datasets used for IWV measurements do not feature the same density of RS launches as those used for in-cloud profiling, since these were the highest-priority measurements. Fortunately, the ARM RL performs measurements of water vapor profiles in clear air every 10 min, and provides a valuable resource for validation of DAR surface-to-cloud-base IWV measurements. However, it is often the case that the RL profiles do not extend all the way to the cloud-base height detected by VIPR. Thus, we adopt an approach that merges the RS and RL water vapor information to interpolate the RS-integrated IWV in time with the RL temporal resolution. Specifically, given an RL humidity profile we first filter out values with uncertainties in excess of 2 g m⁻³, resulting in a maximum RL measurement height z_RL. Then, we integrate the RL and temporally interpolated RS profiles at each 10 min interval up to z_RL, and take the ratio IWV_RL/IWV_RS. This provides a correction factor that can be applied to the temporally interpolated RS profiles for a more accurate comparison with DAR. An example time series of this ratio is shown in Fig. 7a, where we plot both the RL and DAR ratios with respect to the RS IWV. In this case, the nearest RS launches occurred at 1724 and 2039 UTC. The strong agreement between RL and DAR reveal the utility of using this ancillary measurement, as otherwise we would incorrectly determine the DAR IWV to be dry-biased by more than 10% for much of this measurement window.

View Full Size

(a) Time series of the ratio of remotely sensed to RS-integrated IWV for measurements on 6 Apr 2019, where radiosondes were launched at 1724 and 2039 UTC. (b) Comparison of DAR surface-to-cloud-base IWV with RS–RL-merged dataset. The DAR time resolution is 5 s. Measurements from different days are shown in different colors. (c) Cumulative probability distributions for DAR bias (black line) with respect to RS–RL values and 1σ retrieval error (blue line).

Citation: Journal of Atmospheric and Oceanic Technology 37, 6; 10.1175/JTECH-D-19-0122.1

• Download Figure
• Download figure as PowerPoint slide

Using the RS–RL-merged IWV dataset as ground truth, we aggregate DAR measurements from many days between 2 and 14 April 2019, and perform the modified retrieval algorithm described in section 3c. To determine the value of C(f₁)/C(f₂), we calculate the mean bias for each measurement day before averaging these individual biases together to get a final value. This ensures that datasets with many more time points are not weighted disproportionately, which is desirable because most of the variation in DAR IWV is expected to be due to systematic errors that will depend on the type of cloud measured and its height, as well as radar parameters that could vary on the time scale of days. The chosen value for 10 log₁₀C(f₁)/C(f₂) = 3.6 dB is the one that yields an average bias of zero, and is close to those measured on three separate days (4.3, 5.1, and 3.5 dB) with different humidity conditions and two different-sized calibration spheres.

The scatterplot of DAR versus RS–RL IWV values is shown in Fig. 7b, where measurements from different days are depicted with different colors. The large dynamic range of values sampled is a result of measuring clouds with different base levels, and of measuring on different days with varying absolute humidities. The correlation coefficient for this comparison is r = 1.00 with a RMSE of 1.2 mm, and the slope of a best-fit line to the measurements is 1.1. There is a clear trend in the bias in this dataset that is negative for low IWV and positive for high IWV values. We note that as in the case of in-cloud profiling, the sensitivity of the DAR measurement to IWV is set in absolute terms (i.e., in units of mm), and not in terms of relative error. Therefore, there is no reason to expect higher absolute biases for higher IWV values. The resulting cumulative distributions for both the absolute value of measurement bias and the estimated humidity uncertainty from the retrieval (1σ error) are shown in Fig. 7c. Recall that the uncertainty estimates include contributions from the random error associated with the radar power measurements and systematic error associated with the frequency-dependence of the cloud reflectivity. From Fig. 7c we find that the median 1σ error is 0.25 mm and the median bias relative to the RS–RL values is 1.1 mm. Furthermore, biases are less than 1 mm 46% of the time, and less than 2 mm 95% of the time.