Quality Control of Doppler Spectra from a Vertically Pointing, S-Band Profiling Radar
1. Introduction
Radars have been used to study the atmospheric boundary layer (ABL) since at least the 1960s (Atlas et al. 1966; Hardy et al. 1966; Kropfli et al. 1968; Hardy and Katz 1969; Lane 1969; Gossard 1990; Tanamachi et al. 2019; Kotthaus et al. 2023) using backscatter from inhomogeneities in the refractive index. Radars have the ability to detect wind and turbulence and can be used to gauge the relative stability of atmospheric layers (Gage and Balsley 1978). Studies have demonstrated that radars with 10-cm (i.e., S band) or longer wavelengths are superior for the detection of Bragg scattering (Knight and Miller 1998; Ottersten 1969; Ralph 1995). Bragg scattering occurs when “electromagnetic waves impinge on regularly spaced objects or regions of air with different indices of refraction leading to constructive interference between the scattered waves” (Rauber and Nesbitt 2018). Therefore, Bragg scattering signifies turbulent mixing of air with different temperature and moisture characteristics, which cause variation in indices of refraction (Rauber and Nesbitt 2018). Constructive interference of radar signals also occurs when scatterers are located at distances equal to half the radar wavelength (Wolff 1998; Rauber and Nesbitt 2018). Ralph (1995) determined the typical reflectivity thresholds between Bragg scattering and Rayleigh scattering at various radar wavelengths, affirming that Bragg scatter is more frequently detected by S-band radars than by C- and X-band radars.
One type of S-band radar used to monitor the growth of the boundary layer is a frequency-modulated, continuous-wave (FMCW) radar. The phrase “continuous wave” means the radar uses a high duty cycle, transmitting at or near 100% of the time (Richter 1969; İnce et al. 2003; Waldinger 2018). As a result, more energy is incident on scatterers than with a conventional pulsed radar, yielding high radar sensitivity (Richter 1969; İnce et al. 2003; Waldinger 2018). FMCW systems transmit a varying linear frequency-modulated waveform with a long pulse repetition period (Richter 1969; İnce et al. 2003; Waldinger et al. 2017). The received echoes are a delayed version of the transmitted waveform, and when the two signals are combined, the resulting beat frequency is proportional to the range of the target (Richter 1969; İnce et al. 2003; Waldinger et al. 2017). The beat frequency signals are recorded and transformed into pulsed radar-like echoes using a Fourier transform (Eaton et al. 1995; İnce et al. 2003; Waldinger et al. 2017).
S-band FMCW radars have been used to monitor the boundary layer for more than four decades (Gossard 1990 and references therein; İnce et al. 2003). These radars are used to study boundary layer morphology and evolution and to measure the refractive index structure with high spatial and temporal resolution (Chadwick et al. 1976; Waldinger et al. 2017). S-band radars can be used to determine the heights of the convective thermals in the boundary layer (Melnikov and Zrnić 2017), which aids in the detection of the boundary layer height. The use of these types of Doppler radars in field campaigns can aid in understanding the vertical mesoscale structure of the atmosphere (Gage and Balsley 1978).
The radar used in this study is the University of Massachusetts (UMass) FMCW radar (UMass FMCW; İnce et al. 2003). UMass FMCW (Fig. 1), developed at the Microwave Remote Sensing Laboratory (MIRSL; Eaton et al. 1995; İnce et al. 2000, 2003), is an S-band, vertically pointing, single-polarized, pulse compression radar with a 3-dB beam width of 3.5°. This radar is mounted on a truck for mobility and is designed to be deployed at a fixed location continuously collecting observations for long periods of time (Tanamachi et al. 2019). UMass FMCW uses a pair of 2.4-m-diameter parabolic dish antennas, one for transmission and one for reception, each with 34-dB gain (İnce et al. 2003). UMass FMCW has high temporal (∼16 s) and vertical (∼5 m) resolution (Tanamachi et al. 2019).
UMass FMCW at its 2016 VORTEX-SE deployment location near Belle Mina. Photo by the second author.
Citation: Journal of Atmospheric and Oceanic Technology 41, 4; 10.1175/JTECH-D-23-0099.1
UMass FMCW was deployed to monitor the growth of the CBL over northern Alabama during VORTEX–Southeast (VORTEX-SE; Rasmussen 2015; Tanamachi et al. 2019). The purpose of VORTEX-SE was to investigate Southeast U.S. tornadoes, their environments, and their societal impacts (Koch 2016; Rasmussen and Koch 2016). Differences between the Southeast and Great Plains environments have implications for the growth and evolution of the CBL, which in turn affects the atmosphere’s ability to generate and sustain severe thunderstorms. During the 2016 VORTEX-SE field campaign, UMass FMCW was deployed at the Tennessee Valley Research and Extension Center near Belle Mina, Alabama (34.6904°N, 86.8815°W), and operated almost continuously from 7 March to 30 April 2016. This site was selected because it is relatively free from clutter and collocated with other meteorological instruments.
The objective of this deployment was to collect high temporal resolutions of the boundary layer structure over northern Alabama and predict the timing of destabilization and convective initiation. The radar was configured to collect 256 frequency-modulated sweeps over a 1.34-s interval to generate a Doppler spectrum. Twelve spectra were then averaged every 16.1 s. From these averaged spectra, the moments (reflectivity Z, radial velocity Vr, and spectrum width συ) were calculated. Each spectral profile in the 2016 dataset comprises 1024 spectra at 5-m height intervals from 0 to 5.1 km above radar level (ARL). In total, approximately 14 000 spectral profiles were collected over the 2016 field campaign (Frasier et al. 2016). UMass FMCW data from VORTEX-SE 2016 are openly available through the Earth Observing Laboratory (EOL) archive (https://data.eol.ucar.edu/dataset/527.016).
Unfortunately, the Doppler spectra collected in 2016 were contaminated by “spurs,” or spurious spectral peaks, caused by high-voltage switching power supplies in the traveling wave tube amplifier (Waldinger 2018). An example of such a contaminated spectral profile is shown in Fig. 2a. The spurs appear as small, bright, horizontally elongated double peaks. Electronic spurs can be distinguished from naturally occurring peaks in the spectra (e.g., a bird or bug) because they are time continuous and occur at nearly constant heights and amplitudes owing to their electronic origin, whereas echoes from birds and bugs vary widely in height and spectral power. However, spurs vary in frequency from sweep to sweep, defying fixed notch filtering. Spectral peaks at 0 m s−1 Doppler velocity are caused by ground clutter and antenna leakage (Waldinger 2018). Tanamachi et al. (2019) employed a median filtering technique (Fig. 2b) to eliminate most of the spurs, but the largest ones were still present, which degraded the quality of radar moments (e.g., Z, Vr, and συ) and hindered further analysis of these data (e.g., hydrometeor classification and boundary layer height tracking). (The traveling wave tube amplifier was replaced with a solid-state amplifier in 2017, which eliminated the spurs.) This paper describes a unique data quality issue affecting 2016 UMass FMCW Doppler spectra and a novel image processing–based solution that allowed much of the underlying information to be recovered. In this study, we introduce the data rescue method to the meteorological community, in the hopes our solution may be applicable to other similarly afflicted datasets.
Spectral profile for a clear-air case at 0001 UTC 31 Mar 2016, shortly after local sunset: (a) raw spectral power (dBZ); (b) as in (a), but after application of the median filter; and (c) as in (a), but after application of the in-painting method. In (a), two particularly problematic large spurs at 3.9 and 1.3 km AGL are labeled (yellow arrows), as are several examples of smaller spurs (white arrows) and bioscatterers (orange brace). The reader is invited to inspect the corresponding features in (b) and (c).
Citation: Journal of Atmospheric and Oceanic Technology 41, 4; 10.1175/JTECH-D-23-0099.1
2. Methodology
a. Description of in-painting algorithm
The method selected to remove and interpolate across the spurs was the Chan et al. (2017) “in-painting” image restoration method, which was developed to remove noise in 2D images. The in-painting method is a denoising algorithm that converges to a fixed point. This technique is performed over an entire image, simultaneously masking all outliers and filling them in according to surrounding data. The version adapted for the UMass FMCW radar data can be found at https://github.com/sbeverid/inpainting.
Simplified example of the in-painting process using a 5 × 5 image with (a) a peak in the center pixel, (b) implementation of a mask around the peak, and (c) the output image with the peak filled in according to surrounding data. In this example, (a) corresponds to y, a corrupt image, and (c) is
Citation: Journal of Atmospheric and Oceanic Technology 41, 4; 10.1175/JTECH-D-23-0099.1
In essence, each UMass FMCW spectral profile (e.g., Figs. 2a and 5a) was treated as a 256 × 1024 pixel image, that is, the corrupt image y. The in-painting code requires a first guess for the locations of spurs in an image, which were provided by labeling local extrema in the Laplacian of the spectral profile. After these extrema were masked, the range of values in the image (from −50.5 to +47.6 dBZ) was scaled to (0, 1), as required by the in-painting code (Fig. 5b). The in-painting code then filled in the masked area according to surrounding data. In cases in which the holes created by the masking procedure were too large for the in-painting method to fill, the masked 256 × 1024 spectral profile was downsampled to a 128 × 512 pixel image, to which the in-painting code was then applied. The resulting spectral profile was then scaled back up to the original size and dynamic range. Last, regardless of whether the downscaling substep occurred or not, those portions of the spectral profiles that were not masked in the Laplacian filtering step were restored. The resulting spectral profiles are hereafter referred to as the “in-painted” spectral profiles (e.g., Figs. 2c, 4, and 5c).
A flow diagram of the data rescue processes used in this study.
Citation: Journal of Atmospheric and Oceanic Technology 41, 4; 10.1175/JTECH-D-23-0099.1
Example of the in-painting process for a clear-air Doppler spectral profile at 0000 UTC 31 Mar 2016 with (a) the raw spectral profile containing the peaks, (b) implementation of a mask around the peaks with the normalized dynamic range, and (c) the output in-painted image with the peaks filled in according to surrounding data.
Citation: Journal of Atmospheric and Oceanic Technology 41, 4; 10.1175/JTECH-D-23-0099.1
The in-painting method is superior to simplistic methods, such as linear interpolation, because the denoiser is more powerful. Denoisers have an image prior, either implicitly or explicitly defined, that ensures the reconstructed image is close to the distribution of images, whereas simple linear interpolation does not. A limitation regarding the in-painting method is that completely saturated pixels, such as in instances where aircraft pass over the radar, cannot be fixed by the in-painting method because the noise floor is raised. In the 2016 dataset, this was not a common issue and cases were easily eliminated by inspection.
b. Spectral power differences
To qualitatively evaluate the performance of the in-painting method relative to the median filter, spectral power differences were calculated and plotted. Multiple three-panel plots were produced containing spectral power differences for time indices 0, 100, and 200, coinciding with approximately the beginning, middle, and end of a given hour. The authors believe that these time indices were sufficient to give an overview of how well the in-painting method performed compared to the median filter. To visualize the spectral power differences, histograms were produced of spectral profile pixels for each 5-dB spectral power difference bin. The bins (−5, 0 dB) and (0, 5 dB) were masked because their frequencies dominated the histograms by several orders of magnitude.
c. Regenerating UMass FMCW moments using the coherent power (CP) technique
The moments of Doppler spectra are the basis of three primary weather radar variables: Z, Vr, and συ. When the quality of the Doppler spectra is compromised, it impacts the quality of these derived spectral moments. Once the quality of the Doppler spectra was improved, we generated a new version of UMass FMCW spectral moments from the in-painted spectra.
The velocities of targets within a radar sample volume vary due to azimuthal shear of radial velocity, variations in target fall speeds, small-scale turbulent motions (Rauber and Nesbitt 2018 and references therein), and other factors (Doviak and Zrnić 1993). The radial velocity assigned to that sample volume is a representation of the power-weighted average target motion over all targets within the sample volume. When the different phase shifts for all of the different pulse pairs taken within a given sample volume are aggregated, a power-weighted spectrum of velocities, that is, a Doppler spectrum, is produced.
Traditionally, radar reflectivity is calculated using noise-subtracted power measurements (Ulaby et al. 1982; Doviak and Zrnić 1993; Pazmany and Haimov 2018). In the original UMass FMCW moments data, the noise floor was assumed to be the median power of the spectrum in clear-air conditions, or in the case of precipitation, the noise floor was estimated from laboratory measurements. In the 2016 data, the traveling wave tube (TWT) amplifier introduced the large spurs in the Doppler spectra, and their spectral leakage artificially elevated the noise floor. This phenomenon can be seen in the Doppler spectra (Figs. 2a and 6a) as a spectrumwide enhancement of received power, or stripes extending horizontally from the large spurs. Unwanted striping was present in the reflectivity and signal-to-noise ratio (SNR) fields, even after application of in-painting, because the elevated noise floor associated with the largest spurs was not removed by the in-painting method (Figs. 2c and 6c).
(a)–(c) Spectral profiles as in Fig. 2 and (d)–(f) spectral power differences, but for a clear-air case on 0058 UTC 14 Mar 2016, shortly after local sunset. In (a), two large spurs at 3.9 and 1.3 km AGL are labeled, as are several exemplary smaller spurs, bioscatterers, and spread-spectrum turbulence marking the top of the inversion layer at 0.8 km AGL.
Citation: Journal of Atmospheric and Oceanic Technology 41, 4; 10.1175/JTECH-D-23-0099.1
The CP technique has been shown to be particularly useful in low-SNR scenarios with a large number of samples, such as the UMass FMCW detections of Bragg scatter, based on N = 256 samples per profile. The reader is referred to Pazmany and Haimov (2018) for a comprehensive explanation of this technique and proof of convergence.
d. Boundary layer height detection algorithm
Since one of the objectives of VORTEX-SE was to quantitatively assess boundary layer growth, an objective method to measure the boundary layer height in reflectivity observations was needed. Deriving boundary layer height from the UMass FMCW served as a metric to evaluate the improvement of the in-painted Doppler spectra on downstream products.
The method selected to estimate the CBL depth was the boundary layer height detection algorithm of Lange et al. (2015), employing an extended Kalman filter (EKF). The EKF is a recursive adaptive filter that parameterizes the time-changing shape of the CBL-to-free-atmosphere (FT) transition reflectivity profile [hence, the CBL height (CBLH) as well as the CBL accessory parameters] via a state vector and statistical parameters that are updated at each succeeding discrete-time observation tk. The EKF relies on two stochastic models that assimilate the statistics of the estimation problem: 1) the state-vector model and 2) the measurement model.
The first item models the state-vector time transition from tk to tk+1 upon the assumption of a state-vector noise covariance matrix input by the user. In other words, the state-vector model describes the assumed time-changing behavior of the CBL atmospheric parameters described by the state vector.
The second model relates the state vector to the observation vector (i.e., the radar reflectivity measurements). Because of the relatively abrupt CBL-to-FT transition, an error-function-like shape is roughly assumed as the measurement model. Additionally, observation noise is also assimilated into the measurement model via the noise covariance matrix. Measurement noise is essentially related to the SNR.
Similar EKF approaches have also been used for CBLH estimation in lidar (laser radar) and to estimate the CBLH in synergy with, for example, microwave radiometers (Lange et al. 2015; Banks et al. 2015; Da Silva et al. 2022). The processing steps are as follows [the reader is directed to Lange et al. (2015) for additional information regarding the boundary layer height detection algorithm beyond what is discussed below].
The first step in the boundary layer height detection algorithm is a preprocessing step to generate a “clean” time–height reflectivity profile where Bragg scattering dominates: A median filter is used to remove noise caused by Rayleigh scatterers, such as bioscatterers, and instrumentation effects. The resulting reflectivity image is an approximation of what the radar would see if only Bragg scattering was present, from which the CBL depth can then be estimated.
The next step is to run the EKF. The EKF requires initial guesses for the a priori state vector (i.e., the previous time step’s reflectivity profile), a priori state-vector error covariance matrix factor (which models the user’s uncertainty of the state-vector initial guess), atmospheric state-noise covariance matrix factor, and the bounds of the filter, which delineate the estimated start and end ranges of the CBL and the FT intervals at each discrete time tk. Last, the measurement noise covariance matrix is computed over uniformly spaced height intervals from the estimated SNR at each tk under the assumption of ergodicity. With each iteration of the EKF, the filter minimizes the mean-square error over time between the measured and the predicted reflectivity so that a new, a posteriori estimate of the CBL height is generated.
One limitation to the boundary layer height detection algorithm is that—for enhanced convergence—the user assumes relatively slowly varying quantities. For the same reason, the search space of the EKF is limited by error covariances, so there are no large jumps in the boundary layer height. Tracking the stable boundary layer and elevated residual layer (Saeed et al. 2016) was beyond the scope of this study because the focus of VORTEX-SE was on the CBL.
To quantify the improvement of the algorithm with the in-painted moments, histograms were produced of the frequency of CBL height values. These are discussed in the next section.
3. Results
a. Spectral power differences
When examining the spectral power difference plots (Figs. 6d–f), areas that are brighter and have higher spectral power difference values indicate an improvement in the removal of the spurs (i.e., the spurs were more effectively flattened by the in-painting method than the median filter method). Regions that are darker with negative spectral power indicate spurs that were more effectively removed by the median filter than by the in-painting method. Additionally, for the in-painting method to perform better than the median filter method, we expect a right-tailed distribution in the spectral power difference histograms.
1) Clear air
For clear-air (i.e., nonprecipitation containing) spectra, the median filter method did a good job at removing the spurs, particularly the smaller peaks, as can be seen from the raw minus median-filtered panels of the spectral power difference plots (Fig. 6d). However, the in-painting method better removed the larger peaks (Fig. 6e), as indicated by the greater spectral power differences in those areas. There were negative spectral power difference values when the in-painted spectra were subtracted from the median-filtered spectra, indicating the in-painting method did not remove the small peaks as well as the median filter method (Fig. 6f).
Exemplary spectral power difference histograms for the clear-air spectral profile shown in Fig. 6 are presented in Fig. 7. Both the median-filtered and in-painting methods reduce the magnitudes of raw spectral peaks, as evidenced by the right-tailed distributions of spectral power differences (Fig. 7). Both methods change similar numbers of spectral gates (pixels) by >∥5 dB∥ (∼2%). However, the distributions of those changes are slightly different. The median filter removes smaller spectral peaks (i.e., those with maximum differences of ∼<20 dB) more effectively than the in-painting method, whereas the in-painting method more effectively removes larger spectral peaks, including the problematic large spurs (i.e., those with maximum differences > 35 dB) (Fig. 6). This notably different behavior between the two filtering methods is evidenced in Fig. 7 by the fact that the blue bars are taller than the red bars for spectral power differences up to approximately 20 dB. Above this 20-dB threshold, the trend reverses. Negative spectral power differences (<0 dB) indicate that the in-painting method restores some small spurs that would otherwise be smoothed out by the median filter. While the median filter smooths out small spurs, which is desirable, it also smooths out low-amplitude spectral features that we may wish to retain. Examples of such features include monodisperse precipitation spectral peaks as might be encountered in the presence of size sorting, or bioscatterer activity (e.g., Fig. 6a). In contrast, the in-painting method restores some of these features in unmasked regions.
Spectral power difference histograms for the Doppler spectra shown in Fig. 6, for a clear-air case. Histogram bins are each 5 dB wide, but the bars have been plotted with different widths to aid readability. The (−5, 0 dB] and (0, +5 dB] bins have been omitted because they dominate the distribution by two orders of magnitude.
Citation: Journal of Atmospheric and Oceanic Technology 41, 4; 10.1175/JTECH-D-23-0099.1
2) Precipitation
In precipitation cases, an example of which is shown in Fig. 8, the spurs have a smaller impact because power associated with most of the spurs does not exceed that of the rain. As in the clear-air case, the in-painting method better removes the large spurs than the median filtering method, particularly in regions of the spectral profiles unaffected by rain (Fig. 8). Small peaks are not as well removed with the in-painting method because they evade detection during the initial Lagrangian peak-finding step (Fig. 4) and therefore go unmasked.
As in Fig. 6, but for precipitation on 14 Mar 2016. Note that the Doppler velocity has not been dealiased.
Citation: Journal of Atmospheric and Oceanic Technology 41, 4; 10.1175/JTECH-D-23-0099.1
The spectral difference histograms for precipitation cases also have right-tailed distributions (e.g., Fig. 9). However, when compared to the corresponding histograms for clear-air cases, fewer spectral gates (0.5%–1.0%) are changed by >∥5 dB∥. This is because, in many instances, the spurs have lower power than, and are therefore obscured by, the rain signal (e.g., Fig. 8). Negative spectral differences arise mainly from differences along the zero line and the edges of the peaks (Fig. 9). This provides further evidence that the in-painting performs well on large spurs but does not handle smaller spurs as well. However, since the smaller spurs have less detrimental impact on our subsequent analysis than the large spurs, we consider this result acceptable. A comparison of the relative effects of the median filter to those of the in-painting method is summarized in Table 1.
As in Fig. 7, but for the Doppler spectra shown in Fig. 8, during precipitation. Note that the y-axis limits differ from those of Fig. 7.
Citation: Journal of Atmospheric and Oceanic Technology 41, 4; 10.1175/JTECH-D-23-0099.1
Summary of the two spectral processing methods’ effects on Doppler spectra.
b. Moments
The moments generated from the median-filtered Doppler spectra have spurious reflectivity and SNR peaks at constant altitudes caused by the largest spurs (bold horizontal lines at approximately 1.3, 2.6, and 3.9 km in Figs. 10a–d). The in-painting method removed some of the unwanted horizontal striping, resulting in cleaner derived moments (Figs. 10e–h and 11e–h). Removal of the 1.3-km spur is particularly important for correct estimation of the CBL height (see next section). In addition, the use of coherent reflectivity and SNC, which are not noise floor-dependent, helped reduce the effect of receiver saturation between 0 and 400 m AGL that we dubbed “horizon glow” (Figs. 10a,e). The abovementioned effects are also evident when comparing reflectivity (Fig. 12) and coherent reflectivity (Fig. 13) fields for the entire 2016 VORTEX-SE campaign.
Clear-air moments from 14 Mar 2016 that were calculated from the (a)–(d) median-filtered Doppler spectra and (e)–(h) in-painted Doppler spectra. In (a), the spurious echoes at 1.3 and 3.9 km AGL are annotated, as are horizon glow, a layer of bioscatterers, and the decaying residual layer. Note the almost complete removal of the spurious echo at 1.3 km AGL in all fields in (e)–(h).
Citation: Journal of Atmospheric and Oceanic Technology 41, 4; 10.1175/JTECH-D-23-0099.1
As in Fig. 10, but for precipitation on 14 Mar 2016. Note the mean radial (vertical) velocity field has not been dealiased in (b) and (f).
Citation: Journal of Atmospheric and Oceanic Technology 41, 4; 10.1175/JTECH-D-23-0099.1
Reflectivity (dBZ) from UMass FMCW observations taken during the 2016 VORTEX-SE field campaign (from Tanamachi et al. 2019).
Citation: Journal of Atmospheric and Oceanic Technology 41, 4; 10.1175/JTECH-D-23-0099.1
As in Fig. 12 but for coherent reflectivity (dBZ). Note the nearly complete removal of the persistent, spurious reflectivity maximum at 1.3 km AGL and reduction of horizon glow at and below about 400 m AGL, relative to Fig. 12.
Citation: Journal of Atmospheric and Oceanic Technology 41, 4; 10.1175/JTECH-D-23-0099.1
c. Boundary layer height detection
Tanamachi et al. (2019) noted that the Lange et al. (2015) automated boundary layer height detection algorithm struggled to identify the top of the boundary layer in the presence of the spurious reflectivity peaks caused by the spurs in the Doppler spectra, even after median filtering was applied. Because the EKF step of the algorithm relies on the sharp CBL-to-FT transition and operates purely on the reflectivity field (section 2d), the algorithm misidentified the spurious reflectivity peaks, particularly the one at 1.3 km AGL, as the top of the CBL (e.g., Fig. 14a). This feature so confounded the detection algorithm that attempts to apply it to the entire 2016 dataset (Tanamachi et al. 2019) were abandoned. The CBL tracking algorithm tended to “latch on” to the spurious 1.3 km AGL reflectivity peak any time that the top of the Bragg scatter layer approached it (e.g., Fig. 14), requiring manual restart of the filter after the two features diverged and defeating the purpose of automation.
Boundary layer height detection algorithm for a clear-air CBL on 31 Mar 2016 for (a) median-filtered reflectivity factor and (b) in-painted coherent reflectivity factor. Note how the estimated boundary layer height (pink dots) “latches on” to the spurious reflectivity maximum at 1.3 km AGL in the median-filtered case (a) but not the in-painted case (b).
Citation: Journal of Atmospheric and Oceanic Technology 41, 4; 10.1175/JTECH-D-23-0099.1
The combined application of the in-painting and CP technique methods reduced the amplitudes of the large spurs, leaving the Bragg scatter at the top of the CBL as the dominant signal in clear air. A consistent diurnal CBL growth and decay cycle can be seen in the coherent reflectivity time–height series (Fig. 13). The boundary layer height detection algorithm was applied to the coherent reflectivity field. A small modification was made to skip the EKF step of the algorithm if rain was present (i.e., mean reflectivity in the column was >5 dBZ) and instead simply propagate the previous value of boundary layer height to the next time step. Boundary layer heights were flagged if this rain criterion was met, or if the identification occurred after sunset (when, under quiescent conditions, the CBL typically collapses and is replaced by an inversion).
Qualitatively, the results were superior to those obtained using the median-filtered reflectivity on account of the larger Bragg-to-spur reflectivity contrast ratio featured by the in-painting/CP combination: Retrieval of the CBL height was consistently successful on 50 out of 54 days of the 2016 field campaign. Under clear-sky conditions (not cloudy and not rainy), the CBL grew at an average rate of 1.6 m min−1 between 1 and 9 h after sunrise (Fig. 15), leveled off between 9 and 10 h after sunrise, and then decayed at a rate of −1.3 m min−1 until 15 h after sunrise (Fig. 15). For context, the day length varied from 11 h 40 min on 7 March 2016 (the first day of the experiment) to 13 h 33 min on 30 April 2016 (the last day of the experiment).
Composite of clear-sky CBL evolution at Belle Mina for 7 Mar–30 Apr 2016, during the VORTEX-SE 2016 field campaign. Gray lines represent the CBL growth on individual days; the mean CBL height (heavy black line) along with the 25th (dark red line) and 75th (dark blue line) percentiles is plotted. The CBL heights were retrieved from UMass FMCW coherent reflectivity data using the Lange et al. (2015) EKF-based method.
Citation: Journal of Atmospheric and Oceanic Technology 41, 4; 10.1175/JTECH-D-23-0099.1
There were instances where the boundary layer height detection algorithm failed to accurately detect the decaying CBL and instead began tracking the residual layer. It may be possible to tag boundary layer retrievals in the algorithm with a quality flag based on factors such as time of day, net radiative flux sign, or presence of clouds and precipitation. However, such a data quality flag is beyond scope of this study. Synergistic remote sensing CBLH retrievals involving an EKF detection algorithm and avoidance of the residual layer have recently been developed (Da Silva et al. 2022).
To validate the retrieved CBL heights, we compared these data to those derived from radiosonde data using the parcel method (Holzworth 1964; Seibert et al. 2000). The NOAA Atmospheric Turbulence and Diffusion Division (ATDD) group launched Graw DFM-09 radiosondes from Belle Mina during VORTEX–Southeast intensive observation periods (IOPs) (Lee et al. 2016). Of 26 available NOAA ATDD soundings, only 9 were available within 30 min of a valid FMCW-derived boundary layer height during daylight hours (roughly 1200–2359 UTC at Belle Mina). The dearth of available soundings on IOP days is due primarily to rain, whose reflectivity greatly exceeds that of Bragg scatter and thereby prevents EKF-based boundary layer height retrieval from UMass FMCW observations. Radiosondes, on the other hand, can be launched in almost any conditions.
These nine soundings originated on three different IOP days (one on 13 March, three on 31 March, and five on 27 April) (Fig. 16). To account for the presence of turbulent plumes in the CBL and the horizontal drift of the radiosonde away from Belle Mina as it ascended through the boundary layer, we computed a 30-min average of the FMCW-derived boundary layer height, centered around the time of the radiosonde launch. The resulting Pearson correlation coefficient score for these two datasets was r = 0.38. While this correlation is admittedly weak, it is comparable with results from other experiments comparing boundary layer heights derived from remote sensing methods to those from radiosoundings (Seibert et al. 2000). Augmented methods using, for example, fuzzy logic (Bianco and Wilczak 2002) or synergistic combinations of input from multiple sensors (e.g., Seibert et al. 2000) can produce superior correlation coefficients, but these were not incorporated in this study. Separately, a synergistic method incorporating UMass FMCW, EKF-retrieved CBL heights as one of several estimates (Da Silva et al. 2022) resulted in superior CBL tracking across a wider range of conditions and climate regimes.
Scatterplot of UMass FMCW-derived boundary layer heights vs those derived from NOAA ATDD soundings at Belle Mina.
Citation: Journal of Atmospheric and Oceanic Technology 41, 4; 10.1175/JTECH-D-23-0099.1
d. Scatterer classification
As reported in Tanamachi et al. (2019), a rudimentary scatterer classification algorithm, decomposing the radar moments fields into “water-based scatterers” (WBS; i.e., precipitation and bioscatterers) and “clear air” categories, was applied to the original, median-filtered 2016 moments. [Note that Tanamachi et al. (2019) used categories “precipitation” and “nonprecipitation”; in this study, we have renamed these categories “WBS” and “clear air,” respectively, to more accurately reflect the scatterer types present in those gates.] It was found that the horizontal stripes confounded the scatterer classification algorithm, resulting in overclassification of gates as WBS. While only 7.6% of 2017 UMass FMCW observations (which were collected after the TWT amplifier was replaced with a solid-state amplifier, eliminating the spurs) (Waldinger 2018; Villalonga et al. 2020) were classified as WBS, 23.6% of 2016 UMass FMCW observations were classified as WBS. Because the two datasets were collected by the same instrument, in comparable climatic conditions and similar seasonal periods (March–April), we expected the two datasets to exhibit similar percentages of WBS-classified gates. The weights used in the fuzzy logic algorithm were generated from 2017 FMCW data only, after UMass FMCW’s TWT amplifier was replaced with a solid-state amplifier. Overall, the reflectivity associated with precipitation and bioscatterers in the 2017 UMass FMCW data was lower than that in the 2016 data (Waldinger 2018), which partially explains the difference. Applying this same scatterer classification algorithm to the 2016 UMass FMCW observations that were reprocessed as described in this manuscript (i.e., using in-painting on the Doppler spectra and coherent signal processing to calculate the moments), the percentage of gates classified as WBS decreased from 23.6% to 20.4% (Fig. 17). While this reduction in misclassified gates is modest relative to our expectations, it still represents an improvement over Tanamachi et al.’s (2019) original results. Additionally, gates that were reclassified from WBS to clear air (orange regions in Fig. 17) were predominantly in the regions affected by spurs or horizon glow. This result is highly desirable because reducing the impacts of these artifacts was one of our primary motivations for this study.
Revised scatterer classifications, as in Tanamachi et al. (2019) (their Fig. 12), but based upon the coherent reflectivity and coherent SNR. Gates identified as containing WBS (precipitation and bioscatter) are colored blue, and times with no data are colored red. Gates identified as containing relatively scatterer-free (clear) air are colored either white or orange; white indicates gates identified as clear air both in Tanamachi et al. (2019) and the present study. In contrast, orange gates are those that were classified as WBS in Tanamachi et al. (2019) but as clear air in the present study. No gates were reclassified in the other direction (clear air-to-WBS; purple).
Citation: Journal of Atmospheric and Oceanic Technology 41, 4; 10.1175/JTECH-D-23-0099.1
4. Conclusions
We used a combination of two novel techniques, the Chan et al. (2017) in-painting method from image processing and the Pazmany and Haimov (2018) coherent power technique from signal processing, to rectify a data quality issue in UMass FMCW spectra collected during the 2016 VORTEX-SE field campaign. The in-painting method shows promising results when applied to Doppler spectra contaminated by sharp spectral peaks. Specifically, the in-painting method does better than the previously applied median filter at removing the power spurs from the Doppler spectra in both clear-air and precipitation events (Figs. 6 and 8). Owing to the restoration of the raw spectral structure in regions that are not in-painted, our method preserves more of the underlying Doppler spectral structure of the scatterers being sampled (Figs. 2c and 5c). These scatterers are mainly precipitation and bioscatterers, which tend to be smeared out in the median filter (Figs. 2b and 5b).
The radar moments were regenerated from the in-painted spectral profiles using coherent signal processing (Pazmany and Haimov 2018). In combination with the in-painting method, this type of signal processing removes most of the spurious reflectivity and SNR peaks resulting from the spectral peaks in both clear-air and precipitation cases (Figs. 10 and 11). Improvements in products further derived from these moments [automated CBL height detection (Figs. 14 and 15) and scatterer classification (Fig. 17)] were demonstrated.
We share this data rescue study as an example to the community. Our results underscore the importance of retaining “Level I” Doppler spectra and even “Level 0” time series data. Though voluminous, low-level data should be retained whenever possible. Newer research radars and profilers are now routinely archiving their Level 0 and Level I observations. Retention of these low-level data, in addition to moments (Level II) data, allows future researchers to test new data analysis and signal processing techniques. This practice, in turn, enables new knowledge generation. We also emphasize our incorporation of the in-painting technique, borrowed from another discipline (image processing), that enabled the data rescue. We have demonstrated that such interdisciplinary collaborations can yield fruitful scientific results.
We offer this new version (2.0) of the dataset through the 2016 VORTEX-SE data catalog at NCAR EOL (Frasier et al. 2016).
Acknowledgments.
S.B. conducted formal analysis, investigation, methodology, visualization, and writing—original draft. R.T. was responsible for conceptualization, data curation, supervision, funding acquisition, project administration, resources, validation, visualization, writing—original draft, and writing—review and editing. M.A. conducted formal analysis and visualization. A.G. was responsible for methodology and supervision, and S.F. was responsible for data curation and supervision. S.B., M.A., G.D., A.G., S.F., and F.R. were responsible for software. This work was funded by NOAA Grants NA18OAR4590313 and NA19OAR4590209. Dr. Stanley H. Chan and Dr. Andrew Pazmany advised us on the in-painting and CP techniques, respectively. The statements, findings, conclusions, and recommendations are those of the authors and do not necessarily reflect the views of NOAA or the U.S. Department of Commerce. Dr. Rocadenbosch contributed the CBL detection algorithm via project PID2021-126436OB-C21 funded by Ministerio de Ciencia e Investigación (MCIN)/Agencia Estatal de Investigación (AEI) (https://doi.org/10.13039/501100011033) and EU project H2020 ATMO-ACCESS (GA-101008004).
Data availability statement.
UMass FMCW data from VORTEX-SE 2016 are openly available through the Earth Observing Laboratory archive (https://data.eol.ucar.edu/dataset/527.016). The Chan et al. (2017) in-painting code adapted for UMass FMCW data can be found at https://github.com/sbeverid/in_painting.
REFERENCES
Atlas, D., K. R. Hardy, and K. Naito, 1966: Optimizing the radar detection of clear air turbulence. J. Appl. Meteor., 5, 450–460, https://doi.org/10.1175/1520-0450(1966)005<0450:OTRDOC>2.0.CO;2.
Banks, R. F., J. Tiana-Alsina, F. Rocadenbosch, and J. M. Baldasano, 2015: Performance evaluation of the boundary-layer height from lidar and the Weather Research and Forecasting model at an urban coastal site in the north-east Iberian Peninsula. Bound.-Layer Meteor., 157, 265–292, https://doi.org/10.1007/s10546-015-0056-2.
Bianco, L., and J. M. Wilczak, 2002: Convective boundary layer depth: Improved measurement by Doppler radar wind profiler using fuzzy logic methods. J. Atmos. Oceanic Technol., 19, 1745–1758, https://doi.org/10.1175/1520-0426(2002)019<1745:CBLDIM>2.0.CO;2.
Boyd, S., N. Parikh, E. Chu, B. Peleato, and J. Eckstein, 2011: Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers. Now Publishers Inc., 125 pp.
Chadwick, R. B., K. P. Moran, R. G. Strauch, G. E. Morrison, and W. C. Campbell, 1976: Microwave radar wind measurements in the clear air. Radio Sci., 11, 795–802, https://doi.org/10.1029/RS011i010p00795.
Chan, S. H., X. Wang, and O. A. Elgendy, 2017: Plug-and-play ADMM for image restoration: Fixed-point convergence and applications. IEEE Trans. Comput. Imaging, 3, 84–98, https://doi.org/10.1109/TCI.2016.2629286.
da Silva, M. P. A., F. Rocadenbosch, R. L. Tanamachi, and U. Saeed, 2022: Motivating a synergistic mixing-layer height retrieval method using backscatter lidar returns and microwave-radiometer temperature observations. IEEE Trans. Geosci. Remote Sens., 60, 4107418, https://doi.org/10.1109/TGRS.2022.3158401.
Doviak, R. J., and D. S. Zrnić, 1993: Doppler Radar and Weather Observations. Academic Press, 562 pp.
Eaton, F. D., S. A. McLaughlin, and J. R. Hines, 1995: A new frequency-modulated continuous wave radar for studying planetary boundary layer morphology. Radio Sci., 30, 75–88, https://doi.org/10.1029/94RS01937.
Frasier, S. J., J. Waldinger, and NOAA, 2016: UMass S-band FMCW Profiling Radar Data. UCAR, accessed 19 December 2019, https://doi.org/10.5065/D67P8WS3.
Gage, K. S., and B. B. Balsley, 1978: Doppler radar probing of the clear atmosphere. Bull. Amer. Meteor. Soc., 59, 1074–1094, https://doi.org/10.1175/1520-0477(1978)059<1074:DRPOTC>2.0.CO;2.
Gossard, E. E., 1990: Radar research on the atmospheric boundary layer. Radar in Meteorology, Amer. Meteor. Soc., 477–527, https://doi.org/10.1007/978-1-935704-15-7_35.
Hardy, K. R., and I. Katz, 1969: Probing the clear atmosphere with high power, high resolution radars. Proc. IEEE, 57, 468–480, https://doi.org/10.1109/PROC.1969.7001.
Hardy, K. R., D. Atlas, and K. M. Glover, 1966: Multiwavelength backscatter from the clear atmosphere. J. Geophys. Res., 71, 1537–1552, https://doi.org/10.1029/JZ071i006p01537.
Holzworth, G. C., 1964: Estimates of mean maximum mixing depths in the contiguous United States. Mon. Wea. Rev., 92, 235–242, https://doi.org/10.1175/1520-0493(1964)092<0235:EOMMMD>2.3.CO;2.
İnce, T., A. Pazmany, and S. Frasier, 2000: High resolution profiling of the atmospheric boundary layer. Proc. IEEE 2000 Int. Geoscience and Remote Sensing Symp., Honolulu, HI, Institute of Electrical and Electronics Engineers, 209–212, https://doi.org/10.1109/IGARSS.2000.860470.
İnce, T., S. J. Frasier, A. Muschinski, and A. L. Pazmany, 2003: An S-band frequency-modulated continuous-wave boundary layer profiler: Description and initial results. Radio Sci., 38, 1072, https://doi.org/10.1029/2002RS002753.
Knight, C., and L. J. Miller, 1998: Early radar echoes from small warm cumulus: Bragg and hydrometeor scattering. J. Atmos. Sci., 55, 2974–2992, https://doi.org/10.1175/1520-0469(1998)055<2974:EREFSW>2.0.CO;2.
Koch, S., 2016: VORTEX-SE: Program and activities. 28th Conf. on Severe Local Storms, Portland, OR, Amer. Meteor. Soc., 3.1, https://ams.confex.com/ams/28SLS/webprogram/Paper300782.html.
Kotthaus, S., and Coauthors, 2023: Atmospheric boundary layer height from ground-based remote sensing: A review of capabilities and limitations. Atmos. Meas. Tech., 16, 433–479, https://doi.org/10.5194/amt-16-433-2023.
Kropfli, R., I. Katz, T. Konrad, and E. Dobson, 1968: Simultaneous radar reflectivity measurements and refractive index spectra in the clear atmosphere. Radio Sci., 3, 991–994, https://doi.org/10.1002/rds1968310991.
Lane, J., 1969: Radar echoes from clear air in relation to refractive-index variations in the troposphere. Proc. IEE, 116, 1656–1660, https://doi.org/10.1049/piee.1969.0299.
Lange, D., F. Rocadenbosch, J. Tiana-Alsina, and S. Frasier, 2015: Atmospheric boundary layer height estimation using a Kalman filter and a frequency-modulated continuous-wave radar. IEEE Trans. Geosci. Remote Sens., 53, 3338–3349, https://doi.org/10.1109/TGRS.2014.2374233.
Lee, T., M. Buban, and T. Meyers, 2016: NOAA/ATDD mobile radiosonde data, version 1.0. UCAR/NCAR–Earth Observing Laboratory, accessed 9 July 2023, https://doi.org/10.5065/D68K77FN.
Melnikov, V., and D. S. Zrnić, 2017: Observations of convective thermals with weather radar. J. Atmos. Oceanic Technol., 34, 1585–1590, https://doi.org/10.1175/JTECH-D-17-0068.1.
Ottersten, H., 1969: Atmospheric structure and radar backscattering in clear air. Radio Sci., 4, 1179–1193, https://doi.org/10.1029/RS004i012p01179.
Pazmany, A. L., and S. J. Haimov, 2018: Coherent power measurements with a compact airborne Ka-band precipitation radar. J. Atmos. Oceanic Technol., 35, 3–20, https://doi.org/10.1175/JTECH-D-17-0058.1.
Ralph, F. M., 1995: Using radar-measured radial vertical velocities to distinguish precipitation scattering from clear-air scattering. J. Atmos. Oceanic Technol., 12, 257–267, https://doi.org/10.1175/1520-0426(1995)012<0257:URMRVV>2.0.CO;2.
Rasmussen, E., 2015: VORTEX–Southeast Program overview. National Severe Storms Laboratory Tech. Rep., 36 pp.
Rasmussen, E., and S. Koch, 2016: VORTEX-SE: Lessons learned and early results. 28th Conf. on Severe Local Storms, Portland, OR, Amer. Meteor. Soc., 3.2, https://ams.confex.com/ams/28SLS/webprogram/Paper301782.html.
Rauber, R. M., and S. W. Nesbitt, 2018: Radar Meteorology: A First Course. John Wiley & Sons, 496 pp.
Richter, J. H., 1969: High resolution tropospheric radar sounding. Radio Sci., 4, 1261–1268, https://doi.org/10.1029/RS004i012p01261.
Rocadenbosch, F., R. Barragán, S. J. Frasier, J. Waldinger, D. D. Turner, R. L. Tanamachi, and D. T. Dawson, 2020: Ceilometer-based rain-rate estimation: A case-study comparison with S-band radar and disdrometer retrievals in the context of VORTEX-SE. IEEE Trans. Geosci. Remote Sens., 58, 8268–8284, https://doi.org/10.1109/TGRS.2020.2984458.
Saeed, U., F. Rocadenbosch, and S. Crewell, 2016: Adaptive estimation of the stable boundary layer height using combined lidar and microwave radiometer observations. IEEE Trans. Geosci. Remote Sens., 54, 6895–6906, https://doi.org/10.1109/TGRS.2016.2586298.
Seibert, P., F. Beyrich, S.-E. Gryning, S. Joffre, A. Rasmussen, and P. Tercier, 2000: Review and intercomparison of operational methods for the determination of the mixing height. Atmos. Environ., 34, 1001–1027, https://doi.org/10.1016/S1352-2310(99)00349-0.
Tanamachi, R. L., S. J. Frasier, J. Waldinger, A. LaFleur, D. D. Turner, and F. Rocadenbosch, 2019: Progress toward characterization of the atmospheric boundary layer over northern Alabama using observations by a vertically pointing, S-band profiling radar during VORTEX–Southeast. J. Atmos. Oceanic Technol., 36, 2221–2246, https://doi.org/10.1175/JTECH-D-18-0224.1.
Ulaby, F. T., R. K. Moore, and A. K. Fung, 1982: Microwave Remote Sensing: Active and Passive. Vol. II, Radar Remote Sensing and Surface Scattering and Emission Theory. Addision-Wesley, 634 pp.
Villalonga, J., S. L. Beveridge, M. P. A. Da Silva, R. L. Tanamachi, F. Rocadenbosch, D. D. Turner, and S. J. Frasier, 2020: Convective boundary-layer height estimation from combined radar and Doppler lidar observations in VORTEX-SE. Proc. SPIE, 11531, 115310X, https://doi.org/10.1117/12.2576046.
Waldinger, J., 2018: Improvements to the UMass S-band FM-CW vertical wind profiling radar: System performance and data analysis. Ph.D. thesis, University of Massachusetts Libraries, 97 pp.
Waldinger, J., T. Hartley, W. Heberling, S. Frasier, and R. Tanamachi, 2017: S-band FMCW boundary layer profiler: System upgrades and results. 2017 IEEE Int. Geoscience and Remote Sensing Symp. (IGARSS), Fort Worth, TX, Institute of Electrical and Electronics Engineers, 4526–4529, https://doi.org/10.1109/IGARSS.2017.8128008.
Wolff, C., 1998: Radartutorial (English version). http://www.radartutorial.eu/index.en.html.
