1. Introduction

Since the beginning of weather radar, the suppression of ground clutter echoes (GCE) has been a major concern, and much research has focused in the area. Before the advent of fast digital processors in the early 2000s, capable of spectral processing in real time, application of infinite impulse response (IIR) filters to the time series data [in-phase (I) and quadrature (Q) data] was a primary filtering technique. To suppress various power strengths of GCE, IIR filters with different stop bandwidths were used; however, the real-time identification of the presence of GCE and its strength was impractical. Thus, ground clutter filters (GCF) would either be applied everywhere or not applied at all depending on the needs of the weather data users. Additionally, IIR filters had the issue of initialization or filter “warm up.” In the early 2000s, general purpose processors became powerful enough to execute ground clutter filtering on weather Doppler spectra in real time. Siggia and Passarelli (2004) describe one such technique, named Gaussian model adaptive processing (GMAP), which was adopted by the U.S. National Weather Service (NWS) for the Next Generation Weather Radar (NEXRAD) (Ice et al. 2004, 2005). An advantage of the spectral GCF technique is that the spectra of the data at each radar resolution volume can be processed with various adaptive spectrum notch widths in real time. The new computer processing power also provided for real-time GCE identification techniques such as clutter mitigation decision (CMD) (Hubbert et al. 2009a,b; Ice et al. 2009), which is also used by U.S. NWS on NEXRAD. Since these developments, such spectral techniques for ground clutter filtering have become the standard for most weather radars.

For spectral GCF to be effective, it is necessary to apply a window function to the time series before transforming the data to Doppler spectra using the discrete Fourier transform (DFT). This is done to reduce the so-called ground clutter leakage away from zero velocity and near-zero velocities to higher frequencies (Harris 1978). This is best illustrated by example. Figures 1a and 1b show the real (I) and imaginary (Q) parts of a ground clutter signal from S-band polarimetric radar (S-Pol) (UCAR/NCAR Earth Observing Laboratory 1996; Lutz et al. 1995), operated by the National Center for Atmospheric Research (NCAR) for the National Science Foundation (NSF). There are 64 points with 2-ms sample interval, and the data were gathered at a scan rate of 8° s⁻¹. Figure 1c shows the Doppler power spectrum of the time series while Fig. 1d shows the Doppler spectrum after a Blackman window has been applied to the time series. There is a dramatic difference between the two spectra in Figs. 1c and 1d. The spectrum of the nonwindowed data appears tent-like in shape with a peak to minimum power difference of about 39 dB. This tent-like feature, due to ground clutter leakage, is caused by the nature of the DFT. In contrast, the windowed spectrum of Fig. 1d shows a peak to minimum power difference of 65 dB with the ground clutter signal constrained to velocities near and at 0 m s⁻¹. Clearly if the spectrum’s 0-velocity and near-0-velocity components were set to zero power (in linear units), the ground clutter signal would be effectively eliminated. This is the central idea of spectral clutter filtering.¹

Clutter signal gathered by S-Pol at a scan rate of 8° s⁻¹. (a) The I (real part) of the time series, (b) Q (imaginary part) of the time series, (c) power spectrum of raw data, and (d) power spectrum after windowing I and Q with a Blackman window. Note the effect of the Blackman window on the spectrum. From the peak to the noise floor is about 65 dB. In (c), from the peak to lowest power is only about 39 dB.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Clutter signal gathered by S-Pol at a scan rate of 8° s⁻¹. (a) The I (real part) of the time series, (b) Q (imaginary part) of the time series, (c) power spectrum of raw data, and (d) power spectrum after windowing I and Q with a Blackman window. Note the effect of the Blackman window on the spectrum. From the peak to the noise floor is about 65 dB. In (c), from the peak to lowest power is only about 39 dB.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

Clutter signal gathered by S-Pol at a scan rate of 8° s⁻¹. (a) The I (real part) of the time series, (b) Q (imaginary part) of the time series, (c) power spectrum of raw data, and (d) power spectrum after windowing I and Q with a Blackman window. Note the effect of the Blackman window on the spectrum. From the peak to the noise floor is about 65 dB. In (c), from the peak to lowest power is only about 39 dB.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

A disadvantage of applying a window function to the time series is that it attenuates a significant portion of the weather signal that may be present together with the GCE. Widely used window functions are the Hamming, Hann (sometimes called von Hann or Hanning), Blackman, and Blackman–Nuttall, which reduce the average power of a 64-point time series by 4.01, 4.19, 5.23, and 5.89 dB, respectively. This can be calculated by comparing the area under a window function (e.g., Blackman or Hann) with the area of the reference rectangular window function. This gives the average power attenuation due to the applied window function and the power of a windowed time series can be compensated by this attenuation factor. The window functions in effect reduce the equivalent number of independent samples due to the reduced influence of attenuated time series members. This in turn then increases the variance of the radar estimates of an accompanying weather signal (Siggia and Passarelli 2004). This disadvantage has been accepted as a compromise when using the frequency (i.e., Doppler) domain GCF.

A conceptually different approach to mitigating GCE is regression filtering. Regression analysis is frequently used to estimate the trend of time series or to analyze the residuals after the trend has been eliminated (Darlington and Hayes 2017). For weather radar data, GCE varies slowly while the weather data typically vary faster in time, and therefore, a regression fit to the data can track the slowly varying trend due to the clutter. Regression filtering has been investigated in the biomedical field for ultrasound images to suppress stationary and slow-moving tissue signal so that the blood flow signal of interest can be estimated (Bjaerum et al. 2002; Kadi and Loupas 1995). For weather radar, Torres and Zrnić (1999) examined a regression clutter filter for NEXRAD, and it was found to have comparable performance to IIR filters. At that time GMAP also became a viable option and it was adopted in lieu of regression filtering. Here we take an in-depth look at regression clutter filtering.

This paper reexamines and theoretically evaluates regression GCF characteristics and performance. It is shown that the regression filter has comparable or, for some cases, superior clutter rejection performance to a spectral based GCF, which is hereinafter referred to as a window-and-notch (WN) filter. The overwhelming advantage of the regression filter is that windowing the time series is not required and thus the signal statistics of any weather echo overlaid on the GCE are greatly improved, typically by 35%–50% or more depending on the window used, the length of the times series and the inherent signal statistics (Harris 1978; Sachidananda and Zrnić 1999).

This paper is organized as follows. Section 2 provides an analysis of the characteristics and mechanics of the regression filter including its frequency response. Section 3 uses simulated time series data to compare the spectral GFC and regression GCF for three sampling intervals: 1) 1, 2) 2, and 3) 3.1 ms. Section 4 investigates regression filtering of longer length time series where the Savitzky–Golay filter is also considered. Section 5 offers a summary and conclusions. A follow-on paper that is in preparation will analyze and discuss application of the filter to experimental radar data.

2. Regression ground clutter filter

When using a spectral GCF like GMAP, it is well known that a window function needs to be applied to the time series to contain clutter leakage (Harris 1978). The phenomena of ground clutter leakage is a property of the DFT that represents a finite length time series as a periodic function of sinusoids. To represent the “jumps” or discontinuities of the time series at the periodic boundaries with a sum of sinusoids, significant power must be typically placed in the higher frequency sinusoids. This spreading of the power of frequency components, which are smaller than the smallest frequency sinusoidal DFT basis function (i.e., typical clutter signals), to higher DFT basis frequencies is termed clutter leakage. Mathematically, if a signal is composed of frequency components that do not match one of the DFT basis functions, then those components are simply represented by a sum of the available DFT basis functions. This can be thought of as leaking or spreading energy of the non-DFT basis function sinusoid to the available DFT basis functions.

Regression filtering takes a fundamentally different approach to clutter filtering by using a set of polynomials to capture the trend or low frequency variation of the time series. The I and Q signals from GCE in Fig. 1 are smooth and slowly varying, which suggest the trend could be represented by a polynomial of an appropriate order. This polynomial can be subtracted from the I and Q signals such that the higher frequency weather signals remain as residuals. This idea is demonstrated in Fig. 2. Figure 2a gives the real part of an S-Pol time series consisting of ground clutter with weather echo. The red line is a regression fit that tracks the mean trend of the clutter signal. Figure 2b shows the resulting weather signal when the clutter estimate (red line in Fig. 2a) is subtracted from the combined signal.

An example of the regression clutter filter. (a) The I (real part) of a clutter plus weather signal gathered by S-Pol. The red line is a regression fit that follows the slowly varying clutter signal. (b) The residual weather signal resulting when the red line is subtracted from the weather plus clutter signal.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

An example of the regression clutter filter. (a) The I (real part) of a clutter plus weather signal gathered by S-Pol. The red line is a regression fit that follows the slowly varying clutter signal. (b) The residual weather signal resulting when the red line is subtracted from the weather plus clutter signal.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

An example of the regression clutter filter. (a) The I (real part) of a clutter plus weather signal gathered by S-Pol. The red line is a regression fit that follows the slowly varying clutter signal. (b) The residual weather signal resulting when the red line is subtracted from the weather plus clutter signal.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

For low round-off error and reduced computation time, it is important to use orthogonal polynomials for the least squares fit, and in particular we found that the algorithm associated with the Forsythe orthogonal polynomials performed well (Forsythe 1957; Ruckdeschel 1981). The details of polynomial least squares regression have been covered in many textbooks. A brief overview of the polynomial least squares regression technique is provided in appendix A.

Note that a zero-velocity weather echo with a narrow spectrum width would have similar signal characteristics as clutter echo and thus would be eliminated by the regression filter as it would also be by GMAP. Ground clutter identification algorithms, such as CMD, can distinguish between weather and clutter so that a clutter filter is not applied when weather is the predominant echo.

Regression filter frequency response

The effect of a regression clutter filter on a signal can be characterized by the filter’s frequency response, which has been reported before (Bjaerum et al. 2002; Kadi and Loupas 1995; Torres and Zrnić 1999). The regression clutter filtered signal r_i can be written as

$r_i=x_i-{\displaystyle \sum _{j=0}^n{\alpha }_j{P}_j\left(x_i\right)},$(1)

where r is the filtered sequence, x is the signal to be filtered, α are the regression coefficients, and P_j are polynomials. It can be shown that the frequency response H(ω) of the regression filter is

$H\left(\omega \right)=1-{\displaystyle \frac{1}{m}}\hspace{0.17em}{\displaystyle \sum _{i=0}^p{\left|B_i\left(\omega \right)\right|}^2},$(2)

where ω is frequency and B_i(ω) are the Fourier transform of the basis polynomials P_j, p is polynomial order, and m is the filter length (Torp 1997; Torres and Zrnić 1999). Here an alternative approach is taken to calculate the filter frequency response. The frequency response of the regression filter is calculated numerically by using complex white noise sequences as input into the clutter filter Eq. (1). Let x(n) and y(n) be the time domain input and output signals, where n is a time index, and let the cross-correlation function be denoted x(n) ★ y(n). The Fourier cross-correlation transform pair is

$\mathcal{F}\left\{x\left(n\right)\star y\left(n\right)\right\}=\mathcal{F}\left\{x\left(n\right)\right\}\mathcal{F}{\left\{y\left(n\right)\right\}}^{*}={\text{Φ}}_{xy}\left(\omega \right),$(3)

where $\mathcal{F}$ denotes Fourier transform so that Φ_xy(ω) is the frequency domain representation of the cross-correlation function and can be calculated by multiplying the Fourier transform of the input by the conjugate of the Fourier transform of the output. Also, the cross-correlation between the input and output sequences is the convolution of the impulse response of the filter with the autocorrelation of the input (Oppenheim and Schafer 1989); thus,

${\text{Φ}}_{xy}\left(\omega \right)=H\left(\omega \right){\text{Φ}}_{xx}\left(\omega \right),$(4)

where Φ_xx is the Fourier transform of the autocorrelation function of the input signal. If the input is white noise, Φ_xx is a delta function at ω = 0 and is zero elsewhere so that

${\text{Φ}}_{xy}\left(\omega \right)=H\left(\omega \right){\sigma }_W^2,$(5)

where ${\sigma }_W^2$ is the variance of the white noise. However, since we are using finite length white noise sequences, this is not strictly true, i.e., the nonzero lags of the autocorrelation function are not identically zero. Thus, we calculate the frequency response of the filter using

$H\left(\omega \right)={\displaystyle \frac{{\text{Φ}}_{xy}\left(\omega \right)}{{\text{Φ}}_{xx}\left(\omega \right)}}.$(6)

Multiple simulated white noise sequences are filtered and then the corresponding H(ω) are averaged in order for the frequency response to converge to its true theoretical value. See appendix B for details of this algorithm.

Figures 3 and 4 show the frequency response, using the DFT, of the regression filter for 64-point and 16-point time series, respectively, for various polynomial orders. The small circles mark the locations of the discrete points (zero padding was used to increase the resolution and fill in between the discrete sample points). Figures 3a and 4a show the full frequency response from ±π radians while Figs. 3b and 4b show an expanded version to see more filter detail in the transition region. As the polynomial order increases, the stop bandwidth of the regression filter increases. The horizontal axis ±π corresponds to the Nyquist (or folding), velocity which depends on the PRT (pulse repetition time) and operating frequency; that is, Nyquist velocity = λ/(4 × PRT), where λ is the wavelength. For example, for Fig. 3, S-Pol’s wavelength is 0.1067 m and, if the PRT = 2 ms, the Nyquist velocity is 13.34 m s⁻¹. The plots provide a visual comparison of the bandwidths of the regression filter and WN filter frequency responses. The notch in the frequency response of the WN filter can be represented by vertical lines that show the locations of the right and left edges of the points in the spectrum to be set to zero, inclusive. For example, the dashed vertical lines in Fig. 3 show the location of the left and right edges of a 7-point and 9-point spectral notches for a 64-point time series. In Fig. 4a, the vertical lines show the location of the left and right edges of 5-point and 7-point spectral notches for a 16-point time series.

The regression filter frequency response (power) for 64-point time series and various polynomial orders as indicated. (a) Full frequency response. The small circles mark the locations of the discrete 64 points. The dashed vertical lines and dotted lines mark the edges of a 7-point and 9-point spectral notch filter, respectively. (b) expanded version of (a) from −π/2 to 0 radians. The vertical lines are meant to facilitate comparison of the regression frequency response with the window and notch clutter filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

The regression filter frequency response (power) for 64-point time series and various polynomial orders as indicated. (a) Full frequency response. The small circles mark the locations of the discrete 64 points. The dashed vertical lines and dotted lines mark the edges of a 7-point and 9-point spectral notch filter, respectively. (b) expanded version of (a) from −π/2 to 0 radians. The vertical lines are meant to facilitate comparison of the regression frequency response with the window and notch clutter filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

The regression filter frequency response (power) for 64-point time series and various polynomial orders as indicated. (a) Full frequency response. The small circles mark the locations of the discrete 64 points. The dashed vertical lines and dotted lines mark the edges of a 7-point and 9-point spectral notch filter, respectively. (b) expanded version of (a) from −π/2 to 0 radians. The vertical lines are meant to facilitate comparison of the regression frequency response with the window and notch clutter filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

As in Fig. 3, but for 16-point time series. The dashed vertical lines and dotted lines show the locations of the left and right edges of a 5-point and 7-point spectral notch filter, respectively.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

As in Fig. 3, but for 16-point time series. The dashed vertical lines and dotted lines show the locations of the left and right edges of a 5-point and 7-point spectral notch filter, respectively.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

As in Fig. 3, but for 16-point time series. The dashed vertical lines and dotted lines show the locations of the left and right edges of a 5-point and 7-point spectral notch filter, respectively.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

Following the 5-point notch dashed line Fig. 4b, regression filter attenuations where the notch line intersects the polynomial line (circles) for orders 2, 3, 4, and 5 can be estimated as, 0.5, 1.0, 3.0, and 8 dB, respectively. The 5-point WN filter sets this frequency component at the vertical line to zero (linear scale). This is useful when comparing the regression and WN filters’ attenuations.

As a final example, Fig. 5 shows the frequency response for 32-point sequences for odd polynomial orders 3 through 19. We note that the power spectra of clutter signals are usually characterized as Gaussian in shape and the frequency response of the regression filter with its increasing frequency attenuation as the frequency approaches 0 m s⁻¹ appears to provide sufficient attenuation of Gaussian shaped clutter spectra without use of the 0 notch of a WN filter. This will be demonstrated below with simulated clutter signals.

As in Fig. 3, but for 32-point time series.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

As in Fig. 3, but for 32-point time series.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

As in Fig. 3, but for 32-point time series.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

3. Clutter filter performance comparisons via simulated data

To investigate the performance of the regression versus the WN clutter filters, the filters are applied to simulated data that contain both a clutter and a weather signal. The performance of the filters is judged by the bias and standard deviation (Std) of the estimates of power (or SNR of the weather signal), velocity, spectrum width (SW), differential reflectivity Z_DR, copolar differential phase ϕ_DP, and copolar correlation coefficient ρ_HV (Bringi and Chandrasekar 2001). Two simulators are used with nearly identical results: 1) the I&Q simulator described in Frehlich and Yadlowsky (1994), Frehlich (2000), and Frehlich et al. (2001), which is based on the autocorrelation function of a weather echo as defined in Doviak and Zrnić (1993), and 2) the spectral I&Q simulator described in Chandrasekar et al. (1986). Ground clutter echoes are modeled as weather echo at zero velocity with a narrow spectrum width. The following parameters are used: the power spectra shape is Gaussian; the wavelength is 0.1067 m (S band); the sample interval (i.e., PRT) is 1, 2, or 3.1 ms; the resulting Nyquist velocity is respectively 26.7, 13.3, and 8.6 m s⁻¹; the time series length is 16 and 64 points; and the clutter spectrum width is 0.25 and 0.35 m s⁻¹. These parameters are selected to mimic S-Pol and NEXRAD scan strategies for the experimental clutter data that will be analyzed in a follow-on paper.

a. Clutter rejection comparison

The ability of the regression and WN GCFs to reject clutter is compared by filtering simulated clutter signals and linearly averaging the power spectra for 50 000 realizations. Since a critical component of the WN filter is the selection of the window function (Siggia and Passarelli 2004), the effect of the window function on simulated clutter time series is first examined. Figure 6 shows four cases where the simulated I&Q data are processed by four commonly used windows: 1) rectangular, 2) Hamming, 3) Hann, and 4) Blackman. The PRT, SW, and number of points for the four cases are 2-ms samples, 0.25 m s⁻¹ SW, and 64 points (Fig. 6a); 2-ms samples, 0.25 m s⁻¹ SW, and 32 points (Fig. 6b); 1-ms samples, 0.25 m s⁻¹ SW, and 64 points (Fig. 6c); and 3.1-ms samples, 0.35 m s⁻¹ SW, and 16 points (Fig. 6d). The horizontal axis is velocity. The noise floor is at 0 dB so that the clutter leakage when using the rectangular or Hamming window (green and red lines) is clearly seen. The simulated clutter-to-noise ratio (CNR) is selected to be 45 dB. This is chosen to approximate the phase noise performance of S-Pol and NEXRAD radars. A figure of merit for the phase noise of a radar is expressed as dBc/Hz or just dBc defined as the decibels relative to the carrier signal. This can be estimated by pointing the radar at a fixed target, such as a tower, and examining the resulting power spectra. The “carrier” signal is estimated as the peak clutter power and the ratio of the clutter power to the noise floor is an estimate of the dBc. For 64-point spectra the maximum clutter value, due to radar system noise, is roughly 65 dBc. To convert this to a CNR, 10 log₁₀(64) or 18 dB is subtracted, giving 47 dB, which is close to the used 45-dB CNR. For these large CNR clutter signals, a Blackman window is necessary to use as indicated by Fig. 6 for suppressing the clutter leakage.

Averaged windowed clutter spectra for rectangular, Hamming, Hann, and Blackman windows; (a) 2-ms samples, 0.25 m s⁻¹ SW, and 64 points; (b) 2-ms samples, 0.25 m s⁻¹ SW, and 32 points; (c) 1-ms samples, 0.25 m s⁻¹ SW, and 64 points; and (d) 3.1-ms samples, 0.35 m s⁻¹ SW, and 16 points.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Averaged windowed clutter spectra for rectangular, Hamming, Hann, and Blackman windows; (a) 2-ms samples, 0.25 m s⁻¹ SW, and 64 points; (b) 2-ms samples, 0.25 m s⁻¹ SW, and 32 points; (c) 1-ms samples, 0.25 m s⁻¹ SW, and 64 points; and (d) 3.1-ms samples, 0.35 m s⁻¹ SW, and 16 points.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

Averaged windowed clutter spectra for rectangular, Hamming, Hann, and Blackman windows; (a) 2-ms samples, 0.25 m s⁻¹ SW, and 64 points; (b) 2-ms samples, 0.25 m s⁻¹ SW, and 32 points; (c) 1-ms samples, 0.25 m s⁻¹ SW, and 64 points; and (d) 3.1-ms samples, 0.35 m s⁻¹ SW, and 16 points.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

To determine the required notch width for the WN filter, the clutter signal is suppressed down to the noise floor by selecting the appropriate notch width. Table 1 gives the notch width and the accompanying power removed for the data given in Fig. 6a for the Blackman window. For a notch width of 9, the rejected power is 44.678 dB, and increasing the notch width further eliminates negligible clutter power.

Table 1.

Clutter suppression as a function of the spectral notch width for a 64-point sequence using a Blackman window and CNR = 45 dB.

View Table

Next, a regression filter order with equivalent clutter suppression as the selected WN filter is determined. Table 2 gives the average amount of power eliminated from the same dataset, used for Table 1, as a function of polynomial order. As the polynomial order increases, more power is eliminated from the time series. The amount of eliminated power increases significantly for orders 1–8, but for orders 9 and greater the increase of rejected power becomes small, less than 1 dB. This indicates that a ninth-order filter is sufficient to reject the large majority of the clutter power. From Table 1 for a notch width of 9 the rejected power is 44.678 dB, and the rejected power for the ninth-order regression filter is 44.671 dB, which indicates similar clutter rejection performance.

Table 2.

Clutter suppression as a function of regression filter order for a 64-point sequence with CNR = 45 dB.

View Table

Further graphical filter comparison results are shown in Fig. 7 for three cases at S band: 1) 64, 2-ms samples, and 0.25 m s⁻¹ SW; 2) 64, 1-ms samples, and 0.25 m s⁻¹ SW; and 3) 16, 3.1-ms samples, and 0.35 m s⁻¹ SW. The SW is increased to 0.35 m s⁻¹ for the 3.1-ms sample case to reflect the approximate 20° s⁻¹ antenna rotation rates used by NEXRAD for long PRT scans. Figure 7a gives the averaged rectangular windowed spectrum (blue curve) and the averaged Blackman windowed spectrum (orange curve) whereas Fig. 7b shows the WN and regression filter comparisons. The purple curves show the WN filter results for notch widths of 9, 7, and 7 points for cases 1, 2, and 3, respectively. These notch widths are selected to suppress the clutter down to the noise floor. For each of the cases, 3 different orders of regression clutter filter results are also given in Fig. 7b. The regression filter orders that yield equivalent clutter suppression to the given WN filter are orders 9, 5, and 5 for cases 1, 2, and 3, respectively. For case 3 note the WN notch width (purple curve) as compared with the fifth-order regression filter curve (orange). It is clear that the regression filter suppresses the clutter using a smaller filter stop bandwidth. If a weather signal were overlapping the clutter signal, the weather signal would not be attenuated by the regression filter as much as it would be by the WN filter.

A three-case comparison of clutter rejection capabilities of the spectral window-and-notch filter and the regression filter. (left)The averaged rectangular and Blackman windowed spectra, and (right) the comparison of various filters as labeled. Case 1 is 2-ms samples, 64 points, and 0.25 m s⁻¹ SW; case 2 is 1-ms samples, 64 points, and 0.25 m s⁻¹ SW; and case 3 is 3.1-ms samples, 16 points, and 0.35 m s⁻¹ SW.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

A three-case comparison of clutter rejection capabilities of the spectral window-and-notch filter and the regression filter. (left)The averaged rectangular and Blackman windowed spectra, and (right) the comparison of various filters as labeled. Case 1 is 2-ms samples, 64 points, and 0.25 m s⁻¹ SW; case 2 is 1-ms samples, 64 points, and 0.25 m s⁻¹ SW; and case 3 is 3.1-ms samples, 16 points, and 0.35 m s⁻¹ SW.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

A three-case comparison of clutter rejection capabilities of the spectral window-and-notch filter and the regression filter. (left)The averaged rectangular and Blackman windowed spectra, and (right) the comparison of various filters as labeled. Case 1 is 2-ms samples, 64 points, and 0.25 m s⁻¹ SW; case 2 is 1-ms samples, 64 points, and 0.25 m s⁻¹ SW; and case 3 is 3.1-ms samples, 16 points, and 0.35 m s⁻¹ SW.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

Figure 7 then demonstrates the equivalence of the regression and WN clutter filters’ clutter suppression capabilities for the simulated data. This is not surprising since the regression filter frequency response curves also indicate that the regression filter has similar frequency attenuation characteristics as the WN filter. Figure 7 gives a reference by which to select the regression order for filter comparisons in the next section where the performance of the regression and the WN clutter filters are compared via application to simulated data that consist of a combination of both clutter and weather signal.

b. Filtering simulated weather and clutter data

To compare the performance of the WN and regression filters, the filters are applied to six different simulation scenarios with parameters as given by Table 3. Other secondary simulation parameters are weather Z_DR = 0.5 dB, clutter Z_DR = −5 dB, weather ϕ_DP = 45°, clutter ϕ_DP = −45°, weather ρ_HV = 0.997, and clutter ρ_HV = 0.8. Varying these parameters will affect the statistical plots but do not affect the general comparison of the regression and WN filters. The SW of the weather signal is always 2 m s⁻¹, but the ratio of SW to the Nyquist velocity does vary.

Table 3.

A list of simulation parameters used for the statistical plots in Figs. 8–13. The column labels are, from left to right, figure number, weather SNR, CNR, clutter (Clt) spectrum width, PRT, and the window function (WF) and notch width (NW) used (Blk = Blackman, Han = Hann, and Ham = Hamming).

View Table

The various SNRs and CNRs affect the choice of the window function required to suppress the clutter to minimize estimate errors. For example, the scenarios in which SNR = 20 dB and the CNR = 40 dB require a Blackman window. The scenarios in which SNR = CNR = 45 dB require a Hamming or Hann window. Since the SNR is comparable to the CNR, it not necessary to suppress the clutter down to the noise floor with a Blackman window. For the SNR = CNR = 15 dB, a Hamming window is required. The windows are chosen to heuristically optimize the recovery statistics. For a more complete analysis of window selection see Torres and Warde (2014). The dual-polarization variables typically require a more aggressive window selection, as compared with power and velocity variables. It has been shown that clutter with CSRs as low as −10 dB or lower can bias the dual-polarimetric variables (Friedrich and Germann 2009).

1) Interpolation across zero velocity spectrum gap

To reduce the bias in the estimates of SNR, velocity and SW, frequently one interpolates across the 0-velocity gap caused by the clutter filter in the Doppler spectrum. Here we use a simple linear interpolation across the Doppler power spectrum (in logarithm space). For example, for a WN filter with a 7-point notch width, the linear interpolation is done between the first two data points on either side of the 0-velocity gap. For the regression filter, the interpolation points can be determined from where the filter’s frequency response first exceeds a threshold away from 0 velocity. Examples using −2 dB as a threshold are given in Table 4. Though the examples so far have considered linear interpolation, a Gaussian interpolation could be performed on the points similar to GMAP. As is done with NEXRAD data, interpolation is only used for SNR, velocity and SW estimates and not for Z_DR, ϕ_DP, and ρ_HV estimates where interpolation typically increases estimate errors (Zrnić et al. 2008).

Table 4.

A list of 0-velocity centered interpolation widths as a function of regression order for 64- and 16-point time series.

View Table

2) Filtering results

Figures 8–13 show the average bias and Std estimates, as a function of simulated velocity (Sim V), from 50 000 time series realization per simulation velocity increment. The mean and Std are calculated in standard fashion. The mean and Std of the power are calculated in decibel space. The mean and Std of Z_DR are calculated in linear space so that fractional error or uncertainty is used (Taylor 1997),

$F\left(Z_{\text{DR}}\right)=10\hspace{0.17em}{\log }_{10}\left(1+{\sigma }_{z\text{dr}}/\overline{Z_{\text{DR}}}\right),$(7)

where F() denotes fractional error (dB), σ_zdr is the Std of Z_DR, and $\overline{Z_{\mathrm{DR}}}$ is the mean of Z_DR. The biases are defined as the calculated mean minus the simulated value for a radar variable. For Figs. 8–13 the top two rows show SNR, velocity, and SW, with bias and Std in panels a and b, respectively, and the bottom two rows similarly show the polarimetric variables Z_DR, ϕ_DP, and ρ_HV. The magenta curves show results for the WN filtered data, and the green, red, and light-blue curves are for regression filtered data. The middle polynomial order (red curves) in general yields superior error statistics. The used windows and notch widths as well as the polynomial orders are given in the figure keys. The figures can be examined in two regions: 1) around 0 velocity where the effects of the clutter filters on the weather signal are evident and 2) away from 0 velocity where the weather signal is attenuated very little by the filters. In general, the biases in region 2 due to each filter are similar. However, the Stds in region 2 are nearly always significantly higher for the WN filter as compared with the regression filter, and this is the principal advantage of the regression filter over the WN filter. For Figs. 8, 10, and 11 where a Blackman window is required, the Std increase is from about 45% to 55% for the WN data as compared with the regression Stds. The spectrum width estimated from the R0/R1 (the zeroth and first autocorrelation lags; Bringi and Chandrasekar 2001) is more variable but the regression filter in general produces reduced bias and Std. Around the 0-velocity region the results are similar with the regression filter showing reduced Stds as compared with the WN filter.

(a) Bias and (b) standard deviation (Std) comparison statistics for the 64-point and PRT = 2-ms case. The horizontal axis is simulated velocity. The weather SNR is 20 dB, and the CNR is 40 dB. From left to right and then from top to bottom the variables are SNR, velocity, SW, Z_DR, ϕ_DP, and ρ_HV. There is no 0-velocity interpolation for Z_DR, ϕ_DP, and ρ_HV. The Blackman window with a 9-point notch width is used for the WN filter, and three regression filtered curves are shown.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

(a) Bias and (b) standard deviation (Std) comparison statistics for the 64-point and PRT = 2-ms case. The horizontal axis is simulated velocity. The weather SNR is 20 dB, and the CNR is 40 dB. From left to right and then from top to bottom the variables are SNR, velocity, SW, Z_DR, ϕ_DP, and ρ_HV. There is no 0-velocity interpolation for Z_DR, ϕ_DP, and ρ_HV. The Blackman window with a 9-point notch width is used for the WN filter, and three regression filtered curves are shown.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

(a) Bias and (b) standard deviation (Std) comparison statistics for the 64-point and PRT = 2-ms case. The horizontal axis is simulated velocity. The weather SNR is 20 dB, and the CNR is 40 dB. From left to right and then from top to bottom the variables are SNR, velocity, SW, Z_DR, ϕ_DP, and ρ_HV. There is no 0-velocity interpolation for Z_DR, ϕ_DP, and ρ_HV. The Blackman window with a 9-point notch width is used for the WN filter, and three regression filtered curves are shown.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

As in Fig. 8, but the weather SNR is 45 dB and the CNR is 45 dB. A Hamming window with a 9-point notch width is used for the WN filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

As in Fig. 8, but the weather SNR is 45 dB and the CNR is 45 dB. A Hamming window with a 9-point notch width is used for the WN filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

As in Fig. 8, but the weather SNR is 45 dB and the CNR is 45 dB. A Hamming window with a 9-point notch width is used for the WN filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

As in Fig. 8, but for a PRT of 1 ms. A Blackman window with a 7-point notch width is used for the WN filter. The Nyquist velocity is 26.68 m s⁻¹, but the horizontal axis is terminated at 14 m s⁻¹ since the bias and Std do not vary beyond 14 m s⁻¹.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

As in Fig. 8, but for a PRT of 1 ms. A Blackman window with a 7-point notch width is used for the WN filter. The Nyquist velocity is 26.68 m s⁻¹, but the horizontal axis is terminated at 14 m s⁻¹ since the bias and Std do not vary beyond 14 m s⁻¹.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

As in Fig. 8, but for a PRT of 1 ms. A Blackman window with a 7-point notch width is used for the WN filter. The Nyquist velocity is 26.68 m s⁻¹, but the horizontal axis is terminated at 14 m s⁻¹ since the bias and Std do not vary beyond 14 m s⁻¹.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

As in Fig. 8, but for the 16-point and PRT = 3.1-ms case. A Blackman window with a 7-point notch width is used for the WN filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

As in Fig. 8, but for the 16-point and PRT = 3.1-ms case. A Blackman window with a 7-point notch width is used for the WN filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

As in Fig. 8, but for the 16-point and PRT = 3.1-ms case. A Blackman window with a 7-point notch width is used for the WN filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

As in Fig. 11, but the weather SNR is 45 dB and the CNR is 45 dB. A Hann window with a 7-point notch width is used for the WN filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

As in Fig. 11, but the weather SNR is 45 dB and the CNR is 45 dB. A Hann window with a 7-point notch width is used for the WN filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

As in Fig. 11, but the weather SNR is 45 dB and the CNR is 45 dB. A Hann window with a 7-point notch width is used for the WN filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

As in Fig. 8, but the weather SNR is 15 dB and the CNR is 15 dB. A Hamming window with a 7-point notch width is used for the WN filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

As in Fig. 8, but the weather SNR is 15 dB and the CNR is 15 dB. A Hamming window with a 7-point notch width is used for the WN filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

As in Fig. 8, but the weather SNR is 15 dB and the CNR is 15 dB. A Hamming window with a 7-point notch width is used for the WN filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

In all of the simulation scenarios, the green curve for Z_DR, ϕ_DP, and ρ_HV show higher Std as compared with the other regression curves (red and light blue). The green lines are always the lowest-polynomial-order curves and this indicates that the lowest-order polynomial does not reject sufficient clutter power. This is also supported in the ρ_HV bias and Std plots where the green curves show increased bias and Std over the other regression curves. However, for the SNR, velocity, and SW the green curves provide in general reduced bias and Std. This shows that the dual-polarimetric variables are more sensitive to clutter residue as compared with SNR, velocity, and SW (Friedrich and Germann 2009) around the 0-velocity region.

For the PRT = 3.1 ms and 16-samples data of Figs. 11 and 12, the detrimental effect of the WN filter is particularly noticeable in the bias of SNR, velocity, and the Std of Z_DR, ϕ_DP, and ρ_HV as compared with the red regression (order 4) curves. For Fig. 12, even though the SNR = CNR = 45 dB, the Hann window is necessary whereas the less aggressive Hamming window is sufficient for the other scenario with SNR = CNR = 45 dB in Fig. 9.

In Fig. 13 SNR = CNR = 15 dB; however, a Hamming window is still necessary to suppress the clutter—a rectangular window produced increased bias and Std over the Hamming windowed data. The WN filtered data in Fig. 13 have, in general, about 17%–35% increased Std for all variables (the light-blue curve is for ρ_HV).

For Fig. 8 the SNR bias of the WN curve around 0 velocity does show a slightly smaller bias as compared with the regression curves. This is due to the nature of the interpolation used. The same data are shown in Fig. 14 but without interpolation, and in this case, the WN filter eliminates more of the weather signal, thus creating a larger bias. A more sophisticated interpolation scheme may improve the bias of the regression filtered data.

SNR statistics as in Fig. 8, but with no interpolation across the zero-velocity notch. Here the WN filtered data show more bias around zero velocity as compared with the regression filtered data (cf. Fig. 8).

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

SNR statistics as in Fig. 8, but with no interpolation across the zero-velocity notch. Here the WN filtered data show more bias around zero velocity as compared with the regression filtered data (cf. Fig. 8).

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

SNR statistics as in Fig. 8, but with no interpolation across the zero-velocity notch. Here the WN filtered data show more bias around zero velocity as compared with the regression filtered data (cf. Fig. 8).

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

These above six scenarios for PRTs of 1, 2, and 3.1 ms and various SNRs and CNRs clearly demonstrate the advantage of using the regression clutter filter that does not require the use of a window function such as the Blackman, Hann, or Hamming, which effectively reduces the number of independent samples thereby increasing the Stds of the weather variable estimates. The Stds statistics do vary as modeling parameters change. However, we anticipate that the regression statistics will still, in general, be better that those of an equivalent WN filter.

4. Filtering longer length times series

As the length of sequences to be regression filtered increases, high-order polynomial fits are required to maintain similar frequency response filter characteristics, and this could become numerically problematic for very high filter orders. For the sampling strategies considered thus far, this has not been an issue. Nonetheless, we investigate here strategies for filtering longer length sequences where the use of higher-order polynomial fits may be undesirable. The solution is to simply break the sequence into a series of subsequences, filter the subsequences and then rejoin the filtered subsequences. A related filtering technique, termed Savitzky–Golay (Savitzky and Golay 1964; Orfanidis 1996), is also discussed since that technique was investigated by Torres and Zrnić (1998, 1999) for clutter filtering. We examine three types of sequence partitioning using a 64-point sequence as an example as illustrated in Fig. 15: original 64-point sequence (Fig. 15a), four blocks of contiguous 16-point sequences (Fig. 15b), five blocks of overlapping 16-point sequences (overlapping technique) (Fig. 15c), and Savitzky–Golay technique with a 15-point kernel (Fig. 15d).

Partitioning a 64-point sequence: (a) 64 point, (b) four contiguous blocks of 16-point sequences, (c) five 16-point blocks with 4-point overlap, and (d) Savitzky–Golay (SG) with a 15-point kernel (or window). The first and last SG 15-point windows are shown. A polynomial fit is executed, and then the center point of that fit is placed in the filtered sequence as indicated by the green circles. The window is moved forward one point and the process is repeated. This process is akin to FIR filtering.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Partitioning a 64-point sequence: (a) 64 point, (b) four contiguous blocks of 16-point sequences, (c) five 16-point blocks with 4-point overlap, and (d) Savitzky–Golay (SG) with a 15-point kernel (or window). The first and last SG 15-point windows are shown. A polynomial fit is executed, and then the center point of that fit is placed in the filtered sequence as indicated by the green circles. The window is moved forward one point and the process is repeated. This process is akin to FIR filtering.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

Partitioning a 64-point sequence: (a) 64 point, (b) four contiguous blocks of 16-point sequences, (c) five 16-point blocks with 4-point overlap, and (d) Savitzky–Golay (SG) with a 15-point kernel (or window). The first and last SG 15-point windows are shown. A polynomial fit is executed, and then the center point of that fit is placed in the filtered sequence as indicated by the green circles. The window is moved forward one point and the process is repeated. This process is akin to FIR filtering.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

Savitzky–Golay (SG) is a generalized finite impulse response (FIR) filter, where its filtering coefficients are determined from a least squares polynomial fit (Orfanidis 1996) and these coefficients depend only upon the length of the filter (termed the kernel, N) and the polynomial order. Thus, SG uses a sliding window or kernel to select subsequences of length N (N is odd for symmetry about the central point of the filter) across the sequence to be filtered. For the selected 15 points, the central point of the polynomial fit is the filtered version of the original signal at that central point. As with all FIR filters, the beginning and ending points of the sequence to be filtered require special attention. Here, the (N + 1)/2 beginning points and (N + 1)/2 ending points are regression filtered with the N-point kernel and the first and last (N + 1)/2 points are retained in the overall filtered sequence. As with FIR filtering, the SG kernel is moved across the interior points of the sequence to be filtered. Thus, for a 64-point sequence with N = 15, there will be 48 regression fits to the interior points plus two fits on the beginning and ending points of the sequence for a total of 50 polynomial fits to complete the SG filtering process. The SG filtered sequence is then subtracted from the original sequence to obtain the clutter filtered sequence. For a given kernel and given polynomial order, the SG kernel weights can be precomputed independent of the data to be filtered. Once these weights are determined, they are convolved with the data to be filtered just as is done with any FIR filter. See Orfanidis (1996) for details.

For the five-block, 16-point overlapping technique, the regression filtered overlapping points are averaged to smooth the 4-point transition regions. The need for this is clarified below. Consider two consecutive 16-point sequences that have a 4-point overlap as illustrated in Fig. 15. Let the first 16-point filtered sequence be a and the second 16-point filtered sequence be b. Thus, points a₁₃, a_14, a₁₅, and a₁₆ overlap with b₁, b₂, b₃, and b₄ and need to be combined to form the final filtered 64-point sequence. Weights are assigned to the last four members of a denoted Wa₁₃, Wa₁₄, Wa₁₅, and Wa₁₆. Similarly, assign weights to the overlapping b points Wb₁, Wb₂, Wb₃, Wb₄. The weights are Wa₁₃ = 0.8, Wa₁₄ = 0.5, Wa₁₅ = 0.5, Wa₁₆ = 0.2, Wb₁ = 0.2, Wb₂ = 0.5, Wb₃ = 0.5, and Wb₄ = 0.8.

The two 4-point overlapping parts of the two filtered sequences a and b are combined pairwise in a point-by-point fashion by using these weights. This averaging scheme is applied to each overlapping pair of 16-point sequences. As shown in Fig. 15, for the 64-point length sequence with five overlapping 16-point blocks example, there are four overlapping segments of 4 points each. Again, the reason for the weighted average is to provide a smooth transition from 16-point block to 16-point block, and this is illustrated via spectra next. These weights are heuristic and were found via trial and error based on observations of resulting filtered data spectra.

These filtering techniques are now compared via experimental clutter time series data. The data were gathered by S-Pol at 1231:25 UTC 13 March 2019 at a 1.5° elevation angle and 8.625 km, a scan rate of 8° s⁻¹, and a PRT of 2 ms (NCAR/EOL Remote Sensing Facility 2020). Figure 16 compares the effect of the filtering schemes on the S-Pol ground clutter only time series. Figure 16a shows the unfiltered power spectrum. Figure 16c shows an 11th-order regression filter on the 64-point sequence and is considered a reference; Fig. 16b shows the four contiguous blocks of 16 points using a third-order fit on each 16 points (the spectrum due to overlapping filter is nearly identical and is omitted for brevity). Figure 16d shows the spectrum for the SG filtered data using a third-order fit on each 15-point kernel. The SG spectrum does not exhibit a deep attenuation notch at 0 velocity as do the other two filters though the noise floor level is similar. This lack of a deep notch at zero velocity is the usual characteristic for the SG filtered S-Pol clutter spectra.

S-Pol 64-point, clutter only spectra comparison: (a) unfiltered, (b) four contiguous blocks of 16 points, (c) regression order 11, and (d) SG 15-point kernel. Here, (b) and (d) use a third-order regression filter on each block (kernel).

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

S-Pol 64-point, clutter only spectra comparison: (a) unfiltered, (b) four contiguous blocks of 16 points, (c) regression order 11, and (d) SG 15-point kernel. Here, (b) and (d) use a third-order regression filter on each block (kernel).

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

S-Pol 64-point, clutter only spectra comparison: (a) unfiltered, (b) four contiguous blocks of 16 points, (c) regression order 11, and (d) SG 15-point kernel. Here, (b) and (d) use a third-order regression filter on each block (kernel).

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

Next, the same filtering techniques are applied to S-Pol data with weather and clutter echo, which reveals an issue with the filtering technique that uses the contiguous four blocks of 16 points. The data were gathered on 1231:25 UTC 13 March 2019, a scan rate of 8° s⁻¹, and a PRT of 2 ms (NCAR/EOL Remote Sensing Facility 2020). Figure 17 shows spectra for the unfiltered data (Fig. 17a), 11th-order regression filtered data (Fig. 17b), four contiguous blocks of 16 points with a third-order regression filter (Fig. 17c), five overlapping blocks of 16 points with a third-order regression filter (Fig. 17d), and SG filtered data (Fig. 17e). The 11th-order regression filtered spectrum in Fig. 17b shows a deep notch at 0 velocity and can be considered as a reference for the other filters. The contiguous four blocks of 16-point data of Fig. 17c shows odd anomalous oscillations from −13 to −4 m s⁻¹, well above the noise floor of Fig. 17b. This is due to slight discontinuities between the 16-point filtered blocks and is an artifact of the filter. To alleviate this anomalous oscillatory behavior, five blocks of 16 points with 4-point overlap are used, and the resulting spectrum is given in Fig. 17d. The anomalous oscillatory behavior has been mitigated, and the noise floor is similar to the 11th-order regression filtered spectrum; however, the notch at 0 velocity is not as deep. Figure 17e shows the spectrum of the data after filtering using SG (15-point kernel). Again, there is not a deep notch at 0 velocity as compared with Fig. 17b. The SG requires 50 regression fits for this 64-point example, whereas the overlap technique requires only 5. However, since the filtering coefficients for the SG technique can be precomputed, the SG algorithm is relatively fast (Orfanidis 1996).

An S-Pol weather signal spectrum with various filters applied: (a) unfiltered, (b) 11th-order regression, (c) four contiguous blocks of 16 points, (d) five overlapping blocks of 16 points, and (e) SG 16-point kernel. Here, (c), (d), and (e) use a third-order regression filter on each 16-point block.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

An S-Pol weather signal spectrum with various filters applied: (a) unfiltered, (b) 11th-order regression, (c) four contiguous blocks of 16 points, (d) five overlapping blocks of 16 points, and (e) SG 16-point kernel. Here, (c), (d), and (e) use a third-order regression filter on each 16-point block.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

An S-Pol weather signal spectrum with various filters applied: (a) unfiltered, (b) 11th-order regression, (c) four contiguous blocks of 16 points, (d) five overlapping blocks of 16 points, and (e) SG 16-point kernel. Here, (c), (d), and (e) use a third-order regression filter on each 16-point block.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

As was done previously, the above filters’ frequency responses can be calculated and compared using Eq. (6). Figure 18 shows the frequency response of the 11th-order regression on 64 points (dashed green curve), five-block overlap (black), SG using a third-order polynomial, and SG using a fourth-order polynomial, with Fig. 18b being an expanded version of Fig. 18a. The 11th-order regression offers the sharpest transition from the passband to the stopband; it is flat in the passband and thus is considered to have the best performance. The SG filters have a more gradual transition from the pass to stop bands. Also, the SG filters have a sizable ripple in the pass band of about 1 dB. The fourth-order SG filter is given for comparison. Figures 16, 17, and 18 illustrate that using a regression filter over the entire length of the time series yields the best filter characteristics. However, if it is required to divide a long sequence into subsequences, then fewer subsequences are better to minimize the number of subsequence overlaps. The SG filter does not perform as well as the other regression techniques presented here.

(a) Power frequency response comparison for several filters for a 64-point length sequence. (b) An expanded version of (a). Shown are an 11th-order regression on 64 points (dashed green curve); five blocks of 16 points with a 4-point overlap and a third-order regression (black curve); SG, 15-point kernel, and order 3 (blue curve); and SG, 15-point kernel, and order 4 (red curve).

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

(a) Power frequency response comparison for several filters for a 64-point length sequence. (b) An expanded version of (a). Shown are an 11th-order regression on 64 points (dashed green curve); five blocks of 16 points with a 4-point overlap and a third-order regression (black curve); SG, 15-point kernel, and order 3 (blue curve); and SG, 15-point kernel, and order 4 (red curve).

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

(a) Power frequency response comparison for several filters for a 64-point length sequence. (b) An expanded version of (a). Shown are an 11th-order regression on 64 points (dashed green curve); five blocks of 16 points with a 4-point overlap and a third-order regression (black curve); SG, 15-point kernel, and order 3 (blue curve); and SG, 15-point kernel, and order 4 (red curve).

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

• Download Figure
• Download figure as PowerPoint slide

5. Summary and conclusions

The performance of a newly proposed regression clutter filter was compared with spectrum-based window and notch (WN) filters that are currently popular and in widespread use (e.g., GMAP). Regression filters are attractive since they do not require the use of a time domain window function to contain clutter leakage. The use of window functions causes loss of independent samples and the corresponding increase of variance of the radar variable estimates. The window functions used were the Hamming, Hann, and Blackman, which attenuate the time series signal power by 4.01, 4.19, and 5.23 dB, respectively, for 64-point times series. Frequency responses were given for the regression filter for 64-point, 32-point, and 16-point data sequences for various polynomial orders. For a fixed polynomial order, as the sequence length increases, the filter bandwidth decreases, and for a fixed sequence length, as the polynomial order increases, the filter bandwidth increases. The ability of the regression filter to suppress clutter as well as a WN filter was demonstrated via simulated clutter data for various simulation parameters.

The efficacy of the regression filter for recovering radar variables was demonstrated via six simulation scenarios for sequences with 1-, 2-, and 3.1-ms sample intervals (i.e., PRTs) for various combinations of weather SNRs, CNRs, and noise combinations. The resulting simulated time series were processed by three regression filters and a WN filter and the resulting averaged biases and Stds were compared for SNR (power), velocity, SW, Z_DR, ϕ_DP, and ρ_HV. In general, the modeling results showed that Std of the estimated radar variables for the weather signal were about 35%–55% greater for the WN filter as compared with the regression filter. The primary cause of this increase in Std was the window function used for the WN filter, which attenuates the time series thereby reducing the effective number of available independent samples.

The viability of regression filtering longer length sequences that may require very high-order polynomial fits was investigated by dividing the time series into shorter lengths and then regression filtering the shorter length time series so that lower-order regression fits could be employed. The filtered shorter sequences could then be concatenated, thereby yielding a full length filtered time series. However, small discontinuities can be created at the boundaries so that anomalous oscillations can appear in the filtered spectra. The solution to this was to divide the long sequence into shorter overlapping length sequences, which were then filtered. The filtered sequence can be rejoined where a weighted average is used at the overlapping points.

The Savitzky–Golay filtering technique was investigated and compared with the proposed regression filter techniques since it was considered by Torres and Zrnić (1999). The spectra after SG filtering frequently looked anomalous, with shallow or even no discernible notch at 0 velocity. Furthermore, the frequency response of the SG filters was inferior (ripple in the passband and a less sharp transition from the passband to the stopband) to the proposed regression filtering techniques and offered no apparent advantage for clutter filtering.

To successfully implement a regression clutter filter, it is important that a set of orthogonal polynomials, such as the Forsythe polynomials, be used to simplify the matrix inversion in the least squares fit, which significantly reduces round-off error. The filtering coefficients can be precalculated for the length of the time series to be filtered so that execution time is reduced. Source code in C++ and Fortran for the least squares polynomial fit algorithm using Forsythe polynomials can be found at the web page by Moreau (2018). The application of the regression filter to experimental radar data from both S-Pol and NEXRAD will be given in a follow-up paper.