• Bjaerum, S., H. Torp, and K. Kristoffersen, 2002: Clutter filter design for ultrasound color flow imaging. IEEE Trans. Ultrason. Ferroelectr. Freq. Control, 49, 204216, https://doi.org/10.1109/58.985705.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bringi, V., and V. Chandrasekar, 2001: Polarimetric Doppler Weather Radar. Cambridge University Press, 636 pp.

  • Carnahan, B., H. Luther, and J. Wilkes, 1969: Applied Numerical Methods. John Wiley and Sons, 604 pp.

  • Chandrasekar, V., V. Bringi, and P. Brockwell, 1986: Statistical properties of dual-polarized signals. Preprints, 23rd Conf. on Radar Meteorology, Snowmass, CO, Amer. Meteor. Soc., 193–196.

  • Darlington, R., and A. Hayes, 2017: Regression and Linear Models. Guilford Press, 661 pp.

  • Doviak, R. J., and D. S. Zrnić, 1993: Doppler Radar and Weather Observations. 2nd ed. Academic Press, 562 pp.

  • Forsythe, G., 1957: Generation and use of orthogonal polynomials for data fitting with a digital computer. J. Soc. Ind. Appl. Math., 5, 7488, https://doi.org/10.1137/0105007.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Frehlich, R., 2000: Simulation of coherent Doppler lidar performance for space-based platforms. J. Appl. Meteor., 39, 245262, https://doi.org/10.1175/1520-0450(2000)039<0245:SOCDLP>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Frehlich, R., and M. J. Yadlowsky, 1994: Performance of mean-frequency estimators for Doppler radar and lidar. J. Atmos. Oceanic Technol., 11, 12171230, https://doi.org/10.1175/1520-0426(1994)011<1217:POMFEF>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Frehlich, R., L. Cornman, and R. Sharman, 2001: Simulation of three-dimensional turbulent velocity fields. J. Appl. Meteor., 40, 246258, https://doi.org/10.1175/1520-0450(2001)040<0246:SOTDTV>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Friedrich, K., and U. Germann, 2009: Influence of ground clutter contamination on polarimetric radar parameters. J. Atmos. Oceanic Technol., 26, 251269, https://doi.org/10.1175/2008JTECHA1092.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Harris, F., 1978: On the use of windows for harmonic analysis with the discrete Fourier transform. Proc. IEEE, 66, 5183, https://doi.org/10.1109/PROC.1978.10837.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hubbert, J. C., M. Dixon, S. Ellis, and G. Meymaris, 2009a: Weather radar ground clutter. Part I: Identification, modeling and simulation. J. Atmos. Oceanic Technol., 26, 11651180, https://doi.org/10.1175/2009JTECHA1159.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hubbert, J. C., M. Dixon, and S. Ellis, 2009b: Weather radar ground clutter. Part II: Real-time identification and filtering. J. Atmos. Oceanic Technol., 26, 11811197, https://doi.org/10.1175/2009JTECHA1160.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ice, R. L., R. D. Rhoton, D. S. Saxion, N. K. Patel, D. Sirmans, D. A. Warde, D. L. Rachel, and R. G. Fehlen, 2004: Radar Operations Center (ROC) evaluation of the WSR-88D Open Radar Data Acquisition (ORDA) system signal processing. 20th Int. Conf. on Interactive Information Processing Systems for Meteorology, Oceanography, and Hydrology, Seattle, WA, Amer. Meteor. Soc., 5.5, https://ams.confex.com/ams/84Annual/techprogram/paper_70692.htm.

  • Ice, R. L., G. T. McGehee, R. D. Rhoton, D. S. Saxion, D. A. Ward, R. Guenthe, D. Sirmans, and D. L. Rachel, 2005: Radar Operations Center (ROC) evaluation of new signal processing techniques for the WSR-88D. 21 Int. Conf. on Interactive Information Processing Systems for Meteorology, Oceanography, and Hydrology, San Diego, CA, Amer. Meteor. Soc., P1.4, https://ams.confex.com/ams/Annual2005/techprogram/paper_85859.htm.

  • Ice, R. L., and et al. , 2009: Automatic clutter mitigation in the WSR-88D, design, evaluation, and implementation. 34th Conf. on Radar Meteorology, Williamsburg, VA, Amer. Meteor. Soc., P5.3, https://ams.confex.com/ams/34Radar/techprogram/paper_155409.htm.

  • Kadi, A. P., and T. Loupas, 1995: On the performance of regression and step-initialized IIR clutter filters for color Doppler systems in diagnostic medical ultrasound. IEEE Trans. Ultrason. Ferroelectr. Freq. Control, 42, 927937, https://doi.org/10.1109/58.464825.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lutz, J., P. Johnson, B. Lewis, E. Loew, M. Randall, and J. VanAndel, 1995: NCAR S-Pol: Portable polarimetric S-band radar. Preprints, Ninth Symp. on Meteorological Observations and Instrumentation, Charlotte, NC, Amer. Meteor. Soc., 408–410.

  • Moreau, J.-P., 2018: Numerical analysis. Orange, http://jean-pierre.moreau.pagesperso-orange.fr/.

  • NCAR/EOL Remote Sensing Facility, 2020: Marshall S-PolKa radar time series, S-band, version 1.0. UCAR/NCAR Earth Observing Laboratory, accessed 11 February 2020, https://doi.org/10.26023/T8VD-F2C8-900K.

    • Crossref
    • Export Citation
  • Oppenheim, A., and R. Schafer, 1989: Discrete-Time Signal Processing. Prentice Hall, 879 pp.

  • Orfanidis, J., 1996: Introduction to Signal Processing. Prentice Hall, 783 pp., https://www.ece.rutgers.edu/~orfanidi/intro2sp/.

  • Ruckdeschel, F., 1981: Basic Scientific Routines. Vol. II. BYTE/McGraw-Hill, 790 pp.

  • Sachidananda, M., and D. Zrnić, 1999: Systematic phase codes for resolving range overlaid signals in a Doppler weather radar. J. Atmos. Oceanic Technol., 16, 13511363, https://doi.org/10.1175/1520-0426(1999)016<1351:SPCFRR>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Savitzky, A., and M. Golay, 1964: Smoothing and differentiation of data by simplified least squares procedures. Anal. Chem., 36, 16271639, https://doi.org/10.1021/ac60214a047.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Siggia, A., and J. Passarelli, R., 2004: Gaussian model adaptive processing (GMAP) for improved ground clutter cancellation and moment calculation. Proc. Third European Conf. on Radar in Meteorology and Hydrology, Visby, Gotland, ERAD, 67–73.

  • Taylor, J., 1997: An Introduction to Error Analysis. University Science Books, 327 pp.

  • Torp, H., 1997: Clutter rejection filters in color flow imaging: A theoretical approach. IEEE Trans. Ultrason. Ferroelectr. Freq. Control, 44, 417424, https://doi.org/10.1109/58.585126.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Torres, S., and D. Zrnić, 1998: Ground clutter canceling with a regression filter. NSSL Interim Tech. Rep., 35 pp.

  • Torres, S., and D. Zrnić, 1999: Ground clutter canceling with a regression filter. J. Atmos. Oceanic Technol., 16, 13641372, https://doi.org/10.1175/1520-0426(1999)016<1364:GCCWAR>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Torres, S., and D. A. Warde, 2014: Ground clutter mitigation for weather radars using the autocorrelation spectral density. J. Atmos. Oceanic Technol., 31, 20492066, https://doi.org/10.1175/JTECH-D-13-00117.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • UCAR/NCAR Earth Observing Laboratory, 1996: S-PolKa: S-band/Ka-band dual polarization, dual wavelength Doppler radar. UCAR/NCAR Earth Observing Laboratory, accessed 23 January 2017, https://doi.org/10.5065/D6RV0KR8.

    • Crossref
    • Export Citation
  • Zrnić, D., V. Melnikov, and I. Ivic, 2008: Processing to obtain polarimetric variables on the ORDA. NSSL Tech. Rep., 55 pp.

  • View in gallery

    Clutter signal gathered by S-Pol at a scan rate of 8° s−1. (a) The I (real part) of the time series, (b) Q (imaginary part) of the time series, (c) power spectrum of raw data, and (d) power spectrum after windowing I and Q with a Blackman window. Note the effect of the Blackman window on the spectrum. From the peak to the noise floor is about 65 dB. In (c), from the peak to lowest power is only about 39 dB.

  • View in gallery

    An example of the regression clutter filter. (a) The I (real part) of a clutter plus weather signal gathered by S-Pol. The red line is a regression fit that follows the slowly varying clutter signal. (b) The residual weather signal resulting when the red line is subtracted from the weather plus clutter signal.

  • View in gallery

    The regression filter frequency response (power) for 64-point time series and various polynomial orders as indicated. (a) Full frequency response. The small circles mark the locations of the discrete 64 points. The dashed vertical lines and dotted lines mark the edges of a 7-point and 9-point spectral notch filter, respectively. (b) expanded version of (a) from −π/2 to 0 radians. The vertical lines are meant to facilitate comparison of the regression frequency response with the window and notch clutter filter.

  • View in gallery

    As in Fig. 3, but for 16-point time series. The dashed vertical lines and dotted lines show the locations of the left and right edges of a 5-point and 7-point spectral notch filter, respectively.

  • View in gallery

    As in Fig. 3, but for 32-point time series.

  • View in gallery

    Averaged windowed clutter spectra for rectangular, Hamming, Hann, and Blackman windows; (a) 2-ms samples, 0.25 m s−1 SW, and 64 points; (b) 2-ms samples, 0.25 m s−1 SW, and 32 points; (c) 1-ms samples, 0.25 m s−1 SW, and 64 points; and (d) 3.1-ms samples, 0.35 m s−1 SW, and 16 points.

  • View in gallery

    A three-case comparison of clutter rejection capabilities of the spectral window-and-notch filter and the regression filter. (left)The averaged rectangular and Blackman windowed spectra, and (right) the comparison of various filters as labeled. Case 1 is 2-ms samples, 64 points, and 0.25 m s−1 SW; case 2 is 1-ms samples, 64 points, and 0.25 m s−1 SW; and case 3 is 3.1-ms samples, 16 points, and 0.35 m s−1 SW.

  • View in gallery

    (a) Bias and (b) standard deviation (Std) comparison statistics for the 64-point and PRT = 2-ms case. The horizontal axis is simulated velocity. The weather SNR is 20 dB, and the CNR is 40 dB. From left to right and then from top to bottom the variables are SNR, velocity, SW, ZDR, ϕDP, and ρHV. There is no 0-velocity interpolation for ZDR, ϕDP, and ρHV. The Blackman window with a 9-point notch width is used for the WN filter, and three regression filtered curves are shown.

  • View in gallery

    As in Fig. 8, but the weather SNR is 45 dB and the CNR is 45 dB. A Hamming window with a 9-point notch width is used for the WN filter.

  • View in gallery

    As in Fig. 8, but for a PRT of 1 ms. A Blackman window with a 7-point notch width is used for the WN filter. The Nyquist velocity is 26.68 m s−1, but the horizontal axis is terminated at 14 m s−1 since the bias and Std do not vary beyond 14 m s−1.

  • View in gallery

    As in Fig. 8, but for the 16-point and PRT = 3.1-ms case. A Blackman window with a 7-point notch width is used for the WN filter.

  • View in gallery

    As in Fig. 11, but the weather SNR is 45 dB and the CNR is 45 dB. A Hann window with a 7-point notch width is used for the WN filter.

  • View in gallery

    As in Fig. 8, but the weather SNR is 15 dB and the CNR is 15 dB. A Hamming window with a 7-point notch width is used for the WN filter.

  • View in gallery

    SNR statistics as in Fig. 8, but with no interpolation across the zero-velocity notch. Here the WN filtered data show more bias around zero velocity as compared with the regression filtered data (cf. Fig. 8).

  • View in gallery

    Partitioning a 64-point sequence: (a) 64 point, (b) four contiguous blocks of 16-point sequences, (c) five 16-point blocks with 4-point overlap, and (d) Savitzky–Golay (SG) with a 15-point kernel (or window). The first and last SG 15-point windows are shown. A polynomial fit is executed, and then the center point of that fit is placed in the filtered sequence as indicated by the green circles. The window is moved forward one point and the process is repeated. This process is akin to FIR filtering.

  • View in gallery

    S-Pol 64-point, clutter only spectra comparison: (a) unfiltered, (b) four contiguous blocks of 16 points, (c) regression order 11, and (d) SG 15-point kernel. Here, (b) and (d) use a third-order regression filter on each block (kernel).

  • View in gallery

    An S-Pol weather signal spectrum with various filters applied: (a) unfiltered, (b) 11th-order regression, (c) four contiguous blocks of 16 points, (d) five overlapping blocks of 16 points, and (e) SG 16-point kernel. Here, (c), (d), and (e) use a third-order regression filter on each 16-point block.

  • View in gallery

    (a) Power frequency response comparison for several filters for a 64-point length sequence. (b) An expanded version of (a). Shown are an 11th-order regression on 64 points (dashed green curve); five blocks of 16 points with a 4-point overlap and a third-order regression (black curve); SG, 15-point kernel, and order 3 (blue curve); and SG, 15-point kernel, and order 4 (red curve).

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 91 91 63
PDF Downloads 84 84 70

Using a Regression Ground Clutter Filter to Improve Weather Radar Signal Statistics: Theory and Simulations

View More View Less
  • 1 a National Center for Atmospheric Research, Boulder, Colorado
© Get Permissions
Open access

Abstract

Ground clutter filtering is an important and necessary step for quality control of ground-based weather radars. In this paper, ground clutter mitigation is addressed using a time-domain regression filter. Clutter filtering is now widely accomplished with spectral processing where the times series of data corresponding to a radar resolution volume are transformed with a discrete Fourier transform after which the zero and near-zero velocity clutter components are eliminated by setting them to zero. Subsequently for reflectivity, velocity, and spectrum width estimates, interpolation techniques are used to recover some of the power loss due to the clutter filter, which has been shown to reduce bias. The spectral technique requires that the in-phase (I) and quadrature (Q) time series be windowed to reduce clutter power leakage away from zero and near-zero velocities. Unfortunately, window functions such as the Hamming, Hann, and Blackman attenuate the time series signal by 4.01, 4.19, and 5.23 dB for 64-point times series, respectively, and thereby effectively reduce the number of independent samples available for estimating the radar parameters of any underlying weather echo. In this paper, a regression filtering technique is investigated, through simulated data, that does not require the use of such window functions and thus provides for better weather signal statistics. In a follow-on paper that is in preparation the technique will be demonstrated using both S-band polarimetric radar (S-Pol) and NEXRAD data. Here, it is shown that the regression filter rejects clutter as effectively as the spectral technique but has the distinct advantage that estimates of the radar variables are greatly improved. The technique is straightforward and can be executed in real time.

© 2021 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: John C. Hubbert, hubbert@ucar.edu

This article has a companion article which can be found at http://journals.ametsoc.org/doi/abs/10.1175/JTECH-D-20-0184.1.

Abstract

Ground clutter filtering is an important and necessary step for quality control of ground-based weather radars. In this paper, ground clutter mitigation is addressed using a time-domain regression filter. Clutter filtering is now widely accomplished with spectral processing where the times series of data corresponding to a radar resolution volume are transformed with a discrete Fourier transform after which the zero and near-zero velocity clutter components are eliminated by setting them to zero. Subsequently for reflectivity, velocity, and spectrum width estimates, interpolation techniques are used to recover some of the power loss due to the clutter filter, which has been shown to reduce bias. The spectral technique requires that the in-phase (I) and quadrature (Q) time series be windowed to reduce clutter power leakage away from zero and near-zero velocities. Unfortunately, window functions such as the Hamming, Hann, and Blackman attenuate the time series signal by 4.01, 4.19, and 5.23 dB for 64-point times series, respectively, and thereby effectively reduce the number of independent samples available for estimating the radar parameters of any underlying weather echo. In this paper, a regression filtering technique is investigated, through simulated data, that does not require the use of such window functions and thus provides for better weather signal statistics. In a follow-on paper that is in preparation the technique will be demonstrated using both S-band polarimetric radar (S-Pol) and NEXRAD data. Here, it is shown that the regression filter rejects clutter as effectively as the spectral technique but has the distinct advantage that estimates of the radar variables are greatly improved. The technique is straightforward and can be executed in real time.

© 2021 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: John C. Hubbert, hubbert@ucar.edu

This article has a companion article which can be found at http://journals.ametsoc.org/doi/abs/10.1175/JTECH-D-20-0184.1.

1. Introduction

Since the beginning of weather radar, the suppression of ground clutter echoes (GCE) has been a major concern, and much research has focused in the area. Before the advent of fast digital processors in the early 2000s, capable of spectral processing in real time, application of infinite impulse response (IIR) filters to the time series data [in-phase (I) and quadrature (Q) data] was a primary filtering technique. To suppress various power strengths of GCE, IIR filters with different stop bandwidths were used; however, the real-time identification of the presence of GCE and its strength was impractical. Thus, ground clutter filters (GCF) would either be applied everywhere or not applied at all depending on the needs of the weather data users. Additionally, IIR filters had the issue of initialization or filter “warm up.” In the early 2000s, general purpose processors became powerful enough to execute ground clutter filtering on weather Doppler spectra in real time. Siggia and Passarelli (2004) describe one such technique, named Gaussian model adaptive processing (GMAP), which was adopted by the U.S. National Weather Service (NWS) for the Next Generation Weather Radar (NEXRAD) (Ice et al. 2004, 2005). An advantage of the spectral GCF technique is that the spectra of the data at each radar resolution volume can be processed with various adaptive spectrum notch widths in real time. The new computer processing power also provided for real-time GCE identification techniques such as clutter mitigation decision (CMD) (Hubbert et al. 2009a,b; Ice et al. 2009), which is also used by U.S. NWS on NEXRAD. Since these developments, such spectral techniques for ground clutter filtering have become the standard for most weather radars.

For spectral GCF to be effective, it is necessary to apply a window function to the time series before transforming the data to Doppler spectra using the discrete Fourier transform (DFT). This is done to reduce the so-called ground clutter leakage away from zero velocity and near-zero velocities to higher frequencies (Harris 1978). This is best illustrated by example. Figures 1a and 1b show the real (I) and imaginary (Q) parts of a ground clutter signal from S-band polarimetric radar (S-Pol) (UCAR/NCAR Earth Observing Laboratory 1996; Lutz et al. 1995), operated by the National Center for Atmospheric Research (NCAR) for the National Science Foundation (NSF). There are 64 points with 2-ms sample interval, and the data were gathered at a scan rate of 8° s−1. Figure 1c shows the Doppler power spectrum of the time series while Fig. 1d shows the Doppler spectrum after a Blackman window has been applied to the time series. There is a dramatic difference between the two spectra in Figs. 1c and 1d. The spectrum of the nonwindowed data appears tent-like in shape with a peak to minimum power difference of about 39 dB. This tent-like feature, due to ground clutter leakage, is caused by the nature of the DFT. In contrast, the windowed spectrum of Fig. 1d shows a peak to minimum power difference of 65 dB with the ground clutter signal constrained to velocities near and at 0 m s−1. Clearly if the spectrum’s 0-velocity and near-0-velocity components were set to zero power (in linear units), the ground clutter signal would be effectively eliminated. This is the central idea of spectral clutter filtering.1

Fig. 1.
Fig. 1.

Clutter signal gathered by S-Pol at a scan rate of 8° s−1. (a) The I (real part) of the time series, (b) Q (imaginary part) of the time series, (c) power spectrum of raw data, and (d) power spectrum after windowing I and Q with a Blackman window. Note the effect of the Blackman window on the spectrum. From the peak to the noise floor is about 65 dB. In (c), from the peak to lowest power is only about 39 dB.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

A disadvantage of applying a window function to the time series is that it attenuates a significant portion of the weather signal that may be present together with the GCE. Widely used window functions are the Hamming, Hann (sometimes called von Hann or Hanning), Blackman, and Blackman–Nuttall, which reduce the average power of a 64-point time series by 4.01, 4.19, 5.23, and 5.89 dB, respectively. This can be calculated by comparing the area under a window function (e.g., Blackman or Hann) with the area of the reference rectangular window function. This gives the average power attenuation due to the applied window function and the power of a windowed time series can be compensated by this attenuation factor. The window functions in effect reduce the equivalent number of independent samples due to the reduced influence of attenuated time series members. This in turn then increases the variance of the radar estimates of an accompanying weather signal (Siggia and Passarelli 2004). This disadvantage has been accepted as a compromise when using the frequency (i.e., Doppler) domain GCF.

A conceptually different approach to mitigating GCE is regression filtering. Regression analysis is frequently used to estimate the trend of time series or to analyze the residuals after the trend has been eliminated (Darlington and Hayes 2017). For weather radar data, GCE varies slowly while the weather data typically vary faster in time, and therefore, a regression fit to the data can track the slowly varying trend due to the clutter. Regression filtering has been investigated in the biomedical field for ultrasound images to suppress stationary and slow-moving tissue signal so that the blood flow signal of interest can be estimated (Bjaerum et al. 2002; Kadi and Loupas 1995). For weather radar, Torres and Zrnić (1999) examined a regression clutter filter for NEXRAD, and it was found to have comparable performance to IIR filters. At that time GMAP also became a viable option and it was adopted in lieu of regression filtering. Here we take an in-depth look at regression clutter filtering.

This paper reexamines and theoretically evaluates regression GCF characteristics and performance. It is shown that the regression filter has comparable or, for some cases, superior clutter rejection performance to a spectral based GCF, which is hereinafter referred to as a window-and-notch (WN) filter. The overwhelming advantage of the regression filter is that windowing the time series is not required and thus the signal statistics of any weather echo overlaid on the GCE are greatly improved, typically by 35%–50% or more depending on the window used, the length of the times series and the inherent signal statistics (Harris 1978; Sachidananda and Zrnić 1999).

This paper is organized as follows. Section 2 provides an analysis of the characteristics and mechanics of the regression filter including its frequency response. Section 3 uses simulated time series data to compare the spectral GFC and regression GCF for three sampling intervals: 1) 1, 2) 2, and 3) 3.1 ms. Section 4 investigates regression filtering of longer length time series where the Savitzky–Golay filter is also considered. Section 5 offers a summary and conclusions. A follow-on paper that is in preparation will analyze and discuss application of the filter to experimental radar data.

2. Regression ground clutter filter

When using a spectral GCF like GMAP, it is well known that a window function needs to be applied to the time series to contain clutter leakage (Harris 1978). The phenomena of ground clutter leakage is a property of the DFT that represents a finite length time series as a periodic function of sinusoids. To represent the “jumps” or discontinuities of the time series at the periodic boundaries with a sum of sinusoids, significant power must be typically placed in the higher frequency sinusoids. This spreading of the power of frequency components, which are smaller than the smallest frequency sinusoidal DFT basis function (i.e., typical clutter signals), to higher DFT basis frequencies is termed clutter leakage. Mathematically, if a signal is composed of frequency components that do not match one of the DFT basis functions, then those components are simply represented by a sum of the available DFT basis functions. This can be thought of as leaking or spreading energy of the non-DFT basis function sinusoid to the available DFT basis functions.

Regression filtering takes a fundamentally different approach to clutter filtering by using a set of polynomials to capture the trend or low frequency variation of the time series. The I and Q signals from GCE in Fig. 1 are smooth and slowly varying, which suggest the trend could be represented by a polynomial of an appropriate order. This polynomial can be subtracted from the I and Q signals such that the higher frequency weather signals remain as residuals. This idea is demonstrated in Fig. 2. Figure 2a gives the real part of an S-Pol time series consisting of ground clutter with weather echo. The red line is a regression fit that tracks the mean trend of the clutter signal. Figure 2b shows the resulting weather signal when the clutter estimate (red line in Fig. 2a) is subtracted from the combined signal.

Fig. 2.
Fig. 2.

An example of the regression clutter filter. (a) The I (real part) of a clutter plus weather signal gathered by S-Pol. The red line is a regression fit that follows the slowly varying clutter signal. (b) The residual weather signal resulting when the red line is subtracted from the weather plus clutter signal.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

For low round-off error and reduced computation time, it is important to use orthogonal polynomials for the least squares fit, and in particular we found that the algorithm associated with the Forsythe orthogonal polynomials performed well (Forsythe 1957; Ruckdeschel 1981). The details of polynomial least squares regression have been covered in many textbooks. A brief overview of the polynomial least squares regression technique is provided in appendix A.

Note that a zero-velocity weather echo with a narrow spectrum width would have similar signal characteristics as clutter echo and thus would be eliminated by the regression filter as it would also be by GMAP. Ground clutter identification algorithms, such as CMD, can distinguish between weather and clutter so that a clutter filter is not applied when weather is the predominant echo.

Regression filter frequency response

The effect of a regression clutter filter on a signal can be characterized by the filter’s frequency response, which has been reported before (Bjaerum et al. 2002; Kadi and Loupas 1995; Torres and Zrnić 1999). The regression clutter filtered signal ri can be written as
ri=xij=0nαjPj(xi),
where r is the filtered sequence, x is the signal to be filtered, α are the regression coefficients, and Pj are polynomials. It can be shown that the frequency response H(ω) of the regression filter is
H(ω)=11mi=0p|Bi(ω)|2,
where ω is frequency and Bi(ω) are the Fourier transform of the basis polynomials Pj, p is polynomial order, and m is the filter length (Torp 1997; Torres and Zrnić 1999). Here an alternative approach is taken to calculate the filter frequency response. The frequency response of the regression filter is calculated numerically by using complex white noise sequences as input into the clutter filter Eq. (1). Let x(n) and y(n) be the time domain input and output signals, where n is a time index, and let the cross-correlation function be denoted x(n) ★ y(n). The Fourier cross-correlation transform pair is
F{x(n)y(n)}=F{x(n)}F{y(n)}*=Φxy(ω),
where F denotes Fourier transform so that Φxy(ω) is the frequency domain representation of the cross-correlation function and can be calculated by multiplying the Fourier transform of the input by the conjugate of the Fourier transform of the output. Also, the cross-correlation between the input and output sequences is the convolution of the impulse response of the filter with the autocorrelation of the input (Oppenheim and Schafer 1989); thus,
Φxy(ω)=H(ω)Φxx(ω),
where Φxx is the Fourier transform of the autocorrelation function of the input signal. If the input is white noise, Φxx is a delta function at ω = 0 and is zero elsewhere so that
Φxy(ω)=H(ω)σW2,
where σW2 is the variance of the white noise. However, since we are using finite length white noise sequences, this is not strictly true, i.e., the nonzero lags of the autocorrelation function are not identically zero. Thus, we calculate the frequency response of the filter using
H(ω)=Φxy(ω)Φxx(ω).
Multiple simulated white noise sequences are filtered and then the corresponding H(ω) are averaged in order for the frequency response to converge to its true theoretical value. See appendix B for details of this algorithm.

Figures 3 and 4 show the frequency response, using the DFT, of the regression filter for 64-point and 16-point time series, respectively, for various polynomial orders. The small circles mark the locations of the discrete points (zero padding was used to increase the resolution and fill in between the discrete sample points). Figures 3a and 4a show the full frequency response from ±π radians while Figs. 3b and 4b show an expanded version to see more filter detail in the transition region. As the polynomial order increases, the stop bandwidth of the regression filter increases. The horizontal axis ±π corresponds to the Nyquist (or folding), velocity which depends on the PRT (pulse repetition time) and operating frequency; that is, Nyquist velocity = λ/(4 × PRT), where λ is the wavelength. For example, for Fig. 3, S-Pol’s wavelength is 0.1067 m and, if the PRT = 2 ms, the Nyquist velocity is 13.34 m s−1. The plots provide a visual comparison of the bandwidths of the regression filter and WN filter frequency responses. The notch in the frequency response of the WN filter can be represented by vertical lines that show the locations of the right and left edges of the points in the spectrum to be set to zero, inclusive. For example, the dashed vertical lines in Fig. 3 show the location of the left and right edges of a 7-point and 9-point spectral notches for a 64-point time series. In Fig. 4a, the vertical lines show the location of the left and right edges of 5-point and 7-point spectral notches for a 16-point time series.

Fig. 3.
Fig. 3.

The regression filter frequency response (power) for 64-point time series and various polynomial orders as indicated. (a) Full frequency response. The small circles mark the locations of the discrete 64 points. The dashed vertical lines and dotted lines mark the edges of a 7-point and 9-point spectral notch filter, respectively. (b) expanded version of (a) from −π/2 to 0 radians. The vertical lines are meant to facilitate comparison of the regression frequency response with the window and notch clutter filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

Fig. 4.
Fig. 4.

As in Fig. 3, but for 16-point time series. The dashed vertical lines and dotted lines show the locations of the left and right edges of a 5-point and 7-point spectral notch filter, respectively.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

Following the 5-point notch dashed line Fig. 4b, regression filter attenuations where the notch line intersects the polynomial line (circles) for orders 2, 3, 4, and 5 can be estimated as, 0.5, 1.0, 3.0, and 8 dB, respectively. The 5-point WN filter sets this frequency component at the vertical line to zero (linear scale). This is useful when comparing the regression and WN filters’ attenuations.

As a final example, Fig. 5 shows the frequency response for 32-point sequences for odd polynomial orders 3 through 19. We note that the power spectra of clutter signals are usually characterized as Gaussian in shape and the frequency response of the regression filter with its increasing frequency attenuation as the frequency approaches 0 m s−1 appears to provide sufficient attenuation of Gaussian shaped clutter spectra without use of the 0 notch of a WN filter. This will be demonstrated below with simulated clutter signals.

Fig. 5.
Fig. 5.

As in Fig. 3, but for 32-point time series.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

3. Clutter filter performance comparisons via simulated data

To investigate the performance of the regression versus the WN clutter filters, the filters are applied to simulated data that contain both a clutter and a weather signal. The performance of the filters is judged by the bias and standard deviation (Std) of the estimates of power (or SNR of the weather signal), velocity, spectrum width (SW), differential reflectivity ZDR, copolar differential phase ϕDP, and copolar correlation coefficient ρHV (Bringi and Chandrasekar 2001). Two simulators are used with nearly identical results: 1) the I&Q simulator described in Frehlich and Yadlowsky (1994), Frehlich (2000), and Frehlich et al. (2001), which is based on the autocorrelation function of a weather echo as defined in Doviak and Zrnić (1993), and 2) the spectral I&Q simulator described in Chandrasekar et al. (1986). Ground clutter echoes are modeled as weather echo at zero velocity with a narrow spectrum width. The following parameters are used: the power spectra shape is Gaussian; the wavelength is 0.1067 m (S band); the sample interval (i.e., PRT) is 1, 2, or 3.1 ms; the resulting Nyquist velocity is respectively 26.7, 13.3, and 8.6 m s−1; the time series length is 16 and 64 points; and the clutter spectrum width is 0.25 and 0.35 m s−1. These parameters are selected to mimic S-Pol and NEXRAD scan strategies for the experimental clutter data that will be analyzed in a follow-on paper.

a. Clutter rejection comparison

The ability of the regression and WN GCFs to reject clutter is compared by filtering simulated clutter signals and linearly averaging the power spectra for 50 000 realizations. Since a critical component of the WN filter is the selection of the window function (Siggia and Passarelli 2004), the effect of the window function on simulated clutter time series is first examined. Figure 6 shows four cases where the simulated I&Q data are processed by four commonly used windows: 1) rectangular, 2) Hamming, 3) Hann, and 4) Blackman. The PRT, SW, and number of points for the four cases are 2-ms samples, 0.25 m s−1 SW, and 64 points (Fig. 6a); 2-ms samples, 0.25 m s−1 SW, and 32 points (Fig. 6b); 1-ms samples, 0.25 m s−1 SW, and 64 points (Fig. 6c); and 3.1-ms samples, 0.35 m s−1 SW, and 16 points (Fig. 6d). The horizontal axis is velocity. The noise floor is at 0 dB so that the clutter leakage when using the rectangular or Hamming window (green and red lines) is clearly seen. The simulated clutter-to-noise ratio (CNR) is selected to be 45 dB. This is chosen to approximate the phase noise performance of S-Pol and NEXRAD radars. A figure of merit for the phase noise of a radar is expressed as dBc/Hz or just dBc defined as the decibels relative to the carrier signal. This can be estimated by pointing the radar at a fixed target, such as a tower, and examining the resulting power spectra. The “carrier” signal is estimated as the peak clutter power and the ratio of the clutter power to the noise floor is an estimate of the dBc. For 64-point spectra the maximum clutter value, due to radar system noise, is roughly 65 dBc. To convert this to a CNR, 10 log10(64) or 18 dB is subtracted, giving 47 dB, which is close to the used 45-dB CNR. For these large CNR clutter signals, a Blackman window is necessary to use as indicated by Fig. 6 for suppressing the clutter leakage.

Fig. 6.
Fig. 6.

Averaged windowed clutter spectra for rectangular, Hamming, Hann, and Blackman windows; (a) 2-ms samples, 0.25 m s−1 SW, and 64 points; (b) 2-ms samples, 0.25 m s−1 SW, and 32 points; (c) 1-ms samples, 0.25 m s−1 SW, and 64 points; and (d) 3.1-ms samples, 0.35 m s−1 SW, and 16 points.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

To determine the required notch width for the WN filter, the clutter signal is suppressed down to the noise floor by selecting the appropriate notch width. Table 1 gives the notch width and the accompanying power removed for the data given in Fig. 6a for the Blackman window. For a notch width of 9, the rejected power is 44.678 dB, and increasing the notch width further eliminates negligible clutter power.

Table 1.

Clutter suppression as a function of the spectral notch width for a 64-point sequence using a Blackman window and CNR = 45 dB.

Table 1.

Next, a regression filter order with equivalent clutter suppression as the selected WN filter is determined. Table 2 gives the average amount of power eliminated from the same dataset, used for Table 1, as a function of polynomial order. As the polynomial order increases, more power is eliminated from the time series. The amount of eliminated power increases significantly for orders 1–8, but for orders 9 and greater the increase of rejected power becomes small, less than 1 dB. This indicates that a ninth-order filter is sufficient to reject the large majority of the clutter power. From Table 1 for a notch width of 9 the rejected power is 44.678 dB, and the rejected power for the ninth-order regression filter is 44.671 dB, which indicates similar clutter rejection performance.

Table 2.

Clutter suppression as a function of regression filter order for a 64-point sequence with CNR = 45 dB.

Table 2.

Further graphical filter comparison results are shown in Fig. 7 for three cases at S band: 1) 64, 2-ms samples, and 0.25 m s−1 SW; 2) 64, 1-ms samples, and 0.25 m s−1 SW; and 3) 16, 3.1-ms samples, and 0.35 m s−1 SW. The SW is increased to 0.35 m s−1 for the 3.1-ms sample case to reflect the approximate 20° s−1 antenna rotation rates used by NEXRAD for long PRT scans. Figure 7a gives the averaged rectangular windowed spectrum (blue curve) and the averaged Blackman windowed spectrum (orange curve) whereas Fig. 7b shows the WN and regression filter comparisons. The purple curves show the WN filter results for notch widths of 9, 7, and 7 points for cases 1, 2, and 3, respectively. These notch widths are selected to suppress the clutter down to the noise floor. For each of the cases, 3 different orders of regression clutter filter results are also given in Fig. 7b. The regression filter orders that yield equivalent clutter suppression to the given WN filter are orders 9, 5, and 5 for cases 1, 2, and 3, respectively. For case 3 note the WN notch width (purple curve) as compared with the fifth-order regression filter curve (orange). It is clear that the regression filter suppresses the clutter using a smaller filter stop bandwidth. If a weather signal were overlapping the clutter signal, the weather signal would not be attenuated by the regression filter as much as it would be by the WN filter.

Fig. 7.
Fig. 7.

A three-case comparison of clutter rejection capabilities of the spectral window-and-notch filter and the regression filter. (left)The averaged rectangular and Blackman windowed spectra, and (right) the comparison of various filters as labeled. Case 1 is 2-ms samples, 64 points, and 0.25 m s−1 SW; case 2 is 1-ms samples, 64 points, and 0.25 m s−1 SW; and case 3 is 3.1-ms samples, 16 points, and 0.35 m s−1 SW.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

Figure 7 then demonstrates the equivalence of the regression and WN clutter filters’ clutter suppression capabilities for the simulated data. This is not surprising since the regression filter frequency response curves also indicate that the regression filter has similar frequency attenuation characteristics as the WN filter. Figure 7 gives a reference by which to select the regression order for filter comparisons in the next section where the performance of the regression and the WN clutter filters are compared via application to simulated data that consist of a combination of both clutter and weather signal.

b. Filtering simulated weather and clutter data

To compare the performance of the WN and regression filters, the filters are applied to six different simulation scenarios with parameters as given by Table 3. Other secondary simulation parameters are weather ZDR = 0.5 dB, clutter ZDR = −5 dB, weather ϕDP = 45°, clutter ϕDP = −45°, weather ρHV = 0.997, and clutter ρHV = 0.8. Varying these parameters will affect the statistical plots but do not affect the general comparison of the regression and WN filters. The SW of the weather signal is always 2 m s−1, but the ratio of SW to the Nyquist velocity does vary.

Table 3.

A list of simulation parameters used for the statistical plots in Figs. 813. The column labels are, from left to right, figure number, weather SNR, CNR, clutter (Clt) spectrum width, PRT, and the window function (WF) and notch width (NW) used (Blk = Blackman, Han = Hann, and Ham = Hamming).

Table 3.

The various SNRs and CNRs affect the choice of the window function required to suppress the clutter to minimize estimate errors. For example, the scenarios in which SNR = 20 dB and the CNR = 40 dB require a Blackman window. The scenarios in which SNR = CNR = 45 dB require a Hamming or Hann window. Since the SNR is comparable to the CNR, it not necessary to suppress the clutter down to the noise floor with a Blackman window. For the SNR = CNR = 15 dB, a Hamming window is required. The windows are chosen to heuristically optimize the recovery statistics. For a more complete analysis of window selection see Torres and Warde (2014). The dual-polarization variables typically require a more aggressive window selection, as compared with power and velocity variables. It has been shown that clutter with CSRs as low as −10 dB or lower can bias the dual-polarimetric variables (Friedrich and Germann 2009).

1) Interpolation across zero velocity spectrum gap

To reduce the bias in the estimates of SNR, velocity and SW, frequently one interpolates across the 0-velocity gap caused by the clutter filter in the Doppler spectrum. Here we use a simple linear interpolation across the Doppler power spectrum (in logarithm space). For example, for a WN filter with a 7-point notch width, the linear interpolation is done between the first two data points on either side of the 0-velocity gap. For the regression filter, the interpolation points can be determined from where the filter’s frequency response first exceeds a threshold away from 0 velocity. Examples using −2 dB as a threshold are given in Table 4. Though the examples so far have considered linear interpolation, a Gaussian interpolation could be performed on the points similar to GMAP. As is done with NEXRAD data, interpolation is only used for SNR, velocity and SW estimates and not for ZDR, ϕDP, and ρHV estimates where interpolation typically increases estimate errors (Zrnić et al. 2008).

Table 4.

A list of 0-velocity centered interpolation widths as a function of regression order for 64- and 16-point time series.

Table 4.

2) Filtering results

Figures 813 show the average bias and Std estimates, as a function of simulated velocity (Sim V), from 50 000 time series realization per simulation velocity increment. The mean and Std are calculated in standard fashion. The mean and Std of the power are calculated in decibel space. The mean and Std of ZDR are calculated in linear space so that fractional error or uncertainty is used (Taylor 1997),
F(ZDR)=10log10(1+σzdr/ZDR¯),
where F() denotes fractional error (dB), σzdr is the Std of ZDR, and ZDR¯ is the mean of ZDR. The biases are defined as the calculated mean minus the simulated value for a radar variable. For Figs. 813 the top two rows show SNR, velocity, and SW, with bias and Std in panels a and b, respectively, and the bottom two rows similarly show the polarimetric variables ZDR, ϕDP, and ρHV. The magenta curves show results for the WN filtered data, and the green, red, and light-blue curves are for regression filtered data. The middle polynomial order (red curves) in general yields superior error statistics. The used windows and notch widths as well as the polynomial orders are given in the figure keys. The figures can be examined in two regions: 1) around 0 velocity where the effects of the clutter filters on the weather signal are evident and 2) away from 0 velocity where the weather signal is attenuated very little by the filters. In general, the biases in region 2 due to each filter are similar. However, the Stds in region 2 are nearly always significantly higher for the WN filter as compared with the regression filter, and this is the principal advantage of the regression filter over the WN filter. For Figs. 8, 10, and 11 where a Blackman window is required, the Std increase is from about 45% to 55% for the WN data as compared with the regression Stds. The spectrum width estimated from the R0/R1 (the zeroth and first autocorrelation lags; Bringi and Chandrasekar 2001) is more variable but the regression filter in general produces reduced bias and Std. Around the 0-velocity region the results are similar with the regression filter showing reduced Stds as compared with the WN filter.
Fig. 8.
Fig. 8.

(a) Bias and (b) standard deviation (Std) comparison statistics for the 64-point and PRT = 2-ms case. The horizontal axis is simulated velocity. The weather SNR is 20 dB, and the CNR is 40 dB. From left to right and then from top to bottom the variables are SNR, velocity, SW, ZDR, ϕDP, and ρHV. There is no 0-velocity interpolation for ZDR, ϕDP, and ρHV. The Blackman window with a 9-point notch width is used for the WN filter, and three regression filtered curves are shown.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

Fig. 9.
Fig. 9.

As in Fig. 8, but the weather SNR is 45 dB and the CNR is 45 dB. A Hamming window with a 9-point notch width is used for the WN filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

Fig. 10.
Fig. 10.

As in Fig. 8, but for a PRT of 1 ms. A Blackman window with a 7-point notch width is used for the WN filter. The Nyquist velocity is 26.68 m s−1, but the horizontal axis is terminated at 14 m s−1 since the bias and Std do not vary beyond 14 m s−1.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

Fig. 11.
Fig. 11.

As in Fig. 8, but for the 16-point and PRT = 3.1-ms case. A Blackman window with a 7-point notch width is used for the WN filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

Fig. 12.
Fig. 12.

As in Fig. 11, but the weather SNR is 45 dB and the CNR is 45 dB. A Hann window with a 7-point notch width is used for the WN filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

Fig. 13.
Fig. 13.

As in Fig. 8, but the weather SNR is 15 dB and the CNR is 15 dB. A Hamming window with a 7-point notch width is used for the WN filter.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

In all of the simulation scenarios, the green curve for ZDR, ϕDP, and ρHV show higher Std as compared with the other regression curves (red and light blue). The green lines are always the lowest-polynomial-order curves and this indicates that the lowest-order polynomial does not reject sufficient clutter power. This is also supported in the ρHV bias and Std plots where the green curves show increased bias and Std over the other regression curves. However, for the SNR, velocity, and SW the green curves provide in general reduced bias and Std. This shows that the dual-polarimetric variables are more sensitive to clutter residue as compared with SNR, velocity, and SW (Friedrich and Germann 2009) around the 0-velocity region.

For the PRT = 3.1 ms and 16-samples data of Figs. 11 and 12, the detrimental effect of the WN filter is particularly noticeable in the bias of SNR, velocity, and the Std of ZDR, ϕDP, and ρHV as compared with the red regression (order 4) curves. For Fig. 12, even though the SNR = CNR = 45 dB, the Hann window is necessary whereas the less aggressive Hamming window is sufficient for the other scenario with SNR = CNR = 45 dB in Fig. 9.

In Fig. 13 SNR = CNR = 15 dB; however, a Hamming window is still necessary to suppress the clutter—a rectangular window produced increased bias and Std over the Hamming windowed data. The WN filtered data in Fig. 13 have, in general, about 17%–35% increased Std for all variables (the light-blue curve is for ρHV).

For Fig. 8 the SNR bias of the WN curve around 0 velocity does show a slightly smaller bias as compared with the regression curves. This is due to the nature of the interpolation used. The same data are shown in Fig. 14 but without interpolation, and in this case, the WN filter eliminates more of the weather signal, thus creating a larger bias. A more sophisticated interpolation scheme may improve the bias of the regression filtered data.

Fig. 14.
Fig. 14.

SNR statistics as in Fig. 8, but with no interpolation across the zero-velocity notch. Here the WN filtered data show more bias around zero velocity as compared with the regression filtered data (cf. Fig. 8).

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

These above six scenarios for PRTs of 1, 2, and 3.1 ms and various SNRs and CNRs clearly demonstrate the advantage of using the regression clutter filter that does not require the use of a window function such as the Blackman, Hann, or Hamming, which effectively reduces the number of independent samples thereby increasing the Stds of the weather variable estimates. The Stds statistics do vary as modeling parameters change. However, we anticipate that the regression statistics will still, in general, be better that those of an equivalent WN filter.

4. Filtering longer length times series

As the length of sequences to be regression filtered increases, high-order polynomial fits are required to maintain similar frequency response filter characteristics, and this could become numerically problematic for very high filter orders. For the sampling strategies considered thus far, this has not been an issue. Nonetheless, we investigate here strategies for filtering longer length sequences where the use of higher-order polynomial fits may be undesirable. The solution is to simply break the sequence into a series of subsequences, filter the subsequences and then rejoin the filtered subsequences. A related filtering technique, termed Savitzky–Golay (Savitzky and Golay 1964; Orfanidis 1996), is also discussed since that technique was investigated by Torres and Zrnić (1998, 1999) for clutter filtering. We examine three types of sequence partitioning using a 64-point sequence as an example as illustrated in Fig. 15: original 64-point sequence (Fig. 15a), four blocks of contiguous 16-point sequences (Fig. 15b), five blocks of overlapping 16-point sequences (overlapping technique) (Fig. 15c), and Savitzky–Golay technique with a 15-point kernel (Fig. 15d).

Fig. 15.
Fig. 15.

Partitioning a 64-point sequence: (a) 64 point, (b) four contiguous blocks of 16-point sequences, (c) five 16-point blocks with 4-point overlap, and (d) Savitzky–Golay (SG) with a 15-point kernel (or window). The first and last SG 15-point windows are shown. A polynomial fit is executed, and then the center point of that fit is placed in the filtered sequence as indicated by the green circles. The window is moved forward one point and the process is repeated. This process is akin to FIR filtering.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

Savitzky–Golay (SG) is a generalized finite impulse response (FIR) filter, where its filtering coefficients are determined from a least squares polynomial fit (Orfanidis 1996) and these coefficients depend only upon the length of the filter (termed the kernel, N) and the polynomial order. Thus, SG uses a sliding window or kernel to select subsequences of length N (N is odd for symmetry about the central point of the filter) across the sequence to be filtered. For the selected 15 points, the central point of the polynomial fit is the filtered version of the original signal at that central point. As with all FIR filters, the beginning and ending points of the sequence to be filtered require special attention. Here, the (N + 1)/2 beginning points and (N + 1)/2 ending points are regression filtered with the N-point kernel and the first and last (N + 1)/2 points are retained in the overall filtered sequence. As with FIR filtering, the SG kernel is moved across the interior points of the sequence to be filtered. Thus, for a 64-point sequence with N = 15, there will be 48 regression fits to the interior points plus two fits on the beginning and ending points of the sequence for a total of 50 polynomial fits to complete the SG filtering process. The SG filtered sequence is then subtracted from the original sequence to obtain the clutter filtered sequence. For a given kernel and given polynomial order, the SG kernel weights can be precomputed independent of the data to be filtered. Once these weights are determined, they are convolved with the data to be filtered just as is done with any FIR filter. See Orfanidis (1996) for details.

For the five-block, 16-point overlapping technique, the regression filtered overlapping points are averaged to smooth the 4-point transition regions. The need for this is clarified below. Consider two consecutive 16-point sequences that have a 4-point overlap as illustrated in Fig. 15. Let the first 16-point filtered sequence be a and the second 16-point filtered sequence be b. Thus, points a13, a14, a15, and a16 overlap with b1, b2, b3, and b4 and need to be combined to form the final filtered 64-point sequence. Weights are assigned to the last four members of a denoted Wa13, Wa14, Wa15, and Wa16. Similarly, assign weights to the overlapping b points Wb1, Wb2, Wb3, Wb4. The weights are Wa13 = 0.8, Wa14 = 0.5, Wa15 = 0.5, Wa16 = 0.2, Wb1 = 0.2, Wb2 = 0.5, Wb3 = 0.5, and Wb4 = 0.8.

The two 4-point overlapping parts of the two filtered sequences a and b are combined pairwise in a point-by-point fashion by using these weights. This averaging scheme is applied to each overlapping pair of 16-point sequences. As shown in Fig. 15, for the 64-point length sequence with five overlapping 16-point blocks example, there are four overlapping segments of 4 points each. Again, the reason for the weighted average is to provide a smooth transition from 16-point block to 16-point block, and this is illustrated via spectra next. These weights are heuristic and were found via trial and error based on observations of resulting filtered data spectra.

These filtering techniques are now compared via experimental clutter time series data. The data were gathered by S-Pol at 1231:25 UTC 13 March 2019 at a 1.5° elevation angle and 8.625 km, a scan rate of 8° s−1, and a PRT of 2 ms (NCAR/EOL Remote Sensing Facility 2020). Figure 16 compares the effect of the filtering schemes on the S-Pol ground clutter only time series. Figure 16a shows the unfiltered power spectrum. Figure 16c shows an 11th-order regression filter on the 64-point sequence and is considered a reference; Fig. 16b shows the four contiguous blocks of 16 points using a third-order fit on each 16 points (the spectrum due to overlapping filter is nearly identical and is omitted for brevity). Figure 16d shows the spectrum for the SG filtered data using a third-order fit on each 15-point kernel. The SG spectrum does not exhibit a deep attenuation notch at 0 velocity as do the other two filters though the noise floor level is similar. This lack of a deep notch at zero velocity is the usual characteristic for the SG filtered S-Pol clutter spectra.

Fig. 16.
Fig. 16.

S-Pol 64-point, clutter only spectra comparison: (a) unfiltered, (b) four contiguous blocks of 16 points, (c) regression order 11, and (d) SG 15-point kernel. Here, (b) and (d) use a third-order regression filter on each block (kernel).

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

Next, the same filtering techniques are applied to S-Pol data with weather and clutter echo, which reveals an issue with the filtering technique that uses the contiguous four blocks of 16 points. The data were gathered on 1231:25 UTC 13 March 2019, a scan rate of 8° s−1, and a PRT of 2 ms (NCAR/EOL Remote Sensing Facility 2020). Figure 17 shows spectra for the unfiltered data (Fig. 17a), 11th-order regression filtered data (Fig. 17b), four contiguous blocks of 16 points with a third-order regression filter (Fig. 17c), five overlapping blocks of 16 points with a third-order regression filter (Fig. 17d), and SG filtered data (Fig. 17e). The 11th-order regression filtered spectrum in Fig. 17b shows a deep notch at 0 velocity and can be considered as a reference for the other filters. The contiguous four blocks of 16-point data of Fig. 17c shows odd anomalous oscillations from −13 to −4 m s−1, well above the noise floor of Fig. 17b. This is due to slight discontinuities between the 16-point filtered blocks and is an artifact of the filter. To alleviate this anomalous oscillatory behavior, five blocks of 16 points with 4-point overlap are used, and the resulting spectrum is given in Fig. 17d. The anomalous oscillatory behavior has been mitigated, and the noise floor is similar to the 11th-order regression filtered spectrum; however, the notch at 0 velocity is not as deep. Figure 17e shows the spectrum of the data after filtering using SG (15-point kernel). Again, there is not a deep notch at 0 velocity as compared with Fig. 17b. The SG requires 50 regression fits for this 64-point example, whereas the overlap technique requires only 5. However, since the filtering coefficients for the SG technique can be precomputed, the SG algorithm is relatively fast (Orfanidis 1996).

Fig. 17.
Fig. 17.

An S-Pol weather signal spectrum with various filters applied: (a) unfiltered, (b) 11th-order regression, (c) four contiguous blocks of 16 points, (d) five overlapping blocks of 16 points, and (e) SG 16-point kernel. Here, (c), (d), and (e) use a third-order regression filter on each 16-point block.

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

As was done previously, the above filters’ frequency responses can be calculated and compared using Eq. (6). Figure 18 shows the frequency response of the 11th-order regression on 64 points (dashed green curve), five-block overlap (black), SG using a third-order polynomial, and SG using a fourth-order polynomial, with Fig. 18b being an expanded version of Fig. 18a. The 11th-order regression offers the sharpest transition from the passband to the stopband; it is flat in the passband and thus is considered to have the best performance. The SG filters have a more gradual transition from the pass to stop bands. Also, the SG filters have a sizable ripple in the pass band of about 1 dB. The fourth-order SG filter is given for comparison. Figures 16, 17, and 18 illustrate that using a regression filter over the entire length of the time series yields the best filter characteristics. However, if it is required to divide a long sequence into subsequences, then fewer subsequences are better to minimize the number of subsequence overlaps. The SG filter does not perform as well as the other regression techniques presented here.

Fig. 18.
Fig. 18.

(a) Power frequency response comparison for several filters for a 64-point length sequence. (b) An expanded version of (a). Shown are an 11th-order regression on 64 points (dashed green curve); five blocks of 16 points with a 4-point overlap and a third-order regression (black curve); SG, 15-point kernel, and order 3 (blue curve); and SG, 15-point kernel, and order 4 (red curve).

Citation: Journal of Atmospheric and Oceanic Technology 38, 8; 10.1175/JTECH-D-20-0026.1

5. Summary and conclusions

The performance of a newly proposed regression clutter filter was compared with spectrum-based window and notch (WN) filters that are currently popular and in widespread use (e.g., GMAP). Regression filters are attractive since they do not require the use of a time domain window function to contain clutter leakage. The use of window functions causes loss of independent samples and the corresponding increase of variance of the radar variable estimates. The window functions used were the Hamming, Hann, and Blackman, which attenuate the time series signal power by 4.01, 4.19, and 5.23 dB, respectively, for 64-point times series. Frequency responses were given for the regression filter for 64-point, 32-point, and 16-point data sequences for various polynomial orders. For a fixed polynomial order, as the sequence length increases, the filter bandwidth decreases, and for a fixed sequence length, as the polynomial order increases, the filter bandwidth increases. The ability of the regression filter to suppress clutter as well as a WN filter was demonstrated via simulated clutter data for various simulation parameters.

The efficacy of the regression filter for recovering radar variables was demonstrated via six simulation scenarios for sequences with 1-, 2-, and 3.1-ms sample intervals (i.e., PRTs) for various combinations of weather SNRs, CNRs, and noise combinations. The resulting simulated time series were processed by three regression filters and a WN filter and the resulting averaged biases and Stds were compared for SNR (power), velocity, SW, ZDR, ϕDP, and ρHV. In general, the modeling results showed that Std of the estimated radar variables for the weather signal were about 35%–55% greater for the WN filter as compared with the regression filter. The primary cause of this increase in Std was the window function used for the WN filter, which attenuates the time series thereby reducing the effective number of available independent samples.

The viability of regression filtering longer length sequences that may require very high-order polynomial fits was investigated by dividing the time series into shorter lengths and then regression filtering the shorter length time series so that lower-order regression fits could be employed. The filtered shorter sequences could then be concatenated, thereby yielding a full length filtered time series. However, small discontinuities can be created at the boundaries so that anomalous oscillations can appear in the filtered spectra. The solution to this was to divide the long sequence into shorter overlapping length sequences, which were then filtered. The filtered sequence can be rejoined where a weighted average is used at the overlapping points.

The Savitzky–Golay filtering technique was investigated and compared with the proposed regression filter techniques since it was considered by Torres and Zrnić (1999). The spectra after SG filtering frequently looked anomalous, with shallow or even no discernible notch at 0 velocity. Furthermore, the frequency response of the SG filters was inferior (ripple in the passband and a less sharp transition from the passband to the stopband) to the proposed regression filtering techniques and offered no apparent advantage for clutter filtering.

To successfully implement a regression clutter filter, it is important that a set of orthogonal polynomials, such as the Forsythe polynomials, be used to simplify the matrix inversion in the least squares fit, which significantly reduces round-off error. The filtering coefficients can be precalculated for the length of the time series to be filtered so that execution time is reduced. Source code in C++ and Fortran for the least squares polynomial fit algorithm using Forsythe polynomials can be found at the web page by Moreau (2018). The application of the regression filter to experimental radar data from both S-Pol and NEXRAD will be given in a follow-up paper.

Acknowledgments

This work was supported in part by the Radar Operations Center (ROC) of Norman, Oklahoma. The authors acknowledge the EOL/RSF technical staff for their time, effort, and interest in the collection of the experimental data used in this paper. The authors also thank the anonymous reviewers, whose suggestions greatly improved the paper. Doctor Jeff Keeler is also acknowledged for his in-depth reviews. This material is based upon work supported by the National Center for Atmospheric Research, which is a major facility sponsored by the National Science Foundation under Cooperative Agreement 1852977. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the National Science Foundation.

APPENDIX A

Least Squares Fit with Forsythe Polynomials

The set of Forsythe orthogonal polynomials for a least squares regression fit is particularly well suited because of low round-off error and efficiency (Carnahan et al. 1969; Ruckdeschel 1981). An nth-order orthogonal polynomial regression of y against x can be written as
y=j=0nαjPj,
where the Pj are jth-order polynomials and the α are to be determined from a least squares fit. Define the matrix P as
P=[1P1(x1)P2(x1)Pn(x1)1P1(x2)P2(x2)Pn(x2)1P1(xm)P2(xm)Pn(xm)].
Then a matrix equation can be written:
Y=Pα,with
α=(PTP)1PTY,
where Y is a column vector of length m, P is a matrix with n + 1 columns and m rows, and α is the coefficient column matrix of length m, where m is the length of the data to be fitted. If the Pj are orthogonal polynomials that would force the off-diagonal terms of the matrix PTP to be zero so that the solution coefficients can be found with simple summations,
αj=i=1mPj(xi)yii=1mPj2(xi).
Forsythe (1957) found a set of orthogonal polynomials defined by the recursion relationships
Pj(x)=(xgj)Pj1(x)djPj2(x),
with P−1(x) = 0 and P0(x) = 1,
gj=i=1mxiPj12(xi)i=1mPj12(xi),and
dj=i=1mxiPj1(xi)Pj2(xi)i=1mPj22(xi).
The advantage of this technique is that no matrix inverse is required and the number of necessary calculations is significantly reduced so that roundoff error is minimized. Since the Forsythe polynomials depend only on the xi, they can be precomputed, thus making the algorithm more efficient. Also, if a higher-order regression is needed, the higher degree can be added on with the iterative formula. It is also computationally efficient to normalize the Pj,
Pjorth=Pj/[i=1mPj2(xi)]0.5
so that the Pj are orthonormal and (PTP)−1 = I, where I is the identity matrix. Then,
i=1mPj2orth2(xi)=1,
which makes the denominator of Eq. (A5) equal to 1. Once the Forsythe polynomials are calculated for a particular length sequence, Eq. (A5) is used to filter subsequence sequences of the same length.

APPENDIX B

Using White Noise to Determine the Frequency Response

The algorithm to numerically calculate a linear time invariant filter frequency response is based on the correlation of the white noise input and the filter output sequence as well as the autocorrelation function of the input (Oppenheim and Schafer 1989). The algorithm proceeds as follows:

  1. A desired time series length and regression order is first selected.
  2. A complex white noise time series is generated.
  3. Apply the regression filter. A polynomial fit is then calculated for the real and imaginary white noise sequences. The polynomial fit is then subtracted from the noise sequences.
  4. To obtain a more detailed frequency response, the filtered time series and the white noise input time series can be zero padded to arbitrary length.
  5. An FFT algorithm is applied to the input and filter output sequences yielding Φx and Φy, respectively.
  6. Calculate the cross-correlation spectrum, Φxy as Φxy=ΦxΦy*.
  7. Calculate the input autocorrelation spectrum as Φxx=ΦxΦx*.
  8. Calculate the filter frequency response as H = Φxyxx.
  9. Average as many filter frequency responses as needed. The result is the regression filter’s frequency response.

REFERENCES

  • Bjaerum, S., H. Torp, and K. Kristoffersen, 2002: Clutter filter design for ultrasound color flow imaging. IEEE Trans. Ultrason. Ferroelectr. Freq. Control, 49, 204216, https://doi.org/10.1109/58.985705.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bringi, V., and V. Chandrasekar, 2001: Polarimetric Doppler Weather Radar. Cambridge University Press, 636 pp.

  • Carnahan, B., H. Luther, and J. Wilkes, 1969: Applied Numerical Methods. John Wiley and Sons, 604 pp.

  • Chandrasekar, V., V. Bringi, and P. Brockwell, 1986: Statistical properties of dual-polarized signals. Preprints, 23rd Conf. on Radar Meteorology, Snowmass, CO, Amer. Meteor. Soc., 193–196.

  • Darlington, R., and A. Hayes, 2017: Regression and Linear Models. Guilford Press, 661 pp.

  • Doviak, R. J., and D. S. Zrnić, 1993: Doppler Radar and Weather Observations. 2nd ed. Academic Press, 562 pp.

  • Forsythe, G., 1957: Generation and use of orthogonal polynomials for data fitting with a digital computer. J. Soc. Ind. Appl. Math., 5, 7488, https://doi.org/10.1137/0105007.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Frehlich, R., 2000: Simulation of coherent Doppler lidar performance for space-based platforms. J. Appl. Meteor., 39, 245262, https://doi.org/10.1175/1520-0450(2000)039<0245:SOCDLP>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Frehlich, R., and M. J. Yadlowsky, 1994: Performance of mean-frequency estimators for Doppler radar and lidar. J. Atmos. Oceanic Technol., 11, 12171230, https://doi.org/10.1175/1520-0426(1994)011<1217:POMFEF>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Frehlich, R., L. Cornman, and R. Sharman, 2001: Simulation of three-dimensional turbulent velocity fields. J. Appl. Meteor., 40, 246258, https://doi.org/10.1175/1520-0450(2001)040<0246:SOTDTV>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Friedrich, K., and U. Germann, 2009: Influence of ground clutter contamination on polarimetric radar parameters. J. Atmos. Oceanic Technol., 26, 251269, https://doi.org/10.1175/2008JTECHA1092.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Harris, F., 1978: On the use of windows for harmonic analysis with the discrete Fourier transform. Proc. IEEE, 66, 5183, https://doi.org/10.1109/PROC.1978.10837.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hubbert, J. C., M. Dixon, S. Ellis, and G. Meymaris, 2009a: Weather radar ground clutter. Part I: Identification, modeling and simulation. J. Atmos. Oceanic Technol., 26, 11651180, https://doi.org/10.1175/2009JTECHA1159.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hubbert, J. C., M. Dixon, and S. Ellis, 2009b: Weather radar ground clutter. Part II: Real-time identification and filtering. J. Atmos. Oceanic Technol., 26, 11811197, https://doi.org/10.1175/2009JTECHA1160.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ice, R. L., R. D. Rhoton, D. S. Saxion, N. K. Patel, D. Sirmans, D. A. Warde, D. L. Rachel, and R. G. Fehlen, 2004: Radar Operations Center (ROC) evaluation of the WSR-88D Open Radar Data Acquisition (ORDA) system signal processing. 20th Int. Conf. on Interactive Information Processing Systems for Meteorology, Oceanography, and Hydrology, Seattle, WA, Amer. Meteor. Soc., 5.5, https://ams.confex.com/ams/84Annual/techprogram/paper_70692.htm.

  • Ice, R. L., G. T. McGehee, R. D. Rhoton, D. S. Saxion, D. A. Ward, R. Guenthe, D. Sirmans, and D. L. Rachel, 2005: Radar Operations Center (ROC) evaluation of new signal processing techniques for the WSR-88D. 21 Int. Conf. on Interactive Information Processing Systems for Meteorology, Oceanography, and Hydrology, San Diego, CA, Amer. Meteor. Soc., P1.4, https://ams.confex.com/ams/Annual2005/techprogram/paper_85859.htm.

  • Ice, R. L., and et al. , 2009: Automatic clutter mitigation in the WSR-88D, design, evaluation, and implementation. 34th Conf. on Radar Meteorology, Williamsburg, VA, Amer. Meteor. Soc., P5.3, https://ams.confex.com/ams/34Radar/techprogram/paper_155409.htm.

  • Kadi, A. P., and T. Loupas, 1995: On the performance of regression and step-initialized IIR clutter filters for color Doppler systems in diagnostic medical ultrasound. IEEE Trans. Ultrason. Ferroelectr. Freq. Control, 42, 927937, https://doi.org/10.1109/58.464825.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lutz, J., P. Johnson, B. Lewis, E. Loew, M. Randall, and J. VanAndel, 1995: NCAR S-Pol: Portable polarimetric S-band radar. Preprints, Ninth Symp. on Meteorological Observations and Instrumentation, Charlotte, NC, Amer. Meteor. Soc., 408–410.

  • Moreau, J.-P., 2018: Numerical analysis. Orange, http://jean-pierre.moreau.pagesperso-orange.fr/.

  • NCAR/EOL Remote Sensing Facility, 2020: Marshall S-PolKa radar time series, S-band, version 1.0. UCAR/NCAR Earth Observing Laboratory, accessed 11 February 2020, https://doi.org/10.26023/T8VD-F2C8-900K.

    • Crossref
    • Export Citation
  • Oppenheim, A., and R. Schafer, 1989: Discrete-Time Signal Processing. Prentice Hall, 879 pp.

  • Orfanidis, J., 1996: Introduction to Signal Processing. Prentice Hall, 783 pp., https://www.ece.rutgers.edu/~orfanidi/intro2sp/.

  • Ruckdeschel, F., 1981: Basic Scientific Routines. Vol. II. BYTE/McGraw-Hill, 790 pp.

  • Sachidananda, M., and D. Zrnić, 1999: Systematic phase codes for resolving range overlaid signals in a Doppler weather radar. J. Atmos. Oceanic Technol., 16, 13511363, https://doi.org/10.1175/1520-0426(1999)016<1351:SPCFRR>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Savitzky, A., and M. Golay, 1964: Smoothing and differentiation of data by simplified least squares procedures. Anal. Chem., 36, 16271639, https://doi.org/10.1021/ac60214a047.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Siggia, A., and J. Passarelli, R., 2004: Gaussian model adaptive processing (GMAP) for improved ground clutter cancellation and moment calculation. Proc. Third European Conf. on Radar in Meteorology and Hydrology, Visby, Gotland, ERAD, 67–73.

  • Taylor, J., 1997: An Introduction to Error Analysis. University Science Books, 327 pp.

  • Torp, H., 1997: Clutter rejection filters in color flow imaging: A theoretical approach. IEEE Trans. Ultrason. Ferroelectr. Freq. Control, 44, 417424, https://doi.org/10.1109/58.585126.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Torres, S., and D. Zrnić, 1998: Ground clutter canceling with a regression filter. NSSL Interim Tech. Rep., 35 pp.

  • Torres, S., and D. Zrnić, 1999: Ground clutter canceling with a regression filter. J. Atmos. Oceanic Technol., 16, 13641372, https://doi.org/10.1175/1520-0426(1999)016<1364:GCCWAR>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Torres, S., and D. A. Warde, 2014: Ground clutter mitigation for weather radars using the autocorrelation spectral density. J. Atmos. Oceanic Technol., 31, 20492066, https://doi.org/10.1175/JTECH-D-13-00117.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • UCAR/NCAR Earth Observing Laboratory, 1996: S-PolKa: S-band/Ka-band dual polarization, dual wavelength Doppler radar. UCAR/NCAR Earth Observing Laboratory, accessed 23 January 2017, https://doi.org/10.5065/D6RV0KR8.

    • Crossref
    • Export Citation
  • Zrnić, D., V. Melnikov, and I. Ivic, 2008: Processing to obtain polarimetric variables on the ORDA. NSSL Tech. Rep., 55 pp.

1

After the clutter signal is notched, some weather signal can be recovered through an interpolation scheme that is based on the underlying weather spectrum being Gaussian shaped. This is done in the GMAP algorithm.

Save