1. Introduction

With the advent of modern digital receivers, range-oversampling processing has become practical for weather radars. By sampling the signal at a range resolution finer than that of the transmitter pulse and applying a linear transformation, range-oversampling processing techniques can be utilized to reduce the variance of estimates and/or reduce the required observation (dwell) times. The whitening transformation and digital matched filter were the first transformations that were introduced for range-oversampled data (Torres 2001). Later, pseudowhitening transformations were proposed to mitigate the noise enhancement effects of the whitening transformation (Torres et al. 2004). More recently, adaptive pseudowhitening was implemented on the National Weather Radar Testbed Phased-Array Radar (NWRT PAR; Curtis and Torres 2011). This technique applies a nearly optimal transformation at each range location to provide excellent performance for differing conditions. All of these transformations have one thing in common—they directly depend on the normalized range correlation matrix.

A mismeasurement of the normalized range correlation matrix can cause biases in reflectivity estimates and a less-than-expected variance reduction (Torres and Curtis 2013). Thus, accurately measuring the range correlation is an important first step for range-oversampling processing. To accurately measure the range correlation, it is important to understand which parameters determine it and how these change over time. A useful way to think of the range correlation is in terms of the modified pulse. The modified pulse combines the effects of the transmitted pulse and the impulse response of the receiver to capture the two effects that contribute to the range correlation. Hence, hardware changes to the transmitter or receiver and drifts over time can affect the modified pulse (and the range correlation). If these variations over time are significant, then the measurement should be repeated often enough to make sure that performance is not degraded.

One way to measure the modified pulse is to use the returns from a strong point target (Torres and Curtis 2013). This method can give a reasonable measurement of the range correlation, but it suffers from the following issues: finding a perfectly isolated target with sufficiently narrow range extent, obtaining uncorrupted returns from the target, and automating this process for different radar sites. Another possible method is to inject a replica of the transmitter pulse into the receiver front end, but this may be infeasible on some operational radars. The goal is to find a way to easily measure the range correlation in real time to reflect any fluctuations in the modified pulse with minimal effects on data collection. With these constraints, it is natural to look at possible ways to measure the range correlation directly from the data.

An early method for measuring the range correlation from data relied on weather returns that had relatively uniform reflectivity (Ivić et al. 2003). This method required manual selection of data with the specified characteristics, and it was devised primarily to demonstrate the feasibility of range-oversampling processing for a research radar. In this paper, we propose an extension of this method that does not require a particular type of radar return to be effective, making it more suited for automation and operational use. The method uses raw range-oversampled data from normal radar scans in a separate processing chain. It operates on the complex time series data before the ground clutter filter and thus is independent from the typical signal processing chain that produces meteorological data. The method uses radar returns from weather and ground clutter and will work with any type of scenario as long as there are enough samples for an accurate measurement.

The rest of the paper is organized as follows. Section 2 describes the method for range correlation measurement based on a robust estimator of the correlation coefficient. Section 3 introduces simulations to show that the range correlation estimates are only marginally biased and to determine the amount of data needed for an accurate measurement. In section 4, range correlation measurements from real data are examined. The changes in range correlation over time are illustrated on the NWRT PAR along with the effects of using an inaccurate measurement.

2. Range correlation measurement

The range correlation at lag l is defined as

View Expanded

where E[.] is the expected value, the asterisk (*) denotes the complex conjugate, and V is the range-oversampled, complex times series data. Integer m indexes sample time at increments given by the pulse repetition time (T_s); n and l index the range time and range lag at increments given by the time series sampling time (T_r). The elements of the normalized range correlation matrix, , can be obtained from (1) as

View Expanded

where denotes the complex element in the ith row and jth column of matrix , is the range correlation coefficient, and L is the range-oversampling factor. Typically, LT_r is chosen to match the width of the modified pulse (Curtis and Torres 2011), but the analysis herein is valid in general. The normalized range correlation matrix is Hermitian Toeplitz with ones along its main diagonal; thus, only L − 1 entries [] are needed to define it.

As mentioned in the introduction, range-oversampling processing relies on accurate measurement of ; thus, an estimator of the range correlation coefficient in Eq. (2) is needed. The traditional estimator of the correlation coefficient is given by the Pearson product-moment correlation coefficient (a.k.a. the sample correlation coefficient; Pearson 1896), and there are several ways it can be extended to our application. The Pearson product-moment correlation coefficient can be generically defined as the sum of lagged-sample products normalized by the powers in the pairs, which in its most generic form can be written as

View Expanded

where the sample set is partitioned into subsets A_j (1 ≤ j ≤ J). The first stage of averaging is done over each subset A_j, which produces J ratios that will then be averaged in a second stage. Different results can be obtained depending on how these subsets are defined (i.e., subsets may contain only one sample, a group of samples, all samples in one pulse, or all samples in a radial or dwell), leading to different variations on the same estimator. We thoroughly evaluated these variations and found that the one with the overall best performance is the one doing the first stage of averaging at the radial level; that is,

View Expanded

where N and M are the total number of samples in range and sample time in a radial, respectively.

Because the goal is to measure the range correlation from radar data, Eq. (4) cannot be used directly, as it is known to be highly affected by outliers (Huber and Ronchetti 2009). Several techniques have been proposed to construct robust estimators of the correlation coefficient that significantly reduce the effects of outliers (Shevlyakov and Smirnov 2011). However, we are only concerned with the effects of noiselike and saturated samples, so we give preference to simpler variants of the traditional estimator in Eq. (4). Once again, several simple mechanisms to reduce the effects of outliers can be devised, and a few were evaluated. Of these, the one that exhibited the best performance handling saturated and noiselike samples is described next.

A robust version of the estimator in Eq. (4) can be constructed by using only “valid” sample pairs that meet instantaneous signal-to-noise ratio (SNR) and voltage-saturation conditions; that is, a minimum SNR threshold (denoted by SNR_min and expressed in decibels) is set to exclude noiselike samples, and a system-dependent maximum voltage threshold (V_max) is set to exclude potentially saturated samples (e.g., returns from ground targets located very close to the radar). Mathematically, a sample is classified as “invalid” for the correlation coefficient estimator if

View Expanded

where Re and Im stand for real and imaginary parts of a complex number, respectively, and N_lin is the receiver noise power in linear units. Because samples are correlated in range, additional samples in a range–time radius of influence K from invalid samples must be excluded as well (Fig. 1). The radius of influence should be chosen as small as possible to exclude the least number of samples, but large enough so that the effect of invalid samples on valid ones is insignificant. Thus, a good choice for K is L − 1; that is, K is the largest lag before the range correlation becomes negligible. With this, a robust version of Eq. (4) that only uses valid sample pairs is given by

View Expanded

where X(l, m) is the set of indexes to the first sample of pairs that are considered valid for the estimation of the range correlation at lag l. It is important to note that the total number of sample pairs used by this estimator is given by , where |.| denotes the cardinality of a set, and this number may be different for different correlation lags.

View Full Size

Example of valid and invalid sample pairs for the proposed correlation coefficient estimator. A valid sample pair is defined as two samples spaced by lag l, both outside the radius of influence of any invalid sample. Invalid samples are those below SNR_min or reaching V_max. In this figure, solid lines indicate a valid sample pair for l = 1, dashed lines indicate invalid samples, and dotted lines indicate samples in their radius of K.

Citation: Journal of Atmospheric and Oceanic Technology 30, 12; 10.1175/JTECH-D-13-00090.1

• Download Figure
• Download figure as PowerPoint slide

Because it is not known a priori whether there will be enough valid sample pairs to reliably estimate the range correlation matrix, the estimator is extended to work across radial (or dwell) boundaries. To fit typical multiprocessor signal processing architectures where radials may be processed independently by different processors, Eq. (6) is applied on individual radials, and the results are combined in a weighted average. Among all the weights we considered (i.e., uniform, based on the total power of valid pairs, based on the total number of valid pairs), the ones given by the normalized number of valid sample pairs used in each estimate worked the best; that is, for a set of R radials, the range correlation coefficient is finally estimated as

View Expanded

where index r is added to indicate radial number. This weighting prevents less reliable estimates obtained from radials with fewer valid sample pairs from significantly influencing the final result.

It is well known that if there are no outliers and the data are normally distributed, then the sample correlation coefficient in Eq. (4) is asymptotically unbiased and efficient (e.g., Bachman 2004). However, it is important to determine whether the asymptotically unbiased property carries over to the proposed estimator in Eq. (7), and, from a practical standpoint, to establish the total number of valid sample pairs that are required to reliably measure the range correlation. These questions will be answered by evaluating the statistical performance of the proposed estimator using simulations.

3. Simulations

In this section, simulations are used to evaluate the statistical performance of the proposed range correlation measurement technique. It will be shown that, when applied to realistic radar data, the proposed estimator is slightly biased but consistent (i.e., the standard deviation of estimates decreases with increasing amounts of data).

Synthetic and real range profiles of SNR, Doppler velocity, and spectrum width are used to simulate time series data with known range correlation (Torres and Zrnić 2003). Simulation parameters are designed to match operational conditions of the NWRT PAR located in Norman, Oklahoma (Zrnić et al. 2007). Scattering centers are placed in range at 60-m intervals for a range-oversampling factor of L = 4 (the NWRT PAR's pulse width is ~240 m), and time series data for each scattering center are simulated using the procedure by Zrnić (1975) with M = 16 samples, a pulse repetition time of T_s = 3 ms, and a Nyquist velocity of 8.3 m s⁻¹, which are typical acquisition parameters at the lowest antenna elevation angles. This process creates a two-dimensional array of complex samples with the desired sample-time correlation. Finally, a known range correlation is imposed on the data by convolving (along range time) with the measured NWRT PAR modified pulse (Torres and Curtis 2013). To compute statistics, 1000 realizations of 100 radials with 1000 range gates each are simulated and grouped into increasingly larger radial sets (i.e., the first set contains one radial, the second set contains two radials, etc.). The proposed estimator with K = 3, V_max = 25 119 (arbitrarily chosen to be equivalent to an SNR of 90 dB), and SNR_min = 10 dB is applied to each of these radial sets, and biases and standard deviations are computed for each range correlation lag (l = 1, 2, and 3).

Three range profiles are considered: (i) a “constant profile,” (ii) a “random profile,” and (iii) a “real profile” (Fig. 2). The SNR, Doppler velocity, and spectrum width of scattering centers for the constant and random profiles are independent, identically distributed, Gaussian random variables; for the real profile, they are drawn from real measurements obtained with the NWRT PAR containing both ground clutter and weather signals. The scattering centers in the constant profile have the following properties: SNRs with a mean of 40 dB and a standard deviation of 0.01 dB (no saturation is present), Doppler velocities with a mean of 0 m s⁻¹ and a standard deviation of 1 m s⁻¹, and spectrum widths with a mean of 2 m s⁻¹ and a standard deviation of 1 m s⁻¹. The scattering centers in the random profile have similar Doppler velocities and spectrum widths, but the SNRs have a mean of 40 dB and a standard deviation of 23 dB (~1.4% of the samples are saturated). Finally, the scattering centers in the real profile have SNRs, Doppler velocities, and spectrum widths with means obtained from the real radar measurements and standard deviations of 0.01 dB, 1 m s⁻¹, and 1 m s⁻¹, respectively. Figure 2 shows one realization of such range profiles.

View Full Size

Examples of simulated (top) SNR, (middle) Doppler velocity, and (bottom) spectrum width range profiles corresponding to the (left) constant, (middle) random, and (right) real range profiles.

Citation: Journal of Atmospheric and Oceanic Technology 30, 12; 10.1175/JTECH-D-13-00090.1

• Download Figure
• Download figure as PowerPoint slide

The normalized biases and normalized standard deviations for estimates of the relevant entries of the normalized range correlation matrix are shown in Fig. 3 as a function of the number of valid sample pairs used in the estimator for each of the three simulated range profiles. In general, it can be seen that biases are very small and standard deviations decrease as the number of samples used in the estimator increases. The constant and random profiles exhibit the smallest and largest biases, respectively; whereas the real profile, as expected, performs in between these two extremes. In the less favorable scenario of the random profile, biases of range correlation estimates are ~0.4%, ~1.4%, and ~3.2% for lags 1–3, respectively, all of which are negligible as demonstrated later. In terms of standard deviations, both synthetic profiles (constant and random) exhibit almost the same performance, whereas the “real” profile exhibits ~3 times worse performance. This is caused by the uneven distribution of powers in the real profile and the inherent limitation of the Pearson product-moment correlation coefficient estimator when dealing with data from mixed distributions; that is, the estimator in Eq. (6) weights each sample pair according to the strength of the corresponding samples, and, for realistic signal profiles, this effectively reduces the number of samples with a significant contribution to the estimator and increases its standard deviation.

View Full Size

(left) Normalized biases and (right) normalized standard deviations of range correlation coefficient estimates for lags l = 1 (solid lines), 2 (dashed lines), and 3 (dashed–dotted lines) as a function of the number of valid sample pairs used in the estimator for simulated (top) constant, (middle) random, and (bottom) real range profiles.

Citation: Journal of Atmospheric and Oceanic Technology 30, 12; 10.1175/JTECH-D-13-00090.1

• Download Figure
• Download figure as PowerPoint slide

A more meaningful way of analyzing the statistical performance of the proposed estimator is in terms of the reflectivity biases induced by an inaccurate range correlation measurement. These theoretical reflectivity biases can be computed using Eq. (22) in Torres and Curtis (2013) as

View Expanded

where is the true range correlation matrix and is a linear transformation matrix derived from the measured correlation matrix. Figure 4 shows the mean and maximum of the theoretical reflectivity biases as a function of the number of valid sample pairs used in the range correlation estimator for each of the three simulated range profiles. Theoretical reflectivity biases are obtained for two linear transformations: a matched filter () and a whitening transformation (), both derived from the range correlation measured using the proposed estimator. These plots in conjunction with a maximum reflectivity bias requirement can be used to assess the amount of data required to obtain precise measurements of . For example, for a maximum reflectivity bias (resulting from range correlation mismeasurement) less than 0.1 dB, ~60 000 valid sample pairs are needed to achieve acceptable performance (real profile in Fig. 4). This amounts to ~3750 total range gates (recall that M = 16) or an average of just under 11 range gates per radial in a full 360° constant-elevation scan. Even in a clear-air situation, it would not be difficult to get that number of valid sample pairs containing ground clutter at the lowest elevation angles, which makes the proposed technique effective for most operational situations and radar sites.

View Full Size

Mean and maximum of theoretical reflectivity biases as a function of the number of valid sample pairs used in the range correlation estimator for each of the three simulated range profiles: (top) constant, (middle) random, and (bottom) real. Theoretical biases of reflectivity estimates and are obtained for processing with a matched filter and a whitening transformation, respectively.

Citation: Journal of Atmospheric and Oceanic Technology 30, 12; 10.1175/JTECH-D-13-00090.1

• Download Figure
• Download figure as PowerPoint slide

4. Real data

The main motivation for measuring the range correlation in real time is the potential for variations caused by hardware changes and drifts. To see this, the new range correlation measurement method introduced in section 2 was applied to data from several volume scans collected with the NWRT PAR. The time series data from the lowest elevation angle were used for all of the calculations. At the lowest elevation angle of 0.5°, there are 109 beam positions in azimuth spread over a 90° sector with two pulse repetition times (PRT) used at each beam position, 3 and 0.8 ms. Nine pulses were transmitted using the long PRT and 25 using the short PRT. The estimator was used with the same parameters as those used for the simulations (K = 3 and SNR_min = 10 dB), except for a slightly different maximum voltage threshold that was set to the saturation value for the NWRT PAR digital receiver (V_max = 23 028). The results from four scans are shown in Table 1.

Table 1.

Normalized range correlation values (magnitude and phase) measured from four scans and collected over 3 days.

View Table

The cases are labeled from A to D for easier identification. The second column shows the date and time in UTC for all four scans. The next three columns contain the normalized range correlation values at lags l = 1, 2, and 3. The complex numbers are represented in magnitude–phase format to better interpret the differences between them. The number of pairs used to calculate the range correlation varied from a minimum of 376 795 pairs in a scan with only ground clutter to 878 237 pairs in a scan with both weather and ground clutter returns; this clearly meets the threshold of approximately 60 000 sample pairs mentioned in section 3. The scan for case A was collected on 29 March 2012 and consisted of mainly ground clutter returns (as did case B). This collection occurred a couple of days before the failure of a trigger amplifier in the transmitter. The amplifier was replaced on 2 April 2012, and the scan for case B was collected later that day. There is a clear difference in the magnitude and phase values from these two scans. The magnitudes are greater for case B, and the phases are significantly altered from case A. This is an example of a hardware change that results in a large difference in the range correlation that needs to be accurately measured to ensure effective performance of range-oversampling processing. The scan for case C was collected a week later and shows a smaller change in the range correlation, illustrating the effects of drift over a week. This scan was composed of both weather and ground clutter returns (as was the scan from case D). The effect on the performance would be significantly less in this case, but measuring the range correlation in real time ensures that the drift is accounted for and the performance is the best possible. The scan for case D was collected about 10 min after the scan for case C. The range correlation did not change significantly during this period, which shows that the hardware drift is not especially abrupt and that the method for measuring the range correlation is reasonably consistent.

Although this range correlation measurement method was evaluated using simulations in section 3 and shown to be only slightly biased, simulations may not always capture all of the attributes of real radar data. To independently verify the performance of the method, a technique was used to measure the reflectivity bias directly from real data (Torres and Curtis 2013). This technique was applied to the scans for the cases in Table 1. It compares the powers computed using matched-filter-based (MFB) processing and whitening-transformation-based (WTB) processing (e.g., Torres 2001). At each range gate, the ratio of the powers was computed if both the numerator and denominator powers were between 10 and 90 dB above the calculated noise level. The mean of the ratios was then estimated, and this ratio was converted to decibels to obtain an estimate of the reflectivity bias.

Table 2 shows the results of these calculations for three different power ratio comparisons. The values in the MFB/WTB column compare the matched-filter-based powers to the whitening-transformation-based powers using the range correlation measurement for each case. Theoretically, there should be zero reflectivity bias for these measurements, since both the MFB and WTB powers should be unbiased if the range correlation is accurately measured. This lack of bias is shown using zeros in the “theory” column for this comparison. The measured bias is slightly positive in all four cases but is no larger than about 0.25 dB. This could be from a small bias in the range correlation estimator or from inaccuracies in the bias calculation method, but, in either case, the range correlation measurements are reasonable and the biases are relatively small.

Table 2.

Comparisons between the theoretical and measured reflectivity biases for four scans. MFB/WTB biases are computed based on range correlation measurements made from each scan. Mismatched biases illustrate the bias when the range correlation from the previous scan is then applied to the next scan (only valid for cases B–D).

View Table

The last two columns of Table 2 show the results when the range correlation measured from the previous case are applied to the next case. This can cause a mismatch in the range correlation that leads to possibly significant biases. In the MFB_MM/WTB scenario, the MFB powers using the mismatched range correlation (MFB_MM) are compared to the WTB powers using the range correlation measured from the scan itself. For example, when the range correlation measurement from case A is applied to the matched filtered data from case B, the theory predicts a −1.83-dB reflectivity bias. The bias measured from the data was −1.60 dB. This illustrates how a hardware change between case A and case B can lead to a nearly 2-dB reflectivity bias. For cases C and D, the biases are significantly smaller, which is expected since the range correlation measurements are more similar. The final column shows the WTB_MM/WTB scenario comparing mismatched WTB powers (WTB_MM) to WTB powers. Again, the largest discrepancy is in case B when using the measurement from case A. There is a positive bias of nearly 1 dB for both the theoretical and measured biases. The biases are much smaller for cases C and D.

The change in the range correlation over time illustrates the importance of accurate real-time measurements, but we also need to determine how often the measurements need to be updated. As shown in the case from 9 April, the range correlation only changed slightly over about 10 min. A natural way to update the measurement of the range correlation is at the end of a volume scan; that is, the range correlation is measured using data from one volume scan, and the updated results are applied to the next scan. If the measurement is made using data from the lowest elevation scan, then there should be plenty of data to make an accurate measurement and plenty of time to use this result on the next scan. Figure 5 shows the results of this method for the 10-min stretch between 2305:49 and 2316:29 UTC 9 April. There are 12 scans in between cases C and D from Table 1, resulting in a total of 14 scans. The top panel shows the magnitude of the range correlation measurement, and the middle panel shows its phase. In an operational system, each estimate would be used by the succeeding scan. To better understand the effects of this procedure, the reflectivity bias was calculated using the theoretical bias computations described in the previous section. The measurement from a particular scan was treated as the true value of the range correlation, and the measurement from the previous scan was treated as the estimate. The bottom panel in Fig. 5 shows the expected reflectivity bias under these assumptions for both MFB and WTB processing. The maximum bias is less than 0.07 dB and shows that this measurement scheme should not introduce significant bias into the reflectivity estimation process.

View Full Size

(top) Magnitude and (middle) phase of the range correlation measurements from consecutive scans over a period of ~10 min. (bottom) Expected theoretical reflectivity biases if the measurement from the previous scan is applied to the next scan for both MFB and WTB processing. Times are in UTC, and scans were collected on 9 Apr 2012.

Citation: Journal of Atmospheric and Oceanic Technology 30, 12; 10.1175/JTECH-D-13-00090.1

• Download Figure
• Download figure as PowerPoint slide

5. Conclusions

We proposed an automated technique to measure the normalized range correlation matrix based on the Pearson moment-product correlation coefficient estimator, with the caveat that invalid samples in the neighborhood of noiselike and saturated returns are excluded from the computations. Among all possible variations of the classical correlation coefficient estimator, a weighted average of radial-based estimates was determined to have the best overall performance. Simulations showed that measurements are consistent and only slightly biased, leading to reflectivity biases on the order of 0.1 dB. Such biases are small compared to the accepted error of reflectivity estimates on most operational weather radars [e.g., the maximum allowable bias of reflectivity estimates on the Weather Surveillance Radar-1988 Doppler (WSR-88D) is 1 dB].

Analysis on real data showed that hardware changes and drift can affect the range correlation over time. In the case of a hardware change, the resulting reflectivity biases could be significant (~2 dB when using a digital matched filter). This justifies the need for precise and timely measurements of the range correlation matrix, which can be done using the method presented in this paper. Evaluation of the technique's performance demonstrated that measuring the range correlation in real time is feasible on operational weather radars and provides quantifiable benefits for range-oversampling processing.