## 1. Introduction

With the advent of modern digital receivers, range-oversampling processing has become practical for weather radars. By sampling the signal at a range resolution finer than that of the transmitter pulse and applying a linear transformation, range-oversampling processing techniques can be utilized to reduce the variance of estimates and/or reduce the required observation (dwell) times. The whitening transformation and digital matched filter were the first transformations that were introduced for range-oversampled data (Torres 2001). Later, pseudowhitening transformations were proposed to mitigate the noise enhancement effects of the whitening transformation (Torres et al. 2004). More recently, adaptive pseudowhitening was implemented on the National Weather Radar Testbed Phased-Array Radar (NWRT PAR; Curtis and Torres 2011). This technique applies a nearly optimal transformation at each range location to provide excellent performance for differing conditions. All of these transformations have one thing in common—they directly depend on the normalized range correlation matrix.

A mismeasurement of the normalized range correlation matrix can cause biases in reflectivity estimates and a less-than-expected variance reduction (Torres and Curtis 2013). Thus, accurately measuring the range correlation is an important first step for range-oversampling processing. To accurately measure the range correlation, it is important to understand which parameters determine it and how these change over time. A useful way to think of the range correlation is in terms of the modified pulse. The modified pulse combines the effects of the transmitted pulse and the impulse response of the receiver to capture the two effects that contribute to the range correlation. Hence, hardware changes to the transmitter or receiver and drifts over time can affect the modified pulse (and the range correlation). If these variations over time are significant, then the measurement should be repeated often enough to make sure that performance is not degraded.

One way to measure the modified pulse is to use the returns from a strong point target (Torres and Curtis 2013). This method can give a reasonable measurement of the range correlation, but it suffers from the following issues: finding a perfectly isolated target with sufficiently narrow range extent, obtaining uncorrupted returns from the target, and automating this process for different radar sites. Another possible method is to inject a replica of the transmitter pulse into the receiver front end, but this may be infeasible on some operational radars. The goal is to find a way to easily measure the range correlation in real time to reflect any fluctuations in the modified pulse with minimal effects on data collection. With these constraints, it is natural to look at possible ways to measure the range correlation directly from the data.

An early method for measuring the range correlation from data relied on weather returns that had relatively uniform reflectivity (Ivić et al. 2003). This method required manual selection of data with the specified characteristics, and it was devised primarily to demonstrate the feasibility of range-oversampling processing for a research radar. In this paper, we propose an extension of this method that does not require a particular type of radar return to be effective, making it more suited for automation and operational use. The method uses raw range-oversampled data from normal radar scans in a separate processing chain. It operates on the complex time series data before the ground clutter filter and thus is independent from the typical signal processing chain that produces meteorological data. The method uses radar returns from weather and ground clutter and will work with any type of scenario as long as there are enough samples for an accurate measurement.

The rest of the paper is organized as follows. Section 2 describes the method for range correlation measurement based on a robust estimator of the correlation coefficient. Section 3 introduces simulations to show that the range correlation estimates are only marginally biased and to determine the amount of data needed for an accurate measurement. In section 4, range correlation measurements from real data are examined. The changes in range correlation over time are illustrated on the NWRT PAR along with the effects of using an inaccurate measurement.

## 2. Range correlation measurement

*l*is defined as

*E*[.] is the expected value, the asterisk (*) denotes the complex conjugate, and

*V*is the range-oversampled, complex times series data. Integer

*m*indexes sample time at increments given by the pulse repetition time (

*T*

_{s});

*n*and

*l*index the range time and range lag at increments given by the time series sampling time (

*T*

_{r}). The elements of the normalized range correlation matrix,

*i*th row and

*j*th column of matrix

*L*is the range-oversampling factor. Typically,

*LT*

_{r}is chosen to match the width of the modified pulse (Curtis and Torres 2011), but the analysis herein is valid in general. The normalized range correlation matrix is Hermitian Toeplitz with ones along its main diagonal; thus, only

*L*− 1 entries [

_{j}(1 ≤

*j*≤

*J*). The first stage of averaging is done over each subset A

_{j}, which produces

*J*ratios that will then be averaged in a second stage. Different results can be obtained depending on how these subsets are defined (i.e., subsets may contain only one sample, a group of samples, all samples in one pulse, or all samples in a radial or dwell), leading to different variations on the same estimator. We thoroughly evaluated these variations and found that the one with the overall best performance is the one doing the first stage of averaging at the radial level; that is,

*N*and

*M*are the total number of samples in range and sample time in a radial, respectively.

Because the goal is to measure the range correlation from radar data, Eq. (4) cannot be used directly, as it is known to be highly affected by outliers (Huber and Ronchetti 2009). Several techniques have been proposed to construct robust estimators of the correlation coefficient that significantly reduce the effects of outliers (Shevlyakov and Smirnov 2011). However, we are only concerned with the effects of noiselike and saturated samples, so we give preference to simpler variants of the traditional estimator in Eq. (4). Once again, several simple mechanisms to reduce the effects of outliers can be devised, and a few were evaluated. Of these, the one that exhibited the best performance handling saturated and noiselike samples is described next.

_{min}and expressed in decibels) is set to exclude noiselike samples, and a system-dependent maximum voltage threshold (

*V*

_{max}) is set to exclude potentially saturated samples (e.g., returns from ground targets located very close to the radar). Mathematically, a sample is classified as “invalid” for the correlation coefficient estimator if

*N*

_{lin}is the receiver noise power in linear units. Because samples are correlated in range, additional samples in a range–time radius of influence

*K*from invalid samples must be excluded as well (Fig. 1). The radius of influence should be chosen as small as possible to exclude the least number of samples, but large enough so that the effect of invalid samples on valid ones is insignificant. Thus, a good choice for

*K*is

*L*− 1; that is,

*K*is the largest lag before the range correlation becomes negligible. With this, a robust version of Eq. (4) that only uses valid sample pairs is given by

**X**(

*l, m*) is the set of indexes to the first sample of pairs that are considered valid for the estimation of the range correlation at lag

*l*. It is important to note that the total number of sample pairs used by this estimator is given by

*R*radials, the range correlation coefficient is finally estimated as

*r*is added to indicate radial number. This weighting prevents less reliable estimates obtained from radials with fewer valid sample pairs from significantly influencing the final result.

It is well known that if there are no outliers and the data are normally distributed, then the sample correlation coefficient in Eq. (4) is asymptotically unbiased and efficient (e.g., Bachman 2004). However, it is important to determine whether the asymptotically unbiased property carries over to the proposed estimator in Eq. (7), and, from a practical standpoint, to establish the total number of valid sample pairs that are required to reliably measure the range correlation. These questions will be answered by evaluating the statistical performance of the proposed estimator using simulations.

## 3. Simulations

In this section, simulations are used to evaluate the statistical performance of the proposed range correlation measurement technique. It will be shown that, when applied to realistic radar data, the proposed estimator is slightly biased but consistent (i.e., the standard deviation of estimates decreases with increasing amounts of data).

Synthetic and real range profiles of SNR, Doppler velocity, and spectrum width are used to simulate time series data with known range correlation (Torres and Zrnić 2003). Simulation parameters are designed to match operational conditions of the NWRT PAR located in Norman, Oklahoma (Zrnić et al. 2007). Scattering centers are placed in range at 60-m intervals for a range-oversampling factor of *L* = 4 (the NWRT PAR's pulse width is ~240 m), and time series data for each scattering center are simulated using the procedure by Zrnić (1975) with *M* = 16 samples, a pulse repetition time of *T*_{s} = 3 ms, and a Nyquist velocity of 8.3 m s^{−1}, which are typical acquisition parameters at the lowest antenna elevation angles. This process creates a two-dimensional array of complex samples with the desired sample-time correlation. Finally, a known range correlation is imposed on the data by convolving (along range time) with the measured NWRT PAR modified pulse (Torres and Curtis 2013). To compute statistics, 1000 realizations of 100 radials with 1000 range gates each are simulated and grouped into increasingly larger radial sets (i.e., the first set contains one radial, the second set contains two radials, etc.). The proposed estimator with *K* = 3, *V*_{max} = 25 119 (arbitrarily chosen to be equivalent to an SNR of 90 dB), and SNR_{min} = 10 dB is applied to each of these radial sets, and biases and standard deviations are computed for each range correlation lag (*l* = 1, 2, and 3).

Three range profiles are considered: (i) a “constant profile,” (ii) a “random profile,” and (iii) a “real profile” (Fig. 2). The SNR, Doppler velocity, and spectrum width of scattering centers for the constant and random profiles are independent, identically distributed, Gaussian random variables; for the real profile, they are drawn from real measurements obtained with the NWRT PAR containing both ground clutter and weather signals. The scattering centers in the constant profile have the following properties: SNRs with a mean of 40 dB and a standard deviation of 0.01 dB (no saturation is present), Doppler velocities with a mean of 0 m s^{−1} and a standard deviation of 1 m s^{−1}, and spectrum widths with a mean of 2 m s^{−1} and a standard deviation of 1 m s^{−1}. The scattering centers in the random profile have similar Doppler velocities and spectrum widths, but the SNRs have a mean of 40 dB and a standard deviation of 23 dB (~1.4% of the samples are saturated). Finally, the scattering centers in the real profile have SNRs, Doppler velocities, and spectrum widths with means obtained from the real radar measurements and standard deviations of 0.01 dB, 1 m s^{−1}, and 1 m s^{−1}, respectively. Figure 2 shows one realization of such range profiles.

The normalized biases and normalized standard deviations for estimates of the relevant entries of the normalized range correlation matrix are shown in Fig. 3 as a function of the number of valid sample pairs used in the estimator for each of the three simulated range profiles. In general, it can be seen that biases are very small and standard deviations decrease as the number of samples used in the estimator increases. The constant and random profiles exhibit the smallest and largest biases, respectively; whereas the real profile, as expected, performs in between these two extremes. In the less favorable scenario of the random profile, biases of range correlation estimates are ~0.4%, ~1.4%, and ~3.2% for lags 1–3, respectively, all of which are negligible as demonstrated later. In terms of standard deviations, both synthetic profiles (constant and random) exhibit almost the same performance, whereas the “real” profile exhibits ~3 times worse performance. This is caused by the uneven distribution of powers in the real profile and the inherent limitation of the Pearson product-moment correlation coefficient estimator when dealing with data from mixed distributions; that is, the estimator in Eq. (6) weights each sample pair according to the strength of the corresponding samples, and, for realistic signal profiles, this effectively reduces the number of samples with a significant contribution to the estimator and increases its standard deviation.

*true*range correlation matrix and

*measured*correlation matrix. Figure 4 shows the mean and maximum of the theoretical reflectivity biases as a function of the number of valid sample pairs used in the range correlation estimator for each of the three simulated range profiles. Theoretical reflectivity biases are obtained for two linear transformations: a matched filter (

*M*= 16) or an average of just under 11 range gates per radial in a full 360° constant-elevation scan. Even in a clear-air situation, it would not be difficult to get that number of valid sample pairs containing ground clutter at the lowest elevation angles, which makes the proposed technique effective for most operational situations and radar sites.

## 4. Real data

The main motivation for measuring the range correlation in real time is the potential for variations caused by hardware changes and drifts. To see this, the new range correlation measurement method introduced in section 2 was applied to data from several volume scans collected with the NWRT PAR. The time series data from the lowest elevation angle were used for all of the calculations. At the lowest elevation angle of 0.5°, there are 109 beam positions in azimuth spread over a 90° sector with two pulse repetition times (PRT) used at each beam position, 3 and 0.8 ms. Nine pulses were transmitted using the long PRT and 25 using the short PRT. The estimator was used with the same parameters as those used for the simulations (*K* = 3 and SNR_{min} = 10 dB), except for a slightly different maximum voltage threshold that was set to the saturation value for the NWRT PAR digital receiver (*V*_{max} = 23 028). The results from four scans are shown in Table 1.

Normalized range correlation values (magnitude and phase) measured from four scans and collected over 3 days.

The cases are labeled from A to D for easier identification. The second column shows the date and time in UTC for all four scans. The next three columns contain the normalized range correlation values *l* = 1, 2, and 3. The complex numbers are represented in magnitude–phase format to better interpret the differences between them. The number of pairs used to calculate the range correlation varied from a minimum of 376 795 pairs in a scan with only ground clutter to 878 237 pairs in a scan with both weather and ground clutter returns; this clearly meets the threshold of approximately 60 000 sample pairs mentioned in section 3. The scan for case A was collected on 29 March 2012 and consisted of mainly ground clutter returns (as did case B). This collection occurred a couple of days before the failure of a trigger amplifier in the transmitter. The amplifier was replaced on 2 April 2012, and the scan for case B was collected later that day. There is a clear difference in the magnitude and phase values from these two scans. The magnitudes are greater for case B, and the phases are significantly altered from case A. This is an example of a hardware change that results in a large difference in the range correlation that needs to be accurately measured to ensure effective performance of range-oversampling processing. The scan for case C was collected a week later and shows a smaller change in the range correlation, illustrating the effects of drift over a week. This scan was composed of both weather and ground clutter returns (as was the scan from case D). The effect on the performance would be significantly less in this case, but measuring the range correlation in real time ensures that the drift is accounted for and the performance is the best possible. The scan for case D was collected about 10 min after the scan for case C. The range correlation did not change significantly during this period, which shows that the hardware drift is not especially abrupt and that the method for measuring the range correlation is reasonably consistent.

Although this range correlation measurement method was evaluated using simulations in section 3 and shown to be only slightly biased, simulations may not always capture all of the attributes of real radar data. To independently verify the performance of the method, a technique was used to measure the reflectivity bias directly from real data (Torres and Curtis 2013). This technique was applied to the scans for the cases in Table 1. It compares the powers computed using matched-filter-based (MFB) processing and whitening-transformation-based (WTB) processing (e.g., Torres 2001). At each range gate, the ratio of the powers was computed if both the numerator and denominator powers were between 10 and 90 dB above the calculated noise level. The mean of the ratios was then estimated, and this ratio was converted to decibels to obtain an estimate of the reflectivity bias.

Table 2 shows the results of these calculations for three different power ratio comparisons. The values in the MFB/WTB column compare the matched-filter-based powers to the whitening-transformation-based powers using the range correlation measurement for each case. Theoretically, there should be zero reflectivity bias for these measurements, since both the MFB and WTB powers should be unbiased if the range correlation is accurately measured. This lack of bias is shown using zeros in the “theory” column for this comparison. The measured bias is slightly positive in all four cases but is no larger than about 0.25 dB. This could be from a small bias in the range correlation estimator or from inaccuracies in the bias calculation method, but, in either case, the range correlation measurements are reasonable and the biases are relatively small.

Comparisons between the theoretical and measured reflectivity biases for four scans. MFB/WTB biases are computed based on range correlation measurements made from each scan. Mismatched biases illustrate the bias when the range correlation from the previous scan is then applied to the next scan (only valid for cases B–D).

The last two columns of Table 2 show the results when the range correlation measured from the previous case are applied to the next case. This can cause a mismatch in the range correlation that leads to possibly significant biases. In the MFB_{MM}/WTB scenario, the MFB powers using the mismatched range correlation (MFB_{MM}) are compared to the WTB powers using the range correlation measured from the scan itself. For example, when the range correlation measurement from case A is applied to the matched filtered data from case B, the theory predicts a −1.83-dB reflectivity bias. The bias measured from the data was −1.60 dB. This illustrates how a hardware change between case A and case B can lead to a nearly 2-dB reflectivity bias. For cases C and D, the biases are significantly smaller, which is expected since the range correlation measurements are more similar. The final column shows the WTB_{MM}/WTB scenario comparing mismatched WTB powers (WTB_{MM}) to WTB powers. Again, the largest discrepancy is in case B when using the measurement from case A. There is a positive bias of nearly 1 dB for both the theoretical and measured biases. The biases are much smaller for cases C and D.

The change in the range correlation over time illustrates the importance of accurate real-time measurements, but we also need to determine how often the measurements need to be updated. As shown in the case from 9 April, the range correlation only changed slightly over about 10 min. A natural way to update the measurement of the range correlation is at the end of a volume scan; that is, the range correlation is measured using data from one volume scan, and the updated results are applied to the next scan. If the measurement is made using data from the lowest elevation scan, then there should be plenty of data to make an accurate measurement and plenty of time to use this result on the next scan. Figure 5 shows the results of this method for the 10-min stretch between 2305:49 and 2316:29 UTC 9 April. There are 12 scans in between cases C and D from Table 1, resulting in a total of 14 scans. The top panel shows the magnitude of the range correlation measurement, and the middle panel shows its phase. In an operational system, each estimate would be used by the succeeding scan. To better understand the effects of this procedure, the reflectivity bias was calculated using the theoretical bias computations described in the previous section. The measurement from a particular scan was treated as the true value of the range correlation, and the measurement from the previous scan was treated as the estimate. The bottom panel in Fig. 5 shows the expected reflectivity bias under these assumptions for both MFB and WTB processing. The maximum bias is less than 0.07 dB and shows that this measurement scheme should not introduce significant bias into the reflectivity estimation process.

## 5. Conclusions

We proposed an automated technique to measure the normalized range correlation matrix based on the Pearson moment-product correlation coefficient estimator, with the caveat that invalid samples in the neighborhood of noiselike and saturated returns are excluded from the computations. Among all possible variations of the classical correlation coefficient estimator, a weighted average of radial-based estimates was determined to have the best overall performance. Simulations showed that measurements are consistent and only slightly biased, leading to reflectivity biases on the order of 0.1 dB. Such biases are small compared to the accepted error of reflectivity estimates on most operational weather radars [e.g., the maximum allowable bias of reflectivity estimates on the Weather Surveillance Radar-1988 Doppler (WSR-88D) is 1 dB].

Analysis on real data showed that hardware changes and drift can affect the range correlation over time. In the case of a hardware change, the resulting reflectivity biases could be significant (~2 dB when using a digital matched filter). This justifies the need for precise and timely measurements of the range correlation matrix, which can be done using the method presented in this paper. Evaluation of the technique's performance demonstrated that measuring the range correlation in real time is feasible on operational weather radars and provides quantifiable benefits for range-oversampling processing.

## Acknowledgments

The authors thank two anonymous reviewers and I. Ivić and D. Zrnić for providing comments to improve the manuscript. Funding was provided by NOAA's Office of Oceanic and Atmospheric Research under NOAA–University of Oklahoma Cooperative Agreement NA11OAR4320072, U.S. Department of Commerce.

## REFERENCES

Bachman, L. F., 2004:

*Statistical Analyses for Language Assessment.*Cambridge University Press, 364 pp.Curtis, C., and Torres S. , 2011: Adaptive range oversampling to achieve faster scanning on the National Weather Radar Testbed Phased-Array Radar.

,*J. Atmos. Oceanic Technol.***28**, 1581–1597.Huber, P. J., and Ronchetti E. M. , 2009:

*Robust Statistics.*Wiley, 380 pp.Ivić, I., Zrnić D. S. , and Torres S. , 2003: Whitening in range to improve weather radar spectral moment estimates. Part II: Experimental evaluation.

,*J. Atmos. Oceanic Technol.***20**, 1449–1459.Pearson, K., 1896: Contributions to the mathematical theory of evolution. Note on reproductive selection.

,*Proc. Roy. Soc. London***59**, 301–305.Shevlyakov, G., and Smirnov P. , 2011: Robust estimation of the correlation coefficient: An attempt of survey.

,*Austrian J. Stat.***40**, 147–156.Torres, S., 2001: Estimation of Doppler and polarimetric variables for weather radars. Ph.D. dissertation, University of Oklahoma, 158 pp.

Torres, S., and Zrnić D. , 2003: Whitening in range to improve weather radar spectral moment estimates. Part I: Formulation and simulation.

,*J. Atmos. Oceanic Technol.***20**, 1433–1448.Torres, S., and Curtis C. , 2013: The importance of accurately measuring the range correlation for range-oversampling processing.

,*J. Atmos. Oceanic Technol.***30**, 261–273.Torres, S., Curtis C. , and Cruz J. R. , 2004: Pseudowhitening of weather radar signals to improve spectral moment and polarimetric variable estimates at low signal-to-noise ratios.

,*IEEE Trans. Geosci. Remote Sens.***42**, 941–949.Zrnić, D. S., 1975: Simulation of weatherlike Doppler spectra and signals.

,*J. Appl. Meteor.***14**, 619–620.Zrnić, D. S., and Coauthors, 2007: Agile-beam phased array radar for weather observations.

,*Bull. Amer. Meteor. Soc.***88**, 1753–1766.