1. Introduction

In meteorology and oceanography, and other fields, it is often necessary to check whether two quantities x and y are linearly related. Usually this involves the calculation of a correlation coefficient and a regression coefficient. Both coefficients are useful; the sample correlation coefficient provides a test of the linearity of the relationship and the regression coefficient provides the proportionality constant between the two variables. By its sign and size, the regression coefficient often informs us whether the relationship makes sense physically.

In practice, “noise” reduces the correlation coefficient and affects the accuracy of the regression coefficient. Noise can be due to measurement error or to different physical processes in x and y that affect the linearizing process common to both. While measurement error can sometimes be estimated for each variable, error due to possible physical influences in the real data is often not known. So there is a large class of regression problems in meteorology, oceanography, and other fields in which the signal-to-noise ratio in both variables is unknown.

When there is noise in both variables, the regression coefficient between the variables is underestimated even if an infinite sample of data were available. To illustrate this well-known result and to introduce our notation, consider the problem of finding, from a sample of M points, the true linear regression coefficient between two random variables X and Y that are both subject to random noise. Mathematically, we want to find the regression coefficient α in the linear relationship

given the observations

In (2) and (3), the error or noise terms ɛ and δ are taken to be normally distributed zero-mean random variables that are independent of each other and X.

One estimate for is the ordinary least squares estimate

where

is the sample covariance of x and y,

is the sample standard deviation of x, and

are the sample means of x and y, respectively. Since X, ɛ, and δ are all independent of each other, as ,

where σ_X and σ_ɛ are the (true) standard deviations of X and ɛ. Thus it follows from (4) that as ,

where the noise-to-signal ratio n_x is defined as

Therefore, whenever x has nonzero noise and hence n_x > 0, (9) implies the well-known result that underestimates the true regression coefficient α even when the number of observations . If x has zero noise, then n_x = 0 and an unbiased estimate α for the regression of y is available; conversely, if y has zero noise (δ = 0), then a regression of x on y yields an unbiased estimate for α⁻¹. Confidence intervals in either case can then be determined in the finite case using a Student’s distribution with M − 2 degrees of freedom. However, when both x and y have nonzero noise, an unbiased estimate of α or α⁻¹ is not available and confidence intervals for the true regression coefficient are unknown.

Many methods (Ricker 1973; Jolicoeur 1975; Riggs et al. 1978; McArdle 1988, 2003; Frost and Thompson 2000) have been devised to obtain a best estimate for when both variables are noisy. When the relative size of the noise for each variable is known, an explicit estimate for can be given (Kendall and Stuart 1973). However, as discussed above, often this ratio is unknown and then the way forward is murky and “definitive recommendations are difficult to make” (Sokal and Rohlf 1995).

In this paper, we overcome this difficulty by noting that when the relative size of the noise for each variable is unknown, we have no basis for choosing between the variables. Consequently, we must assume that the noise in each variable is equally likely subject to the constraint that the sample correlation coefficient for the given set of data is known. This equal likelihood noise assumption, subject to the known sample correlation coefficient, enables us to determine explicitly the confidence intervals in the case analytically and in the finite M case numerically.

The rest of the paper is organized as follows: in the next section we discuss an unbiased estimate for the true regression coefficient when nothing is known about the noise in each variable. Then, in section 3, we examine the case and determine a probability density function for the true regression coefficient analytically. Confidence intervals for the finite M case are found numerically in section 4, and an example of their use is then given in section 5. A final section 6 contains some concluding remarks.

2. An unbiased estimate for the true regression coefficient

Consider the problem of estimating the true regression coefficient between x and y when nothing is known or assumed about the noise in x and y except for the constraint provided by the sample correlation coefficient. One estimate for the true regression coefficient α of y on x is , where the nondimensional factor f₁ is used to correct the bias in discussed in the introduction. We could equally as well have regressed x on y to obtain , where s_y is the sample standard deviation of y defined analogously to s_x in (6). The bias in can similarly be corrected by writing using another nondimensional factor f₂. But note that there is no basis for distinguishing f₁ and f₂. In both regression cases there are the same number of points M and the same sample correlation coefficient. Also in both cases we are carrying out the same procedure, namely, using a factor to scale the ordinary least squares regression coefficient so that it gives an unbiased estimate of the true regression coefficient. Therefore, the factors f₁ and f₂ should be the same and we write f₁ = f₂ = f.

Since is our estimate for the true regression coefficient α and is our estimate for α⁻¹, and we have no way of distinguishing these estimates, we must have and hence

that is, , where is the sample correlation coefficient . Thus an appropriate , and from (4) and the formula for we have

where sgn is +1 if is positive and −1 if is negative. From (12), is simply the ratio of the standard deviations of y and x. In the oceanography literature, has been called the “neutral” regression coefficient (Garrett and Petrie 1981), but in other fields, for example, the life sciences, fisheries, and statistics (Ricker 1973; Riggs et al. 1978; Sprent and Dolby 1980; Barker et al. 1988; Sokal and Rohlf 1995), it is known as the geometric mean regression coefficient. The latter nomenclature follows because is the geometric mean of the estimates and . Henceforth, we write

Note that is also the regression coefficient obtained when x is normalized by s_x, y is normalized by s_y, and the perpendicular distance from the regression line is minimized rather than the vertical distance as in an ordinary least squares fit. A principal component analysis of the normalized variables also has its principal axis as the regression line. There has been considerable controversy and confusion over the use of this regression coefficient (Jolicoeur 1975; Ricker 1975; Sprent and Dolby 1980; Barker et al. 1988; Emery and Thomson 2001; McArdle 2003). Emery and Thomson (2001) comment that the geometric mean coefficient, “though appealing, rests on shaky statistical ground and its use remains controversial.” As suggested by (12), one way to think of is as an appropriate estimate for when signal-to-noise ratios are not known (see also Barker et al. 1988).

3. The probability density function for α/α_GMR for the limiting M case

Given M data pairs, we can calculate both a correlation coefficient and the regression coefficient above using the sign of and the ratio of the standard deviations. In what follows, we analyze a large data point limiting case using values r and .

As , the sample coefficient

approaches

where is the true standard deviation of δ. By dividing both numerator and denominator by , (15) may also be written in the form

where

Note that is the (true) variance of the signal Y in y [see (1) and (3)], and so, analogous to the noise-to-signal ratio n_x, the parameter n_y in (16) is the noise-to-signal ratio for y. Squaring r gives, from (16), the constraint

on n_x and n_y since r can be determined from the data.

As n_x ≥ 0 and n_y ≥ 0, it follows from (18) that n_x and n_y both vary between 0 and r⁻² − 1; geometrically, the points (n_x, n_y) lie along a hyperbolic curve (see Fig. 1). Because we know nothing about the relative sizes of n_x and n_y, we assume that the points (n_x, n_y) are uniformly distributed along the curve between the limiting values at A (n_x = 0, n_y = r⁻² − 1) and B (n_x = r⁻² − 1, n_y = 0). If l denotes arc length and l_AB is the length of the curve AB, then the required uniformly distributed probability density function (pdf) is

since this is the same value for any point on the curve AB and satisfies .

• Download Figure
• Download figure as PowerPoint slide

The hyperbolic curve segments defined by and for |r| in increments of 0.1. The points A and B are shown when r = 0.9; in general the curves intersect the axis at A and the axis at B.

Citation: Journal of Atmospheric and Oceanic Technology 30, 1; 10.1175/JTECH-D-12-00067.1

Note that other assumptions about the distribution of n_x and n_y along the curve (18) are not justifiable. For example, if we assume n_x is uniformly distributed along the curve, then because of the hyperbolic form of (18), n_y is not uniformly distributed along the curve. Different distributions would be inappropriate, since we have no way of distinguishing the noise in x and y; all we have is the knowledge that the noise pair (n_x, n_y) lies somewhere along the curve defined by (18) and illustrated in Fig. 1.

The cumulative distribution function corresponding to (19) is

or, in terms of n_x,

From (18) we can find n_y in terms of n_x and r and hence calculate dn_y/dn_x to obtain, from (21),

From (9),

and from (13), . Therefore,

where we have introduced the variable ζ in (24) for notational convenience. Changing variables in (22) to ζ, we have

Thus,

Since from (24), g in (26) is the pdf for . It enables us to calculate confidence intervals for α given a knowledge of the parameters r and α_GMR, which we can calculate from the data.

Figure 2 shows examples of g for various values of r. It follows from (24) and that

so plots of g as a function of vary over this domain. The density function g has a long tail for small |r|; that is, the confidence intervals are wide, because for small |r| not much of the variance is explained by the regression fit. From (20), the median of g occurs when , that is, halfway along the hyperbolic curve AB (see Fig. 1). By symmetry this corresponds to n_x = n_y, or by (18), when (1 + n_x)|r| = 1. From (24) this is equivalent to . This differs from the most likely value for , which occurs when g is a maximum. Since g is a monotonically decreasing function of [see (26)], from (27) its maximum value occurs when

• Download Figure
• Download figure as PowerPoint slide

The pdf for five values of . As noted in the text, the domain of g in each case is .

Citation: Journal of Atmospheric and Oceanic Technology 30, 1; 10.1175/JTECH-D-12-00067.1

Note from (18) that the curve AB in Fig. 1 of length l_AB depends on r². As , the intercepts at A(0, r⁻² − 1) and B(r⁻² − 1, 0) approach 0 and so the length of the curve approaches zero; as , . Figure 3 shows l_AB as a function of r². Note that if we approximate the length of l_AB using a straight line between A and B, then l_AB is the length of the hypotenuse of a right-angled isosceles triangle with equal sides r⁻² − 1, and so

Figures 1 and 3 show that this approximation becomes increasingly accurate as , and that it is a reasonable approximation even when 0.3 ≤ r² ≤ 1. Thus (26) can also be written approximately, with increasing accuracy as , as

• Download Figure
• Download figure as PowerPoint slide

The length of the hyperbolic curves in Fig. 1 as a function of r² (solid line). The dashed line is , the length of the straight line approximation to those curves.

Citation: Journal of Atmospheric and Oceanic Technology 30, 1; 10.1175/JTECH-D-12-00067.1

While it is useful to have obtained results for the limiting large M case, in practice we are usually faced with the problem of determining α given a finite number M of hard-won data points and finite M estimates and of the correlation and regression coefficients. In the case, the confidence intervals follow directly from the pdf (26), but in the finite M case we know only and rather than r and α_GMR. In the next section, we show how to calculate the relevant finite M pdf and corresponding confidence intervals numerically.

4. Confidence intervals for the true regression coefficient for finite M

To calculate the confidence intervals for finite M and a given and , begin by defining

The sample correlation coefficient for x_* and y_* is the same as that for x and y while the sample regression coefficient of y_* on x_* is

Noting from (2) and (3) that and , we have

and

If we now divide (33) by and (34) by and use (31), we may write (33) and (34) as

and

The right-hand sides of (35) and (36) may be written in terms of n_x, n_y, and the zero-mean, unit variance variables

and

where is the true mean of X. To see that (35) and (36) can indeed be written in this way, first note that

Also, from (10), (17), (37), and (38) we have

and

The substitution of (40), (41), and (42) into (35) and (36) yields

and

We found the required confidence intervals numerically from (43) and (44), sampling the zero-mean, unit-variance variables X_*, ɛ_*, and δ_* from independent zero-mean, unit-variance normal distributions. The numerical calculations proceed by first obtaining a random r from a uniform distribution over the interval [−1, 1]. For that r we then randomly sample a point (n_x, n_y) from a uniform distribution along the curve AB corresponding to r². Now that we have n_x and n_y we can use the independent zero-mean, unit-variance normal distributions of X_*, ɛ_*, and δ_* to obtain M random samples of x_* and y_* from (43) and (44) and hence obtain the correlation coefficient , which, as we noted earlier, is the same as that for x and y. We can also calculate the regression coefficient , which, from (32), is . But since we know r for this sample, we also know sgn(α) and hence . Proceeding in this way we calculated 100 million samples of and thus calculated the probability density function of as a function of .

This pdf can be used to find the lower and upper limits L and U for confidence intervals for the true regression coefficient α. For example, for a 95% confidence interval we can find L and U such that

Once L and U are known, then for a given sample (x, y) of M points we can calculate and then use (45) to estimate the 95% confidence intervals for as for and for . Other confidence intervals can similarly be obtained.

Since we have an analytical solution for and numerical results for finite M, it is of interest to see how large M has to be for the confidence limits to be approximately correct. Figure 4 shows the 95% L and U as a function of for various M. For small M it is only when that the finite M case resembles the case. As for the pdf (see Fig. 2), distributions are skewed right so that, as decreases, U increases more rapidly than L decreases. In the finite case, plots end when L = 0, because for smaller even the sign of α is not known with 95% confidence.

• Download Figure
• Download figure as PowerPoint slide

Upper (U) and lower (L) 95% confidence interval limits as a function of for the ratio for M = 5 (dotted line), M = 10 (short-dashed line), M = 25 (dashed–dotted line), M = 100 (long-dashed line), M = 500 (gray line), and M = ∞ (black solid line). The upper confidence intervals U are all ≥1 and lower confidence intervals L are all ≤1. In the finite cases the plots end when L = 0 because for lower , even the sign of α is not known with 95% confidence. The curves approach the analytical M = ∞ limit as or .

Citation: Journal of Atmospheric and Oceanic Technology 30, 1; 10.1175/JTECH-D-12-00067.1

5. An example

Tables for L and U corresponding to the 80%, 90%, 95%, and 99% confidence intervals for various M and are provided in the supplemental material that is online (http://dx.doi.org/10.1175/JTECH-D-12-00067.s1.). These tables can be used on a wide variety of scientific data and we here illustrate their use in an example relating monthly atmospheric anomaly time series strongly influenced by El Niño. These time series are anomalous in that they are monthly time series that are departures from the seasonal cycle. Theory (Bunge and Clarke 2009) for the equatorial Pacific atmospheric boundary layer suggests that the monthly surface equatorial atmospheric pressure anomalies in the western equatorial Pacific p(o, t) and eastern equatorial Pacific p(L, t) should be related to the zonal integral of the eastward equatorial wind stress anomaly τ by

In (46), t is the time in months, ξ = 0 refers to the western equatorial Pacific boundary, ξ = L to the eastern equatorial Pacific boundary, and h(L) is a measure of the height of the turbulent atmospheric boundary layer in the eastern equatorial Pacific. Note that both monthly time series on the left- and right-hand sides of (46) contain noise. This noise is due to the measurement error in the thousands of ships of opportunity wind and pressure observations, the measurement error in the station pressure measurements at Darwin, Australia, and their adjustment to the equator, and the noise due to theoretical approximations in the derivation of (46). The noise-to-signal ratios n_x and n_y are thus nonzero and unknown and this must be taken into account when estimating h(L) by regression.

Bunge and Clarke (2009) tested the validity of (46) using monthly pressure and wind stress data from 1978 to 2003 inclusive. Since the data are autocorrelated, the number of degrees of freedom in the data is not the number of months of data, but rather the number of years of data because El Niño time series can be thought of as independent 12-month segments [see, e.g., Fig. 2.14 of Clarke (2008)]. As there are 26 yr of data, M = 26. Also, Bunge and Clarke found from the correlation of the time series in (46) and their standard deviations that = 0.91 and = 188 m.

The required 95% confidence interval limits L and U for α can be obtained for M = 26 by linearly interpolating the values for M = 25 and M = 35 in the supplement. For example, from Table S17, the 95% entry for L for = 0.91 is 0.7845 for M = 25 while the corresponding value from Table S18 for the M = 35 case is 0.8149. Linear interpolation gives L = 0.7875 for M = 26 and a similar analysis gives U = 1.2706. This implies, from the discussion associated with (45), that the 95% confidence interval for the true value of α = h(L) is

Note that the above confidence interval takes into account both the finite number of points M and our uncertainty about the noise. By comparison, the standard ordinary least squares regression of the left-hand side of (46) and on the right-hand side gives a 95% confidence interval (138 m, 204 m). This interval is smaller than that in (47), but it is a confidence interval for the M = regression coefficient α_OLS rather than the true regression coefficient α. The confidence interval for the true regression coefficient is larger because, in addition to the finite M limitation, the confidence interval for the required true regression coefficient takes into account the unknown noise.

6. Concluding remarks

Two reviewers’ comments made us think that we should point out here the difference between linear prediction and our goal of estimating the true regression coefficient . In linear prediction the aim is to predict y from x linearly. In that case, the ordinary least squares regression coefficient is appropriate because it matches y as best it can (in the least squares sense) given past data. When the noise is large, r² is small [see (18)], and such a linear predictor will only explain a small percentage of the variance. This results in all predictions being near the mean. Because of this, in some prediction circles the regression fit is “inflated” so that the regression estimates have the same variance as y (i.e., effectively is used as the regression coefficient). It is recognized that this is at the expense of the regression coefficient no longer giving the best least squares fit to y.

The preceding is related to, but separate from, our goal to find the true regression coefficient α between the variables Y and X given noisy realizations y and x as stated in (2) and (3). In our case, if it is known that the “noise”-to-signal ratio in at least one of the variables x and y (say x) is small, then the ordinary least squares regression coefficient is nearly unbiased and can be used [see (9) with small n_x]. But when the noise-to-signal ratio is not known in both variables, the ordinary least squares regression coefficient is biased. In that case the confidence intervals that are calculated for just inform you about the true least squares regression coefficient rather than the regression coefficient α, the coefficient we really want. When the noise-to-signal ratio is not known in both variables, and the tables in the supplement provide a simple practical way to estimate confidence intervals for the true regression coefficient. All that is needed is the number M of independent samples, and, from the data, the standard deviations of each variable and the correlation coefficient between them.