a. Introduction

There are two types of errors occurring in the retrievals. The first occurs because the precise atmospheric conditions present at the actual moment of irradiance observation are unknown at the time of the retrieval. Cloud height, thickness, aerosol content, surface albedo, water vapor, and ozone content are often unknown. However, with a high-density radiosonde network and satellite information of ozone content, the latter two parameters can be determined with sufficient accuracy to pose no obstacles in the retrievals. Also, excluding regions in the mid- and high latitudes in the winter surface albedo is only slowly varying in time. We make the assumption that water vapor and ozone concentrations are known at the time of retrieval, and, for simplicity, the surface albedo is set to zero throughout the remainder of the calculations.

The second type of error is introduced by the common, but mistaken, assumption that clouds are uniformly plane-parallel. Clouds vary in three dimensions on small scales, while a pyranometer receives radiation from the full hemisphere. The fundamental question of this paper is thus, to what extent do the cloud inhomogeneities influence the retrieval of optical depth?

b. Atmospheric parameters

Before simulations with inhomogeneous clouds were attempted, the sensitivity of the plane-parallel retrievals to variations in aerosol content, cloud height, and cloud microphysics was investigated. The existence of an aerosol layer introduces an error of at most 7 W m⁻² in the irradiances, which yields an error in cloud optical depth of about 1%. Changing the cloud-base height from 500 to 2500 m introduces an error of less than 1%. Changing the droplet concentration N causes the largest error. Calculations were performed using concentrations ranging from 50 to 400 cm⁻³ (see Fig. 1). The cosine of the solar zenith angle μ₀ = 0.8. The upper (lower) most curve corresponds to irradiance for N = 50 (400) cm⁻³. While we varied the droplet concentration for each value of the optical depth, the liquid water path remains the same. The three curves almost coincide, although a weak dependence on N is clearly evident.

The reason for the sensitivity of irradiance to cloud microphysics is that a cloud is not a conservative scattering medium. There are several wavelength regions in the near-infrared portion of the solar spectrum where cloud droplets absorb radiation. However, the contribution of solar radiation in the near-infrared spectral region to the total irradiance is small. For the single scatter albedo ω̃₀ < 1, it can be shown that ω̃₀ = H₁(r_eff), where r_eff is the effective radius of the cloud droplet distribution or equivalently that ω̃₀ = H₂(N), where H₁ and H₂ are monotonic functions decreasing in r_eff/increasing in N (Boers and Mitchell 1994).

Depending on the solar zenith angle the retrieval error is 10% at most and typically 5% for optical depth τ = 9. Leontyeva et al. (1994) discussed a similar calculation but used the effective radius as the free parameter rather than droplet concentration. They found errors of up to 15%, which is somewhat larger than in our case, possibly due to differences in the transfer calculations.

c. Inhomogeneous clouds

Reflectance and transmittance are highly nonlinear functions of optical depth. For example, suppose that a cloud field consists of a part 1 with optical depth τ₁, reflectance r₁, and transmittance t₁, and a part 2 with τ₂, r₂, t₂. Then, from the functional shape of Fig. 1 it follows that

View Expanded

This is the companion equation for the reflectance (see, e.g, Stephens et al., 1991; Marshak et al. 1998):

View Expanded

The consequences of inequalities such as (1) and (2) are far reaching. In general circulation models (GCMs) the albedo calculated for a cloud field with a specified mean liquid water content will be negatively biased from the real albedo if the subgrid variability of the optical depth is not taken into account. Similarly, the transmittance will be positively biased. So, if the average transmittance is positively biased, a retrieval of cloud optical depth from an averaged transmittance yields negatively biased optical depth values. However, it is impossible to be precise without extra information about the cloud structure, which is usually unavailable. It will be necessary to rely on models of the cloud variability to estimate the retrieval errors.

a. Fractal models

Through the pioneering work of Cahalan and Joseph (1989), Cahalan et al. (1994), and others we know that the horizontal variability of stratocumulus clouds can be captured by fractal models. In these models the horizontal structure and variance of liquid water content and optical depth are governed by a small number of parameters. The value of these parameters can be deduced from the power spectrum of optical depth or liquid water path fluctuations. The radiative structure of an inhomogeneous cloud can, up to a point, be described by simulating the inhomogeneous cloud field by a two-dimensional (x, y) field of adjacent pixels. For each of these pixels the radiative properties are calculated by plane-parallel theory. The only provision must be that the structural inhomogeneity of the cloud field is governed by fractal parameters conforming to values found in nature.

For the next cascade step we first multiply the value of f₁ by a fixed value smaller than one to reduce the variance at each cascade step. We get (see, e.g., Marshak et al. 1998)

View Expanded

where 0 ⩽ p ⩽ ½, H > 0, and i represents the cascade step. The value of p determines the ratio of the variance to the mean of the signals, while H determines the power β of the scalar spectrum S(k) ∼ k^−β as follows:

βHH,(5)

independent of p (Marshak et al. 1998). For many geophysical parameters, including the liquid water content and hence the optical depth, the value of β ∼ 1.62–1.65. This means that H ∼ 0.33. Figure 2 shows the downward irradiance at the surface as a function of f = (1 − 2p). Nine cascade steps were used, and for each of the 512 pixels the irradiance was calculated and averaged. The average value of the liquid water content is the same for each value of the fractal parameter. For f = 0, the solution corresponds to that for a horizontally homogeneous cloud. As the cloud liquid water, and hence the optical depth, is progressively redistributed according to the bounded-cascade model, the irradiance at the surface increases monotonically in f, which is consistent with the results presented by Stephens et al. (1991).

The implication is clear: if the mean optical depth is retrieved from the observed irradiance, then it will be smaller than the real mean optical depth. It is also clear that the measure by which the optical depth is underestimated can only be known if the fractal parameters f and p are known a priori. In other words, if f or p is not known, all that we know is that the optical depth is underestimated, but not by how much. This means that to provide adequate retrievals of optical depth from pyranometer observations, it is necessary to provide an independent means to ascertain the horizontal variability of clouds.

b. An adaptation to the bounded-cascade model

In order to quantify the errors associated with optical depth retrievals from pyranometer measurements, a two-dimensional (x, y) cloud field is generated with optical depth characteristics that are found in nature. We used an adaptation of the Marshak et al. (1995) two-dimensional extension of the bounded-cascade model to generate such a field. The adaptation consists of two parts described below; both adaptations involve the parameter p. For the cascade, a 50 × 50 km² cloud field was assumed. In our simulations it will be necessary to obtain realizations of cloud fields with varying optical depth.

The original bounded-cascade model is somewhat unrealistic: a stratocumulus cloud is normally bounded in the vertical by a temperature inversion. This restricts the depth of a cloud, implying a bound on the optical depth as well as a bound on the variance of optical depth at large scales. Yet, the final cascade steps of the original cascade model can occasionally distribute very high values of liquid water (and thus optical depth) in very small regions. There are different ways to overcome this problem. Our choice for a first adaptation is to restrict the variance in the first two steps of the cascade (i.e., scales down to about 12.5 km) by using the value of p = 0.43. Not only does it restrict the variance at large scales but also eliminates the excessively large values of optical depth at small scales. So, the redistribution of optical depth (or liquid water) is severely restricted from 50 km down to 12.5 km. For the other cascade steps the value of p = 0.30 is used. The absence of large optical depth values is in line with the analysis of LANDSAT imagery by Davis et al. (1997). They reveal that, at scales larger than 10–15 km, the spectrum flattens somewhat, which indicates that beyond those scales there is little increase in optical depth variance. Although the principal manner to simulate a change of the slope of the spectrum is a change to the value of H, our crude model is sufficient provided it is only used on the first two cascade steps.

The second adaptation also involves the value of p. As mentioned above, p determines the variance-to-mean ratio of the distribution. So, for the original fractal model the variance increases with the value of the average. This can easily be simulated by generating cascades with increasing mean values of the parameter field. Once again, this is unrealistic because real clouds are restricted in the vertical, providing a natural upper limit to the optical depth. To simulate this restriction the variance is allowed to grow according to the fractal model above for mean values of the optical depth below 12.8, that is, p and H remain the same while τ_mean increases. However, for τ_mean > 12.8, new optical depth distributions are generated by adding a fixed optical depth value to all pixels generated by the fractal model with τ_mean = 12.8. In essence this is a simple linear transformation of the entire distribution. The results are new distributions that are identical in shape to the ones generated at τ_mean = 12.8, but the variance remains constant and the variance-to-mean ratio reduces with optical depth.

We demonstrate our model by computing eight cascades to obtain a grid of 256 × 256 pixels. The unit of 200 m, which roughly correspond to the minimum size for which the IPA is still valid, used to obtain the grid implies that the area of the cloud field is a little over 50 × 50 km². The mean value of the optical depth of our demonstration case was 12.8. The value of the fractal parameter H = 0.33 determines the wave number dependence of the power spectrum, while p determines the patchiness of the distribution shown in Fig. 3a. The bounded-cascade model generates distribution functions that closely follow the lognormal distribution (Cahalan et al. 1994). The lognormal distribution function is plotted on top of the histogram in Fig. 3b, from which it is concluded that, despite our changes to the cascade model, the new distribution still closely follows the lognormal function. The peak value of optical depth of the lognormal distribution (i.e., the median of the distribution) is 11.08, which is considerably smaller than the mean.

Figure 4 shows two averaged spectra derived from the two-dimensional optical depth fields generated by the cascade model. The upper spectrum is calculated from an optical depth field generated with the unmodified cascade model, while the lower one is generated from our adapted cascade model. The spectra demonstrate that at larger scales the variance in our model is reduced by 50% when compared to the original bounded-cascade model. Toward lower scales the desired −5/3 behavior is found, which is indicated in Fig. 4 by a separate straight line. Figure 5 shows two one-dimensional cuts in the y direction of the two-dimensional optical depth field at the very left of Fig. 3a, separated by 10 pixels in the x direction. As expected the large-scale structure of the two signals is very similar, while at small scales the fractal redistribution of liquid water leads to small differences.

The principal advantage is of our model is that it is able to simulate a realistic flattening of the spectrum at large wavenumbers. The simulated optical depth field using the adapted fractal model is thus probably more realistic than that of the original bounded-cascade model, although we cannot be certain due to the absence of verification by independent data.

In the following sections we will use different realizations of the adapted bounded-cascade model to illustrate the effect of cloud inhomogeneities on optical depth retrievals. Three-dimensional cloud effects caused by multiple reflections from the side of clouds cannot be simulated in this manner, as the IPA model inherently assumes that the cloud field is infinitely thin in the vertical.

a. Theory

If θ is the angle between the incident radiance and the pyranometer normal, and ϕ is the azimuth angle in the plane perpendicular to the pyranometer normal, then the total irradiance incident at the pyranometer is

View Expanded

Here I(θ, ϕ) is the angle-dependent radiance falling onto the pyranometer, which has a direct and a diffuse part. Equation (6) may be recast in the form

View Expanded

where I(θ) is the average of the radiance over azimuth

View Expanded

In practice, the irradiance F can be decomposed into a direct and a diffuse component,

FF_dirF_diff(9)

and F_diff computed by integrating over cloud base in concentric annuli, whose radii increase at 200-m intervals. The diffuse radiance emerging from cloud base was assumed to be isotropic, and was computed from the diffuse irradiance via

I_diffF_diffπ.(10)

Using Eqs. (7)–(10) we find that

View Expanded

where (I_diffA)_i = Σ_j I_ijA_ij, the area A_ij is the area of the pixel contributing to the ith annulus, j is the pixel index in the ith annulus, θ_i is the mean zenith angle subtended by the ith annulus, A_i is the area of an individual annulus, Δθ_i is an angle sector, and F_dir is the direct irradiance from the single pixel in the direct line between the Sun and the pyranometer. Here the summation over angle θ_i is an approximation to the angle integral in Eq. (7), while the summation over index j is an approximation to the azimuth integral in Eq. (8). Equation (11) is instructive as it demonstrates that the individual rings are unequally weighted in the calculations for the diffuse irradiance by the sine factor. For equally thick rings, the dominant contribution is at angles near 45°.

b. Potential problems with the IPA approach

It is well known that the diffuse radiance downwelling from the cloud is not strictly angle independent, as we have assumed with our two-stream model. However, we rely on results presented in several recent studies that demonstrate that the IPA assumption is good enough for our purposes (Cahalan et al. 1994; Marshak et al. 1997, 1998; Stammes 1995), although it is recognized that at high spatial resolution (less than 100 m) the results of IPA diverge significantly from the Monte Carlo results.

Potential errors can occur because a small amount of water vapor absorption takes place between cloud base and the pyranometer in selected regions of the electromagnetic spectrum. Such absorption was not taken into account in our calculations. However, the clouds were embedded in a springtime atmosphere at an altitude of 1000 m at the top of the boundary layer with a mean water vapor content of 3 g kg⁻¹. So, water vapor absorption is expected to be small. Nevertheless, the presence of water vapor absorption implies that for individual pixels F_{ij,diff,cloud} > F_{ij,diff,surface} by a small but unknown amount. To partly avoid this problem, it was decided to substitute the downward irradiances F_ij at cloud base by those generated at the surface when originally performing the plane-parallel irradiance calculations for the individual pixels. This means that, if we were to substitute the fractal cloud field for a horizontally homogeneous cloud field, the calculated irradiance using Eq. (11) would correspond exactly to that of a single pixel using plane-parallel theory, as it should. The alternative approach is to use the downward irradiances at cloud base and assume an atmosphere without water vapor between the cloud base and the surface.

Note that, because the surface albedo was set to zero, the complex situation arising from multiple reflections of radiation from the surface and from cloud base is avoided. With these caveats in mind we show that the results below present a physically consistent picture of the influence of cloud inhomogeneities on the pyranometer signals.

c. Retrievals

Once the irradiance F at the pyranometer level was calculated using Eq. (11) the value of F was used to retrieve optical depth (τ_ret) using plane-parallel theory. In turn τ_ret was then compared to the real optical depth of the cloud field (τ_real)

View Expanded

where (τA)_i = Σ_j τ_ijA_ij; we note the structural difference between (11) and (12). Where (12) provides weighting over area only, in (11) the weighting is done over area and over angle. To simulate the variability in time due to wind drift, the entire cloud field was shifted one line of pixels to the left per time step after which the calculation was repeated. As the cloud field generated was around 50 × 50 km², and the area of the cloud field over which the irradiance calculations were performed was 12 × 12 km², the maximum translation distance of the cloud field was 25 km before time calculations had to be stopped. Assuming a wind speed of 10 m s⁻¹, this corresponds to a total time of 40 min.

The solar zenith angle was 60° and the mean optical depth over the full 50 × 50 km² grid was 12.8. Figure 6a shows typical variability of the direct and diffuse contributions to the total irradiance. For this case the contribution of the direct irradiance is at most 40%, only for time steps near time step indicator 80. For the rest it is considerably smaller, and, up to indicator 45, it can be entirely neglected. Figure 6b shows that the optical depth retrieved at individual time steps can be quite different from the real optical depth. Note that the optical depth is different from the mean value of 12.8. The reason is that the value using Eq. (12) is an average over that part of the cloud field that is within the view of the pyranometer, not the entire cloud field.

The variations in fluxes and retrievals are the result of cloud inhomogeneities passing by. The overestimation of optical depth in the first 45 time steps is the result of the unequal weighting of the irradiance with solid angle. Because the cloud located within the angle sector that has a large contribution to the irradiance at the pyranometer is thicker than the rest, the retrieved optical depth is larger than the real optical depth. However, near time indicator 80, the retrieved optical depth is less than 50% of the real optical depth. This is the result of the contribution of the direct irradiance to the total irradiance, which is large because a thin cloud section passed by in the direct line between the sun and the pyranometer.

Clearly, large errors can be incurred when cloud optical depth is retrieved from pyranometer data. The errors can easily exceed 50% and are mainly due to the inhomogeneity of the cloud field. They far exceed the errors associated with the assumption that the cloud droplet concentration is single valued; these errors, as we recall, were of the order of only 5%.

We proceed by generating many realizations of the adapted bounded-cascade model described above, and retrieving the optical depth according to

View Expanded

Here, the index k is a summation over time, K is the total number of time steps, and G is the retrieval procedure that converts the calculated irradiance F_k at time k into an optical depth. The procedure G relies on a straightforward tabulation of F versus τ at the solar zenith angle μ₀. So, knowing the zenith angle and the irradiance the table is searched for the corresponding value of optical depth. In the graphs to be shown, the comparisons are plotted as pluses. The retrievals according to (13) were calculated for averaging times of 10, 20, and 40 min, the results of which are shown in Fig. 7. The figure demonstrates that single-valued optical depths determined from 10-min averaged irradiance incur a large error, probably too large to be of any value to the observer. The error has little to do with errors in the plane-parallel theory caused by any problematic model assumption associated with the use of set values of cloud-base height, droplet concentration, or aerosol optical depth. They are entirely due to the sampling imprecision of the pyranometer. Such observations are thus only useful as part of a large collection of data points for which statistics can be computed. Also shown (as dots) are the averages in optical depth intervals of 2.5, including a bar that is the standard deviation of the collection of optical depth values used to determine the average point.

When the averaging time is increased, the point spread is reduced. However, there is no discernible improvement in the overall characteristic of the plot. For the 40-min average, the overall bias is the same as for the 10-min average, but the standard deviation is reduced by a factor of 3. Note further that the averaging time to compute the mean irradiance cannot be increased indiscriminately because in reality the solar zenith angle is changing with time. So, for long averaging times, it becomes progressively more difficult to assign a “mean” solar zenith angle to the mean irradiance necessary to retrieve optical depth.

All three plots show that the retrieved value of the optical depth is negatively biased. The bias varies somewhat but has a value of between −0.5 and −1.2, or about 10% of the optical depth value. The value of the bias depends on the choice of fractal model to simulate the cloud field. A set of sample calculations was performed by increasing the factor (1−2p) to 0.5. For these conditions the negative optical depth bias is doubled.

d. Broken clouds

Cloud fields are often broken, and under such conditions the pyranometer may view a portion of the clear sky, or even direct sunlight. From a 10-min averaged pyranometer signal, as is typically available for analysis, it is not obvious how much of the time the pyranometer was exposed to a fractionally covered sky. One possible way to simulate cloud clearing is to increase the value of the fractal parameter p and then implement a pixel clearing when the optical depth becomes smaller than a prescribed value for individual pixels.

For six such realizations the downward irradiance was simulated as a function of time. The results are shown in a series of panels in Fig. 8. In the last three panels (Figs. 8d,e,f) cloud clearing was invoked for progressively larger values of the pixel optical depth. Turning first to Figs. 8a,b,c, the decrease of the parameter p from 0.40 to 0.15 results in an increase in the variability of the irradiance. In particular, beyond time step 40, the cloud between the Sun and the pyranometer becomes progressively more transparent so that the direct contribution to the irradiance increases rapidly. As the cloud clearing option is invoked, irradiances exceed the clear-sky irradiance for a series of time steps, despite the fact that the total cloud cover in the field of view of the pyranometer well exceeds 80%. Clearly, when the irradiance of a partly cloudy sky exceeds that of a clear sky, the retrieval of cloud optical depth is impossible.

Note further that the diffuse irradiance decreases rapidly as patches of clear sky enter the field of view of the pyranometer (cf., e.g., Fig. 8a, Fig. 8f). The reason is that the diffuse irradiance from a clear sky is smaller than that of a cloudy sky. This means that an optical depth retrieval from diffuse irradiances alone can yield erroneous results because diffuse irradiances will decrease not only with thickening clouds but also with increasing clear patches.

Cloud clearing also can be obtained by uniformly subtracting a prescribed value of the optical depth from an entire predetermined field. Any negative values of the optical depth are set to zero and the retrieval procedure can be repeated. So, the cloud fraction reduces to values less than one, as the optical depth offset is increased. We proceeded by using this slightly different technique and implemented three retrieval methods. For method 1, the values of F were averaged to 10, 20, or 40 min before inverting them to values of optical depth.

For method 2, each individual measurement of the irradiance was used to retrieve an optical depth, and next, the time series of optical depth was averaged over 10, 20, or 40 min:

View Expanded

The comparisons are plotted as open diamonds (Fig. 9).

For method 3, it was assumed that the Sun is eclipsed at the pyranometer by a mechanical device, so that the irradiance comprises only diffuse irradiance. This can easily be done by omitting the direct irradiance in the calculation of (11):

View Expanded

The comparisons are plotted as crosses (Fig. 9). The reason for including this method in our calculation is that, in this way, the high variability of the optical depth retrievals is reduced because the diffuse irradiance is only a function of the total cloud cover, not of the illumination of the Sun.

The cosine of the solar zenith angle was set to 0.5. The results are shown in Fig. 9. As the optical depth is decreased by subtracting different values of a fixed optical depth from the entire 256 × 256 pixel grid, the three retrievals start to diverge from each other. The general negative bias at large optical depth is due to inequality (1); the bias is directly related to the value of the fractal parameters. The retrievals based on the 10-, 20-, and 40-min averaged values of the irradiance (pluses on the graph) become progressively lower, until at optical depth values around 5 retrievals become very small or impossible to compute. This is the result of the fact that cloud clearing often results in direct illumination of the pyranometer (cf. Fig. 6), so that the observed irradiance exceeds that of a clear sky.

The inclusion of clear-sky values in the time average will drive the irradiance high so that no credible optical depth values can be obtained. Nevertheless, in the absence of alternative information about cloudiness, or about the amount of direct sunlight striking the pyranometer, such a low retrieval value may appear realistic. Note that there are still several points in the vicinity of the line indicating a one-to-one correspondence between the retrieved and observed optical depth. These are the result of averages for which a cloud persisted between the pyranometer and the Sun. Cloud fraction for the retrievals near τ_real = 5 it still well over 70%, so that the large errors occur even though the sky is still well covered by clouds.

In contrast, the values derived from the high-resolution irradiance record (method 2) do not appear to be negatively biased at the low end of the optical depth scale. In fact, for lower optical depth values, they appear to become positively biased. The reason is that, if for individual times the measured irradiance exceeds that of a clear sky, no optical depth can be retrieved. This implies that there will be no contribution to the average optical depth for that particular time step. As a result of these omissions (or “retrieval errors”) the average retrieved optical depth is higher than the true optical depth.

The positive bias is even larger if only the diffuse irradiance is used for retrieval. The reason is that, for increased cloud clearing or thinner clouds, the direct contribution to the irradiance becomes progressively larger, while at the same time the diffuse irradiance becomes progressively smaller. The diffuse irradiance does not become smaller because clouds become thicker, but because the diffuse irradiance emanating from the clear-sky patches is smaller than that of a corresponding cloudy sky. Therefore, the retrieval procedure would correspondingly tend to overestimate the optical depth, as it cannot distinguish between clear-sky diffuse and cloudy-sky diffuse irradiances.

Although there seems to be a threshold value of about 5 below which no optical depths can be retrieved, this is an artifact based on the modeled spread of the lognormal distributions of optical depth used to generate cloud clearing. Other distributions would generate different thresholds.

e. Summary of errors

The largest errors in the retrievals are due to the inhomogeneity of the cloud layer. This introduces a transmittance bias, similar but opposite to the well-known albedo bias. The result is that the retrieved optical depths are generally undervalued. Great care needs to be taken to ensure that cloud cover is 100%. If cloud cover cannot be measured by other independent means, retrievals for optical depths smaller than about 8 become progressively more error prone, while below values of the optical depth of 5 retrievals are impossible, or otherwise have an unacceptably large error, due to the sun striking the pyranometer. These errors occur even though the cloud cover is over 70%. The optical depth retrieval will be negatively biased with the mean bias near −1 amounting to roughly 10% of the optical depth values. Individual retrievals based on 10-min averages of irradiance can easily have 50% to 80% errors, which make them largely unsuitable for the observer. They are useful only in a statistical sense.