1. Introduction

The study by Koshak (2010) showed that the distributions of ground and cloud flash optical characteristics, as seen from the Optical Transient Detector (OTD), overlap appreciably. Therefore, space-based flash-type discrimination (on a flash-by-flash basis) is fundamentally difficult. However, Koshak (2010) also indicated that the mean values of the optical characteristics for ground and cloud flashes are distinct, so that an analysis of a sample of N flashes could possibly provide information about the fraction of ground flashes (i.e., the ground flash fraction) within the sample.

The follow-on study by Koshak and Solakiewicz (2011) confirmed this hypothesis. They established a mathematical formalism for analyzing the mean optical data and provided a straightforward approach for estimating the ground flash fraction in a sample of flashes. In numerical tests, they retrieved the ground flash fraction for 52 regions that were widely distributed across the conterminous United States (CONUS). The retrieval errors were under 11.1%, and as low as 6.1%. However, their method is based on employing CONUS-based estimates of the population means of the optical characteristics. This implies that the ground flash fraction retrieval accuracy could degrade if the method is applied outside the CONUS where the CONUS mean estimates might no longer be accurate. In other words, if the optical characteristics that are used [i.e., the maximum number of events in a group (MNEG), or the maximum group area (MGA)] vary greatly from their respective CONUS-mean values, then retrieval error will increase to unacceptable levels.

Though Koshak and Solakiewicz (2011) demonstrated that the ground and cloud flash values of MNEG and MGA did not vary greatly across CONUS (a region of highly variable lightning, highly variable thunderstorm structures, and highly variable cloud morphologies with diverse cloud-scattering properties), it is still uncertain how variable MNEG and MGA are across the globe. To determine the global variability, satellite-based total lightning observations would need to be compared with independent ground flash observations so that the satellite observations could be separated into ground and cloud flashes. Statistics of the spatial variation of MNEG and MGA for both the ground and cloud flashes could then be computed. However, because of the lack of ground flash detection systems across the globe (with particularly poor ground flash detection over the oceans), the present ability to make detailed assessments of the global variability of MNEG and MGA is somewhat limited; and no studies have yet been conducted to assess the variability given the existing coverage. Fortunately, this will not always be the case because improvements to global ground flash detection/coverage are in progress [e.g., the Vaisala Corporation has introduced a new Global Lightning Dataset (GLD360) product with plans for continued network expansion (F. DeFina, Vaisala Corporation, 2009, personal communication)].

Because of the lack of knowledge of the global variability of MNEG and MGA, and because relying on CONUS-based estimates of these optical characteristics is fundamentally restrictive, it is highly desirable to devise a retrieval algorithm that not only retrieves the unknown ground flash fraction for a particular geographical region, but also the population means of the unknown optical characteristics (either MNEG or MGA) for that region. Such an algorithm would obviously be more robust than that provided in Koshak and Solakiewicz (2011) because it would not depend on using CONUS-based estimates of lightning optical characteristics.

Other motivations for this study are of a general nature and have been previously discussed. The ground flash fraction (or the closely related parameter called the “Z ratio,” given by the ratio of the number of cloud flashes to ground flashes) is thought to be particularly useful in a number of areas, for example, severe weather warning, lightning–convection relationships, lightning nitrogen oxide (NO_x) production, the contribution of lightning to the global electric circuit, and cross-sensor validation (Koshak 2010; Boccippio et al. 2001). In addition, a satellite-based ground flash fraction retrieval product would obviously be valuable for cross comparison/validation with global ground flash measurements, such as the GLD360 dataset mentioned above.

In this study, a method is introduced for retrieving the ground flash fraction of a sample of N flashes observed from space [e.g., from the OTD, the Lightning Imaging Sensor (LIS), or the future Geostationary Operational Environmental Satellite (GOES)-R Geostationary Lightning Mapper (GLM)]. The method is more sophisticated and more general than that provided in Koshak and Solakiewicz (2011) because retrievals of the unknown population means of optical characteristics (i.e., of MGA) are obtained. Hence, the method is useful for global application.

The study begins by introducing a general form for the mixture of two optical distributions (section 2). The mixture consists of a linear combination of two distributions—one distribution represents the distribution of a ground flash optical characteristic and the other distribution represents the distribution of the associated cloud flash optical characteristic. The mean and variance of the mixture distribution are provided. A Bayesian method for retrieving the ground flash fraction and attributes of the ground and cloud flash optical distributions is provided in section 3. Next, section 4 introduces a special case of the mixture distribution, called the “mixed exponential distribution model,” which is the primary focus of this study. Section 5 details the “label switching” ambiguity associated with mixed exponential distribution models, and section 6 introduces a useful approach for initializing the numerical search associated with the Bayesian retrieval process. Section 7 shows how population statistics can be used as constraints to the retrieval process. Finally, numerical tests of the Bayesian retrieval method (applied to the mixed exponential distribution model) are provided in section 8; the method is also applied in section 9 as a first attempt to retrieve the ground flash fraction on a global scale using OTD data. Concluding remarks are provided in section 10.

2. The distribution of a mixture

Consider a set of i = 1, … , N flashes that are observed over a time period Δt by a satellite lightning imager (e.g., a low-earth-orbiting sensor like the LIS, or a geostationary sensor like the future GLM). For each of the N flashes, the sensor measures a particular optical characteristic x. For example, this characteristic could represent flash radiance, flash area, flash duration, the number of optical groups in the flash, or the number of optical events in the flash. One could consider several other optical characteristics, such as the maximum number of events in a 2-ms sensor frame time for a given flash, radiance of the first event in the flash, radiance of the brightest group, MNEG, and MGA. Note that the common nomenclature for describing the OTD/LIS data is used; that is, a flash is composed of optical groups, and each optical group is composed of optical events (see Mach et al. 2007).

In general, the distribution of the optical characteristic x will be different for ground and cloud flashes. See examples of the differences (for various optical characteristics) in Koshak (2010). The distribution of x for ground flashes is denoted by p_g(x), and the distribution of x for cloud flashes is denoted by p_c(x). Hence, the distribution of a mixture of ground and cloud flashes, called the mixture distribution p(x), is given by

View Expanded

where α is the ground flash fraction, as discussed in Koshak and Solakiewicz (2011). For the set of N flashes, αN of them are ground flashes and (1 − α)N are cloud flashes. Formally, note that p(x), as well as p_g(x) and p_c(x), are probability density functions (pdfs); the terms “distribution” and “density” are used interchangeably. Hence, the population mean and variance of the mixture distribution are

View Expanded

Here, the population mean and variance of the ground and cloud flashes are, respectively,

View Expanded

Obtaining the results in (2) are straightforward, and the second result in (2) requires a little algebra.

3. Bayesian inference of model parameters

a. The MAP solution

As discussed above, the satellite lightning imager will observe N flashes over a particular region during some time interval Δt. Considering the optical characteristic x, this means that one has the N vector of observations x = (x₁, x₂, … , x_N) from which to retrieve the ground flash fraction α. In general, however, a particular retrieval algorithm might not use every piece of information in x to retrieve α. To include all classes of Bayesian-type retrievals, one can consider a vector d that is derived from x. For example, the components of d could be the first several moments of the sample observations in x; that is, the first component of d could be the sample mean x of the N observations, the second component could be the sample variance s² of the N observations, and so on. The entire vector d might even degenerate into a single scalar, for example, . The components of d could even represent various sample survival values (complementary cumulative distribution values), that is, the first component of d could be the fraction of the elements in x that exceed some value X, and the second component of d could be the fraction that exceed some value Y, and so on. Finally, if all of the raw data are used directly, then one simply has d = x. This is the preferred approach because one intuitively expects more accurate retrievals when all of the observational information is used. In the case where x might be a very large vector (i.e., N very large), practical limitations might warrant using the other forms of d described so that the resulting dimensions of d are less than N.

With all of these possibilities in mind for the choices of d, one can write Bayes’ law as

View Expanded

where v = (υ₁, … , υ_η) = (α, υ₂, … , υ_η). The η − 1 components (υ₂, … , υ_η) are population statistics of p_g(x) and p_c(x). The attributes v define the η parameters of a model that, in turn, are used to described the mixture distribution p(x) in (1). The sample data vector d is used to infer the model parameters v. Note that (4) provides an expression for determining the probability of getting the model parameters v given the sample data d. The notation in (4) follows that provided in Rodgers (2000), that is, P(v|d) is the posterior pdf, P(v) is the prior pdf, P(d|v) is the measurement pdf, and P(d) is a pdf that serves as a normalization constant. Essentially, prior to any measurement one has the prior knowledge of the state v given by P(v). After specific measurements are made, this prior knowledge is updated by P(d|v), which describes the knowledge of d if the state were v; that is, P(d|v) describes the forward problem. The forward problem describes how one maps the unknown model parameters v into measurement space d. Carrying out the algebra in Bayes’ law given in (4), one obtains P(v|d), that is, the probability density of the state v when the measurement d is given.

The approach here is to determine the most probable v given the sample data d. Because d is given, it can be treated as a constant vector. The maximum a posteriori (MAP) solution described in Rodgers (2000) is the value of v that maximizes P(v|d). Formally, the maximization is expressed by obtaining the critical point solution to the following set of equations:

View Expanded

b. Model parameter independence

The attributes (model parameters) are commonly assumed to be independent. For example, one does not expect the ground flash fraction itself to depend on the population mean optical statistics, and the population statistics of ground and cloud flashes are expected to be reasonably independent. The condition for independence can be expressed as

View Expanded

As a simple illustration of independence, consider the case of three model parameters. In general, one has P(υ₁, υ₂, υ₃) = P(υ₁|υ₂, υ₃) × P(υ₂, υ₃) = P(υ₁|υ₂, υ₃) × P(υ₂|υ₃) × P(υ₃). However, if υ₁ is independent of υ₂ and υ₃, then P(υ₁|υ₂, υ₃) = P(υ₁); and if υ₂ is independent of υ₃, then P(υ₂|υ₃) = P(υ₂). Hence, independence implies that P(υ₁, υ₂, υ₃) = P(υ₁) × P(υ₂) × P(υ₃).

c. Uniform and normal priors

When there is little or no knowledge about a model parameter and/or one does not want to bias solution retrieval in any way, the ignorance prior is often assumed. The ignorance prior for the jth model parameter is that of a uniform distribution given by

View Expanded

where the distribution is defined over the interval I[a_j, b_j], and c_j is a jth constant.

In some case, however, one might have data or other information that suggests a particular model parameter is normally (or approximately normally) distributed. A normally distributed prior takes on the form

View Expanded

where μ_j and σ_j are the population mean and standard deviation, respectively.

d. The MAP solution assuming independence and mixed uniform and normal priors

Suppose that model parameter independence is assumed. Also suppose that the first j = 1, … , η_u model parameters are assumed to follow a uniform distribution, and the remaining η − η_u model parameters are assumed to follow a normal distribution. In this case, (6) reduces to

View Expanded

The probability density of drawing the vector d given the model parameters v is simply the product of the probability densities associated with drawing the ith component of d given v; that is,

View Expanded

Note that p(d_i|v) is just a way of representing the mixture distribution p(d) when the conditionality on v is explicitly given; see section 2 for more on the mixture distribution.

Substituting (9) and (10) into the last equation set given in (5) gives a nonlinear system of η equations in η model parameter unknowns, where d is regarded as a constant m vector, with m ≤ N. It is normally either difficult or impossible to solve such a system analytically, so the MAP solution is typically found by using numerical optimization methods such as those provided in Press et al. (1992, chapter 10). Because the product of small numbers (probabilities) is quite small and often results in computational underflow errors, it is customary to take the natural logarithm of the function to be maximized. The natural logarithm removes the underflow errors by converting very small fractions into “regular range” negative numbers. Hence, in practice, the MAP solution can be found by maximizing the function S given by

View Expanded

4. The mixed exponential distribution model

The preceding section provided a general framework for obtaining the MAP solution when model parameter independence is assumed, and when the model parameter prior distributions can be assumed to be either uniform distributions, normal distributions, or a combination of both. In this section, a particular form for the model mixture distribution p(d_i|v) is chosen, and the prior distributions are specified.

The primary approach taken in this study is to thoroughly examine the known observations composing p_g(x) and p_c(x), and then build a physically meaningful model based on these observations. By virtue of the mathematical form of the model invoked, physical constraints are implicitly added to the retrieval process. These “model constraints” ultimately help remove solution ambiguity. [In addition, one should note that it is possible to avoid model constraints altogether by invoking the Central Limit Theorem of statistics (see the appendix for additional details).]

In Koshak (2010), two closely related optical characteristics were suggested as being valuable in the retrieval of α; one was the MNEG and the other was the MGA. Because the MNEG distribution contains only integer numbers that lead to less smooth distributions, the MGA optical characteristic is used here. Hence, in all that follows, note that x = MGA.

Figure 1 shows the distribution of x (i.e., MGA) for ground and cloud flashes, as provided in Koshak (2010). These data were derived from an analysis of 5 yr of OTD data. Note that the distribution of x ramps up quickly to a peak frequency value, and then gradually decays for larger values of x. The low end of the distribution gets complicated/truncated by the OTD pixel resolution, which varies across the OTD field of view (and is about 8 km at nadir). Rather than attempting to accommodate the low-end peculiarities using a complicated multiparameter model, it was better to simply focus on all x ≥ 64 km². That is, a new random variable y ≡ x − 64 km² is used.

View Full Size

Distribution of MGA for (left) ground flashes and (right) cloud flashes as found in Koshak (2010).

Citation: Journal of Atmospheric and Oceanic Technology 28, 4; 10.1175/2010JTECHA1438.1

• Download Figure
• Download figure as PowerPoint slide

Figure 2 shows plots of the distribution of y for the OTD ground and cloud flashes. The red analytic curve in each plot is an exponential distribution of the form , where y takes on the data average shown in the upper-right corner of the plot. For a mixture of ground and cloud flashes, one can consider a superposition of exponential distributions. This gives, in general, the following mixed exponential distribution model:

View Expanded

where the population means are given by

View Expanded

The population mean and variance of the mixture distribution p(y) are

View Expanded

Mixed exponential distribution models are commonly used in a variety of disciplines [e.g., such models are employed in the medical literature to analyze patient “length of stay” in the hospital (Keatinge 1999)].

View Full Size

The distribution of y ≡ x − 64 km² for the OTD (left) ground flashes and (right) cloud flashes. In each plot is shown an exponential distribution (red curve) having as a mean the OTD CONUS average of y, given in the upper-right hand corner of the plot. Note how well the exponential model fits the two datasets. The statistics shown are based only on values of y ≥ 0.

Citation: Journal of Atmospheric and Oceanic Technology 28, 4; 10.1175/2010JTECHA1438.1

• Download Figure
• Download figure as PowerPoint slide

Note that the mixed exponential model in (12) has three model parameters, that is, v = (α, μ_g, μ_c), and p(y) could be rewritten in the more explicit fashion as that of p(y|v). In addition, d = y; that is, y is derived from x and consists of all of the elements of x that are greater than or equal to 64 km² (i.e., the instrument nadir pixel ground footprint area). Next, assumptions are made about the forms of the model parameter prior distributions.

The prior distribution P(α) is assumed to be uniform. Because 0 ≤ α ≤ 1, this implies that P(α) = P(υ₁) = 1/(b₁ − a₁) = 1/(1 − 0) = 1. One could argue that lower values of α are more likely (i.e., cloud flashes generally out number ground flashes), but because the main interest of this study is to retrieve α, extra caution has been taken to avoid prebiasing retrieval results.

For P(μ_g) and P(μ_c) it is useful to consider the data in Table 3 of Koshak and Solakiewicz (2011), where the statistics of the spatial variability in MGA across CONUS are provided. The mean (and standard deviation) of the ground flash MGA over 52 widely distributed locations across CONUS was 488.7 km² (39.5 km²); the cloud flash MGA values were 213.7 km² (16.8 km²). These are only rough estimates because the statistics depend on how many flashes are analyzed in the vicinity of each of the 52 locations; nonetheless, the standard deviation values give a “ballpark” idea of the spatial variability. In addition, given that CONUS is a region of highly variable lightning, highly variable thunderstorm structures, and highly variable cloud morphologies with diverse cloud scattering properties, the spatial variability of MGA across the globe is probably not too much larger than the CONUS standard deviation values indicated here. In addition, the mean y values shown in Fig. 2 are likely to be reasonably close in value to the respective (μ_g, μ_c) values for a particular geographic region over the globe, plus or minus some random variability. Taking all of this into account, it is assumed that P(μ_g) is normally distributed with a mean given by the CONUS mean (i.e., 431.521 70 km² from the left-hand distribution in Fig. 2) and with a standard deviation given by 50 km² (i.e., something on the order of, or a bit larger than, the spatial variability of y across CONUS given by the values 39.5 km² and 16.8 km² mentioned above). Similarly, it is assumed that P(μ_c) is normally distributed with a mean given by the CONUS mean (i.e., 152.949 93 km² from the right hand distribution in Fig. 2) and again a standard deviation of 50 km². [Note that the standard deviation of MGA across CONUS is identical to the standard deviation of y across CONUS since y is just shifted in magnitude; i.e., y = MGA – 64 km².]

With these assumptions imposed on the prior distributions, and utilizing the mixed exponential model in (12), the function S in (11) simplifies to

View Expanded

where μ₂ ± σ₂ = 431.521 70 km² ± 50 km², and μ₃ ± σ₃ = 152.949 93 km² ± 50 km².

To obtain the MAP solution, that is, the value of v that maximizes S in (15), the Broyden–Fletcher–Goldfarb–Shannon variant of the Davidon–Fletcher–Powell (DFP) algorithm is used to find the minimum of −S. (see chapter 10 of Press et al. 1992 for details on numerical optimization).

5. Label switching

Note that if all of the model parameter prior distributions are defined as uniform distributions, then S(v) depends only on the first sum in (15) plus a constant. This makes the retrieval process highly susceptible to solution nonuniqueness because of the so-called “label switching” problem (Redner and Walker 1984; Diebolt and Robert 1994; Richardson and Green 1997; Celeux 1998; Stephens 2000; Jasra et al. 2005). Label switching can be clearly defined by considering the following transformation:

View Expanded

For y > 0, the mixture distribution is

View Expanded

In other words, the mixture distribution is invariant under the transformation given in (16). This means that a mixture with the properties v = (α, μ_g, μ_c) is indistinguishable from a mixture with the properties v = (1 − α, μ_c, μ_g). This is a fundamental ambiguity related to the fact that one can always suppose that the ground flashes are really all cloud flashes, and that the cloud flashes are really all ground flashes; hence, the population means interchange and the ground flash fraction α converts to its complement 1 − α.

However, because the MGA distributions for the conterminous United States provided in Koshak (2010) have the property μ_g > μ_c, it is reasonable to consider only those solutions where μ_g > μ_c holds. This approach for helping to mitigate the effects of label switching is called an identifiability constraint (Stephens 2000).

Even after invoking the identifiability constraint, label switching is still a problem when the true value of α is near (or at) zero or unity. For example, when α = 0, μ_g is undefined so that the retrieved value of μ_g is meaningless and therefore not comparable with μ_c. When α = 1, the converse is true; that is, μ_c is undefined so that the retrieved value of μ_c is meaningless and therefore not comparable with μ_g.

Nonetheless, one could still try comparing the meaningful μ_g value (case α = 1) or the meaningful μ_c value (case α = 0) with the CONUS means in Fig. 2. For example, if one assumes that the case α = 0 is correct, then one would expect the retrieved μ_c value to be reasonably close to the cloud flash CONUS mean of 152.949 93 km² given in Fig. 2. If, instead, μ_c was closer to the ground flash CONUS mean of 431.521 70 km², then one would suspect that the case α = 1 is actually the truth. Similarly, if one assumes that the case α = 1 is correct, then one would expect the retrieved μ_g to be reasonably close to the ground flash CONUS mean of 431.521 70 km² given in Fig. 2. If, instead, μ_g was closer to the cloud flash CONUS mean of 152.949 93 km², then one would suspect that the case α = 0 is actually the truth. However, the author has found that such comparisons still are not sufficiently effective in removing solution ambiguity.

Fortunately, assuming that the prior distributions P(μ_g) and P(μ_c) are normally distributed helps to mitigate the label-switching problem because the symmetry in S that leads to the label-switching problem is broken [see section 8b, which quantifies the benefits of invoking the normal priors]. Note that the means of these normal distributions (i.e., μ₂ = 431.521 70 km², and μ₃ = 152.949 93 km²) given in the previous section are consistent with the identifiability constraint because μ₂ > μ₃.

6. Initializing the search using constraints based on population statistics

Recall that the DFP minimization algorithm mentioned in section 4 is used to find the minimum of −S, where S = S(v) = S(α, μ_g, μ_c) is given in (15). The DFP algorithm requires an initialization (i.e., “first guess”) of v. A good first guess improves the chance of getting successful convergence to the global minimum of the function. In this section, constraints that are based on population statistics are introduced, and are shown to be very useful in obtaining good initializations of v. These constraints also clarify what is (mathematically) involved with picking solutions that have μ_g > μ_c.

For m large, the sample mean and sample variance provide good estimates of the associated populations statistics in (14); that is,

View Expanded

where the sample mean and variance are given by

View Expanded

For a fixed (α, μ, σ²), and viewing (μ_g, μ_c) as coordinate axes, the first equation in (18) is an equation of a line and the second equation in (18) is the equation of an ellipse that is not in standard form; that is, it is a rotated ellipse. The equation of the line allows one to eliminate μ_c as follows:

View Expanded

Substituting this expression for μ_c into the variance equation in (18) gives a result that is quadratic in μ_g with solution

View Expanded

Because a = a(α), b = b(α; μ), and c = c(α; μ, σ²), note that one obtains the following functional dependencies: μ_g = μ_g(α; μ, σ²) and μ_c = μ_c(α; μ, σ²). Simplifying the positive root of (21) and substituting it into the first equation of (20) gives the positive root solutions as

View Expanded

By similar manipulations, the negative root solutions are

View Expanded

In addition, the results in (18) imply that

View Expanded

Hence, the expressions under the radicals in (22) and (23) are nonnegative. This means that the positive root solution in (22) necessarily results in the condition μ_g ≥ μ_c, whereas the negative root solution results in the condition μ_g ≤ μ_c. In other words, because this study only picks solutions with μ_g > μ_c, this means that only the positive root solution is considered; that is, this study uses (22), not (23). This is not an arbitrary strategy because all of the analyses of CONUS data provided by Koshak (2010) have in fact suggested that μ_g > μ_c. (As discussed in section 5, one is initially pressed with the issue of picking solutions either of type μ_g > μ_c or μ_g < μ_c in order to address the label-switching issue in a consistent way.)

Moreover, the result in (22) is used in this study to help initialize v. Using the estimates in (18), and making the initialization α_initial = 0.5, (22) gives the initializations

View Expanded

Note that this is usually an excellent initialization because the estimates in (18) are excellent when the sample size N (i.e., the number of flashes observed) is large.

7. Aggressive application of population statistic constraints (reduced search spaces)

For completeness, it is worth mentioning that it is possible to more aggressively apply the population statistics provided in (18) when obtaining a solution. For example, one can substitute (22) into (15) to eliminate (μ_g, μ_c) altogether so that the function S is converted into a function of just one variable, that is, S(v) → S(α). In this case, is used to estimate (σ² − μ²) with some estimation error ɛ. This means that one would just need to scan through the values of α between 0 and 1 to find the maximum of S(α); that is, the optimum ground flash fraction would just be the value of α at which the S(α) curve has its maximum. This in fact does work for values of α between about 0.2 and 0.8. However, note in (22) that when one scans for values of α near zero the value of the factor (1 − α)/α approaches infinity, and when one scans for values of α near unity the factor α/(1 − α) approaches infinity. This results in unreasonable error magnification of the estimation error ɛ, and hence a poor retrieval results when the true ground flash fraction is near zero or unity.

Another approach is to use the first equation in (18) to eliminate α; that is, the expression can be substituted into (15) in order to convert S(v) → S(μ_g, μ_c). This conversion reduces down the dimensionality of the problem, which has practical benefits (smaller search space, less CPU time to obtain a solution, and a more constrained solution), but this approximation of α is nonoptimum if the approximation happens to be poor.

The author has experimented with the two above retrieval methods, and finds that, although they have some beneficial features, a numerical maximization of the full three-dimensional function S(v) given in (15) is preferred. The main reason for this decision is that the basic objective of finding the v that maximizes P(v|y) is not compromised in any way.

8. Numerical tests

a. Dependence of retrieval error on sample size

To begin, a simple test is performed that illustrates the effect of finite sampling on retrieval errors. A known mixed exponential distribution is assumed; that is, the population means are fixed at μ_g = 400 km² and μ_c = 170 km², and the ground flash fraction is fixed at α = 0.3. The three parameters v = (α, μ_g, μ_c) fully define the mixture distribution p(y). Note that the values of μ_g and μ_c chosen here are physically reasonable; that is, the CONUS y means based on the 5-yr OTD data analysis of Koshak (2010) are μ_g = 431.521 70 km² and μ_c = 152.949 93 km². Next, the known mixture distribution is used to randomly generate a set of m = N simulated measurements. These simulated measurements are inverted using the Bayesian inference methodology, and the retrieval error in each of the three parameters (α, μ_g, μ_c) is computed and stored. Another set of N measurements are randomly generated from the mixture distribution, and again the retrieval errors are computed and stored. A total of 4000 simulated retrievals were performed in this way.

To appreciate the advantages of employing (22) in the initialization of v, Fig. 3 shows the distribution of the deviation of the sample mean from the population mean, that is, and the distribution of the deviation between the sample standard deviation and the population standard deviation s − σ. Here, the sample size N = 2000. As expected, the values of y are close to μ and the values of s are close to σ. Hence, the approximations in (18) are quite reasonable for the initialization provided in (22).

View Full Size

The distribution of (left) and (right) s − σ for the case N = 2000. Because the difference between the sample and population value is reasonably small, the initialization methodology introduced in (22) is highly useful.

Citation: Journal of Atmospheric and Oceanic Technology 28, 4; 10.1175/2010JTECHA1438.1

• Download Figure
• Download figure as PowerPoint slide

Figure 4 shows the statistics of the retrieval errors for the cases with N = 500, 1000, 2000, and 5000 measurements. The retrieval error is given by e = |α_r − α|, where α_r is the retrieved ground flash fraction and α is the true ground flash fraction (i.e., 0.3). As expected, the retrieval errors decrease as N increases because the estimates in (22) improve, and also because more information is used in the Bayesian retrieval method. Tests of this type were also performed for the case α = 0.7, and similar results were found.

View Full Size

The distribution of ground flash fraction retrieval errors e = |α_r − α| as a function of the number of flashes N in the mixture. Retrieval errors decrease as N increases.

Citation: Journal of Atmospheric and Oceanic Technology 28, 4; 10.1175/2010JTECHA1438.1

• Download Figure
• Download figure as PowerPoint slide

Average errors in μ_g were (25.5, 24.3, 23.6, and 20.8 km²) for the respective values (N = 500, 1000, 2000, and 5000), and average errors in μ_c were (14.7, 11.7, 9.5, and 7.0 km²). All of these errors are a reasonably small percentage of the known values (μ_g = 400 km² and μ_c = 170 km²).

b. Dependence of retrieval error on α and the prior distributions

Next, it is illustrated how the mean retrieval error in α varies as a function of α, and on the type of prior distributions used. With the same fixed values (μ_g = 400 km², μ_c = 170 km²), 100 simulated retrievals were performed in each α bin of width 0.05. That is, 100 known values of α were randomly chosen from the first bin (α, 0.0–0.05), 100 known values from the second bin (α, 0.05–0.1), and so on up to the last bin (α, 0.95–1.0). These tests provide the mean retrieval errors provided in Fig. 5. The vertical lines indicate the standard deviations about the mean error values. As always, a uniform distribution is used for the prior P(α). In the left-hand plot of Fig. 5, uniform distributions are used for the priors P(μ_g) and P(μ_c), but in the right-hand plot of Fig. 5 the usual normal priors are used. The large errors that occur for values of α near zero or unity in the left-hand plot (here called “tail errors”) are a consequence of the label-switching ambiguity discussed in section 5. Note that the use of the normal priors nicely removes these large errors. Similar improvements have been found for simulations that employed other known values of (μ_g, μ_c).

View Full Size

Average ground flash fraction retrieval error as a function of the known ground flash fraction when (left) uniform priors are assumed for P(μ_g) and P(μ_c), and (right) normal priors are assumed for P(μ_g) and P(μ_c). The vertical lines indicate the standard deviation about the average errors shown. Use of the normal priors substantially reduces errors.

Citation: Journal of Atmospheric and Oceanic Technology 28, 4; 10.1175/2010JTECHA1438.1

• Download Figure
• Download figure as PowerPoint slide

c. Full error analysis

In addition to the sample size N, retrieval errors will depend on the true values of all three variables (α, μ_g, μ_c) because these define the specific form of the mixture distribution p(y). Therefore, thorough simulated retrievals were run across the full range of each of these three variables. Based on the typical variability in μ_g and μ_c found across CONUS in Koshak and Solakiewicz (2011, their Table 3), the following ranges were chosen: μ_g = 256–608 km² and μ_c = 98–208 km². The range on α was of course 0–1. As before, random draws from the known mixture distribution p(y) are used to generate N simulated measurements y.

Figure 6 shows the mean retrieval error in α across these full ranges; the sample size in this simulation is N = 2000. In each cell shown, 100 retrievals were performed; that is, 100 different simulated measurement vectors y were mathematically inverted using the Bayesian methodology for each cell. For example, in Fig. 6a, a particular cell is defined by the ranges α = 0–0.1, μ_g = 320–352 km², and μ_c = 108–118 km². This means that the known values of (α, μ_g, μ_c) were randomly selected from these respective ranges and the resulting mean retrieval error (and standard deviation) in α was 0.022 (0.015), as given in the cell.

View Full Size

The mean retrieval error (standard deviation) in α as a function of the indicated ranges of the model parameters; these plots cover the range (a) α = 0–0.2, (b) α = 0.2–0.4, α = 0.4–0.6, (d) α = 0.6–0.8, and (e) α = 0.8–1. Note: low, midlevel, and high retrieval errors are in shades of green, blue, and red, respectively.

Citation: Journal of Atmospheric and Oceanic Technology 28, 4; 10.1175/2010JTECHA1438.1

• Download Figure
• Download figure as PowerPoint slide

To help interpret the plan view of the errors, each cell is color coded according to its mean error in α; low-level errors are shades of green, midlevel errors are shades of blue, and high-level errors are shades of red. Blank (white) cells represent regions where a mean value could not be computed over all 100 retrievals because one or more retrievals of the 100 had nonconvergent solutions; that is, the Bayesian method did not provide a physically meaningful answer because of poor convergence to the minimum of −S. However, these cases were rare and the mean errors for the remaining good retrievals in the affected cell were generally similar to the mean errors in adjacent cells.

Because the actual value of α over the globe is expected to be on the low end (0 ≤ α ≤ 0.4), it is encouraging to see that the Bayesian retrieval method performs well in this range. Excessive retrieval errors are only found for large values of α when μ_g is under about 320 km². The central cell, which has a square box drawn around it for emphasis, contains the CONUS mean values (μ_g = 431.521 70 km² and μ_c = 152.949 93 km²). Because the normal priors employed use these CONUS values as population means, retrieval errors are small in this cell as expected.

It is important to note, as emphasized previously, that retrieval errors tend to decrease as the sample size of flashes N increases. Hence, the same analysis was run again, but with N = 5000. As expected, the retrieval errors generally decreased. For example, the central cell had an average reduction in ground flash retrieval error of 28.6%.

Finally, the retrievals of μ_g and μ_c presented here represent an improvement over Koshak and Solakiewicz (2011) wherein CONUS estimates of these two parameters were used; that is, the Bayesian retrieval scheme provides a comprehensive methodology for obtaining the most likely parameters (α, μ_g, μ_c) for an arbitrary region of the globe and therefore does not depend on estimates derived from CONUS observations.

9. Preliminary application on a global scale

As a preliminary application of the Bayesian retrieval method using the mixed exponential distribution model, the entire 5-yr OTD dataset has been analyzed to estimate the ground flash fraction on a global scale. The globe is divided into 4° × 4° latitude–longitude bins, and no retrieval is performed if a bin contains less than 2000 flashes. The retrieval results are provided in Fig. 7.

View Full Size

(a) The number of flashes in each 4° × 4° latitude–longitude bin. White areas indicate zero flashes. (b) The retrieval of the ground flash fraction α on a global scale using the 5-yr OTD dataset. Only 4° × 4° bins with 2000 or more flashes are analyzed. (c) The Z ratio associated with the retrieved ground flash fraction shown in (b).

Citation: Journal of Atmospheric and Oceanic Technology 28, 4; 10.1175/2010JTECHA1438.1

• Download Figure
• Download figure as PowerPoint slide

Figure 7a provides the number of flashes in each latitude/longitude bin. Note that the number of flashes in central Africa is quite large (i.e., the global maximum number of flashes in a bin occurred in this region, and had a value of 32 173 flashes). Hence, based on the simulation results in section 8, which showed that retrieval errors decrease with increasing sample size N, ground flash fraction retrievals in central Africa are expected to be quite good.

Figure 7b provides the spatial distribution of the retrieved ground flash fraction, and Fig. 7c provides the associated Z ratio [number of cloud flashes/number of ground flashes = (1 − α)/α]. One must be cautious in making comparisons of these values to others in the literature. For example, these results do not strictly account for the effects of the diurnal cycle; that is, the OTD data were not averaged over a 55-day cycle, as was done in Boccippio et al. (2001). The results shown are just a Bayesian analysis of all of the flashes that happen to occur in the particular latitude–longitude bin, no matter what time of day these flashes may have occurred. Also, the results shown are obviously not at the storm scale. However, as long as the flash sample size is maintained (i.e., roughly 2000 or more flashes, depending on the retrieval accuracy desired) the Bayesian retrieval method is fully applicable when both the space and time scales are made smaller and shorter, respectively.

For this global-scale analysis, the 4° bin size was used so that not too many bins across the globe would fall below the N = 2000 flash threshold. In fact, with the 4° bin size, there were a total of 562 bins across the globe that met this threshold. The spatial average (and standard deviation) of α across the domain analyzed was 0.151 (0.081) with a median of 0.140, and the range was 0.019–0.739. For Z, the spatial average (standard deviation) was 7.58 (5.45) with a median of 6.14, and a range of 0.353–51.6. Finally, note algebraically that , in general.

10. Summary

This paper has introduced a Bayesian inversion technique (section 3) for retrieving the optimum ground flash fraction and other statistical parameters of a set of N flashes observed by a satellite lightning imager (such as OTD, LIS, or the future GLM). The method is based on describing the observed flashes as a mixture of two distributions of a particular flash optical characteristic (section 2); one distribution describes the ground flashes and the other distribution describes the cloud flashes.

The flash optical characteristic analyzed by the Bayesian retrieval algorithm is the maximum group area (MGA); selection of this variable was motivated by Koshak (2010), who identified MGA as a useful “return stroke” detector for a large sampling of flashes. Using 5 yr of CONUS OTD flashes, it was shown that when the MGA data for ground flashes is slightly shifted (i.e., decremented by the instrument nadir footprint, ∼64 km² for OTD), the resulting “shifted MGA” variable is exponentially distributed. Similarly, the shifted MGA data for cloud flashes is also exponentially distributed. However, the sample means of each of these exponential distributions were shown here to be significantly distinct (Fig. 2). The best estimates of the CONUS population means were found to be μ_g = 431.521 70 km² and μ_c = 152.949 93 km², respectively.

Because the shifted MGA data for ground and cloud flashes are each exponentially distributed, with statistically distinct means, a mixed exponential distribution model (section 4) has been introduced to describe the statistical distribution of an arbitrary mixture of ground and cloud flashes. In total, the mixed exponential distribution model contains the following three model parameters: 1) the ground flash fraction α, 2) the population mean μ_g of the shifted MGA distribution for ground flashes, and 3) the population mean μ_c of the shifted MGA distribution for cloud flashes.

Given a set of N shifted MGA observations from an arbitrary geographical region over the globe, the Bayesian retrieval algorithm finds the optimum values of (α, μ_g, μ_c) for that region. From exhaustive numerical simulations (section 8), it is shown that reasonable retrieval accuracy is obtained if N is about 2000 or larger, and that improved retrieval accuracy can be obtained by increasing N. [For GLM application, one could consider, for example, 10 thunderclouds in a region, with each producing a nominal flash rate of 10 flashes per minute. This means that one would only need to wait 20 min (=2000 flashes per 100 flashes per minute) before applying the Bayesian algorithm.]

The importance of applying normally distributed prior distributions within the Bayesian framework as a way of minimizing the deleterious effects of the infamous label-switching problem (section 5) is demonstrated; the normal priors effectively help to minimize retrieval errors when the true value of the ground flash fraction is either near zero or unity (Fig. 5). In addition, because the Bayesian algorithm relies on a numerical maximization of a log-likelihood function to obtain the optimum model parameters, it is important to obtain good initial estimates of the model parameters to improve the chances that the parameter search will converge to the global maximum. To serve this purpose, a highly useful initialization scheme that is based on fundamental population statistic constraints is introduced and applied (see section 6).

The Bayesian retrieval algorithm (using the mixed exponential distribution model) has been applied to obtain a first estimate of the global distribution of the ground flash fraction (Fig. 7b), as well as a closely related variable, the Z ratio (number cloud flashes/number of ground flashes). As expected, the spatially averaged ground flash fraction over the analyzed global-scale domain was small (0.151), reflecting the fact that cloud flashes normally outnumber ground flashes. Some retrieved values of the ground flash fraction are notably large in a few geographic locations, and this merits detailed follow-on studies to determine the specific cause(s). Owing to the coarse (4°) spatial resolution, the transient nature of OTD view times, and biases resulting from diurnal variability in lightning, care must be taken in intercomparing these global retrievals with results from other studies. In any case, the Bayesian technique is quite versatile, and the present limitations stem more from observational limitations rather than retrieval algorithm limitations.

Moreover, the global retrievals presented here represent an important first step in the task of partitioning the global OTD/LIS total lightning climatology into separate global ground and cloud flash distributions. Because the basic physical characteristics [peak current, multiplicity (number of strokes per flash), channel altitude, channel length, and presence of continuing currents] of ground and cloud flashes differ, they produce, in general, different amounts of nitrogen oxides (NO_x = NO₂ + NO). This has been explicitly shown in a recent detailed study by Koshak et al. (2010) that employed north Alabama Lightning Mapping Array (LMA) data and the National Aeronautics and Space Administration (NASA) Marshall Space Flight Center (MSFC) Lightning Nitrogen Oxides Model (LNOM). Therefore, obtaining separate ground and cloud flash distributions will allow one to make better estimates of global lightning NO_x production for chemistry/climate model predictions. Similarly, the partitioning of flashes at the regional level will directly assist in making better estimates of regional lightning NO_x production for regional air quality models.

Note that a recent study by Ott et al. (2010) suggested NO_x production from ground and cloud flashes might be similar; it also identified other studies that suggested the same. However, it is important to note that there is still healthy debate on the issue, and further studies are needed to clarify the situation. In any case, what is certain is that the physical characteristics of lightning mentioned above are highly variable. They are so variable, in fact, that the NO_x is certain to vary not only between ground and cloud flashes, but from one ground flash to the next, or from one cloud flash to the next. Hence, without question, partitioning total lightning distributions into separate ground and cloud flash distributions is highly desirable and informative.

Finally, this effort has demonstrated that ground flash fraction retrieval is a very fascinating problem from a theoretical point of view. There exists a multitude of different ways to approach the inverse problem; section 7 on “reduced search spaces” alluded to this fact. In addition, the appendix illustrates that one can consider removing all reference to the use of any specific model (such as a mixed exponential distribution model) by simply employing the Central Limit Theorem (CLT) of statistics. The benefit of such an approach is that one does not have to assume anything about the specific form of the shifted MGA distributions. However, a disadvantage is that one has to retrieve two additional population statistical parameters (making a total of five unknowns to be retrieved). Because the entire retrieval process is inherently less constrained, it will be highly susceptible to solution nonuniqueness issues. Therefore, actively imposing normal distribution priors (or other external constraints) will be required. In this sense, the “no free lunch” principle appears to be at work. Nonetheless, testing the CLT-based approach described in the appendix would make for some interesting future work.