Historically, researchers explore the effectiveness of one lightning detection system with respect to another system; that is, the probability that system A detects a discharge given that system B detected the same discharge is estimated. Since no system detects all lightning, a more rigorous comparison should include the reverse process—that is, the probability that system B detects a discharge given that system A detected it. Further, the comparison should use the fundamental physical process detected by each system. Of particular interest is the comparison of ground-based radio frequency detectors with space-based optical detectors. Understanding these relationships is critical as the availability and use of lightning data, both ground based and space based, increases. As an example, this study uses Bayesian techniques to compare the effectiveness of the Earth Networks Total Lightning Network (ENTLN), a ground-based wideband network, and the Lightning Imaging Sensor (LIS), a space-based optical detector. This comparison is completed by matching LIS groups and ENTLN pulses, each of which correspond to stroke-type discharges. The comparison covers the period from 2009 to 2013 over several spatial domains. In 2013 LIS detected 52.0% of the discharges ENTLN reported within the LIS field of view globally and 53.2% near North America. Conversely, ENTLN detected 5.9% of the pulses detected by LIS globally and 26.9% near North America in 2013. Using these results in the Bayesian-based methodology outlined, the study finds that LIS detected 80.1% of discharges near North America in 2013, while ENTLN detected 40.1%.
In part because lightning radiates across a broad spectrum of frequencies (Cummins and Murphy 2009), there exists a wide variety of instruments and instrument arrays used to detect lightning. Each instrument detects the part of the lightning discharge that radiates in the band in which it operates. For example, very high-frequency (VHF) instruments are well suited to detect radiation associated with small-scale (~1 m) breakdown, while very low-frequency/low-frequency (VLF/LF) instruments detect radiation associated with larger-scale (~1 km) discharges. Instruments that detect radiation in the optical band respond to energetic breakdowns such as return strokes and K changes.
Some instruments are single detectors, such as the Lightning Imaging Sensor (LIS), while others, such as the National Lightning Detection Network (NLDN) and the Earth Networks Total Lightning Network (ENTLN), detect the radiation at several sensors as an array. Herein, we will refer to a lightning locating system (LLS), whether it is a single instrument or an array of instruments.
The last 20 years have seen a notable increase in the number of systems that detect lightning. Besides sensitivity in different frequency bands, LLSs also vary in how well they detect lightning and what type of lightning is detected. Lightning Mapping Arrays (LMAs) (e.g., Thomas et al. 2004; Goodman et al. 2005) that operate in the VHF band and have relatively small (~10 km) baselines have a high detection efficiency of all lightning flashes in a limited spatial domain; however, they have limited capability to directly detect return strokes and in-cloud energetic discharges such as K changes. Various VLF/LF LLSs with baselines from ~10 km such as the Huntsville Alabama Marx Meter Array (HAMMA) (Bitzer et al. 2013) and Broadband Observation Network for Lightning and Thunderstorm (BOLT) (Yoshida et al. 2014) can also detect most lightning, including identification of return strokes and in-cloud discharges, but they also have limited range for accurate three-dimensional locations. As the baseline for a VLF/LF network increases to ~100 km as for NLDN (Cummins and Murphy 2009, and references therein) and ENTLN (Liu and Heckman 2012), the spatial domain in which lightning is detected increases but the detection efficiency decreases, particularly for in-cloud discharges. Longer-range VLF/LF networks such as the World Wide Lightning Location Network (WWLLN) (e.g., Rodger et al. 2005) and the Global Lightning Dataset GLD360 (Said et al. 2013) with baselines of ~1000 km can detect lightning over much of the globe but primarily detect return strokes. Optical systems currently deployed in low-Earth orbit—for example, LIS (Christian et al. 1999)—provide high detection efficiency but limited temporal sampling and have their latitude range limited by the orbital inclination. Future missions, such as the International Space Station Lightning Imaging Sensor (ISS-LIS), will expand the latitude range but will also have limited temporal sampling. The Geostationary Lightning Mapper (GLM) (Goodman et al. 2013), scheduled for launch in 2016, will provide continuous detection of total lightning over one hemisphere.
Notably, no operational LLS detects a lightning flash. Instead, each system detects a subprocess of a flash. The detected process varies among systems and is a function of the frequency range in which the system operates. In addition, the name of the process used by each LLS can be different. For example, an LMA detects small-scale breakdown processes and names these “sources,” while LIS detects optical emission from lightning and names these “groups.” Regardless of the name used, every system has a measurement basis, that is, a fundamental measurement related to a physical process that characterizes the system. The measurement bases can be algorithmically grouped together into flashes, using spatial and temporal constraints that vary from system to system, depending on the nature of the discharge detected and variations of the detection efficiency within the system.
To assess the performance of an LLS, researchers can use a ground truth dataset, for example, using rocket-triggered lightning or high-speed video. The performance of various networks covering Europe has been documented using high-speed video (Poelman et al. 2013). The absolute detection efficiency and other attributes of NLDN have been found using both high-speed video (e.g., Biagi et al. 2007) and rocket-triggered lightning (e.g., Jerauld et al. 2005; Nag et al. 2011). Recently, rocket-triggered lightning has been used to determine performance characteristics of ENTLN (Mallick et al. 2015). However, these studies are limited in the domain in which the performance of an LLS can be assessed. Large-scale characterization of LLS performance usually uses two (or more) systems.
Because of the varying types of LLSs, there is a growing need to understand how these different systems perform relative to each other and the best way to combine different systems in order to maximize the amount of lightning detected and the physical information associated with the detection. Previous work has compared ground-based systems to other ground-based systems (e.g., Said et al. 2010; Abarca et al. 2010; Pohjola and Mäkelä 2013) and ground-based systems to space-based optical systems (e.g., Boccippio et al. 2001; Rudlosky and Shea 2013; Thompson et al. 2014). Generally speaking, these works assume the dataset of one of the LLSs is truth—that is, all lightning is detected—or assume an absolute detection efficiency for one or both systems. Then, the data from each LLS are matched using spatial and temporal constraints that are dependent on the nature of the detecting systems. For example, the detection efficiency of LIS and NLDN is assumed to estimate the ratio of intracloud to cloud-to-ground lightning over the contiguous United States (CONUS) (Boccippio et al. 2001). A comprehensive overview of intercomparisons between various LLSs can be found in Nag et al. (2015).
However, neither LLS needs to be treated as truth when comparing the performance of two systems in lieu of any ground truth data. There will be lightning discharges detected by one system but not the other. This has been considered when comparing GLD360 and NLDN (Said et al. 2010), as well as performance assessment between GLD360 and the European Cooperation for Lightning Detection (EUCLID) (Pohjola and Mäkelä 2013). However, these examples do not exploit the full information available when finding the relative performance of each LLS.
In general, the problem of detection efficiency using two LLSs can be treated in a Bayesian manner based on conditional probabilities: given that one LLS detects a discharge, the probability that the other LLS detects the discharge is found. When combined with probability theory, a rigorous assessment of the performance of each LLS can be found without resorting to any a priori assumptions on performance.
Some previous comparisons between two LLSs use the algorithmically derived entity of a flash for the comparison, not the fundamental unit of measurement of the system. If the measurement bases of two systems correspond to different physical processes, then they cannot be easily directly compared and using flashes facilitates the comparison. For example, because of the intrinsic difference in the type of discharge that is detected, there is no direct comparison between a VLF/LF LLS and a VHF LLS. Hence, it is reasonable for the comparison of these two LLSs to use a flash for the comparison (e.g., Thomas et al. 2000).
However, comparing flashes is dependent, at least in part, on the parameters of the flash sorting algorithm utilized by each LLS, which convolves the LLS comparison with algorithmic effects. Further, the grouping of the measurement basis into flashes can yield different flash counts depending on the parameters used; this effect can be pronounced for VHF systems (Murphy 2006). There is little research exploring the effect of flash sorting parameters on the resulting flash counts from VLF/LF systems.
There is no fundamental reason to compare VLF/LF and optical LLSs using flashes as the basis. Optical-based LLSs primarily detect large-scale, optically bright discharges, such as return strokes and in-cloud discharges (e.g., K changes). These processes are often, if not always, accompanied by a rapid change in the electric field (Goodman et al. 1988; Bitzer 2011). This is precisely what many ground-based VLF/LF LLSs use to detect lightning. Hence, the measurement basis of optical-based systems and ground-based VLF/LF systems largely correspond to the same physical process; these can be directly compared without resorting to artificial sorting of detected processes into flashes.
In this paper, we develop a method to compare a ground-based VLF system to a space-based optical system using the measurement basis of each. The methodology is developed in the context of two LLSs: LIS and ENTLN. Using the algorithm described herein, other LLSs can be compared in which the measurement bases correspond to similar physical processes. Further, we outline how to carry out a comparison between two lightning detection systems in a Bayesian manner. This provides a general framework by which an estimate of the relative and absolute detection efficiency can be found directly from the data. In addition, a quantitative assessment of the added value of assimilating lightning datasets together can be found.
Two recent papers (Thompson et al. 2014; Rudlosky 2015) focus on the comparison of ENTLN relative to LIS; neither considered the reverse problem, namely, how LIS performs relative to ENTLN. Further, neither uses the measurement basis of each LLS as the comparison, although LIS groups were used in each to facilitate the matching. Hence, there remains more information to be ascertained from comparing these two LLSs.
A rigorous treatment of the comparison of various lightning detection systems, in particular space-based optical LLSs and ground-based VLF LLSs, is particularly important with the launch of GLM, scheduled for 2016 (Goodman et al. 2013; Christian et al. 1989). Assessing the performance of GLM and determining the added value of assimilating ground-based lightning systems will be critical needs for the best understanding and operational use of lightning data.
a. Lightning Imaging Sensor
LIS is a staring imager on board the Tropical Rainfall Measuring Mission (TRMM) satellite that detects transient optical events from space. Its design is driven by the goal of detecting weak optical lightning pulses during daytime when the sunlight reflecting off the cloud tops can be orders of magnitude greater than the lightning events. LIS detects lightning-generated optical emissions from the atomic oxygen line at 777.4 nm. It does so measuring the number of photons in a particular time period, known as the frame integration time. LIS uses 559 frames per second (Bitzer and Christian 2015).
LIS uses a 128 × 128 pixel array to provide a field of view on the ground of about 600 km × 600 km; each pixel has a footprint of 4.5 km. Because TRMM is in low-Earth orbit, LIS views a specific point on the ground for approximately 90 s. Previous ground-based comparisons suggest that LIS has a total flash detection efficiency of about 70% during times of peak cloud-top brightness and about 88% at nighttime (Boccippio et al. 2002, and references therein).
When the optical emission in a single pixel is above a particular threshold, LIS classifies this as an event. Events in a single frame that are in adjacent pixels are classified as a group, which roughly corresponds to a lightning stroke [intracloud (IC) or cloud to ground (CG)]. This is considered to be the fundamental measurement made by LIS, that is, the measurement basis. The LIS group location that is reported is the radiance-weighted centroid of the constituent events. Groups that occur within 330 ms and 5 km are classified as belonging to the same flash (Christian et al. 2000; Mach et al. 2007).
Besides the basic lightning data of events, groups, and flashes, LIS data also contain pertinent information from the spacecraft concerning location, telemetry issues, platform issues, etc. This housekeeping data are received every second; hence, it is referred to as 1-s data. This portion of the LIS data record is not typically used in lightning studies but contains several important attributes.
First, the field of view (FOV) of LIS can be determined by using 1-s data to find the “look vector” for each LIS pixel. The look vector defines the orientation, relative to the instrument, of each pixel to Earth. This is constant for a particular pixel throughout time. Next, the position vector of the spacecraft and transformation matrix are found from the 1-s data. The position vector defines the position of the spacecraft, while the transformation matrix defines the rotations necessary to get from the spacecraft reference frame to the instrument reference frame. These are applied to the pixels that are on the edge of the charge-coupled device (CCD) to define the LIS FOV at each second. (It should be noted that code to carry out the procedures discussed herein is provided by the LIS science team; however, we have written our own updated and optimized algorithm, based on the provided code.)
Further, the 1-s data contain fields that provide information concerning the quality of the data. These are referred to as alert flags and are associated with the instrument, the platform (spacecraft), or an external factor. Alerts can be classified as fatal or warning; a fatal alert indicates that no data are obtained in the 1-s time period the flag is high. For example, a fatal alert occurs if the spacecraft fails to send valid position data. A warning flag indicates that some of the data in the 1-s time period may be compromised. For example, the external warning flag is high when the satellite is within the South Atlantic anomaly. In addition, the instrument warning flag is high for all times since the TRMM boost in August 2003 and is ignored for this analysis.
b. Earth Networks Total Lightning Network
More information about ENTLN can be found in two conference papers (Liu and Heckman 2010, 2012), but relevant information is summarized here. ENTLN is an array of electric field change meters that detect radiation from lightning in the VLF/LF band. This system primarily uses the radiation component of the electric field, proportional to the time derivative of the current, to detect discharges. The measurement basis for ENTLN is a pulse that corresponds to a return stroke or an in-cloud discharge, for example, a K change. Groups of pulses are classified as a flash if they are within 700 ms and 10 km. In the United States, the baseline between sensors varies primarily between 100 and 200 km, with denser spacing east of the Mississippi River (S. Heckman 2014, personal communication). The archive for ENTLN pulses goes back to February 2009; however, the system has been continually upgraded since then.
Instead of proceeding in the usual manner in which one LLS is assumed to detect all discharges (or an absolute detection efficiency is assumed) to compare systems, we instead cast the comparison in Bayesian terms: find the probability that one LLS detects a lightning discharge, given that another LLS detected the discharge; to do so we review elements of probability theory and apply them to the problem at hand.
Let the set of all lightning discharges be S. If we denote the set of discharges detected by one LLS as A and discharges detected by the other as B, then and are the unconditional probabilities (i.e., absolute detection efficiency) of each system. From a frequentist standpoint, if the number of lightning discharges N that comprise S is known, then can be determined:
where is the number of discharges detected by system A. A similar expression holds for .
In general, N is not known; hence, the absolute detection efficiency of either LLS cannot be found directly. Strictly speaking, previous research that compares lightning detection systems finds the conditional probability; for example, given that system A detects a discharge, the probability that system B detects the discharge is found. Then the absolute probability of system B, , is estimated using , where ρ is the assumed absolute detection efficiency of system A. In the case , the assumption implies system A detects all discharges; that is, one dataset constitutes S. Under this assumption, the conditional probability is directly related to the absolute probability.
In practice, the assumption is not true. Instead, a more complete assessment of the two LLSs determines the conditional probability that system B detects a discharge given that system A detects a discharge, , and the conditional probability that LLS A detects a discharge given that LLS B detects the discharge, . The latter can be loosely referred to as the reverse probability and is generally not calculated. However, this value is easily ascertained directly from the data.
The conditional probabilities are related to the absolute (or unconditional) by Bayes’s theorem:
Hence, calculation of both conditional probabilities leads directly to a relative detection efficiency .
The problem remains how to identify if two systems detect the same discharge. This is usually done by using various temporal and spatial constraints to identify discharges detected by both LLSs (e.g., Boccippio et al. 2001). Since previous comparisons use flash data as the basis, these works generally use large constraints. Because we use stroke-level data (i.e., LIS groups and ENTLN pulses), we are able to use much more stringent spatial and temporal characteristics to match discharges.
Specifically, we consider the two LLSs to have detected the same discharge if they are coincident to 10 ms and 20 km. The temporal constraint is motivated in part from the physical process that produces a discharge; the light associated with a discharge should occur very close in time to VLF/LF radiation. The spatial constraint is also physically motivated, but additional effects for system performance are accounted for. For example, the LIS group location, a radiance-weighted centroid, may not be exactly the same as the VLF-derived location; that is, the brightest part of the discharge as viewed from space may not be the same as the strike location. However, we consider a range of values in time and space and explore the influence of this choice on the results. Locations and distances in this work use the World Geodetic System 1984 (WGS-84) ellipsoid.
We first start with the ENTLN dataset and find the LIS group associated with each pulse. We then do the reverse problem: start with each LIS group and find the associated ENTLN pulse. We do this for data starting in February 2009 (the beginning of the ENTLN data archive) through the end of 2013. Hence, we can analyze the evolution of the performance of each LLS over the time period.
Since LIS is on board a low-Earth-orbiting satellite, some preprocessing of the data is necessary to ensure a rigorous comparison. In particular, only ENTLN pulses in the LIS field of view are considered in the comparison. This is done by utilizing the 1-s data associated with LIS. Further, we do not consider ENTLN data that occur when an LIS alert flag is fatal, which occurs for 4.7% of ENTLN pulses. This is because we are generally concerned with the general capability of space-based optical measurements and not necessarily the operational performance of the two systems. It is worth noting that including time periods when the LIS alert flag is fatal minimally affects the final results; the probabilities are within a couple percentage points. In addition, we used only the operational processing algorithm for LIS, which may miss weak events. It should be noted that no such quality control is provided with ENTLN data, so the ENTLN dataset is taken as is.
As an example of the matching algorithm, consider Fig. 1. This is a CG flash, with eight return strokes identified by HAMMA (Bitzer et al. 2013) (the black squares along the bottom of the time–altitude plot). There are 14 LIS groups and five ENTLN pulses. The LIS groups, located by the radiance-weighted centroid of the constituent LIS events, are 9.5 km due north of the ENTLN pulse locations. For completeness, NLDN (not shown) detects the first six return strokes and are located within 500–1400 m of the ENTLN pulses.
There are LIS groups directly associated with the first six return strokes. A seventh stroke, at ms, was not directly detected by LIS. However, approximately 14 ms after this stroke LIS detects a group, coincident with a HAMMA-detected in-cloud discharge. Another in-cloud discharge is located by HAMMA 11 ms later, which also produces an LIS group. ENTLN detects the first five return strokes, although the return stroke at ms is misidentified as an intracloud discharge. In addition, LIS detects an in-cloud discharge at ms (also detected by HAMMA) that is not detected by ENTLN. Neither LIS nor ENTLN detects the eighth return stroke at ms.
Given an ENTLN pulse, corresponding to a return stroke or in-cloud discharge, we identify any LIS groups that occur within the spatial and temporal constraints. For this particular example, there is an LIS group associated with all five ENTLN discharges. An LIS group can be matched only to a single ENTLN pulse, as each pulse should produce independent optical emission. This provides a straightforward manner in which to match an LIS group to an ENTLN pulse.
Given an LIS group, the identification of the corresponding ENTLN pulse is more complicated because, strictly speaking, each group may not be a separate discharge. The light pulse from a single return stroke can be split across frames, resulting in two groups, yet one physical discharge. This occurs for the return strokes at ms and ms in Fig. 1. A similar manifestation of this phenomenon can be found in Østgaard et al. (2013, their Fig. 3; and related discussion). Since the physical process would result in a single ENTLN pulse, we allow an ENTLN pulse to be associated with every LIS group that satisfies the temporal and spatial constraints. The identification of a light pulse that is split across frames requires detailed millisecond-scale analysis and is not practical for the research presented herein. Hence, this methodology tends to overestimate the relative probability that ENTLN detects a discharge, as our analysis indicates most discharges result in a single group. In this example, there is an ENTLN pulse associated with 8 of the 14 LIS groups.
In this manner, we ensure the same discharge is compared, despite slight differences in the LLSs. It should be noted that we do not account for false alarms from either LLS. It is expected that this has an insignificant effect, given the false alarm rate for LIS is ~5%. The false alarm rate for ENTLN is unknown, but it is expected to be on the same order or smaller as that of LIS.
To assess the relative performance of the detection of lightning between a ground-based VLF LLS and a space-based optical sensor LLS, we match LIS groups and ENTLN pulses in the manner discussed. We then analyzed several attributes of the matched and unmatched discharges. We discuss the results for the entire dataset and explore the relative performance for the year 2013 in more depth. Further, we explore variations in the relative performance as a function of spatial domain. We use three domains: global, the Western Hemisphere, and North America. The global domain spans a latitude range of and a longitude range . The Western Hemisphere domain spans the same latitude range and a longitude range of and roughly corresponds to the expected field of view of GLM. The North American domain spans a latitude range of and a longitude range , and roughly corresponds to the region of the globe in which ENTLN has the highest sensor density (Liu and Heckman 2012). The latitude range of each domain is ultimately limited by the LIS field of view.
a. Relative probability
The probability that LIS detects a discharge given that ENTLN detected the discharge, , is 0.528 globally. There is little spatial variation in this quantity (Fig. 2). However, one notable feature in Fig. 2 is the reduced value of for latitudes greater than 37° in CONUS. This latitude is at the very edge of the FOV of LIS. It is expected that the LIS detection efficiency is not as high here because of the reduced optical throughput near the lens edge. This limitation is due to the orbit inclination of the TRMM satellite and the wide-angle lens used for LIS; this artifact is not expected to be present for a system such as GLM. The field of view for GLM is designed to span nearly the full disk, and GLM uses a lens with a smaller half angle. For completeness, Fig. 3 shows the spatial variation for the Western Hemisphere.
There is very little change in over the years (Table 1). Further, the spatial pattern is similar for each year (not shown). The variation in for North America for the year 2013 is shown in Fig. 4. In addition, there is very little variation in in the three domains in the year 2013 (Table 2); the other years exhibit the same consistency across the domains.
Next, we consider the reverse process—the conditional probability that ENTLN detects a discharge given that LIS detected the discharge, . Globally, ENTLN detects 3.7% of LIS groups. ENTLN detects more discharges near North America, where ENTLN sensor density is greatest (Figs. 5 and 6).
As discussed, an ENTLN pulse may be matched to more than one LIS group in the reverse process. For the year 2013, of the 15 405 568 LIS groups (5.9%) that were detected globally 908 271 were matched to an ENTLN pulse. Of the 908 271 matches, 435 813 (48.0%) are unique ENTLN pulses. For the unique ENTLN pulses, 215 487 (49.4%) were matched to one LIS group. Another 108 479 (24.9%) are matched to two LIS groups, and 48 230 (11.1%) are matched to three LIS groups. Since typically the light from a discharge would be split over a maximum of two frames, instances in which ENTLN matches to three or more LIS groups suggests multiple discharges. Hence, our methodology overestimates the probability that ENTLN detects a discharge, given LIS detects the discharge.
The conditional probability has generally increased over the years (Table 1). For all three domains, the relative detection efficiency of ENTLN has increased since first becoming operational, except for a slight decrease in 2011. This is spatially variant; there is a ~27% probability that ENTLN detects a discharge (given that LIS did) near North America, while it is ~6% for global discharges (Table 2). While these results are for 2013, a similar trend holds for all years. To better explore the spatial variation in the conditional probability, Fig. 7 provides the variation in for the North American domain. Again, ENTLN system performance is best for higher sensor densities. Curiously, ENTLN performance is relatively constant extending out over the Atlantic Ocean and Gulf of Mexico. Even in this region of increased performance, the conditional probability that ENTLN detects a discharge given an LIS detection lags behind the converse.
b. Timing differences
To further analyze the differences in the two LLSs, we also examine the timing differences between discharges detected by both systems. The distribution of the time difference for LIS groups given an ENTLN pulse () is sharply peaked, with a mean of −1.7 ms (Fig. 8). The full statistics for the year 2013 are given in Table 3. An ENTLN pulse occurs first, on average, but the mean and median difference are on the order of the LIS frame integration time. Other years exhibit similar differences; further, there is very little variation across the three domains on average. There is a larger spread in the time differences for ENTLN pulses, given LIS detects a discharge (Table 3). This is due to the manner in which we assign an ENTLN pulse to an LIS group; since an ENTLN pulse can match to more than one LIS group, there is larger spread in the data. However, the mean and median are not significantly different by any reasonable statistical test. It should be noted that we do not apply the timing correction to the LIS data discussed in Bitzer and Christian (2015). The correction would tend to bring the mean and median closer to zero but does not otherwise significantly impact the results herein.
c. Distance differences
We also find the differences in the location of discharges detected by both systems. Although the brightest part of the discharge (given by the LIS group location) may not be the same as the strike point (given by the ENTLN pulse), the mean difference in the locations is ~8 km (Table 4). This is roughly twice the footprint of an LIS pixel. This offset is consistent with other results comparing LIS to ground-based systems (e.g., Thomas et al. 2000, and discussion therein). The distribution of the differences is shown in Fig. 9.
Unlike for the timing difference, the distance difference is smaller over the North American domain. The mean and median are ~400 m smaller; the standard deviation is ~300 m smaller. Despite the differences in how the locations are determined by each system, this suggests, on average, the locations returned by ENTLN and LIS are more similar where the ENTLN sensor density is highest.
The distance difference for each conditional comparison is the same, unlike the timing difference. This indicates that when an ENTLN pulse is matched to more than one LIS group, the locations are similar but are relatively separated in time. This further implies that the phenomenon of multiple groups for a single discharge is rare. Hence, when a single ENTLN pulse is matched to more than one LIS group, the multiple LIS groups are separate discharges on average. This further supports the claim that our method of finding matches tends to overestimate the number of matches for ENTLN, given LIS detects a discharge.
d. LIS performance
To explore the limitations of a space-based optical sensor, we also investigate how the conditional probability varies relative to the CCD. For LIS groups that detect a given ENTLN discharge, we use the reported CCD location of the LIS group. For ENTLN pulses that do not have an associated LIS group, we find the footprint of all pixels in the CCD and determine which pixel should have been associated with the pulse. We then take the simple ratio of all the pulses in each pixel and the pulses that have an associated group.
This ratio is largely constant over the CCD (Fig. 10). However, the ratio for pixels near the edge of the array is lower than in the center. This effect is especially pronounced for pixel locations greater than 120. Because a wide-angle lens (~45°) is used by LIS, the edge of the lens suffers from distortion and reduced throughput. This reduces sensitivity at the edge of the CCD. Indeed, the relative detection efficiency for the entire CCD is 0.520; this roughly linearly increases as we exclude more edge pixels. When the outer 10 pixels are excluded, the relative detection efficiency is 0.574. We do not find any variation of detection efficiency in the reverse case—for the pixels in which an LIS group occurs, there is no significant variation that ENTLN detects the group (except for the single-edge pixel of the CCD).
e. Effect of matching constraints
We explored the impact of the choice of the spatial and temporal matching constraints had on our results. As might be expected, smaller constraints resulted in fewer matches. However, the effect of restricting the spatial constraints was minimal. For 10 ms and 20 km, and for the year 2013 in North America; using 15 km (and keeping 10 ms) resulted in and . Increasing the distance to 25 km increased the relative probabilities by similar nominal amounts.
When the time constraint is varied, changes only by two or fewer percentage points. However, there is a larger effect on : for 5 ms and 20 km, for the year 2013 in North America; for the same year and spatial domain, for 15 ms and 20 km.
This larger effect is due to the multiple matches we allow when matching ENTLN pulses to LIS groups. When increasing the temporal constraint, more matches occur. However, this also increases the chance ENTLN pulses are matched to LIS groups that are not physically related. Despite this, the similarity of the probabilities with various constraints suggests that the results reported herein are robust. For the following discussion, we will use the results for 10 ms and 20 km.
The results indicate that in a relative manner, LIS detects more discharges than ENTLN. While this is a meaningful result, the question remains: how do these two systems compare in a quantitative manner? Further, what does this mean about either system’s ability to detect all lightning discharges? Finally, what is the quantitative benefit in assimilating these two systems?
To further explore the use of conditional probabilities in the context of our methodology, we can use two well-known results from probability theory. The probability one or both systems detect a lightning discharge is given by the additive law of probability:
The probability that both LLSs detect a discharge can be determined with the multiplicative law:
Finally, applying (2) and rearranging,
Similarly, if we start with Eq. (5) and factor out instead, we arrive at the complementary relationship:
Equations (7) and (8) yield a robust, quantitative manner in which to compare the performance of two lightning detection systems. These two equations yield the relative probability . Using the results discussed herein, these provide a framework to quantitatively compare the two systems without assuming an estimate for the absolute detection efficiencies.
Table 5 summarizes the relative probabilities for the entire dataset and for the year 2013. Globally, the absolute detection efficiency of LIS is a factor of 14 larger than the absolute detection efficiency of ENTLN for all years and is a factor of almost 9 larger for the year 2013. In North America (where the sensor density is highest for ENTLN), the absolute detection efficiency of LIS is a factor of almost 3 greater than that of ENTLN for all years. While ENTLN performance has improved relative to LIS, the absolute detection efficiency is still a factor of 2 smaller than that of LIS in the year 2013.
These results can be extended further by assuming . Then, Eqs. (7) and (8) can be solved for and , respectively. In this manner, we can find the probability that a discharge will be detected by one LLS or the other. This assumption yields an estimate of the absolute detection efficiency of the set of discharges detected by either (or both) LLSs. One interpretation of the assumption is that the combined dataset fully comprises the set S. Under this assumption, and provide an upper limit on the absolute detection efficiency of each system, since it assumes all lightning is detected.
If we consider the global dataset for all years, then of the discharges detected by one or both LLSs, ENTLN detects 6.8%, while LIS detects 96.8% (Table 6). Hence, ENTLN detects a small percentage of global lightning discharges.
However, the number of discharges detected by ENTLN has improved over the years and is spatially dependent. If we consider just the year 2013, then ENTLN detects 10.8% of global discharges in the LIS field of view and LIS detects 94.8%. In the Western Hemisphere domain, ENTLN detects 22.7% of the discharges, while LIS detects 89.1%. Finally, in the North American domain, where the ENTLN sensor density is highest, ENTLN detects 40.9% of all discharges, and LIS detects 80.9%. Hence, of the discharges detected by either or both LIS and ENTLN, twice as many are detected by LIS.
These results suggest care should be taken when discussing a comparison of two datasets. The commonly used assumption that one dataset comprises the entire set of lightning discharges is often invalid. This is particularly true if one system is more likely to detect a particular type of discharge (e.g., cloud-to-ground strokes). Although this assumption is usually acknowledged by assuming an absolute detection efficiency, the implications are rarely discussed. If this assumption was used in the analysis herein, then the conclusion would be the upper limit of the detection efficiency of ENTLN is 26.9%, while a more rigorous analysis indicates this value is 40.9%.
Further, this analysis suggests that as the ENTLN LLS has evolved and detects more discharges, these discharges are not necessarily detected by LIS. Using Bayes’s theorem [(2)] and noting is relatively constant, if we assume that absolute LIS performance—that is, —is constant, then the increase in the conditional probability in time can be attributed to an increase in in time.
We can use these results to estimate the added benefit when assimilating lightning data from multiple systems. As an example, consider the data from 2013 in North America. Of the lightning detected by either LIS or ENTLN (or both), LIS detects 80.9% of all discharges. Assimilating ENTLN data, accounting for simultaneous discharges, adds 23.6% more discharges detected. Conversely, assimilating LIS data into ENTLN data yields 96.2% more discharges detected. Hence, the methodology provides a manner in which to quantitatively assess the added benefit when combining lightning datasets.
In this paper, we introduce a new methodology in which two lightning locating systems (LLSs) are compared by using Bayesian principles. We are able to quantitively assess the performance of two LLSs without resorting to assumptions of the absolute detection efficiency of either system. In addition, this method provides an upper limit on the absolute detection efficiency.
Further, we compare a space-based optical LLS and ground-based VLF LLS in terms of the common physical process detected by each. Each system primarily detects return strokes and in-cloud discharges, and this forms the measurement basis. We do not compare flashes, which are an algorithmically derived entity. This allows for direct comparison of the performance of each LLS by deconvolving any biases introduced by a flash sorting algorithm.
This methodology is applied to two systems: the Lightning Imaging Sensor (LIS) and the Earth Networks Total Lightning Network (ENTLN). We find that LIS detects ~52% of ENTLN pulses, and this is relatively invariant in space and time. Conversely, ENTLN detects ~4% of LIS groups globally. In 2013, ENTLN detects ~6% of LIS groups globally and 27% of LIS groups near North America. ENTLN performance relative to LIS has increased, yet LIS still detects more lightning discharges. By casting the comparison as a Bayesian problem, we also find that, of all the discharges detected by either LIS or ENTLN, LIS detects 81% near North America in 2013, while ENTLN detects 41%.
There is great interest in how to best assimilate various LLSs to maximize the amount of lightning detected. With the impending launch of GLM, there is a particular desire to find how much information is added by ground-based systems to space-based optical measurements. We quantitatively assess the added benefit of assimilating ENTLN into LIS data and find this adds 23.6% more discharges. Conversely, assimilating LIS data into ENTLN yields 96.2% more discharges.
This work can also be extended so that one-third of LLSs could be compared. Intercomparisons of each LLS should be made, following the probabilistic methods outlined herein, to yield the conditional probabilities between each system. Then, compare each dataset to the union set of the other two. In this manner, a more complete set S will be formed, and hence a more robust estimate for the absolute detection efficiency of several LLSs can be found. Further, this provides a manner in which a researcher can more rigorously assess the added contribution of assimilating different lightning datasets by analyzing the various conditional probabilities.
This study was partially supported by NOAA Grant Z7813005 [Cooperative Institute for Climate and Satellites (CICS) at the University of Maryland (ESSIC) as part of GOES-R Risk Reduction Research] and NASA Grant NNM05AA22A. The Lightning Imaging Sensor (LIS) science data were obtained from the NASA EOSDIS Global Hydrology Resource Center (GHRC) DAAC, Huntsville, Alabama (http://lightning.nsstc.nasa.gov/). ENTLN data were provided by Stan Heckman and Christopher Sloop.