1. Introduction
In part because lightning radiates across a broad spectrum of frequencies (Cummins and Murphy 2009), there exists a wide variety of instruments and instrument arrays used to detect lightning. Each instrument detects the part of the lightning discharge that radiates in the band in which it operates. For example, very high-frequency (VHF) instruments are well suited to detect radiation associated with small-scale (~1 m) breakdown, while very low-frequency/low-frequency (VLF/LF) instruments detect radiation associated with larger-scale (~1 km) discharges. Instruments that detect radiation in the optical band respond to energetic breakdowns such as return strokes and K changes.
Some instruments are single detectors, such as the Lightning Imaging Sensor (LIS), while others, such as the National Lightning Detection Network (NLDN) and the Earth Networks Total Lightning Network (ENTLN), detect the radiation at several sensors as an array. Herein, we will refer to a lightning locating system (LLS), whether it is a single instrument or an array of instruments.
The last 20 years have seen a notable increase in the number of systems that detect lightning. Besides sensitivity in different frequency bands, LLSs also vary in how well they detect lightning and what type of lightning is detected. Lightning Mapping Arrays (LMAs) (e.g., Thomas et al. 2004; Goodman et al. 2005) that operate in the VHF band and have relatively small (~10 km) baselines have a high detection efficiency of all lightning flashes in a limited spatial domain; however, they have limited capability to directly detect return strokes and in-cloud energetic discharges such as K changes. Various VLF/LF LLSs with baselines from ~10 km such as the Huntsville Alabama Marx Meter Array (HAMMA) (Bitzer et al. 2013) and Broadband Observation Network for Lightning and Thunderstorm (BOLT) (Yoshida et al. 2014) can also detect most lightning, including identification of return strokes and in-cloud discharges, but they also have limited range for accurate three-dimensional locations. As the baseline for a VLF/LF network increases to ~100 km as for NLDN (Cummins and Murphy 2009, and references therein) and ENTLN (Liu and Heckman 2012), the spatial domain in which lightning is detected increases but the detection efficiency decreases, particularly for in-cloud discharges. Longer-range VLF/LF networks such as the World Wide Lightning Location Network (WWLLN) (e.g., Rodger et al. 2005) and the Global Lightning Dataset GLD360 (Said et al. 2013) with baselines of ~1000 km can detect lightning over much of the globe but primarily detect return strokes. Optical systems currently deployed in low-Earth orbit—for example, LIS (Christian et al. 1999)—provide high detection efficiency but limited temporal sampling and have their latitude range limited by the orbital inclination. Future missions, such as the International Space Station Lightning Imaging Sensor (ISS-LIS), will expand the latitude range but will also have limited temporal sampling. The Geostationary Lightning Mapper (GLM) (Goodman et al. 2013), scheduled for launch in 2016, will provide continuous detection of total lightning over one hemisphere.
Notably, no operational LLS detects a lightning flash. Instead, each system detects a subprocess of a flash. The detected process varies among systems and is a function of the frequency range in which the system operates. In addition, the name of the process used by each LLS can be different. For example, an LMA detects small-scale breakdown processes and names these “sources,” while LIS detects optical emission from lightning and names these “groups.” Regardless of the name used, every system has a measurement basis, that is, a fundamental measurement related to a physical process that characterizes the system. The measurement bases can be algorithmically grouped together into flashes, using spatial and temporal constraints that vary from system to system, depending on the nature of the discharge detected and variations of the detection efficiency within the system.
To assess the performance of an LLS, researchers can use a ground truth dataset, for example, using rocket-triggered lightning or high-speed video. The performance of various networks covering Europe has been documented using high-speed video (Poelman et al. 2013). The absolute detection efficiency and other attributes of NLDN have been found using both high-speed video (e.g., Biagi et al. 2007) and rocket-triggered lightning (e.g., Jerauld et al. 2005; Nag et al. 2011). Recently, rocket-triggered lightning has been used to determine performance characteristics of ENTLN (Mallick et al. 2015). However, these studies are limited in the domain in which the performance of an LLS can be assessed. Large-scale characterization of LLS performance usually uses two (or more) systems.
Because of the varying types of LLSs, there is a growing need to understand how these different systems perform relative to each other and the best way to combine different systems in order to maximize the amount of lightning detected and the physical information associated with the detection. Previous work has compared ground-based systems to other ground-based systems (e.g., Said et al. 2010; Abarca et al. 2010; Pohjola and Mäkelä 2013) and ground-based systems to space-based optical systems (e.g., Boccippio et al. 2001; Rudlosky and Shea 2013; Thompson et al. 2014). Generally speaking, these works assume the dataset of one of the LLSs is truth—that is, all lightning is detected—or assume an absolute detection efficiency for one or both systems. Then, the data from each LLS are matched using spatial and temporal constraints that are dependent on the nature of the detecting systems. For example, the detection efficiency of LIS and NLDN is assumed to estimate the ratio of intracloud to cloud-to-ground lightning over the contiguous United States (CONUS) (Boccippio et al. 2001). A comprehensive overview of intercomparisons between various LLSs can be found in Nag et al. (2015).
However, neither LLS needs to be treated as truth when comparing the performance of two systems in lieu of any ground truth data. There will be lightning discharges detected by one system but not the other. This has been considered when comparing GLD360 and NLDN (Said et al. 2010), as well as performance assessment between GLD360 and the European Cooperation for Lightning Detection (EUCLID) (Pohjola and Mäkelä 2013). However, these examples do not exploit the full information available when finding the relative performance of each LLS.
In general, the problem of detection efficiency using two LLSs can be treated in a Bayesian manner based on conditional probabilities: given that one LLS detects a discharge, the probability that the other LLS detects the discharge is found. When combined with probability theory, a rigorous assessment of the performance of each LLS can be found without resorting to any a priori assumptions on performance.
Some previous comparisons between two LLSs use the algorithmically derived entity of a flash for the comparison, not the fundamental unit of measurement of the system. If the measurement bases of two systems correspond to different physical processes, then they cannot be easily directly compared and using flashes facilitates the comparison. For example, because of the intrinsic difference in the type of discharge that is detected, there is no direct comparison between a VLF/LF LLS and a VHF LLS. Hence, it is reasonable for the comparison of these two LLSs to use a flash for the comparison (e.g., Thomas et al. 2000).
However, comparing flashes is dependent, at least in part, on the parameters of the flash sorting algorithm utilized by each LLS, which convolves the LLS comparison with algorithmic effects. Further, the grouping of the measurement basis into flashes can yield different flash counts depending on the parameters used; this effect can be pronounced for VHF systems (Murphy 2006). There is little research exploring the effect of flash sorting parameters on the resulting flash counts from VLF/LF systems.
There is no fundamental reason to compare VLF/LF and optical LLSs using flashes as the basis. Optical-based LLSs primarily detect large-scale, optically bright discharges, such as return strokes and in-cloud discharges (e.g., K changes). These processes are often, if not always, accompanied by a rapid change in the electric field (Goodman et al. 1988; Bitzer 2011). This is precisely what many ground-based VLF/LF LLSs use to detect lightning. Hence, the measurement basis of optical-based systems and ground-based VLF/LF systems largely correspond to the same physical process; these can be directly compared without resorting to artificial sorting of detected processes into flashes.
In this paper, we develop a method to compare a ground-based VLF system to a space-based optical system using the measurement basis of each. The methodology is developed in the context of two LLSs: LIS and ENTLN. Using the algorithm described herein, other LLSs can be compared in which the measurement bases correspond to similar physical processes. Further, we outline how to carry out a comparison between two lightning detection systems in a Bayesian manner. This provides a general framework by which an estimate of the relative and absolute detection efficiency can be found directly from the data. In addition, a quantitative assessment of the added value of assimilating lightning datasets together can be found.
Two recent papers (Thompson et al. 2014; Rudlosky 2015) focus on the comparison of ENTLN relative to LIS; neither considered the reverse problem, namely, how LIS performs relative to ENTLN. Further, neither uses the measurement basis of each LLS as the comparison, although LIS groups were used in each to facilitate the matching. Hence, there remains more information to be ascertained from comparing these two LLSs.
A rigorous treatment of the comparison of various lightning detection systems, in particular space-based optical LLSs and ground-based VLF LLSs, is particularly important with the launch of GLM, scheduled for 2016 (Goodman et al. 2013; Christian et al. 1989). Assessing the performance of GLM and determining the added value of assimilating ground-based lightning systems will be critical needs for the best understanding and operational use of lightning data.
2. Instrumentation
a. Lightning Imaging Sensor
LIS is a staring imager on board the Tropical Rainfall Measuring Mission (TRMM) satellite that detects transient optical events from space. Its design is driven by the goal of detecting weak optical lightning pulses during daytime when the sunlight reflecting off the cloud tops can be orders of magnitude greater than the lightning events. LIS detects lightning-generated optical emissions from the atomic oxygen line at 777.4 nm. It does so measuring the number of photons in a particular time period, known as the frame integration time. LIS uses 559 frames per second (Bitzer and Christian 2015).
LIS uses a 128 × 128 pixel array to provide a field of view on the ground of about 600 km × 600 km; each pixel has a footprint of 4.5 km. Because TRMM is in low-Earth orbit, LIS views a specific point on the ground for approximately 90 s. Previous ground-based comparisons suggest that LIS has a total flash detection efficiency of about 70% during times of peak cloud-top brightness and about 88% at nighttime (Boccippio et al. 2002, and references therein).
When the optical emission in a single pixel is above a particular threshold, LIS classifies this as an event. Events in a single frame that are in adjacent pixels are classified as a group, which roughly corresponds to a lightning stroke [intracloud (IC) or cloud to ground (CG)]. This is considered to be the fundamental measurement made by LIS, that is, the measurement basis. The LIS group location that is reported is the radiance-weighted centroid of the constituent events. Groups that occur within 330 ms and 5 km are classified as belonging to the same flash (Christian et al. 2000; Mach et al. 2007).
Besides the basic lightning data of events, groups, and flashes, LIS data also contain pertinent information from the spacecraft concerning location, telemetry issues, platform issues, etc. This housekeeping data are received every second; hence, it is referred to as 1-s data. This portion of the LIS data record is not typically used in lightning studies but contains several important attributes.
First, the field of view (FOV) of LIS can be determined by using 1-s data to find the “look vector” for each LIS pixel. The look vector defines the orientation, relative to the instrument, of each pixel to Earth. This is constant for a particular pixel throughout time. Next, the position vector of the spacecraft and transformation matrix are found from the 1-s data. The position vector defines the position of the spacecraft, while the transformation matrix defines the rotations necessary to get from the spacecraft reference frame to the instrument reference frame. These are applied to the pixels that are on the edge of the charge-coupled device (CCD) to define the LIS FOV at each second. (It should be noted that code to carry out the procedures discussed herein is provided by the LIS science team; however, we have written our own updated and optimized algorithm, based on the provided code.)
Further, the 1-s data contain fields that provide information concerning the quality of the data. These are referred to as alert flags and are associated with the instrument, the platform (spacecraft), or an external factor. Alerts can be classified as fatal or warning; a fatal alert indicates that no data are obtained in the 1-s time period the flag is high. For example, a fatal alert occurs if the spacecraft fails to send valid position data. A warning flag indicates that some of the data in the 1-s time period may be compromised. For example, the external warning flag is high when the satellite is within the South Atlantic anomaly. In addition, the instrument warning flag is high for all times since the TRMM boost in August 2003 and is ignored for this analysis.
b. Earth Networks Total Lightning Network
More information about ENTLN can be found in two conference papers (Liu and Heckman 2010, 2012), but relevant information is summarized here. ENTLN is an array of electric field change meters that detect radiation from lightning in the VLF/LF band. This system primarily uses the radiation component of the electric field, proportional to the time derivative of the current, to detect discharges. The measurement basis for ENTLN is a pulse that corresponds to a return stroke or an in-cloud discharge, for example, a K change. Groups of pulses are classified as a flash if they are within 700 ms and 10 km. In the United States, the baseline between sensors varies primarily between 100 and 200 km, with denser spacing east of the Mississippi River (S. Heckman 2014, personal communication). The archive for ENTLN pulses goes back to February 2009; however, the system has been continually upgraded since then.
3. Methodology
Instead of proceeding in the usual manner in which one LLS is assumed to detect all discharges (or an absolute detection efficiency is assumed) to compare systems, we instead cast the comparison in Bayesian terms: find the probability that one LLS detects a lightning discharge, given that another LLS detected the discharge; to do so we review elements of probability theory and apply them to the problem at hand.





In general, N is not known; hence, the absolute detection efficiency of either LLS cannot be found directly. Strictly speaking, previous research that compares lightning detection systems finds the conditional probability; for example, given that system A detects a discharge, the probability that system B detects the discharge is found. Then the absolute probability of system B,
In practice, the assumption

The problem remains how to identify if two systems detect the same discharge. This is usually done by using various temporal and spatial constraints to identify discharges detected by both LLSs (e.g., Boccippio et al. 2001). Since previous comparisons use flash data as the basis, these works generally use large constraints. Because we use stroke-level data (i.e., LIS groups and ENTLN pulses), we are able to use much more stringent spatial and temporal characteristics to match discharges.
Specifically, we consider the two LLSs to have detected the same discharge if they are coincident to 10 ms and 20 km. The temporal constraint is motivated in part from the physical process that produces a discharge; the light associated with a discharge should occur very close in time to VLF/LF radiation. The spatial constraint is also physically motivated, but additional effects for system performance are accounted for. For example, the LIS group location, a radiance-weighted centroid, may not be exactly the same as the VLF-derived location; that is, the brightest part of the discharge as viewed from space may not be the same as the strike location. However, we consider a range of values in time and space and explore the influence of this choice on the results. Locations and distances in this work use the World Geodetic System 1984 (WGS-84) ellipsoid.
We first start with the ENTLN dataset and find the LIS group associated with each pulse. We then do the reverse problem: start with each LIS group and find the associated ENTLN pulse. We do this for data starting in February 2009 (the beginning of the ENTLN data archive) through the end of 2013. Hence, we can analyze the evolution of the performance of each LLS over the time period.
Since LIS is on board a low-Earth-orbiting satellite, some preprocessing of the data is necessary to ensure a rigorous comparison. In particular, only ENTLN pulses in the LIS field of view are considered in the comparison. This is done by utilizing the 1-s data associated with LIS. Further, we do not consider ENTLN data that occur when an LIS alert flag is fatal, which occurs for 4.7% of ENTLN pulses. This is because we are generally concerned with the general capability of space-based optical measurements and not necessarily the operational performance of the two systems. It is worth noting that including time periods when the LIS alert flag is fatal minimally affects the final results; the probabilities are within a couple percentage points. In addition, we used only the operational processing algorithm for LIS, which may miss weak events. It should be noted that no such quality control is provided with ENTLN data, so the ENTLN dataset is taken as is.
As an example of the matching algorithm, consider Fig. 1. This is a CG flash, with eight return strokes identified by HAMMA (Bitzer et al. 2013) (the black squares along the bottom of the time–altitude plot). There are 14 LIS groups and five ENTLN pulses. The LIS groups, located by the radiance-weighted centroid of the constituent LIS events, are 9.5 km due north of the ENTLN pulse locations. For completeness, NLDN (not shown) detects the first six return strokes and are located within 500–1400 m of the ENTLN pulses.
An 8-stroke CG flash. (left) The temporal evolution of the flash and (right) the spatial evolution. The top seven plots on the left are the electric field waveforms recorded at different HAMMA sensors. VHF sources located by the North Alabama Lightning Mapping Array (NALMA) are indicated throughout the figure by colored circles. The black diamonds (associated with IC discharges) and black squares (associated with CG return stroke discharges) are VLF/LF sources located by HAMMA. VLF/LF sources located by ENTLN are given by green crosses (CG) and green diamonds (IC). In the bottom-left plot, LIS groups are represented by gray diamonds and gray bars; the height of the bar yields the radiance of the group. The right panels, from upper right and proceeding counterclockwise, contain the altitude histogram of the number of sources in the flash for LMA (red) and HAMMA (blue), the x–z projection, the plan view, and the z–y projection. LIS events are yellow to orange squares, with orange colors indicating higher-radiance events. The black and red squares are two of the HAMMA sensors and correspond to the waveforms of the same color on the left side.
Citation: Journal of Atmospheric and Oceanic Technology 33, 3; 10.1175/JTECH-D-15-0032.1
There are LIS groups directly associated with the first six return strokes. A seventh stroke, at
Given an ENTLN pulse, corresponding to a return stroke or in-cloud discharge, we identify any LIS groups that occur within the spatial and temporal constraints. For this particular example, there is an LIS group associated with all five ENTLN discharges. An LIS group can be matched only to a single ENTLN pulse, as each pulse should produce independent optical emission. This provides a straightforward manner in which to match an LIS group to an ENTLN pulse.
Given an LIS group, the identification of the corresponding ENTLN pulse is more complicated because, strictly speaking, each group may not be a separate discharge. The light pulse from a single return stroke can be split across frames, resulting in two groups, yet one physical discharge. This occurs for the return strokes at
In this manner, we ensure the same discharge is compared, despite slight differences in the LLSs. It should be noted that we do not account for false alarms from either LLS. It is expected that this has an insignificant effect, given the false alarm rate for LIS is ~5%. The false alarm rate for ENTLN is unknown, but it is expected to be on the same order or smaller as that of LIS.
4. Results
To assess the relative performance of the detection of lightning between a ground-based VLF LLS and a space-based optical sensor LLS, we match LIS groups and ENTLN pulses in the manner discussed. We then analyzed several attributes of the matched and unmatched discharges. We discuss the results for the entire dataset and explore the relative performance for the year 2013 in more depth. Further, we explore variations in the relative performance as a function of spatial domain. We use three domains: global, the Western Hemisphere, and North America. The global domain spans a latitude range of
a. Relative probability
The probability that LIS detects a discharge given that ENTLN detected the discharge,
Relative detection efficiency of LIS, given ENTLN detected a discharge for all years in the dataset. Bins with fewer than 10 ENTLN pulses are not shown. LIS detects 0.528 of ENTLN pulses in this domain.
Citation: Journal of Atmospheric and Oceanic Technology 33, 3; 10.1175/JTECH-D-15-0032.1
Relative detection efficiency of LIS, given ENTLN detected a discharge for all years in the dataset. Bins with fewer than 10 ENTLN pulses are not shown.
Citation: Journal of Atmospheric and Oceanic Technology 33, 3; 10.1175/JTECH-D-15-0032.1
There is very little change in
Conditional detection efficiency statistics for the entire dataset by year.
Relative detection efficiency of LIS, given ENTLN detected a discharge for 2013. Bins with fewer than 10 ENTLN pulses are not shown. LIS detects 0.532 of ENTLN pulses in this domain.
Citation: Journal of Atmospheric and Oceanic Technology 33, 3; 10.1175/JTECH-D-15-0032.1
Conditional detection efficiency statistics for 2013 by spatial domain.
Next, we consider the reverse process—the conditional probability that ENTLN detects a discharge given that LIS detected the discharge,
Relative detection efficiency of ENTLN, given LIS detected a discharge for all years in the dataset. Bins with fewer than 10 LIS groups are not shown. Contrast this figure with Fig. 2. Note that there are different levels than in Fig. 2 to better show variations in the data. ENTLN detects 0.037 of LIS groups in this domain.
Citation: Journal of Atmospheric and Oceanic Technology 33, 3; 10.1175/JTECH-D-15-0032.1
Relative detection efficiency of ENTLN, given LIS detected a discharge for all years in the dataset. Bins with fewer than 10 LIS groups are not shown. Contrast this figure with Fig. 3.
Citation: Journal of Atmospheric and Oceanic Technology 33, 3; 10.1175/JTECH-D-15-0032.1
As discussed, an ENTLN pulse may be matched to more than one LIS group in the reverse process. For the year 2013, of the 15 405 568 LIS groups (5.9%) that were detected globally 908 271 were matched to an ENTLN pulse. Of the 908 271 matches, 435 813 (48.0%) are unique ENTLN pulses. For the unique ENTLN pulses, 215 487 (49.4%) were matched to one LIS group. Another 108 479 (24.9%) are matched to two LIS groups, and 48 230 (11.1%) are matched to three LIS groups. Since typically the light from a discharge would be split over a maximum of two frames, instances in which ENTLN matches to three or more LIS groups suggests multiple discharges. Hence, our methodology overestimates the probability that ENTLN detects a discharge, given LIS detects the discharge.
The conditional probability
Relative detection efficiency of ENTLN, given LIS detected a discharge for 2013. Bins with fewer than 10 LIS groups are not shown. ENTLN detects 0.269 of LIS groups in this domain.
Citation: Journal of Atmospheric and Oceanic Technology 33, 3; 10.1175/JTECH-D-15-0032.1
b. Timing differences
To further analyze the differences in the two LLSs, we also examine the timing differences between discharges detected by both systems. The distribution of the time difference for LIS groups given an ENTLN pulse (
Distribution of the difference in times between LIS groups and given ENTLN pulses, for N = 460 212. The mean time difference is −1.698 ms, indicating the ENTLN pulse time tag is before the LIS group time tag, on average.
Citation: Journal of Atmospheric and Oceanic Technology 33, 3; 10.1175/JTECH-D-15-0032.1
Summary of statistics for the timing difference between simultaneous LIS groups and ENTLN pulses for 2013. Negative values indicate ENTLN pulse occurred first. All values are in milliseconds.
c. Distance differences
We also find the differences in the location of discharges detected by both systems. Although the brightest part of the discharge (given by the LIS group location) may not be the same as the strike point (given by the ENTLN pulse), the mean difference in the locations is ~8 km (Table 4). This is roughly twice the footprint of an LIS pixel. This offset is consistent with other results comparing LIS to ground-based systems (e.g., Thomas et al. 2000, and discussion therein). The distribution of the differences is shown in Fig. 9.
Summary of statistics for the difference in location between simultaneous LIS groups and ENTLN pulses for 2013. All values are in kilometers.
Distribution of the difference in locations between LIS groups and ENTLN pulses, for N = 460 212. The mean distance difference is 8.082 km.
Citation: Journal of Atmospheric and Oceanic Technology 33, 3; 10.1175/JTECH-D-15-0032.1
Unlike for the timing difference, the distance difference is smaller over the North American domain. The mean and median are ~400 m smaller; the standard deviation is ~300 m smaller. Despite the differences in how the locations are determined by each system, this suggests, on average, the locations returned by ENTLN and LIS are more similar where the ENTLN sensor density is highest.
The distance difference for each conditional comparison is the same, unlike the timing difference. This indicates that when an ENTLN pulse is matched to more than one LIS group, the locations are similar but are relatively separated in time. This further implies that the phenomenon of multiple groups for a single discharge is rare. Hence, when a single ENTLN pulse is matched to more than one LIS group, the multiple LIS groups are separate discharges on average. This further supports the claim that our method of finding matches tends to overestimate the number of matches for ENTLN, given LIS detects a discharge.
d. LIS performance
To explore the limitations of a space-based optical sensor, we also investigate how the conditional probability
This ratio is largely constant over the CCD (Fig. 10). However, the ratio for pixels near the edge of the array is lower than in the center. This effect is especially pronounced for pixel locations greater than 120. Because a wide-angle lens (~45°) is used by LIS, the edge of the lens suffers from distortion and reduced throughput. This reduces sensitivity at the edge of the CCD. Indeed, the relative detection efficiency for the entire CCD is 0.520; this roughly linearly increases as we exclude more edge pixels. When the outer 10 pixels are excluded, the relative detection efficiency is 0.574. We do not find any variation of detection efficiency in the reverse case—for the pixels in which an LIS group occurs, there is no significant variation that ENTLN detects the group (except for the single-edge pixel of the CCD).
Relative detection efficiency of LIS, given ENTLN detected a discharge as function of location in the LIS CCD. Pixels are aggregated into 4 × 4 bins.
Citation: Journal of Atmospheric and Oceanic Technology 33, 3; 10.1175/JTECH-D-15-0032.1
e. Effect of matching constraints
We explored the impact of the choice of the spatial and temporal matching constraints had on our results. As might be expected, smaller constraints resulted in fewer matches. However, the effect of restricting the spatial constraints was minimal. For 10 ms and 20 km,
When the time constraint is varied,
This larger effect is due to the multiple matches we allow when matching ENTLN pulses to LIS groups. When increasing the temporal constraint, more matches occur. However, this also increases the chance ENTLN pulses are matched to LIS groups that are not physically related. Despite this, the similarity of the probabilities with various constraints suggests that the results reported herein are robust. For the following discussion, we will use the results for 10 ms and 20 km.
5. Discussion
The results indicate that in a relative manner, LIS detects more discharges than ENTLN. While this is a meaningful result, the question remains: how do these two systems compare in a quantitative manner? Further, what does this mean about either system’s ability to detect all lightning discharges? Finally, what is the quantitative benefit in assimilating these two systems?

Equations (7) and (8) yield a robust, quantitative manner in which to compare the performance of two lightning detection systems. These two equations yield the relative probability
Table 5 summarizes the relative probabilities for the entire dataset and for the year 2013. Globally, the absolute detection efficiency of LIS is a factor of 14 larger than the absolute detection efficiency of ENTLN for all years and is a factor of almost 9 larger for the year 2013. In North America (where the sensor density is highest for ENTLN), the absolute detection efficiency of LIS is a factor of almost 3 greater than that of ENTLN for all years. While ENTLN performance has improved relative to LIS, the absolute detection efficiency is still a factor of 2 smaller than that of LIS in the year 2013.
Relative probabilities of detection.
These results can be extended further by assuming
If we consider the global dataset for all years, then of the discharges detected by one or both LLSs, ENTLN detects 6.8%, while LIS detects 96.8% (Table 6). Hence, ENTLN detects a small percentage of global lightning discharges.
Upper limit of absolute detection efficiency.
However, the number of discharges detected by ENTLN has improved over the years and is spatially dependent. If we consider just the year 2013, then ENTLN detects 10.8% of global discharges in the LIS field of view and LIS detects 94.8%. In the Western Hemisphere domain, ENTLN detects 22.7% of the discharges, while LIS detects 89.1%. Finally, in the North American domain, where the ENTLN sensor density is highest, ENTLN detects 40.9% of all discharges, and LIS detects 80.9%. Hence, of the discharges detected by either or both LIS and ENTLN, twice as many are detected by LIS.
These results suggest care should be taken when discussing a comparison of two datasets. The commonly used assumption that one dataset comprises the entire set of lightning discharges is often invalid. This is particularly true if one system is more likely to detect a particular type of discharge (e.g., cloud-to-ground strokes). Although this assumption is usually acknowledged by assuming an absolute detection efficiency, the implications are rarely discussed. If this assumption was used in the analysis herein, then the conclusion would be the upper limit of the detection efficiency of ENTLN is 26.9%, while a more rigorous analysis indicates this value is 40.9%.
Further, this analysis suggests that as the ENTLN LLS has evolved and detects more discharges, these discharges are not necessarily detected by LIS. Using Bayes’s theorem [(2)] and noting
We can use these results to estimate the added benefit when assimilating lightning data from multiple systems. As an example, consider the data from 2013 in North America. Of the lightning detected by either LIS or ENTLN (or both), LIS detects 80.9% of all discharges. Assimilating ENTLN data, accounting for simultaneous discharges, adds 23.6%
6. Conclusions
In this paper, we introduce a new methodology in which two lightning locating systems (LLSs) are compared by using Bayesian principles. We are able to quantitively assess the performance of two LLSs without resorting to assumptions of the absolute detection efficiency of either system. In addition, this method provides an upper limit on the absolute detection efficiency.
Further, we compare a space-based optical LLS and ground-based VLF LLS in terms of the common physical process detected by each. Each system primarily detects return strokes and in-cloud discharges, and this forms the measurement basis. We do not compare flashes, which are an algorithmically derived entity. This allows for direct comparison of the performance of each LLS by deconvolving any biases introduced by a flash sorting algorithm.
This methodology is applied to two systems: the Lightning Imaging Sensor (LIS) and the Earth Networks Total Lightning Network (ENTLN). We find that LIS detects ~52% of ENTLN pulses, and this is relatively invariant in space and time. Conversely, ENTLN detects ~4% of LIS groups globally. In 2013, ENTLN detects ~6% of LIS groups globally and 27% of LIS groups near North America. ENTLN performance relative to LIS has increased, yet LIS still detects more lightning discharges. By casting the comparison as a Bayesian problem, we also find that, of all the discharges detected by either LIS or ENTLN, LIS detects 81% near North America in 2013, while ENTLN detects 41%.
There is great interest in how to best assimilate various LLSs to maximize the amount of lightning detected. With the impending launch of GLM, there is a particular desire to find how much information is added by ground-based systems to space-based optical measurements. We quantitatively assess the added benefit of assimilating ENTLN into LIS data and find this adds 23.6% more discharges. Conversely, assimilating LIS data into ENTLN yields 96.2% more discharges.
This work can also be extended so that one-third of LLSs could be compared. Intercomparisons of each LLS should be made, following the probabilistic methods outlined herein, to yield the conditional probabilities between each system. Then, compare each dataset to the union set of the other two. In this manner, a more complete set S will be formed, and hence a more robust estimate for the absolute detection efficiency of several LLSs can be found. Further, this provides a manner in which a researcher can more rigorously assess the added contribution of assimilating different lightning datasets by analyzing the various conditional probabilities.
Acknowledgments
This study was partially supported by NOAA Grant Z7813005 [Cooperative Institute for Climate and Satellites (CICS) at the University of Maryland (ESSIC) as part of GOES-R Risk Reduction Research] and NASA Grant NNM05AA22A. The Lightning Imaging Sensor (LIS) science data were obtained from the NASA EOSDIS Global Hydrology Resource Center (GHRC) DAAC, Huntsville, Alabama (http://lightning.nsstc.nasa.gov/). ENTLN data were provided by Stan Heckman and Christopher Sloop.
REFERENCES
Abarca, S. F., Corbosiero K. L. , and Galarneau T. J. , 2010: An evaluation of the Worldwide Lightning Location Network (WWLLN) using the National Lightning Detection Network (NLDN) as ground truth. J. Geophys. Res., 115, D18206, doi:10.1029/2009JD013411.
Biagi, C., Cummins K. , Kehoe K. , and Krider E. , 2007: National Lightning Detection Network (NLDN) performance in southern Arizona, Texas, and Oklahoma in 2003–2004. J. Geophys. Res., 112, D05208, doi:10.1029/2006JD007341.
Bitzer, P. M., 2011: New revelations on lightning initiation and evolution using a newly developed array of wideband electric field sensors. Ph.D. thesis, University of Alabama in Huntsville, 273 pp.
Bitzer, P. M., and Christian H. J. , 2015: Timing uncertainty of the Lightning Imaging Sensor. J. Atmos. Oceanic Technol., 32, 453–460, doi:10.1175/JTECH-D-13-00177.1.
Bitzer, P. M., and Coauthors, 2013: Characterization and applications of VLF/LF source locations from lightning using the Huntsville Alabama Marx Meter Array. J. Geophys. Res. Atmos., 118, 3120–3138, doi:10.1002/jgrd.50271.
Boccippio, D. J., Cummins K. , Christian H. , and Goodman S. , 2001: Combined satellite-and surface-based estimation of the intracloud–cloud-to-ground lightning ratio over the continental United States. Mon. Wea. Rev., 129, 108–122, doi:10.1175/1520-0493(2001)129<0108:CSASBE>2.0.CO;2.
Boccippio, D. J., Koshak W. , and Blakeslee R. , 2002: Performance assessment of the Optical Transient Detector and Lightning Imaging Sensor. Part I: Predicted diurnal variability. J. Atmos. Oceanic Technol., 19, 1318–1332, doi:10.1175/1520-0426(2002)019<1318:PAOTOT>2.0.CO;2.
Christian, H. J., Blakeslee R. J. , and Goodman S. J. , 1989: The detection of lightning from geostationary orbit. J. Geophys. Res., 94, 13 329–13 337, doi:10.1029/JD094iD11p13329.
Christian, H. J., and Coauthors, 1999: The Lightning Imaging Sensor. 11th International Conference on Atmospheric Electricity, H. J. Christian, Ed., NASA Conf. Publ. NASA/CP-1999-209261, 746–749.
Christian, H. J., Blakeslee R. J. , Goodman S. J. , and Mach D. , 2000: Algorithm theoretical basis document (ATBD) for the Lightning Imaging Sensor (LIS). NASA Marshall Space Flight Center Tech. Rep., 53 pp. [Available online at http://eospso.gsfc.nasa.gov/sites/default/files/atbd/atbd-lis-01.pdf.]
Cummins, K. L., and Murphy M. J. , 2009: An overview of lightning locating systems: History, techniques, and data uses, with an in-depth look at the U.S. NLDN. IEEE Trans. Electromagn. Compat., 51, 499–518, doi:10.1109/TEMC.2009.2023450.
Goodman, S. J., Buechler D. E. , Wright P. D. , and Rust W. D. , 1988: Lightning and precipitation history of a microburst-producing storm. Geophys. Res. Lett., 15, 1185–1188, doi:10.1029/GL015i011p01185.
Goodman, S. J., and Coauthors, 2005: The North Alabama Lightning Mapping Array: Recent severe storm observations and future prospects. Atmos. Res., 76, 423–437, doi:10.1016/j.atmosres.2004.11.035.
Goodman, S. J., and Coauthors, 2013: The GOES-R Geostationary Lightning Mapper (GLM). Atmos. Res., 125–126, 34–49, doi:10.1016/j.atmosres.2013.01.006.
Jerauld, J., Rakov V. A. , Uman M. A. , Rambo K. J. , Jordan D. M. , Cummins K. L. , and Cramer J. A. , 2005: An evaluation of the performance characteristics of the U.S. National Lightning Detection Network in Florida using rocket-triggered lightning. J. Geophys. Res., 110, D19106, doi:10.1029/2005JD005924.
Liu, C., and Heckman S. , 2010: The application of total lightning detection and cell tracking for severe weather prediction. TECO-2010—WMO Tech. Conf. on Meteorological and Environmental Instruments and Methods of Observation, Helsinki, Finland, World Meteorological Organization, P2(7). [Available online at https://www.wmo.int/pages/prog/www/IMOP/publications/IOM-104_TECO-2010/P2_7_Heckman_USA.pdf.]
Liu, C., and Heckman S. , 2012: Total lightning data and real-time severe storm prediction. TECO-2012—WMO Tech. Conf. on Meteorological and Environmental Instruments and Methods of Observation, Brussels, Belgium, World Meteorological Organization, P5(10). [Available online at https://www.wmo.int/pages/prog/www/IMOP/publications/IOM-109_TECO-2012/Session5/P5_10_Liu_Total_Lightning_Data_and_Real-Time_Severe_Storm_Prediction.pdf.]
Mach, D., Christian H. , Blakeslee R. , Boccippio D. , Goodman S. , and Boeck W. , 2007: Performance assessment of the Optical Transient Detector and Lightning Imaging Sensor. J. Geophys. Res., 112, D09210, doi:10.1029/2006JD007787.
Mallick, S., and Coauthors, 2015: Performance characteristics of the ENTLN evaluated using rocket-triggered lightning data. Electr. Power Syst. Res., 118, 15–28, doi:10.1016/j.epsr.2014.06.007.
Murphy, M., 2006: When flash algorithms go bad. Proc. First Int. Lightning Meteorology Conf., Tucson, AZ, Vaisala. [Available online at http://www.vaisala.com/en/events/ildcilmc/Documents/When%20Flash%20Algorithms%20Go%20Bad.pdf.]
Nag, A., and Coauthors, 2011: Evaluation of U.S. National Lightning Detection Network performance characteristics using rocket-triggered lightning data acquired in 2004–2009. J. Geophys. Res., 116, D02123, doi:10.1029/2010JD014929.
Nag, A., Murphy M. J. , Schulz W. , and Cummins K. L. , 2015: Lightning locating systems: Insights on characteristics and validation techniques. Earth Space Sci., 2, 65–93, doi:10.1002/2014EA000051.
Østgaard, N., Gjesteland T. , Carlson B. E. , Collier A. B. , Cummer S. , Lu G. , and Christian H. J. , 2013: Simultaneous observations of optical lightning and terrestrial gamma ray flash from space. Geophys. Res. Lett., 40, 2423–2426, doi:10.1002/grl.50466.
Poelman, D. R., Schulz W. , and Vergeiner C. , 2013: Performance characteristics of distinct lightning detection networks covering Belgium. J. Atmos. Oceanic Technol., 30, 942–951, doi:10.1175/JTECH-D-12-00162.1.
Pohjola, H., and Mäkelä A. , 2013: The comparison of GLD360 and EUCLID lightning location systems in Europe. Atmos. Res., 123, 117–128, doi:10.1016/j.atmosres.2012.10.019.
Rodger, C. J., Brundell J. B. , and Dowden R. L. , 2005: Location accuracy of VLF World-Wide Lightning Location (WWLL) network: Post-algorithm upgrade. Ann. Geophys., 23, 277–290, doi:10.5194/angeo-23-277-2005.
Rudlosky, S. D., 2015: Evaluating ENTLN performance relative to TRMM/LIS. J. Oper. Meteor., 3, 11–20, doi:10.15191/nwajom.2015.0302.
Rudlosky, S. D., and Shea D. T. , 2013: Evaluating WWLLN performance relative to TRMM/LIS. Geophys. Res. Lett., 40, 2344–2348, doi:10.1002/grl.50428.
Said, R. K., Inan U. S. , and Cummins K. L. , 2010: Long-range lightning geolocation using a VLF radio atmospheric waveform bank. J. Geophys. Res., 115, D23108, doi:10.1029/2010JD013863.
Said, R. K., Cohen M. B. , and Inan U. S. , 2013: Highly intense lightning over the oceans: Estimated peak currents from global GLD360 observations. J. Geophys. Res. Atmos., 118, 6905–6915, doi:10.1002/jgrd.50508.
Thomas, R., Krehbiel P. , Rison W. , Hamlin T. , Boccippio D. , Goodman S. , and Christian H. , 2000: Comparison of ground-based 3-dimensional lightning mapping observations with satellite-based LIS observations in Oklahoma. Geophys. Res. Lett., 27, 1703–1706, doi:10.1029/1999GL010845.
Thomas, R., Krehbiel P. , Rison W. , Hunyady S. , Winn W. , Hamlin T. , and Harlin J. , 2004: Accuracy of the lightning mapping array. J. Geophys. Res., 109, D14207, doi:10.1029/2004JD004549.
Thompson, K. B., Bateman M. G. , and Carey L. D. , 2014: A comparison of two ground-based lightning detection networks against the satellite-based Lightning Imaging Sensor (LIS). J. Atmos. Oceanic Technol., 31, 2191–2205, doi:10.1175/JTECH-D-13-00186.1.
Yoshida, S., Wu T. , Ushio T. , Kusunoki K. , and Nakamura Y. , 2014: Initial results of LF sensor network for lightning observation and characteristics of lightning emission in LF band. J. Geophys. Res. Atmos., 119, 12 034–12 051, doi:10.1002/2014JD022065.