Abstract

This paper evaluates the use of precipitable water (PW) from the global positioning system (GPS) in lightning prediction. Additional independent verification of an earlier model is performed. This earlier model used binary logistic regression with the following four predictor variables optimally selected from a candidate list of 23 candidate predictors: the current precipitable water value for a given time of the day, the change in GPS PW over the past 9 h, the K index, and the electric field mill value. The K index was used as a measure of atmospheric stability, which, of the traditional stability measures, has been shown to work best in the area and season under study. This earlier model was not optimized for any specific forecast interval, but showed promise for 6- and 1.5-h forecasts. Two new models were developed and verified. These new models were optimized for two operationally significant forecast intervals. The first model was optimized for the 0.5-h lightning advisories issued by the U.S. Air Force’s 45th Weather Squadron. An additional 1.5 h was allowed for sensor dwell, communication, calculation, analysis, and advisory decision by the forecaster. Therefore, the 0.5-h advisory model became a 2-h forecast model for lightning within the 45th Weather Squadron advisory areas. The second model was optimized for major ground processing operations supported by the 45th Weather Squadron, which can require lightning forecasts with a lead time of up to 7.5 h. Using the same 1.5-h lag as in the other new model, this became a 9-h forecast model for lightning within 37 km (20 n mi) of the 45th Weather Squadron advisory areas. The two new models were built using binary logistic regression and a list of 26 candidate predictor variables: the current GPS PW value, the K index, and 24 candidate variables of the change in GPS PW levels over 0.5-h increments up to 12 h. The new 2-h model found the following four predictors to be statistically significant, listed in decreasing order of contribution to the forecast: the 0.5-h change in GPS PW, the 7.5-h change in GPS PW, the current GPS PW value, and the K index. The new 9-h forecast model found the following five independent variables to be statistically significant, listed in decreasing order of contribution to the forecast: the current GPS PW value, the 8.5-h change in GPS PW, the 3.5-h change in GPS PW, the 12-h change in GPS PW, and the K index. In both models, the GPS PW parameters had better correlation to the lightning forecast than did the K index, a widely used thunderstorm index. Possible future improvements to this study are discussed.

1. Background

The U.S. Air Force’s 45th Weather Squadron (45 WS) provides comprehensive weather services to America’s space program at Cape Canaveral Air Force Station (CCAFS) and Kennedy Space Center (KSC). CCAFS and KSC are collectively known as Spaceport Canaveral. These facilities are located in east-central Florida near the highest lightning flash densities in North America (Fig. 1). The most frequent products of the 45 WS are lightning advisories for 13 different points (Fig. 2). These lightning advisories are issued for the personnel safety of over 25 000 people and for resource protection of facilities worth over $17 billion. A two-tiered advisory process is used. A phase 1 lightning advisory is issued for a point if lightning of any type is expected within 5 n mi of any of the points with a desired lead time of 30 min. A phase 2 lightning advisory is issued for a point if lightning of any type is imminent or occurring within 9.3 km (5 n mi) of the point. Lightning forecasting is important to other operations supported by the 45 WS, especially major ground processing. For example, transporting the space shuttle from the Vehicle Assembly Building to the launch pad requires a less than 10% probability of lightning within 37.0 km (20 n mi) during the approximate 6 h that the shuttle is being moved with the briefing for the final decision occurring about 2 h before transport begins. Lightning is also vitally important to space launches, but a special set of launch-commit criteria are used for space launches and lightning support for these operations are not covered in this paper (Roeder and McNamara 2006). The 45th WS has several techniques for forecasting lightning to support their lightning advisory and ground processing requirements. But the 45th WS is always trying to refine their current techniques and develop new methods to improve their lightning forecasts.

Fig. 1.

Average cloud-to-ground lightning flash density for the contiguous United States (1989–98). The largest flash densities occur in central FL. (Courtesy of Dr. R. Orville, Texas A&M University)

Fig. 1.

Average cloud-to-ground lightning flash density for the contiguous United States (1989–98). The largest flash densities occur in central FL. (Courtesy of Dr. R. Orville, Texas A&M University)

Fig. 2.

The 45 WS’s lightning advisory areas.

Fig. 2.

The 45 WS’s lightning advisory areas.

This paper explores the use of global positioning satellite (GPS)-based precipitable water (PW) in lightning prediction at Spaceport Canaveral. A previous GPS PW model for lightning prediction at Spaceport Canaveral (Mazany et al. 2002) received performance verification on a larger set of independent data. Two new models were developed for this research project. The first of the new models was optimized for the relatively short timelines of the 45 WS lightning advisories. The second new model was optimized for the longer timelines of 45 WS major ground processing operations. The performance of the previous GPS PW lightning prediction model was not duplicated. However, the two new GPS PW models did show promise.

2. Introduction

a. Description of GPS PW

PW is traditionally calculated from data obtained by weather balloons. However, it was discovered over a decade ago that PW can be calculated from GPS satellites (Bevis et al. 1992, 1994). Previous researchers have called this the GPS integrated water vapor (GPS IWV), but the authors use the term precipitable water rather than IWV since the two terms are equivalent, with precipitable water being the older and better established term. Applications of GPS PW have been explored by Businger et al. (1996), Bauman et al. (1997), and Wolfe and Gutman (2000). The phase delay of GPS signals passing through the atmosphere depends on the electron density in the ionosphere, the mass of the atmosphere, the amount of hydrometeors in the atmosphere, and the total amount of water vapor in the atmosphere. The delay due to the ionospheric electron density along each GPS line of sight can be calculated from the total electron count, which can be calculated by comparing the L1 and L2 GPS signals. The mass of the atmosphere can be calculated from the surface pressure measured by a barometer at the surface. The GPS phase delay due to hydrometeors is usually insignificant and not considered. Therefore, any GPS propagation delay remaining after accounting for these three delays is attributed to water vapor. GPS PW is normally measured by averaging the GPS propagation delays over all of the GPS satellites more than 15° above the horizon over a 30-min period. GPS PW has several important advantages over weather balloons. GPS PW is as accurate, if not more so, than weather balloons; is available every 30 min as compared with twice a day, which is typical of weather balloons; provides a remote all-weather capability; and can be automated, thereby avoiding the costs of human-operated weather balloons.

b. Mazany model

A model to forecast lightning from GPS PW was first developed in 2002 (Mazany et al. 2002). This model used binary logistic regression to predict the probability of lightning at Spaceport Canaveral using GPS PW, the K index to incorporate atmospheric instability, and the largest value from the network of 31 surface electric field mills at Spaceport Canaveral to include the electric signal from developing thunderstorms.

The output of the model is a lightning index between 0 (lightning) and 1 (no lightning) that indicates conditions for lightning. The lightning index is compared with thresholds to determine if and when lightning will occur. The model is as follows:

 
formula

where x1 is the electric field mill reading (V m−1), x2 is the PW (mm), x3 is the 9-h ΔPW (mm), x4 is the K index, and ŷ is the lightning index. The lightning index was then compared with the onset of lightning and the following thresholds were determined:

  • 0.7–1.0, no lightning in the next ∼6 h;

  • 0.6–0.7, lightning expected in the next ∼6 h; and

  • 0.0–0.6, lightning expected in the next ∼1.5 h.

The accuracy measures applied to the initial test results of the Mazany model forecasts were false alarm rate (FAR), probability of detection (POD), and hit rate (HR). The follow-on independent verification in this new study also included the Kuipers skill score (KSS; Wilks 1995) and the operational utility index (OUI). The OUI is a nonstandard metric developed by 45 WS for comparing lightning forecast tools that gives POD a weight of three, KSS a weight of two, and FAR a weight of negative one, and then normalizes the sum of the weighted metrics by the absolute value of the sum of the weights for easier interpretation; an OUI of one is perfect forecasting, and an OUI of zero is worthless forecasting. The weights were set by the operational importance of the metrics to lightning forecasting by the 45 WS. Because personnel safety is involved, POD is most important. A good level of skill is desired, to provide good service to the customers, but it is less important than POD. A low FAR is also desired, to also provide good service to the customers, but is least important of the three metrics:

 
formula

c. Data sources and validation

Data from four thunderstorm seasons, 1 May–30 September 2000–03, were used in the reverification of the Mazany model. Quality control was performed on the PW and K index data. Scatterplots were used to visually identify potential outliers. Statistical process control (SPC) charts were also used to more objectively identify potential outliers. The SPC charts identified points outside three standard deviations of the mean of the dataset. Specifically, x charts were used to identify x, upper control limits (UCL), and lower control limits (LCL) for each year. SPC x charts were created for both the PW and K index datasets for each year. Points that fell below the LCL and above the UCL were flagged as potential outliers and further examined against meteorological conditions surrounding the data points. One of the authors (WPR), the meteorologist on the team, reviewed the data from a meteorological perspective and determined that the potential outliers were actually valid data points.

The K index was calculated from the CCAFS weather balloons, which are usually available at 1000, 1500, and 2300 UTC during the summer thunderstorm season (May–September). The time series of the K index could be used in two different methods. To account for changes in the K index between weather balloon observations, the K index could be interpolated linearly between existing observations. This linear interpolation of the K index was used in the revalidation of the Mazany model. However, to mimic how the information would be used operationally, the future value would not be known at forecast time; therefore, the last K index was used unchanged until the next weather balloon observation in developing the two new forecasting tools.

As discussed above, the electric field mill data were not important to the relatively large forecast intervals being verified, when the electric fields could be fair-field values. At these fair-field values, scale analysis shows that the Mazany model is insensitive to typical variations in electric fields. Therefore, because it would not be important to the forecast intervals being verified, a constant typical fair field value of 300 V m−1 was used to validate the Mazany model (Marshall et al. 1999).

3. Categorical verification of the Mazany model

Actual observations of the independent and dependent variables for the four seasons (2000–03) were used to validate the Mazany model. Categorical forecasts were created from the continuous predictand of the Mazany model. If the model predictand was 0.7 or less, it was considered to be a “yes” forecast for the 6-h forecast interval. If the model predictand was greater than 0.7, it was considered a “no” forecast for the 6-h interval. If the model predictand was 0.6 or less, it was considered a yes forecast for the 1.5-h forecast interval. If the model predictand was greater than 0.6, it as considered a no forecast for the 1.5-h interval. Two forecast intervals were verified. A 6-h forecast was used to match the verification in the original Mazany study (Mazany et al. 2002). A 1.5-h forecast was also verified to match part of the operationally focused verification of the new models. Standard 2 × 2 contingency tables and the metrics discussed above were used to analyze performance. Table 1 shows the results of the predictive capability of the current Mazany model for each year and a combined performance for years 2000–03.

Table 1.

(a) Accuracy measure and skill scores of forecasts of the original Mazany model. The 2 × 2 contingencies tables for the 1.5- and 6-h forecast intervals for each year (2000–03) and all years are presented. (b) A summary table.

(a) Accuracy measure and skill scores of forecasts of the original Mazany model. The 2 × 2 contingencies tables for the 1.5- and 6-h forecast intervals for each year (2000–03) and all years are presented. (b) A summary table.
(a) Accuracy measure and skill scores of forecasts of the original Mazany model. The 2 × 2 contingencies tables for the 1.5- and 6-h forecast intervals for each year (2000–03) and all years are presented. (b) A summary table.

The results of the accuracy measures for the 1.5-h forecast period were compared with the accuracy measures of the test results of the Mazany model forecasts performed for the time identified as period B from the original study (10 June–26 September 1999). The period B forecasts were based upon the index value falling below 0.7 and 1.5 h prior to the first strike. Table 2 compares the 1999 period B with the four thunderstorm seasons from 1 May to 30 September 2000–03 and the combined performance.

Table 2.

Comparison of accuracy measurements of the GPS lightning index.

Comparison of accuracy measurements of the GPS lightning index.
Comparison of accuracy measurements of the GPS lightning index.

The GPS lightning index accuracy measure results for all four thunderstorm seasons were below expectations when compared with the results of the 1999 period B. On average, the FAR increased by 58%, the POD decreased by 31%, and the HR decreased by 20%. Both the development and test datasets came from different periods of the same thunderstorm season, resulting in more accurate performance of the Mazany model during 1999 period B.

Even though the accuracy measures indicate that the Mazany model is not reliable for forecasting lightning events 1.5 h prior to the first strike, the encouraging aspect of the model results was the consistency of the measures. This suggests there is some useful signal in the GPS PW timelines for forecasting lightning in the study area and that perhaps better performance could be obtained with alternate regression models with other predictors and other lead times.

As was determined in Mazany (2002), the performance of the GPS lightning index lead time with regard to the timing of the first strike follows approximately a normal distribution. With the range of lead times before the first strike varying between 0 and 12 h, the typical lead time is approximately 6 h. Based upon this finding, the GPS lightning index was also applied to the 6-h forecast period. When compared with the 1.5-h forecast period, the Mazany model performed better overall. Table 3 compares the average 1.5-h forecast period accuracy measures with the 6-h forecast period accuracy measures for the four thunderstorm seasons from 1 May to 30 September 2000–03 and the combined performance.

Table 3.

Comparison of accuracy measurements for lead times of 1.5 and 6 h for the Mazany model.

Comparison of accuracy measurements for lead times of 1.5 and 6 h for the Mazany model.
Comparison of accuracy measurements for lead times of 1.5 and 6 h for the Mazany model.

On average, the Mazany model performed better in the longer 6-h time period in both FAR and HR. On average, the FAR for the 6-h forecast period was lower by 15%, and the HR was higher by 7%. However, the Mazany model performed better in the shorter 1.5-h time period in POD, which was on average higher by 16% over the 6-h time period.

a. Timeline

The project sought to develop two new forecasting tools for Spaceport Canaveral. The objective of the first tool was to provide a desired 0.5-h lead time prior to a lightning event; the second sought a desired 7.5-h lead time. These lead times were chosen to meet operational requirements. The 0.5-h lead time is for lightning advisories. The 7.5-h lead time is for major ground processing operations, such as roll-out of the space shuttle to the launch pad, transport of major components, and others.

To achieve the desired lead times for both tools, consideration was paid to the process that would be used in the operational implementation of the tools. A 1.5-h delay or lag time had to be built into the model to account for the process. The initial lag is caused by the PW 30-min dwell time. When the operator receives the PW reading, it is already 30 min behind the center time stamp and 15 min behind the end of the PW dwell time. Another 15-min lag time was added to account for communication of the most recent PW data to the operator. A 45-min delay was added for processing of the model, communication of the results to the 45 WS, and comparison of the model output with other weather data to make the forecast decision. This process is depicted in Fig. 3. This turns the desired 0.5-h lead time into a 2-h forecast tool and the desired 7.5-h lead time into a 9-h forecast tool.

Fig. 3.

Time line for sensing, receiving, and interpreting the GPS PW observations to meet the desired lead time of 30 min for 45th Weather Squadron lightning advisories..

Fig. 3.

Time line for sensing, receiving, and interpreting the GPS PW observations to meet the desired lead time of 30 min for 45th Weather Squadron lightning advisories..

b. Tool development

1) Logistic regression

Logistic regression was chosen as the tool to develop both models for several reasons. Previous studies conducted for the 45 WS indicated the applicability of logistic regression in modeling meteorological data. Logistic regression is constrained to be between zero and one, as are the probabilities. Linear regression for probability forecasting can predict probabilities greater than one and less than zero. Logistic regression can also model rapid changes in probability as thresholds of predictors are exceeded, as often happens in meteorology. This is depicted in Fig. 4.

Fig. 4.

Logistical regression is usually preferred over linear regression in probability forecasting. Logistic regression can model threshold values with rapid transition in predictands better than linear regression and avoids the problem linear regression can have of predicting undefined probabilities less than zero and greater than one.

Fig. 4.

Logistical regression is usually preferred over linear regression in probability forecasting. Logistic regression can model threshold values with rapid transition in predictands better than linear regression and avoids the problem linear regression can have of predicting undefined probabilities less than zero and greater than one.

Finally, the output of the logistic regression model is the probability that the predictand is equal to 1. In the case of both the 2- and 9-h forecast tools, the output of the model is the probability of lightning. This allowed for the creation of a lightning index threshold. The lightning index threshold is the point at which lightning is predicted when the model output falls above the threshold and not predicted when model output falls below the threshold.

2) Candidate predictors

The dataset consisted of 26 candidate independent variables that are shown in Table 4. These were chosen to match the three basic requirements for thunderstorms: 1) moisture, 2) instability, and 3) a trigger of upward motion. The current values of GPS PW and the K index directly measure moisture and instability. The K index was chosen because it is one of the best of the traditional stability indexes for forecasting thunderstorms at Spaceport Canaveral (Kelly et al. 1998) and to match the Mazany model. The change in GPS PW indirectly measures thunderstorm triggers by the moisture convergence from vertical motions. The change in GPS PW over a 30-min interval up to 12 h is chosen to exceed the lead times of the forecast tool and thus represent the triggers that apply to those lead times.

Table 4.

Candidate regressors.

Candidate regressors.
Candidate regressors.

3) Methodologies for regressor selection

Two basic methodologies were used for regressor selection: a forward method and a backward method. In the forward method, one variable was added at each iteration of the model. In the backward method, all variables were initially entered into the model, with one variable removed at each iteration. In both cases, a regressor was selected only when there was a 95% probability that the regressor was significant in predicting the model outcome.

Goodness-of-fit statistics were initially calculated to evaluate the model fit as different sets of predictors were used. Specifically, Cox–Snell R squared, Nagelkerke R squared, and the Hosmer–Lemeshow goodness of fit were produced by the Statistical Package for the Social Sciences (SPSS) tool that was used for model development. For both tools, there appeared to be a lack of fit based on the values of the Cox–Snell R-squared, Nagelkerke R-squared, and Hosmer–Lemeshow goodness-of-fit statistics. While this seemed to indicate a poor fit of the model, further investigation eliminated this concern for several reasons. First, the R-squared values from a logistic regression are not the same as the R-squared values calculated in a linear regression model. Therefore, they are not proven to be good measures of model fit. Second, these values are not relevant in measuring the model’s utility because they are evaluated against a threshold of 0.5. This means that outputs above 0.5 were predicted as lightning, whereas values below 0.5 were predicted as no lightning. The utility of the model will be evaluated based on an optimized threshold.

In selecting the model, the most weight was given to the values of the accuracy measurements and skill scores discussed above. These performance metrics included HR, FAR, POD, KSS, and OUI.

The forward and backward methods of model selection chose different regressors for both tools. For the 2-h forecast tool, the most recent PW measurement, the latest reading of the K index, and the 0.5-h ΔPW were all selected to be significant in predicting lightning in a 0.5-h time period by both methods. However, the forward model selection process selected an additional regressor, the 7.5-h ΔPW, while the backward model selection process picked two additional regressors, the 4.5- and the 5.5-h ΔPW. For the 9-h forecast tool, both the forward and backward model selection methodologies chose the most recent PW measurement, the latest reading of the K index, and the 3.5-, 8.5-, and 12.0-h ΔPW. The backward model selection method picked an additional regressor, the 6.5-h ΔPW, that was not chosen by the forward model selection method.

The predictors selected as having the most independent signal in predicting lightning may provide insight as to the physical mechanisms causing the lightning for the respective time periods being forecast. The 2-h forecast tool selected the 0.5-h change in GPS PW as the most important predictor. The authors speculate that this predictor represents the local moisture convergence of the developing thunderstorms. Indeed, detecting this mechanism was the original inspiration by one of the authors (WPR) for using timelines of GPS PW in local lightning forecasting and as hypothesized as the top predictor for such short-term forecasts. The second most important predictor for the 2-h forecast tool is the 7.5-h change in GPS PW. The mechanism associated with this predictor is not very obvious. Possibilities may include general convergence over the Florida peninsula due to solar heating (sunrise to typical thunderstorms forming at 2000 UTC is about 8 h), or perhaps a dynamic trigger in the asynoptic upward motion in the right-entrance and left-exit regions of weak jet streaks over the forecast area (Uccellini and Kocin 1987), or moisture convergence under flow with a southerly component, or other mechanisms. The 2-h forecast tool selected the current PW and the K index as the third and fourth most important predictors, respectively. These choices of predictors are likely because thunderstorms require moisture and instability to form.

The 9-h forecast tool selected four predictors in the following order of statistical importance: current GPS PW, 3.5-h change in GPS PW, 8.5-h change in GPS PW, and K index. The current GPS PW and K index likely have the same meteorological explanation as that for the 2-h forecast tool. The 8.5-h change in GPS PW may be a result of the same reasons speculated for the 2-h forecast tool. The 3.5-h change in GPS PW is also not obvious. The authors speculate it may be due to some trigger of upward motion, perhaps from approaching sea-breeze fronts that are known to be important in thunderstorm formation in east-central Florida.

It is interesting that the K index was the least important predictor for both tools. The K index has been shown to be one of the best performing of the traditional thunderstorm indexes in east-central Florida (Kelly et al. 1998).

4) Model optimization

For the 2-h forecast tool, the model was optimized based on the value of the OUI. The OUI is a locally developed performance metric used to optimize personnel safety. The OUI is considered the most critical factor because this metric applies more weight to the POD, which is critical when personnel safety is at stake. The 2-h forecast tool is meant to support the decision-making process involved with phase 1 lightning advisories, which are issued to ensure personnel working outdoors have adequate time to seek shelter. The equation for the OUI is

 
formula

For the 9-h forecast tool, the model was optimized based on the KSS. The 9-h forecast tool supports major ground processing operations where personnel safety is less of an issue. The KSS is a more traditional measure of skill, which is appropriate for this application.

5) Lightning index threshold

As mentioned previously, the output of the logistic regression model for both the 2- and 9-h forecast tools is the probability of lightning. The statistical software package used in model development defaults to a threshold of 0.5, meaning that when the probability is 0.5 (or 50%) or greater, lightning is predicted. Conversely, when the probability falls below 0.5, lightning is not predicted. A lightning index threshold was established by varying the default value and recalculating the various skill scores and accuracy measures. The lightning index threshold was established to optimize the OUI for the 2-h forecast tool and the KSS for the 9-h forecast tool.

6) Test and verification

A key last step to developing the two forecast tools was to independently test and verify the model. This ensures that the model results are repeatable. A verification dataset was created for each model using a random sampling of 10% of the initial data points. This verification dataset was not used in the development of the model and, thus, was not included as part of the development dataset. Its sole purpose was to serve as a check after the models were developed to compare the results of the verification dataset to the development dataset.

4. The 2-h forecast tool

The 2-h forecast tool was designed to support the current phase 1 lightning advisory system at Spaceport Canaveral by providing a 0.5-h lead time prior to a lightning event. The accuracy measurements and skill scores were calculated based on varying the lightning index threshold from 0.0 to 1.0. Table 5 depicts the accuracy measures and skill scores of both forward and backward model selection methodologies at various levels of the lightning index threshold. Again, the objective is to maximize the OUI. The model and lightning index that produced the highest OUI is in boldface in Table 5.

Table 5.

Comparison of accuracy measurements and skill scores for the 2-h forecast tool. The model and lightning index that produced the highest OUI is in boldface.

Comparison of accuracy measurements and skill scores for the 2-h forecast tool. The model and lightning index that produced the highest OUI is in boldface.
Comparison of accuracy measurements and skill scores for the 2-h forecast tool. The model and lightning index that produced the highest OUI is in boldface.

Figure 5 shows how the various accuracy measurements change as the lightning index threshold changes. Setting the lightning index threshold at 0.0 means that when the model outputs the probability of lightning at greater than 0.0, the model predicts that lightning will occur. Because the output of a logistic regression equation is between 0.0 and 1.0, lightning will always be predicted at a lightning index threshold of 0.0. Conversely, setting the lightning index threshold at 1.0 means that when the model outputs the probability of lightning at greater than 1.0, the model predicts that lightning will occur. The latter case is impossible, because the output of a logistic regression is always between 0.0 and 1.0. Therefore, lightning will never be predicted when the lightning index threshold equals 1.0. POD is at its highest when the lightning index threshold equals 0.0 because lightning is predicted every time. This results in a very high OUI because the OUI places the most weight on POD. A lower lightning index threshold drives a higher FAR because lightning is falsely predicted more often. At a lightning index threshold of 1.0, the POD becomes 0.0 because lightning is never detected. This also produces a 0.0% FAR because if lightning is never predicted, naturally it is never falsely predicted.

Fig. 5.

OUI vs lightning index for the 2-h forecast tool.

Fig. 5.

OUI vs lightning index for the 2-h forecast tool.

The forward and backward model selection methods performed similarly at all levels of the lightning index threshold; however, the model developed using the forward selection process produced the highest OUI. The OUI is maximized at 45.8% at a lightning index threshold of 0.3. However, thresholds of 0.2 or 0.4 both provide a favorable OUI ranging from 43.2% to 43.9%. The OUI was calculated for indexes in 0.01 increments around the optimal range of 0.25–0.45 to fine-tune the optimal index.

Figure 5 shows that the OUI increases slightly up to a peak of 0.32 and then begins to fall steadily at a threshold of 0.38. The highest OUI results from the forward selection model process at a lightning index threshold of 0.32. This OUI is 46.3%. Figure 3 also shows that there is not much sensitivity in the OUI in this range of lightning index thresholds. The OUI ranges between 38.5% and 46.3% when the lightning index threshold is varied between 0.25 and 0.45.

Lowering the lightning index threshold from the default value of 0.5 to 0.32 will adjust the mix of lightning forecast–not forecast and lightning observed–not observed, as shown in Table 6.

Table 6.

Contingency tables for the 2-h forecast tool.

Contingency tables for the 2-h forecast tool.
Contingency tables for the 2-h forecast tool.

Decreasing the lightning index threshold increases the number of forecast lightning strikes from 896 to 1405 and decreases the number of times that lightning is not forecasted from 709 to 200. The number of missed lightning events decreases from 286 to 30, yet the number of falsely predicted lightning events increases from 384 to 637. Changing the lightning index threshold from the default 0.5 to 0.32 changes accuracy measurements as shown in Table 7.

Table 7.

Comparison of accuracy measurements and skill scores at lightning index thresholds of 0.5 and 0.32 for the 2-h forecast tool.

Comparison of accuracy measurements and skill scores at lightning index thresholds of 0.5 and 0.32 for the 2-h forecast tool.
Comparison of accuracy measurements and skill scores at lightning index thresholds of 0.5 and 0.32 for the 2-h forecast tool.

Lowering the lightning index threshold means that more lightning is detected. This results in a higher HR, POD, KSS, and OUI. However, this also increases the FAR. This result is acceptable because detecting lightning is much more important than falsely warning of a lightning strike when lives are at stake. Whereas a lightning index threshold of 0.32 maximizes the OUI, other thresholds provide a higher HR and KSS and a lower FAR while still maintaining a relatively good OUI.

a. New logistic regression equation

The models perform differently at different lightning index threshold levels; therefore, the model selected will vary based on the lightning index. At a lightning index threshold of 0.32, the model generated using the forward model selection process was selected as the new logistic regression equation because this model maximized the OUI. The logistic regression takes the form of

 
formula

where

 
formula

and α′ = −2.366, β1 = 2.053, x1 = Δ0.5 h PW, β2 = –0.538, x2 = Δ7.5 h PW, β3 = 0.031, x3 = K index, β4 = 0.322, and x4 = PW (cm). This translates to

 
formula

The most significant independent variable in the model is the 0.5-h change in PW. A 0.5-h change in PW will have the greatest effect on the outcome of the model. The least significant variable in the model is the K index, which is surprising since it is a traditional tool for forecasting thunderstorms and their associated lightning.

b. Test and verification

An independent dataset was used to validate the model. The results are shown in Table 8. For all skill scores and accuracy measures, the verification dataset performed closely to the development dataset, validating the model results.

Table 8.

Development and verification results for the 2-h forecast tool (lightning index threshold=0.32).

Development and verification results for the 2-h forecast tool (lightning index threshold=0.32).
Development and verification results for the 2-h forecast tool (lightning index threshold=0.32).

5. The 9-h forecast tool

The 9-h forecast tool seeks to provide a 7.5-h lead time prior to a lightning event to support major outdoor operations, such as space shuttle roll-out from the vehicle assembly building to the launch pad. Prior to beginning an extended outdoor activity, it is essential to know the probability of lightning. If lightning has a high probability of occurrence, the outdoor operations will be postponed or rescheduled until weather conditions are more favorable.

The accuracy measurements and skill scores were calculated based on varying the lightning index threshold from 0.0 to 1.0. Table 9 depicts the accuracy measures and skill scores of both forward and backward model selection methodologies at various levels of the lightning index threshold. For this tool, the objective is to maximize the KSS. The models and lightning index thresholds that produce the highest KSS are shown in boldface in Table 9.

Table 9.

Comparison of accuracy measurements and skill scores for the 9-h forecast tool. The models and lightning index thresholds that produce the highest KSS are shown in boldface

Comparison of accuracy measurements and skill scores for the 9-h forecast tool. The models and lightning index thresholds that produce the highest KSS are shown in boldface
Comparison of accuracy measurements and skill scores for the 9-h forecast tool. The models and lightning index thresholds that produce the highest KSS are shown in boldface

Lowering the lightning index threshold from the default value of 0.5 to 0.36 will adjust the mix of lightning forecast–not forecast and lightning observed–not observed, as shown in Table 10.

Table 10.

Contingency tables for the 9-h forecast tool.

Contingency tables for the 9-h forecast tool.
Contingency tables for the 9-h forecast tool.

The KSS varies significantly with changes in the lightning index threshold, although the values at thresholds of 0.3 and 0.4 are similar. The KSS is maximized at 35.9% at a lightning index threshold of 0.3 for the forward method and 0.4 for both the forward and backward methods. Both forward and backward selection models performed similarly at all levels of the threshold.

The KSS is at its highest at lightning index thresholds of 0.3 and 0.4; therefore, the lightning index threshold was further refined around these two points by calculating KSS for 0.01 increments of the index from 0.30 to 0.45. Figure 6 shows how the KSS varies with the lightning index threshold for both forward and backward models between 0.25 and 0.45.

Fig. 6.

KSS vs lightning index for the 9-h forecast tool.

Fig. 6.

KSS vs lightning index for the 9-h forecast tool.

Figure 6 shows that KSS increases slightly up to a maximum of 0.35 and begins to fall steadily at a lightning index threshold of 0.38. The maximum KSS of 36.8% results from the forward selection model process at lightning index thresholds of 0.35 and 0.37. In this range of the lightning index threshold, the KSS changes minimally, with the largest value being 36.8% and the smallest being 33.4%.

a. Lightning index threshold

The KSS was maximized at lightning index thresholds of both 0.35 and 0.37, indicating an optimal range for the lightning index threshold between 0.35 and 0.37. Because the thresholds are being refined to an accuracy of 0.01, a lightning index threshold of 0.36 was selected as optimal.

The output of the logistic regression model is the probability that the outcome is equal to 1. In this case, the output is interpreted as the probability of lightning in a continuous 7.5-h period. The model is designed to predict lightning when the probability of lightning is greater than 50%. Lowering the lightning index threshold from 0.5 to 0.36 will increase the amount of lightning that is detected, which will adjust the mix of lightning forecast–not forecast and lightning observed–not observed.

Decreasing the lightning index threshold increases the number of forecast lightning strikes from 4120 to 7147, and decreases the number of times that lightning is not forecasted from 7904 to 4877. The number of missed lightning events decreases from 2227 to 830, yet the number of falsely predicted lightning events increases from 1727 to 3357. When the lightning index threshold is decreased from 0.5 to 0.36, the skill and accuracy measurements of the model change as shown in Table 11.

Table 11.

Comparison of accuracy measurements and skill scores at lightning index thresholds of 0.5 and 0.36 for the 9-h forecast tool.

Comparison of accuracy measurements and skill scores at lightning index thresholds of 0.5 and 0.36 for the 9-h forecast tool.
Comparison of accuracy measurements and skill scores at lightning index thresholds of 0.5 and 0.36 for the 9-h forecast tool.

Lowering the lightning index threshold means that more lightning is detected. This results in a higher POD, KSS, and OUI. However, this also increases the FAR. Whereas a lightning index threshold of 0.36 maximizes the KSS, other thresholds provide a lower FAR while still providing an acceptable KSS.

b. New logistic regression equation

The model generated using the forward model selection process was selected as the new logistic regression equation because this model produced the highest KSS. The logistic regression takes the form of Eq. (3), where z is calculated as in Eq. (4), and α′ = −4.885, β1 = 0.541, x1 = PW (cm), β2 = 0.346, x2 = Δ3.5-h PW, β3 = –0.446, x3 = Δ8.5-h PW, β4 = 0.235, x4 = Δ12-h PW, β5 = 0.071, and x5 = K index. This translates to

 
formula

The most significant independent variable in predicting lightning is the current PW level. The second most significant variable is the 8.5-h change in the PW level in the atmosphere. This is similar to the Mazany model, which determined that the 9-h change in PW was most significant in predicting lightning. The K index was the least statistically significant variable.

c. Test and verification

An independent dataset was used to validate the model. The results are shown in Table 12. For all skill scores and accuracy measures, the verification dataset performed closely to the development dataset, validating the model results.

Table 12.

Development and verification results for the 9-h forecast tool.

Development and verification results for the 9-h forecast tool.
Development and verification results for the 9-h forecast tool.

6. Recommendations for future research

This project lays the foundation for demonstrating the utility of GPS PW timelines in forecasting lightning during the east-central Florida summer season. Several opportunities exist to build upon this research and continually improve forecasting accuracy.

This research focused on only two main variables: PW (current level as well as the change in 0.5-h increments over a 12-h period) and the current reading of the K index. A multitude of meteorological factors (e.g., lifted index and Thompson index) could be added to the list of candidate predictors and tested for significance. While the K index is among the best indexes for the summer lightning season, more recent studies have indicated that indexes optimized by month provide better skill (Lambert et al. 2006). This same research showed that the monthly indexes, flow regime, 1-day persistence, and daily climatology contributed significantly to the probability of lightning. Perhaps the lightning probability that combines these other predictors should be used as a predictor with GPS PW–based parameters. In addition, the influence of changes in PW over extended periods (beyond 12 h) on lightning could be tested. Evidence of this impact is shown in the 9-h forecast tool, in which a significant model variable was the 12-h change in PW. This suggests that the real optimal change in GPS PW likely occurs at a larger time increment.

The research in this paper very specifically targeted east-central Florida during the summer thunderstorm season. The potential exists to extend the application of this model to other areas and other seasons. Future research should investigate GPS PW timelines for lightning prediction at Spaceport Canaveral during the winter frontal regime. Also, future research includes investigation of GPS PW timelines for lightning prediction at other locations across Florida and especially areas outside the subtropics.

Neural network forecast modeling is a sophisticated tool that provides the potential to support forecasting. The selection of nonlinear optimal predictor variables, plus the eventual integration of all lightning precursors, such as electric field mills, local boundary layer convergence, flow regime, daily persistence, daily climatology, and numerical model inputs, into the final answer, will lead to improvements in modeling capabilities.

Older methods exist to estimate the total water vapor in the atmosphere from surface-based parameters such as surface dewpoint (Reitan 1963; Bolsenga 1965; Viswanadham 1981), absolute humidity (Reber and Swope 1972), and temperature and cloudiness (Adem 1967). While presumably not as accurate as GPS PW, these methods are less costly and could be implemented at current surface observation stations. Future work could evaluate the cost-effectiveness of these techniques, the trade-off in terms of accuracy for lower cost, with a special consideration for the PW accuracy required to improve the numerical weather prediction models. Likewise, there is some evidence that GPS PW can work with useful accuracy when incorporating the surface pressure from numerical models, so that an on-site barometer is not needed (Bai and Feng 2003; Quinn and Herring 1996). This significantly reduces the cost of GPS PW. Future work could also evaluate the cost-effectiveness of this model-based barometer-less type of GPS PW.

7. Conclusions

In conclusion, the extended validation of the Mazany model underperformed expectations in POD, HR, FAR, and KSS. The two new tools developed show promise in supporting forecasting at Spaceport Canaveral during the summer thunderstorm season. The new 2-h forecast tool will support the current phase 1 lightning advisory system, and the new 9-h forecast tool will support major, extended outdoor operations. Both tools will help to improve forecasting accuracy, thus improving personnel safety and reducing costs.

This research concludes that GPS PW timelines may have utility in forecasting lightning in east-central Florida during the summer thunderstorm season. While this is not the final answer to the task of lightning forecasting at Spaceport Canaveral, it is another key tool in the forecaster’s toolbox.

Acknowledgments

The data resources and collection efforts that went into this research were substantial and could not have been accomplished without the assistance of several outside parties. The Air Force Combat Climatology Center provided the K indexes from the Cape Canaveral Air Force Station radiosondes used in this research. Dr. Seth Gutman from the National Oceanic and Atmospheric Administration/Forecast Systems Laboratory (now Earth System Research Laboratory Global Systems Division) provided the GPS PW data from the Cape Canaveral Coast Guard Station. This research partially satisfied the M.S. in industrial and systems engineering degree from the University of Florida for two of the authors (Graf and Kehrer). Dr. H. Edwin Romeijn from the University of Florida supported the project as the advisor and provided invaluable support regarding statistical analysis and model development. This research was supported by NASA Kennedy Space Center through the funding of the degree and the statistical software used, as well as providing time and space to conduct the research.

REFERENCES

REFERENCES
Adem
,
J.
,
1967
:
Parameterization of atmospheric humidity using cloudiness and temperature.
Mon. Wea. Rev.
,
95
,
83
88
.
Bai
,
Z.
, and
Y.
Feng
,
2003
:
GPS water vapor estimation using interpolated surface meteorological data from Australian automatic weather stations.
J. Global Prediction Syst.
,
2
,
83
89
.
Bauman
,
W. H.
,
S.
Businger
, and
M. L.
Kaplan
,
1997
:
Nowcasting convective activity for space shuttle landings during easterly flow regimes.
Wea. Forecasting
,
12
,
78
107
.
Bevis
,
M.
,
S.
Businger
,
T. A.
Herring
,
C.
Rocken
,
R. A.
Anthes
, and
R. H.
Ware
,
1992
:
GPS meteorology: Remote sensing of atmospheric water vapor using the global positioning system.
J. Geophys. Res.
,
97
,
15787
15801
.
Bevis
,
M.
,
S.
Businger
,
S.
Chriswell
,
T. A.
Herring
,
R. A.
Anthes
,
C.
Roken
, and
R. H.
Ware
,
1994
:
GPS meteorology: Mapping zenith wet delay onto precipitable water.
J. Appl. Meteor.
,
33
,
1282
1299
.
Bolsenga
,
S.
,
1965
:
The relationship between total atmospheric water vapor and surface dew point on a mean hourly basis.
J. Appl. Meteor.
,
4
,
430
432
.
Businger
,
S.
, and
Coauthors
,
1996
:
The promise of GPS in atmospheric monitoring.
Bull. Amer. Meteor. Soc.
,
77
,
5
17
.
Kelly
,
J. L.
,
H. E.
Fuelberg
, and
W. P.
Roeder
,
1998
:
Thunderstorm predictive signatures for the East Coast sea breeze at Cape Canaveral Air Station and the Kennedy Space Center. Preprints, 19th Conf. on Severe Local Storms, Minneapolis, MN, Amer. Meteor. Soc., 677–680
.
Lambert
,
W.
,
M. M.
Wheeler
, and
W. P.
Roeder
,
2006
:
Lightning probability forecasts at Kennedy Space Center/Cape Canaveral Air Force Station, Florida. First Int. Lightning Meteorology Conf., Tucson, AZ
.
Marshall
,
T. C.
,
W. D.
Rust
,
M.
Stolzenberg
,
W. P.
Roeder
, and
P. R.
Krehbiel
,
1999
:
A study of enhanced fair-weather electric fields occurring soon after sunrise.
J. Geophys. Res.
,
104
.
(D20), 24 455–24 469
.
Mazany
,
R. A.
,
S.
Businger
,
S. I.
Gutman
, and
W.
Roeder
,
2002
:
A lightning prediction index that utilizes GPS integrated precipitable water vapor.
Wea. Forecasting
,
17
,
1034
1047
.
Quinn
,
K. J.
, and
T. A.
Herring
,
1996
:
GPS atmospheric water vapor measurements without the use of local barometers. Abstracts, Amer. Geophys. Union Fall Meeting, San Francisco, CA, Amer. Geophys. Union, Poster G12A-06
.
Reber
,
E.
, and
J.
Swope
,
1972
:
On the correlation of the total precipitable water in a vertical column and absolute humidity at the surface.
J. Appl. Meteor.
,
11
,
1322
1325
.
Reitan
,
C. H.
,
1963
:
Surface dew point and water vapor aloft.
J. Appl. Meteor.
,
2
,
776
779
.
Roeder
,
W. P.
, and
T.
McNamara
,
2006
:
A survey of the lightning launch commit criteria. Preprints, Second Conf. on Meteorological Applications of Lightning Data, Atlanta, GA, Amer. Meteor. Soc., 1.1
.
Uccellini
,
L. W.
, and
P. J.
Kocin
,
1987
:
The interaction of jet streak circulations during heavy snow events along the east coast of the United States.
Wea. Forecasting
,
2
,
289
308
.
Viswanadham
,
Y.
,
1981
:
The relationship between total precipitable water and surface dew point.
J. Appl. Meteor.
,
20
,
3
8
.
Wilks
,
D. S.
,
1995
:
Statistical Methods in the Atmospheric Sciences.
Academic Press, 467 pp
.
Wolfe
,
D. E.
, and
S. I.
Gutman
,
2000
:
Development of the NOAA/ERL ground-based GPS water vapor demonstration network: Design and initial results.
J. Atmos. Oceanic Technol.
,
17
,
426
440
.

Footnotes

Corresponding author address: Kristen Kehrer, National Aeronautics and Space Administration, Mail Stop EA-C, Kennedy Space Center, FL 32899. Email: kristen.c.kehrer@nasa.gov