a. Study areas

Statistical guidance was developed for 11 coastal areas of the Florida peninsula that are serviced by FPL. The name of each forecast area (FA) and its boundaries are shown in Figs. 2a–d. These irregularly shaped areas were specified by FPL based primarily on the location and number of customers, although some meteorological factors also were considered.

b. Lightning data

The study utilized CG lightning data from the National Lightning Detection Network (NLDN). This network, in operation since 1989, detects and records CG lightning flashes across the contiguous United States. The NLDN is owned and operated by Vaisala, Inc., and provides both real-time and historical data to government, educational, and commercial users. A complete description of sensors and methods of detection is given in Cummins et al. (1998).

The study period was the warm season months of June–August for the years 1989–2004. The location accuracy and detection efficiency of the NLDN has changed during this time due to system upgrades. Prior to 1994, detection efficiencies across the United States ranged from 65% to 85%, with location accuracies between 8 and 16 km. A system upgrade in 1995 allowed a greater number of flashes to be detected, as well as improved location accuracy. Since the upgrade, the NLDN has a location accuracy of ∼0.5 km over most of the United States, and an estimated flash detection efficiency of 80%–90% (Cummins et al. 1998). Detection efficiencies over Florida currently range from ∼80% over most of the peninsula to only 60% over the extreme southern part of the state. In this study, no corrections were applied to account for these variations in detection efficiency or location accuracy. Thus, actual flash counts are underestimated.

Due to the improved detection efficiency of the NLDN, the same flash can be sensed multiple times, and non-CG discharges can be detected (Cummins et al. 1998). Following the recommendation of Cummins et al. (1998), weak positive flashes with signal strengths less than +10 kA were removed from the dataset. In addition, multiple flashes occurring during the same second and within 10 km of each other were assumed to be duplicate flashes, and were combined into a single flash by retaining the first flash’s time and location and adding the multiplicities.

The number of daily CG flashes in each FA (Figs. 2a–d) was counted over the period of interest, NM eastern daylight time (EDT) (1600–0359 UTC). A morning flash count for 0600–1159 EDT (1000–1559 UTC) also was calculated as a potential predictor of afternoon lightning.

Figure 3 shows the hourly distribution of flash count for the Miami–Dade domain for all warm season days (June–August) during the 16-yr period. A diurnal peak occurs between 1400 and 1500 EDT, with a rapid decrease after 1900 EDT. Similar diurnal variations have been observed in previous studies (e.g., Maier et al. 1984; Reap and MacGorman 1989; Livingston et al. 1996; Lericos et al. 2002; Mazany et al. 2002). Distributions for the other 10 areas are very similar, with only slight variations in the time of peak lightning. The NM forecast period considered in this study accounts for 85%–95% of the daily total in each FA. Although early afternoon is the most active lightning period, the evening hours were included so that FPL officials can better plan whether to retain extra crews after normal business hours.

c. Radiosonde data

Morning radiosonde data for Miami (MFL), Jacksonville (JAX), Tampa (TBW), and Cape Canaveral (XMR) were used to calculate various wind, moisture, and stability parameters to serve as candidate predictors for the regression models. Data for the years 1989–1999 were obtained from the Radiosonde Data of North America CD-ROM prepared by the Forecast Systems Laboratory (FSL) and the National Climatic Data Center (NCDC) (FSL and NCDC 1999). Data for the remaining years (2000–2004) were obtained directly from FSL’s “Radiosonde Database Access” Web site (FSL 2004).

A total of 597 parameters was calculated from the radiosonde data (Table 1), many of which have been found in previous studies to be useful predictors of thunderstorms and lightning during the warm season. They include variables describing wind direction and speed, moisture, temperature, and stability. Pressure-weighted layer averages of these variables also were calculated, as well as layer thickness and temperature lapse rate. This list of potential predictors will be greatly reduced in subsequent procedures described in section 3.

The sounding closest to each FA generally was used under the assumption that the closest sounding would be most representative of the conditions in that area. Correlations between the sounding parameters and lightning showed that this was indeed the case, with only a few exceptions. Specifically, parameters calculated from the MFL sounding generally were better correlated with lightning activity in the Charlotte and Lee areas than parameters from the closer TBW sounding. Since the subtropical ridge axis usually is located north of MFL, the low-level flow in these areas usually is from the southeast. As a result, atmospheric properties in the Charlotte and Lee areas tend to be more similar to MFL than TBW. Thus, better results would be achieved using the MFL sounding for Charlotte and Lee instead of TBW.

Figure 4 shows the sounding used for each forecast area. Prior to 1995, the JAX site was located in Waycross, GA (AYS). Since the JAX soundings are more representative of conditions in the Flagler–St. Johns area than AYS, it was decided that only JAX data from 1995 onward would be used for that area. In addition, due to the poor availability of XMR soundings prior to 1992, only the 1000–1200 UTC data from 1992 onward were used for the Volusia, Martin–St. Lucie–Indian River, and Brevard FAs. For all other areas, MFL–PBI or TBW data for 1989–2004 were used.

The equations being derived are for situations when the sea breeze is the dominant forcing mechanism for convection; they are not meant for days when large-scale forcing leads to thunderstorms. Therefore, an effort was made to remove these synoptically influenced days before equation development. This was done by deleting any day whose 1000–700-hPa layer average wind speed was greater than 3σ from the climatological mean. This simple procedure does not guarantee that every synoptically disturbed day was removed. Approximately 2% of the days during the June–August period were removed by this procedure.

d. Statistical software

Two statistical software packages were used. Most of the exploratory work was done using S-PLUS, version 6.1 for Windows, distributed by Insightful Corporation. Final model development and testing were performed using the Statistical Package for the Social Sciences (SPSS), version 11.5 for Windows, distributed by SPSS, Inc. Both are powerful, state-of-the-art software packages with a wide range of capabilities.

a. Predictands

The first objective of the study was to develop statistical guidance to predict whether at least one CG flash would occur during the NM period in each FA. The output from this equation is intended to aid FPL personnel in determining which areas are most likely to experience at least some lightning during the NM period, so that appropriate preparedness measures can be taken. Since a forecast of “yes” or “no” was sought, a binary indicator was assigned to each day in the dataset; “1” if at least one CG flash was observed during NM anywhere within each area, or “0” if no activity. This binary indicator served as the predictand for the yes–no equations.

The second objective was to develop equations to estimate the amount of lightning that would occur during the NM period, conditional on at least one flash occurring. A major decision was to determine the form of the predictand, that is, whether to forecast the actual flash count or to transform the counts into discrete categories and predict a range of counts. Our initial efforts focusing on eastern Miami–Dade and Broward Counties in south Florida showed that using multiple linear regression to estimate a flash count produced comparatively poor results. Instead, results indicated that predicting a range of flash counts was the best option. Therefore, the flash counts in each FA were grouped into four quartile categories based on climatology, with the quartiles used as the predictand. Flash count ranges for each quartile are shown in Table 2.

Rather than developing one model to forecast the quartile, the best results were achieved using separate equations to distinguish the lowest quartile of activity (Q1)134 from all other days, the highest quartile (Q4) from other days, and an equation to differentiate the upper two quartiles (Q3, Q4) from the lower two (Q1, Q2). Again, we sought a “yes” or “no” forecast for each of these outcomes, so three binary indicators were assigned to each lightning day (days with one or more flashes). That is, 1 was assigned to Q1 lightning events and 0 otherwise, 1 for Q4 events and 0 otherwise, and 1 for events in the upper two quartiles (Q3 or Q4) and 0 otherwise. These three equations then could be used to forecast the most likely quartile (explained in more detail in section 4).

b. Binary logistic regression

For situations when the outcome is binary or dichotomous (i.e., 1 for yes or 0 for no), the most often used technique is “binary logistic regression” (BLR; Hosmer and Lemeshow 1989). Let π denote the probability of a success for some outcome of interest (e.g., the occurrence of at least one CG flash). BLR relates this probability to a linear combination of predictor variables, X_K, by the following relations:

View Expanded

View Expanded

The term on the left side of (1) is the “logit link function,” which may be continuous and can range from −∞ to +∞ depending on the range of X_K (Hosmer and Lemeshow 1989). The probability of a success is then given by

View Expanded

and the probability of a failure (i.e., not observing at least one CG flash) is 1 − π.

BLR has less stringent assumptions than linear regression. Unlike multiple linear regression, BLR does not assume a linear relationship between the independent variables and the dependent (binary) outcome. Rather, the logit function in (1) is assumed to be linear in its parameters, although explicit interaction and power terms can be added as additional variables on the right side of (2). In addition, the form of (3) guarantees that BLR will always produce probability estimates bounded between zero and one inclusive (Hosmer and Lemeshow 1989).

c. Principal component analysis

It is clear that several of the parameters in Table 1 are highly correlated; in other words, they contain redundant information. For example, precipitable water is closely related to the 1000–500-hPa-layer average relative humidity, and the mean cross-shore wind component in a layer is highly correlated with the sine of the mean wind direction in that layer. Wilks (1995) cautions that estimates of the coefficients and standard errors can become unreliable and model performance can be adversely affected when highly correlated predictors compose the model. Thus, a method was needed to reduce the candidate predictors to only the most important variables without much loss of information. This was accomplished by performing a principal component analysis (PCA; Wilks 1995) on all potential sounding predictors (Table 1) using the SPSS software. PCA is a mathematical procedure that transforms a number of correlated variables into a smaller number of uncorrelated variables called principal components (PCs). In this study, the PCs were used as a classification method to cluster the highly correlated predictors into groups having physical meaning. As described in Wilks (1995), only components with eigenvalues >1 were extracted, and the sounding parameters having the greatest weights (or “loadings”) on each component were grouped together.

A total of five groups generally were formed through this process. The five groups contained parameters that described either wind direction, wind speed, moisture, or stability, with a final “miscellaneous” group containing variables that were not highly correlated with those in any other group. Finally, to determine which predictors to retain for the regression analysis, one parameter was chosen from each group that was the most physically relevant and had the greatest correlation with each of the four binary predictands (described in section 3a). This procedure ensures that only the most important and nonredundant predictors are retained in the dataset for possible selection by the BLR procedure.

Table 3 lists the final set of candidate physical predictors for each FA resulting from the PCA. The most common wind parameters are the sine of the layer-averaged wind direction, the layer-averaged cross-shore wind component, and the layer-averaged speed in the low levels. The K index (KI) often was the most important moisture-related parameter, while total totals (TT), Showalter stability index (SSI), layer temperature lapse rate, and low-level temperature were the most important stability parameters. The physical relevance of these parameters to lightning occurrence will be discussed in section 4a.

d. Additional candidate predictors

Table 4 (top) is a 2 × 2 contingency table for the number of days when at least one flash was observed in the Miami–Dade domain versus what occurred the previous day. A similar 4 × 4 contingency table is shown for the quartiles (bottom). Nearly 82% of lightning events are correctly forecast by persistence in the Miami–Dade area, while persistence correctly forecasts the quartile nearly 36% of the time. These results show that persistence is a powerful predictor of lightning during the warm season in Florida and must be included as a candidate predictor. In addition to the parameters listed in Table 3, we included a same day morning (0600–1159 EDT) and a previous day NM indicator of at least one flash, as well as the previous day’s lightning quartile and an indicator for the upper two or lower two quartiles. Since persistence typically produces a more accurate forecast than climatology, persistence will be used as the standard of reference for assessing the overall skill of the prediction equations derived in this study.

To incorporate possible nonlinear and interaction effects, power terms up to the fourth degree and two-way cross products were included as additional candidate predictors (e.g., Neumann and Nicholson 1972; Reap 1994). The power terms were calculated only for the PCA-selected physical variables, while the two-way cross products were calculated between all first-order parameters including persistence. To avoid collinearity problems among these terms, the physical variables first were normalized (i.e., subtract the mean and divide by the standard deviation) before raising them to a power or computing cross products (Wilks 1995).

e. Model building

Four logistic regression equations were derived for each of the 11 FAs. The first gave the probability of at least one CG flash occurring during the NM period in each area. Three additional logistic equations were derived to determine the most likely quartile of lightning, conditional on at least one flash occurring. One equation gave the probability of an event in the lowest quartile (Q1), one for the probability of the upper two quartiles (Q3 or Q4), and a final equation for the probability of the greatest quartile (Q4). Rather than having one equation for each quartile (four total), this three equation approach combined with a decision tree (described later) produced the best results. The logistic regression algorithm in SPSS was used to derive the equations and screen the variables (Table 3) for selection into each model.

A procedure combining forward stepwise screening and cross validation was used to derive each of the four equations. The process began by randomly dividing the dataset into two parts. One set, containing 75% of the cases, served as a “learning” sample for screening the predictors for selection. The remaining 25% were reserved as an “evaluation” sample to test the model each time a new variable was added or removed during the stepwise selection process.

The screening procedure in SPSS uses “forward conditional” stepwise selection, with a test for backward elimination. The first independent variable selected is that which produces the greatest reduction in the residual sum of squares (or residual deviance, RD), that is, the variable that explained the most variation in observed lightning. The algorithm then selected the next variable, which, together with the first, further reduced the RD by the greatest amount. At each step, the algorithm performed a backward check to determine if the addition of a variable caused any previously selected variable to become insignificant (i.e., if the p value of the variable became >0.1), in which case the insignificant variable was removed. This process continued until the RD could no longer be reduced by a significant amount, or until no other variables remained.

At each step in the sequence, a 2 × 2 contingency table was produced to show the percentage of correctly classified days for both the 75% learning sample and the 25% evaluation sample. The predictors composing the model at the step with the highest percentage of correctly classified days in the 25% evaluation sample were noted. A new random sample then was generated, and the stepwise screening was repeated. This resampling and rescreening continued until the combination of predictors was identified that most likely generalizes to independent data, and does not overfit the dependent sample. These “best” predictors identified by the resampling then were reentered for stepwise screening on the entire working dataset (no 25% and 75% random sampling) to obtain the final four logistic equations.

After final equations were obtained for the probability of the lowest quartile, upper two quartiles, and highest quartile, a decision tree was constructed to determine the most likely quartile using probability thresholds for the three equations. To produce an unbiased scheme, the thresholds were chosen so an equal number of cases was partitioned to the left and right at each split of the decision tree. This guarantees that the scheme will not have a prediction bias toward any one quartile (i.e., a tendency to forecast a particular quartile more often than another). Further details about the decision tree scheme are given in section 4.

a. Final logistic equations

The final equations for the 11 FAs generally are a variation on the same theme; therefore, this section only presents results for the Miami–Dade area. Table 5 displays the final equations giving the probability of at least one CG flash [Eq. (1)] and the conditional probability of the lowest quartile [Eq. (2)], the upper two quartiles [Eq. (3)], and the greatest quartile [Eq. (4)] for the Miami–Dade domain. The predictors in each equation are given along with their coefficient (B) and standard error (S.E.), as well as the p value, which indicates the significance of each term and its relative predictive importance. The p value tests the null hypothesis that there is no relationship between the independent variable and the log-likelihood of observing a 1 in the binary dependent variable (also called the log-odds) (Hosmer and Lemeshow 1989). For example, a p value of 0.001 indicates that there is only a 1/1000 chance that the relationship found in the sample would not also be true in the population, indicating that the parameter has “statistical significance” (Wilks 1995). The p values in Table 5 show that all of the coefficients exceed the 95% significance level, and all but a few exceed the 99% level, providing strong evidence that the parameters are significant and belong in the equations.

It is informative to describe the physical significance of each parameter in the equations (Table 5) and their relationships to lightning activity. The most often selected physical predictors are the sine of the vector-averaged wind direction in the 1000–800-hPa layer (SINDIR) and the average wind speed in the 1000–900-hPa layer (MNSPD). Their selection is not surprising, since previous studies have documented that the magnitude and direction of the prevailing low-level wind with respect to the coastline have a significant influence on the strength and inland penetration of the sea breeze (e.g., López and Holle 1987; Reap 1994; Lericos et al. 2002).

In all equations except Eq. (2) the coefficient of SINDIR is negative (Table 5). Since the sine of angles between 180° and 360° is negative, an offshore, low-level wind increases the probability of afternoon lightning and increases the likelihood of a Q3 or Q4 event in the Miami–Dade area. Conversely, the positive coefficient in Eq. (2) indicates that Q1 events are less likely for offshore flow (SINDIR < 0) and more likely for onshore flow (SINDIR > 0). The coefficients for MNSPD in Eqs. (1) and (3) also are negative, suggesting that as the low-level wind speed increases, the probability of at least one flash and the likelihood of upper two quartile events decreases. This result is consistent with Camp et al. (1998) and Arritt (1993) who found that onshore wind speeds exceeding several meters per second and offshore speeds greater than 11 m s⁻¹ suppress sea-breeze development in areas near the coastline. Conversely, weak offshore flow produces a strong sea-breeze circulation whose leading edge remains near the coastline. In eastern Miami–Dade County this offshore scenario can produce extensive, slow-moving thunderstorms and high flash count events if adequate moisture and instability are present.

It is interesting that a nonlinear (cubic) term with a positive coefficient was selected for SINDIR in Eqs. (3) and (4) (Table 5), in addition to the first-order term. This relationship is depicted in Fig. 5a, which plots the log-odds of a Q4 lightning event versus SINDIR if only the first-order and cubic terms in Eq. (4) are considered, that is, setting all other variables in the equation equal to zero. The figure indicates that the log-odds of a Q4 event is maximized for SINDIR between −0.65 and −0.45, with diminishing log-odds as SINDIR increases. This maximum corresponds to wind directions between 205°–220° (SW) and 320°–335° (NW). Northwest flow is uncommon during June–August in south Florida. Therefore, SW flow likely is the greatest contributor to the maximum in the log-odds of Q4 events. A SW (offshore) flow transports subtropical moisture northward into south Florida and opposes the sea breeze, producing enhanced convergence along the sea breeze and widespread thunderstorm and lightning activity in eastern Miami–Dade County.

The Showalter stability index (SSI) was selected in Eqs. (1) and (3). SSI is similar to the lifted index except the parcel is lifted from 850 hPa instead of the surface, with values becoming more negative as instability increases. The negative coefficients indicate that as instability increases, the likelihood of at least one flash and a Q3 or Q4 event increases. Studies by Livingston et al. (1996) and Lambert et al. (2005) also found SSI to be a useful predictor of afternoon lightning.

The KI appears only as a quadratic term in the three quartile equations. Figure 5b plots this quadratic relationship between the log-odds of a Q4 event and KI for the Miami–Dade domain, if all other parameters in Eq. (4) are set to zero. Clearly, the likelihood of a Q4 event increases with increasing KI until a peak is reached between 25° and 30°C. Then, the likelihood of a Q4 event decreases for larger values of KI. Since KI increases with more unstable midlevel lapse rates and greater middle-tropospheric moisture, it is reasonable that convection and lightning also will increase. The reason for decreasing log-odds for KI values greater than ∼30°C is uncertain, but may be due to excess midlevel moisture and cloud cover from early morning convection (i.e., at or near the sounding time), which would tend to suppress surface heating and strong afternoon activity. The decreasing odds for larger KI values may also be due to lightning being the predictand, as opposed to updraft speed or rainfall, which generally increase with instability.

As expected, persistence was selected as a predictor of afternoon lightning in the Miami–Dade area (Table 5). For the probability of at least one flash [Eq. (1)], both the morning and previous day indicators were chosen. The positive coefficients suggest that the likelihood of at least one flash during the NM period increases if at least one flash occurred the previous day, or if at least one flash occurred from 0600 to 1159 EDT in the morning. The morning persistence indicator also appears in Eq. (2) for the probability of a Q1 event. The negative coefficient implies that Q1 events become less likely, and thus Q2 or greater events become more likely, if there was at least one flash during the morning. The previous day quartile indicator was selected in Eqs. (2) and (4), and the persistence indicator for the upper two or lower two quartiles was chosen for Eq. (3). The signs of their respective coefficients indicate that higher flash count events are more likely if the previous day also had a high flash count. Meteorological conditions in south Florida during the warm season often change little from day to day. Thus, if conditions were favorable for lightning on the previous day, conditions on the current day often are similar. Lightning activity during the morning suggests that outflow boundaries may be present during the afternoon. These boundaries can enhance low-level convergence by interacting with the sea-breeze circulation.

Interaction terms were selected in three of the four guidance equations (Table 5). Such terms appear when the effect that one independent variable has on the response (i.e., lightning) is modulated by changes in another independent variable. For example, in Eq. (4) the effect of persistence (previous day quartile) on the likelihood of Q4 events is modulated by MNSPD and SSI. The negative coefficients suggest that decreasing values of MNSPD and SSI reinforce the positive relationship between persistence and the likelihood of Q4 events. Conversely, an increase in MNSPD or SSI counteracts the positive effect of persistence. Thus, these interaction terms serve to prevent persistence from having undue influence on the forecast if current atmospheric conditions are unfavorable for a Q4 event, or enhance its contribution to the forecast if conditions are favorable.

b. Results for dependent data

1) Yes–no equations

The BLR equations provide a probability ranging between zero and one. To forecast whether at least one CG flash will occur during the NM period, a threshold probability must be determined. Then, if the calculated probability exceeds this threshold, at least one flash is forecast to occur; otherwise, no lightning is forecast. The optimum threshold was determined using verification scores from a 2 × 2 contingency table giving the number of days when at least one flash was observed compared to the number predicted using varying trial thresholds. These scores include the probability of detection (POD), hit rate (HR), false alarm ratio (FAR), bias, critical success index (CSI), and the percentage of nonlightning events correctly forecast. The POD is the ratio of the number of events correctly predicted by the model to the total number of observed events in the sample. The HR is the most direct measure of accuracy for categorical (yes–no) forecasts, indicating the percentage of correctly predicted events and nonevents. The FAR is a measure of the forecast events that fail to occur. The bias B indicates the degree of overforecasting (B > 1) or underforecasting (B < 1) an event. Finally, the CSI combines attributes of the POD and FAR, and can be viewed as a HR for the event being forecast after removing correct no forecasts from consideration (Wilks 1995). These quantities are further defined in Reap (1994) and Mazany et al. (2002). Table 6 shows a sample contingency table with formulas used in computing these scores.

Figure 6 shows how these statistics vary using different thresholds for eastern Miami–Dade County. Except for CSI, HR, and the percentage of nonevents correctly forecast, values decrease as the threshold is increased. Based on Reap (1994), we sought to maximize the CSI and POD while minimizing the FAR and capturing as many of the nonevents as possible. This latter consideration was used because results showed that the BLR scheme better forecast days with observed lightning than days without lightning. We found that the hit rate was improved by sacrificing some accuracy forecasting days with lightning in order to improve the forecasts of days without lightning. Based on the above considerations, a threshold of 50% was chosen for the Miami–Dade model. Thresholds for the other 10 forecast areas (not shown) ranged from 45% to 60%.

Table 7 shows a 2 × 2 contingency table (top) and statistics (bottom) for all 16 warm seasons of dependent data for eastern Miami–Dade County, using the optimum probability threshold of 50%. The scores are quite good, with a CSI of 77%, POD of 92%, a bias near 1, and a low FAR of 17%. Also shown is the CSI based on a persistence forecast (Table 3), along with a skill score (SS_mod) calculated from the model CSI and the persistence CSI,

View Expanded

The skill score is positive (25.4%), indicating that forecasts made by the model are superior to persistence. Table 8 contains CSI and skill scores for all 11 FAs. All of the skill scores are positive, ranging from 15.6% in the Charlotte FA to 31.9% in the Volusia area. Thus, all model forecasts based on the dependent data are superior to persistence. The variations in skill score can be attributed to factors such as differences in the skill of persistence in each FA, the size of each FA, and the proximity of the radiosonde site being used.

c. Cross validation

The results presented in section 4b (and those in Tables 7 –11) are for all 16 warm seasons of dependent data. That is, the results show the predictive accuracy of the equations when applied to the same data that were used to derive them. These results do not assess how well the guidance equations will predict cases that were not involved in equation development. To estimate the performance of the equations on independent data, a k-fold cross-validation (CV) procedure was followed. This involved withholding one warm season of data at a time for testing, while using the remaining 15 warm seasons to rederive the equations (following the same procedure outlined in section 3e). The process was repeated 16 times, once for each warm season. Since the CV procedure is both tedious and time consuming, it was performed only for the Flagler–St. Johns (FSJ) and Charlotte FAs. Since the FSJ models achieved one of the best skill scores of the 11 areas, while those for Charlotte were among the least skillful (Tables 10 and 11), it is reasonable to assume that the CV skill scores for the remaining nine areas will lie somewhere in between.

For the yes–no equations, the CV results (Table 12) for both areas produce only a slight reduction in SS_mod of between 0.6% and 1.2% compared to the dependent data. These scores range from 31.0% in the FSJ area to 14.4% in the Charlotte area. The quartile equations (Table 13) exhibit a somewhat larger reduction in SS_mod compared to the dependent data. For the hit rate, skill scores range from 16.5% in FSJ to only 1.4% in the Charlotte area, a reduction of between 3.5% and 5.0%. For the percentage correct to within one quartile of the observed, skill scores range from 26.4% in the FSJ area to 21.7% in Charlotte, a reduction ranging between 4.6% and 5.9%. These results are surprisingly good for independent data and are likely a consequence of the random sampling and testing procedure that was used to derive the original equations (section 3e). The CV results suggest that the guidance equations are statistically robust, and can be expected to provide useful guidance when implemented operationally.