1. Introduction
Three out of every four storm-based tornado warnings issued during 2008 were false alarms, warnings unable to be confirmed as having produced a tornado (NWS 2009). Recognizing tornado false-alarms as a serious problem, the false-alarm ratio (FAR), the ratio of tornado warnings with no confirmed tornadoes to the total number of tornado warnings, is one of several important performance measures that can be used for tornado warning verification (Brooks 2004). Since the National Weather Service (NWS) verification program began in 1986, only a slight improvement in the FAR is evident (McCarthy and Schaefer 2004; Brooks 2004), despite marked improvements with the probability of detection (POD, the fraction of confirmed tornadoes that are warned) in association with the introduction of the Weather Surveillance Radars-1988 Doppler (WSR-88Ds; Crum and Alberty 1993) and NWS modernization (Friday 1994).
There has been some debate as to the importance of the FAR as a warning metric. Barnes et al. (2007) cite numerous studies that showed false alarms to be harmless and, in fact, educational. Barnes et al. also point out several limitations of the FAR, including its inability to reflect “close calls” and the inexactness of the metric due to poor verification across sparsely populated areas. Nevertheless, a study by Simmons and Sutter (2009) identified a direct causal link between the average tornado false-alarm ratio of an area and the average rate of death and injury caused by tornadoes. Thus, a reduction in the FAR is accompanied by a reduction in the number of tornado fatalities and injuries and is subsequently an important measure to understand.
One of the National Oceanic and Atmospheric Administration’s (NOAA’s) goals for 2025 is to provide an average 60-min lead time for tornadoes (Berchoff 2009). This goal of providing advance warning prior to tornadogenesis (i.e., “warn-on forecast”) requires that warnings be issued based upon the anticipation of tornado development. However, warning a priori to the detection of well-known tornado precursors may well increase the likelihood of increased numbers of false alarms. Thus, limiting false alarms may become an even greater challenge in the years ahead if warning lead time is extended.
At least three common factors lead to the issuance of tornado false alarms. First, a tornado could be ongoing upstream, but as an existing warning is extended for an area downstream, the tornado dissipates and no confirmed touchdown is identified in the newly warned area. Second, a circulation signature is identified by weather radar or confirmed by spotters, and a warning is issued, but the storm fails to produce a tornado in the warned area. Third, a tornado does occur within the warning area, but the tornado is never verified. The first and second scenarios may be caused by a combination of a lack of detailed observations at the miso- and mesoscales and/or low levels, and a lack of storm-scale knowledge and predictability at those scales. The third scenario reflects the difficulty and resources required for thorough tornado verification.
The purpose of this study is to analyze a 5-yr climatology of tornado false alarms as one step toward better understanding some of the underlying causes on why these warnings were issued. This study examines the trends in tornado warning FARs according to the diurnal and seasonal influences, geographic region and weather forecast office (WFO), distance from radar, county population density, and county area. Similar to Brotzge and Erickson (2009, 2010; hereafter, BE09 and BE10, respectively), tornado and tornado warning data from 2000 to 2004 were analyzed to identify factors influencing the issuance of unconfirmed tornado warnings. This is a complementary study to those of BE09 and BE10, which (using the same dataset) investigated tornado warning lead times and unwarned tornadoes.
2. Data
A database containing the tornado warning and event information was provided by the Performance Branch of the NWS. This database contains information regarding tornado warning issuance and expiration dates and times, the WFO that issued the warning, the county or parish warned, and, when applicable, the corresponding tornado reports (including F-scale rating, damage, and casualties). A total of 18 763 tornado warnings were analyzed from the 5-yr period, of which 13 593 (72.4%) were false alarms. Estimates of county population density were provided by the Population Division of the U.S. Census Bureau and were obtained from the 1 July 2000 population estimates (U.S. Census Bureau 2008).
As with all tornado studies, several caveats should be mentioned. First, the tornadoes recorded in this study are only those verified by the NWS WFO, and it can be assumed that a significant number of tornadoes did occur but were never documented (BE10). Thus, an unknown percentage of false-alarm warnings may have been erroneously classified as “false.” Second, Trapp et al. (2006) have shown numerous errors in the postevent reporting of severe winds, and it may be safe to assume at least some amount of error from the tornado reporting as well. Thus, there is some amount of uncertainty associated with the data used in this study.
During the 2000–04 period, tornado warnings were issued by county. A false alarm counts as any tornado warning issued for a county in which no tornado is known to have occurred. Thus, a warning issued for multiple counties is verified separately for each county. A tornado occurring in only a subset of those counties would be verified as a false alarm for the remaining counties. The current warning system is now storm based (Waters et al. 2005), which could lead to significantly different results than those presented herein.
3. Climatology
In an effort to understand some of the possible factors contributing to false alarms, a climatology of the 5-yr dataset of tornado warnings was constructed. The false alarms were sorted by hour of the day, month of the year, geographical region and WFO, number of tornadoes observed on a day in which a false alarm was issued, distance of the warned area from the nearest NWS radar, and county population density and area.
a. Diurnal characteristics
The false alarms were sorted by time of day (Fig. 1a), and a mean percentage of these false alarms was calculated hourly. The hourly averages were subtracted from a daily mean of 72.4% (Fig. 1b). A vast majority of false alarms occurred between 1300 and 2100 local time (LT), which is when most of the tornadoes during the 5-yr period occurred. However, interestingly, the percentage of false alarms was less than the daily mean during these peak tornado hours (1500 and 1700–2100 LT). During nonpeak hours for tornado occurrence (2200–1300 LT), the hourly false-alarm ratios were above the daily FAR mean at 75%–80%. The relatively noisy deviations from the daily FAR average during the overnight and early morning hours are indicative of the relatively small sample size per hour.
(a) Hourly totals of the number of false alarms, confirmed tornadoes, and those tornadoes without NWS tornado warning. (b) The hourly FAR expressed as a percentage minus the daily FAR average of 72.4%.
Citation: Weather and Forecasting 26, 4; 10.1175/WAF-D-10-05004.1
As Fig. 2a indicates, the hourly PODs are strongly associated with the hourly FARs. The hourly POD increases during the peak hours of tornado occurrence, while the hourly FAR decreases. Outside of this time, the hourly PODs are lower and the FARs higher. This relationship is best revealed by plotting the diurnal oscillation of hourly POD and success ratio [SR, where the SR is defined as 1 FAR; Roebber (2009)], as shown in Fig. 2b. The POD is at a maximum at 2000 LT and at a minimum at 0500 LT, whereas the SR is maximized at 1900 LT and at a minimum at 0800 LT. Both POD and SR are highest during late afternoon and at their lowest during the early morning hours.
(a) Hourly FAR and POD calculated as a percentage. (b) Hourly estimates of POD plotted as a function of SR.
Citation: Weather and Forecasting 26, 4; 10.1175/WAF-D-10-05004.1
b. Monthly characteristics
The data were sorted by month (Fig. 3a), and monthly average percentages of false alarms were subtracted from the annual average percentage (Fig. 3b). A majority of false alarms occur during the spring storm-season months of April–June. Similar to the hourly analysis, a strong association exists between the total number of false alarms and the total number of tornadoes each month. However, an examination of monthly anomalies (Fig. 3b) show that the FAR during May, June, September, and October was actually below the annual average.
(a) Monthly totals of the number of false alarms, confirmed tornadoes, and those tornadoes without NWS tornado warning. (b) The monthly FAR expressed as a percentage minus the monthly FAR average of 72.4%.
Citation: Weather and Forecasting 26, 4; 10.1175/WAF-D-10-05004.1
Relatively high monthly FARs are associated with lower monthly PODs (Fig. 4a). Monthly plots of POD and SR (Fig. 4b) show a biannual trend, with POD and SR reaching relative minima during the coldest and hottest periods of the year.
(a) Monthly FAR and POD calculated as a percentage. (b) Monthly estimates of SR plotted as a function of POD.
Citation: Weather and Forecasting 26, 4; 10.1175/WAF-D-10-05004.1
c. Geographical distribution
Tornado warnings and tornado warning lead times vary as a function of the region in which they occur (BE09, BE10). As a first step toward understanding the impacts of geography on false alarms, the data were sorted into four geographical regions: Southeast, Midwest/East, Plains, and West (Fig. 5). A review of the false-alarm statistics by region (Table 1) indicates that the Plains and West regions have relatively lower FARs while the Midwest and Southeast have significantly higher values. As described in BE10, some variation among regional warning statistics is likely due to meteorological variability in storm type and seasonal and diurnal climatologies. However, there are several contributing nonmeteorological factors for these regional differences, and these reasons are discussed in detail in successive sections.
All data were divided among four geographic regions: Southeast, Midwest/East, Plains, and West.
Citation: Weather and Forecasting 26, 4; 10.1175/WAF-D-10-05004.1
Distribution by region of all confirmed tornadoes and false alarms between 2000 and 2004 with and without NWS tornado warnings, listed by column. Data from HI and AK are not included. Confidence intervals (CIs) are shown for POD and FAR.
A review of POD and SR statistics from individual WFOs (Fig. 6a) shows that the mean office statistics vary widely, with POD ranging from 10% to 90%+ and SR ranging from 10% to 60%. As section 3d will discuss in greater detail, the mean POD (and to a lesser extent the FAR) is largely determined by the total number of tornadoes observed by a WFO (Fig. 6b). Those WFOs with less than 10 tornadoes confirmed during the 5-yr period recorded a POD less than 20% and an SR less than 20%, while those WFOs with greater than 125 tornadoes during the period recorded a POD of nearly 80% and an SR greater than 30%. In general, when more tornadoes occurred within a WFO’s county warning area (CWA), the POD (and to a lesser extent the SR) was significantly higher. As will be shown, this trend is most likely due to the difficulties involved with warning (and not warning) on marginal, weak, and isolated events.
(a) The 5-yr estimates of SR and POD approximated for each WFO and region. (b) The 5-yr estimates of SR and POD approximated per WFO cluster, sorted by the number of tornadoes confirmed during the 5-yr period.
Citation: Weather and Forecasting 26, 4; 10.1175/WAF-D-10-05004.1
d. Impact of number of tornadoes per day
BE09 showed that the percentage of tornadoes with zero and negative lead times decreased as the number of tornadoes per day increased. Furthermore, seasonal variations in the number of zero and negative lead-time warnings were related to the number of monthly multitornado days. BE10 showed that the relative number of unwarned tornadoes typically increased on days with fewer reports of tornadoes. Such relationships between the number of tornadoes and certain warning statistics suggest a similar association for false alarms. Several root causes may be responsible for a lower POD and higher FAR during less intense, more isolated events: weaker storm environment, less clear conceptual model, and, often times, a greater variety of circumstances (e.g., off season, late night or early morning, rare geographic locations).
The total number of tornado false alarms and the FAR were estimated as a function of the number of tornadoes per tornado day (Table 2). For clarification, each separate county tornado event (as reported in the NWS verification records) counted as a single tornado. For example, if on one day a single tornado crossed three counties all within a single WFO, then that day would be counted as three tornadoes reported within that WFO for that day.
Tornado statistics listed as a function of the number of confirmed tornadoes per tornado day per WFO with and without NWS tornado warning.
On days with no reported tornadoes, 4300 tornado warnings were issued during the 5-yr period, accounting for nearly a third of the false alarms in the dataset. When one tornado was observed within a CWA on a given day, the FAR was 78.2%. Not only was the FAR particularly high for these days, the number of unwarned tornadoes (739) was also particularly high, accounting for well over 50% of the tornadoes observed on these days and almost 40% of the total number of unwarned tornadoes during the 5-yr period.
On days with two, three, or four tornadoes reported per WFO, the FAR dropped to between 65% and 70%, while the POD rose from 66.7% for two tornadoes per day to 78.1% for four tornadoes per day. The FAR and POD continued to improve with increased numbers of tornadoes per day per WFO, with the FAR estimated at less than 40% for days with at least 20 tornadoes and the POD estimated at over 90%. Five-year estimates of POD and SR sorted by the average number of tornadoes per day (Fig. 7) showed clear improvement in their verification statistics during larger-scale outbreak events.
The 5-yr estimates of SR and POD approximated per WFO cluster, sorted by the average number of tornadoes per tornado day as confirmed during the 5-yr period.
Citation: Weather and Forecasting 26, 4; 10.1175/WAF-D-10-05004.1
The number of tornadoes per tornado day impacts the geographical distribution of false alarms. PODs and FARs were calculated for each region for days with zero to four tornadoes and for days with five or more tornadoes (Table 3). As expected, the POD was relatively low and the FAR rather high for days with zero to four tornadoes. As discussed in BE10, the POD was lowest across the West (47.2%) and highest across the Plains (69.8%). The FAR varied less among the geographic regions, ranging from 77.3% to 86.1%. However, a different pattern emerged when considering days with five or more tornadoes. The FAR improved for all areas, but with greater variability between regions, ranging from only 40.1% in the West to 65.0% in the Southeast. Interestingly, the POD was nearly the same across all regions on days with five or more tornadoes, with POD ranging from 82.7% in the Midwest to 87.5% across the Plains. It appears that once an outbreak was recognized, the POD was very high regardless of geographic region.
Distribution by region of all confirmed tornadoes and false alarms between 2000 and 2004 with and without NWS tornado warnings, listed by column. Data from HI and AK are not included.
e. Impact of distance from radar
Warning performance degrades with distance from radar because of limited low-level coverage due to terrain blockage and the earth’s curvature (BE09, BE10). To ascertain the impact of radar distance on FAR, the POD and SR were calculated as a function of distance from radar (Fig. 8). The mean POD decreased slowly with increasing distance from the radar at a rate of 0.3% (10 km)−1 within 150 km of the radar (Fig. 8a), with a much faster drop beyond 150 km. The SR also was relatively steady within 150 km of the radar but rapidly increased thereafter.
(a) The POD and SR plotted as a function of radar distance. (b) Data binned by distance from radar and plotted as a function of SR with 95% CIs marked based upon the sample sizes.
Citation: Weather and Forecasting 26, 4; 10.1175/WAF-D-10-05004.1
To better understand the statistical significance of the increase in SR with distance, the data were binned into four categories of radar range (Fig. 8b). For those tornado warnings ≥150 km, the SR rose to 46.5%, a statistically higher SR value than was found from warnings made within 150 km of radar.
Several factors may be limiting the FAR beyond the 150-km radar range. First, the lower POD and FAR values indicate that fewer warnings were issued for areas far from radar. BE10 showed that POD dropped significantly beyond 150 km from radar, likely due to poor postevent verification in sparsely populated counties far from radar. Second, at distances beyond 150 km forecasters have limited low-level radar coverage and are forced to rely on alternative information from storm spotters, environmental parameters, and in situ and satellite data. This may lead to greater reluctance to issue warnings or perhaps even improved decision making in some circumstances. Other factors that may contribute to a lower FAR with distance include the dependence on county population density and county size. These factors are discussed next.
f. Impact of county population density
BE10 recognized county population density as having a significant impact on tornado verification and subsequent warning percentages (see Fig. 7 and Table 6; BE10). A similar analysis was applied in this study to tornado false alarms. The false-alarm warning data were binned according to the county population density and then plotted as a function of SR (Fig. 9a). Tornado warning false alarms were relatively high (77.0%) for very high-density population counties (≥400 persons km−2) and statistically significantly lower (65.9%) for counties with fewer than 5 persons km−2. This suggests that forecasters may be more willing to issue warnings over densely populated counties due to the threat to public safety. On the other hand, NWS forecasters may be less inclined to issue tornado warnings for sparsely populated areas, possibly because of the difficulty involved in the verification process. Binned data also were plotted as a function of SR and POD (Fig. 9b). As shown in BE10, the POD decreased with increasing population density likely because of more thorough postevent verification. King (1997) found from a historical study across southern Ontario that a population density threshold of at least 6 persons km−2 is needed in order to provide a comprehensive tornado climatology. However, Anderson et al. (2007) found such population influences vary geographically.
(a) Data binned by county population density and plotted as a function of SR with 95% CIs marked based upon the sample sizes. (b) Data binned by county population density, and then plotted as a function of POD and SR.
Citation: Weather and Forecasting 26, 4; 10.1175/WAF-D-10-05004.1
As discussed, the impacts of county population density on the false-alarm ratio vary with distance from radar. To quantify this relationship, the false-alarm data were binned according to the warning distance from radar and county population density (Table 4). For counties within 50 km of radar, the FAR was high (~80%) for densely populated counties and low (~60%) for rural counties. For counties within 50–150 km of radar, the FAR generally decreased for counties with ≥50 persons km−2 and increased for counties with <50 persons km−2. At distances ≥150 km from radar, FAR dropped significantly for all counties. As explained above, these results indicate that for radar signatures observed within 50 km, warnings were readily issued over densely populated counties and less frequently over rural counties. At distances greater than 150 km from radar, the impact of population density appears secondary to radar distance.
Percentage of tornado warnings issued without a confirmed tornado touchdown (false alarms), as a function of radar distance (km) and county population density (persons km−2). Sample size for each category is shown in parenthesis. Percentages are not shown for sample size less than 25. Category averages are highlighted in bold.
g. Impact of county area
The false-alarm rate is sensitive to the ability to detect and/or predict tornado formation within a specified geographic area. As such, the smaller the county size, the more difficult it is to predict tornado formation within that area. To quantify this relationship, the SR was estimated as a function of county size (Fig. 10a). The SR decreased significantly for counties less than 2000 km2 in area. The SR was less than 20% for counties less than 1000 km2.
(a) Data binned by county area and plotted as a function of SR with 95% CIs marked based upon the sample sizes. (b) The median county population density (persons km−2) and median county area (km2) plotted as a function of distance (km) from the nearest WSR-88D.
Citation: Weather and Forecasting 26, 4; 10.1175/WAF-D-10-05004.1
Most WSR-88D radars were originally placed near major metropolitan areas, leaving low-level coverage gaps across some rural areas and particularly across portions of the western United States. This placement philosophy introduces some dependency between county population density, county area, and distance from radar (Fig. 10b). To explore this relationship further, FAR was calculated as a function of county area and distance from radar (Table 5). FAR statistics revealed that the smallest counties had a high FAR (~80%) regardless of the distance from radar. For larger counties, the FAR remained relatively steady at above 70% within 150 km of radar. For radar ranges beyond 150 km, the FAR remained near 50% for counties greater than 2000 km2.
Percentage of tornado warnings issued without a confirmed tornado touchdown (false alarms), as a function of radar distance (km) and county area (km2). Sample size for each category is shown in parentheses. Category averages are highlighted in bold.
County area plays a significant role in explaining the FAR differences across geographic regions. To quantify this impact, FAR was estimated as a function of county size for each of the four geographic regions (Table 6). As expected, FAR was highest for the smallest area counties for each of the four regions. The lowest FAR estimates across the Southeast and West were observed with the larger counties. Overall, however, it is the distribution of county size across each region that skewed the regional FAR averages. The relatively large average county sizes across the Plains and West appear to have improved those regions’ FAR statistics when compared to the much smaller county sizes found across the Midwest/East and Southeast.
Percentage of tornado warnings issued without a confirmed tornado touchdown (false alarms), as a function of geographic region and county area (km−2). Sample size for each category is shown in parentheses. Percentages are not shown for sample size <25. Data from HI and AK are not included. Category averages are highlighted in bold.
4. Discussion and conclusions
Tornado warning false alarms from 2000 to 2004 were analyzed. The warnings were sorted by time of day, month, geography, WFO, total number of tornadoes per day, distance from radar, county population density, and county area. Analysis of these data yielded several findings.
The false-alarm ratio was highest during nonpeak storm periods and at its lowest during peak storm times and days (Figs. 1 and 2). For example, the FAR was highest during the overnight hours and morning (2300–1200 LT) and lowest during the late afternoon hours (1700–2100 LT). Similarly, the FAR was at its highest during the winter (December–February) and late summer (August) (Figs. 3 and 4). These statistics reflect the difficulty in warning at unusual times and seasons and warning on a greater number of isolated (more marginally severe, nonoutbreak) events.
The FAR varied geographically, with lower ratios in the Plains and West and higher ratios across the Midwest and Southeast (Table 1).
WFOs with a relatively large (small) number of tornadoes during the 5-yr period generally had higher (lower) PODs and slightly lower (higher) FARs (Fig. 6). The more isolated and rare the tornadoes, the greater was the fraction of warnings that were false alarms. However, some regional differences remained even once the effects of outbreaks were removed (Table 3).
Tornado warning POD and SR improved with the number of tornadoes per tornado day (Fig. 7, Table 2). Nearly one-third of all false alarms were issued on days when no tornadoes were confirmed within the WFO’s CWA. False alarms accounted for over 86.5% of the total number of warnings issued on days with two or fewer observed tornadoes in the WFO’s CWA during the 5-yr period. In comparison, false alarms dropped to 56.3% of all warnings issued for days with five or more tornadoes in the WFO’s CWA.
The tornado FAR decreased significantly beyond 150 km from the nearest WSR-88D (Fig. 8). A drop in POD and FAR beyond 150 km suggests a decrease in the number of tornado warnings issued at these distances. Forecasters may be more hesitant to issue warnings for storms in areas with limited or blocked low-level radar coverage or may be reluctant to issue warnings over areas where postevent verification is difficult (BE10).
The tornado FAR was high (>80%) over densely populated counties (≥400 persons km−2) and statistically significantly lower over sparsely populated counties (<5 persons km−2) within 50 km of radar (Fig. 9, Table 4). While FAR increased with increasing population density within 50 km of radar, FAR decreased for all counties regardless of population density for radar ranges greater than 150 km.
SR improved as county area increased (Fig. 10, Table 5). As expected, more precision is needed for a smaller target area. For the smallest counties (<1000 km2), FAR was ~80% regardless of distance from radar. FAR was generally lower for larger counties, particularly for those counties ≥150 km from radar (~50%). The much larger average county size across the Plains and West contributed toward a lower average FAR for those regions (Table 6).
Several implications can be inferred from these results. First, the relative number of false alarms was greatest during those same periods of time when the greatest ratio of unwarned tornadoes occurred (BE10). In other words, there are certain periods of time (e.g., at night, during the winter, during marginally severe storm events) when tornado detection and warning is much more challenging and simultaneously more dangerous to the general public. Put yet another way, there are periods of limited tornado predictability where low POD and high FAR can be expected. Positively, the FAR was lowest during peak tornado periods, such as the late afternoon and during outbreak scenarios. These variations in tornado predictability are intricately tied to technological capabilities and limitations, spotter availability, and the state of scientific knowledge.
Second, these findings highlight a significant limitation of current weather radar technology and our national warning capability. With increased distance from radar, forecasters must rely more heavily on other environmental information (e.g., spotters, surface observations, and satellite imagery). However, this lack of low-level radar data appears to lead to fewer warnings being issued, which in turn leads to a lower FAR but also a lower POD. While a low FAR is ideal and shown by Simmons and Sutter (2009) to be tied to reduced tornado fatality and injury rates, it is also shown by BE10 that the numbers of tornadoes far from radar, across relatively sparsely populated regions, are greatly underestimated due to the difficulties associated with postevent verification. This hesitation for issuing warnings far from radar (in the absence of low-level coverage) reflects the gaps in the current weather radar network and highlights the value of low-level weather radar coverage to the issuance of tornado warnings.
Third, some of these identified trends can be expected to change with the move from county-based warnings to storm-based warnings. The NWS transitioned from county-based warnings to a storm-based warning system on 1 October 2007. Results from Fig. 10 suggest that false-alarm rates may remain correlated to the size of the warning area with smaller polygon warnings having a higher false-alarm rate. On the other hand, storm-based warnings may better encompass projected storm motion and thereby reduce the number of “near misses,” which could result in an overall decrease in the false-alarm rate. A detailed analysis of the false-alarm ratio from storm-based warnings is recommended.
Acknowledgments
We thank Brent Macaloney at NWS Headquarters for supplying us with the tornado record data used in this study, Liz Quoetone at WDTB for insightful discussions, and three anonymous reviewers for their helpful comments. This work is supported by the Engineering Research Centers Program of the National Science Foundation under NSF Award 0313747. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the National Science Foundation.
REFERENCES
Anderson, C. J., Wikle C. K. , Zhou Q. , and Royle J. A. , 2007: Population influences on tornado reports in the United States. Wea. Forecasting, 22, 571–579.
Barnes, L. R., Gruntfest E. C. , Hayden M. H. , Schultz D. M. , and Benight C. , 2007: False alarms and close calls: A conceptual model of warning accuracy. Wea. Forecasting, 22, 1140–1147.
Berchoff, D., cited 2009: Mesoscale probability forecast capability: A national imperative. [Available online at http://www.dtcenter.org/ensemble_presentations/1-2_Berchoff_mesoscale%20workshop_boulder_9_21_09v7.pdf.]
Brooks, H. E., 2004: Tornado-warning performance in the past and future: A perspective from signal detection theory. Bull. Amer. Meteor. Soc., 85, 837–843.
Brotzge, J., and Erickson S. , 2009: NWS tornado warnings with zero or negative lead times. Wea. Forecasting, 24, 140–154.
Brotzge, J., and Erickson S. , 2010: Tornadoes without NWS warning. Wea. Forecasting, 25, 159–172.
Crum, T. D., and Alberty R. L. , 1993: The WSR-88D and the WSR-88D Operational Support Facility. Bull. Amer. Meteor. Soc., 74, 1669–1687.
Friday, E. W., 1994: The modernization and associated restructuring of the National Weather Service: An overview. Bull. Amer. Meteor. Soc., 75, 43–52.
King, P., 1997: On the absence of population bias in the tornado climatology of southwestern Ontario. Wea. Forecasting, 12, 939–946.
McCarthy, D., and Schaefer J. , 2004: Tornado trends over the past thirty years. Preprints, 14th Conf. Applied Meteorology, Seattle, WA, Amer. Meteor. Soc., 3.4. [Available online at http://ams.confex.com/ams/pdfpapers/72089.pdf.]
National Weather Service, cited 2009: Storm-based warnings: A presentation for NOAA’s NWS managers. [Available online at http://www.nws.noaa.gov/cfo/program_planning/doc/FY-2009%20NOAA%27s%20NWS%20National%20Performance%20Measures%20-%20Graph%20Update.pdf.]
Roebber, P. J., 2009: Visualizing multiple measures of forecast quality. Wea. Forecasting, 24, 601–608.
Simmons, K. M., and Sutter D. , 2009: False alarms, tornado warnings, and tornado casualties. Wea. Climate Soc., 1, 38–53.
Trapp, R. J., Wheatley D. M. , Atkins N. T. , Przybylinski R. W. , and Wolf R. , 2006: Buyer beware: Some words of caution on the use of severe wind reports in postevent assessment and research. Wea. Forecasting, 21, 408–415.
U.S. Census Bureau, cited 2008: United States Census 2000. [Available online at http://www.census.gov/main/www/cen2000.html.]
Waters, K., and Coauthors, 2005: Polygon weather warnings—A new approach for the National Weather Service. Preprints, 21st Int. Conf. on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, San Diego, CA, Amer. Meteor. Soc., 14.1. [Available online at http://ams.confex.com/ams/pdfpapers/86326.pdf.]
