Abstract

Knowing the benefits of creating or expanding programs is important for determining optimal levels of investment. Yet estimates of the benefits of weather warning systems are sparse, perhaps because there is often no clear counterfactual of how individuals would have fared without a particular warning system. This paper enriches the literature and informs policy decisions by using conditional variation in the initial broadcast dates of the National Oceanic and Atmospheric Administration’s Weather Radio All Hazards (NWR) transmitters to produce both cross-sectional and fixed effects estimates of the causal impact of expanding the NWR transmitter network. Results suggest that from 1970 to 2014, expanding NWR coverage to a previously untreated county was associated with an almost 40% reduction in injuries and as much as a 50% reduction in fatalities. The benefits associated with further expansion of this system have likely declined over time.

1. Introduction

From 2006 to 2015, weather-related hazards caused an average of over 500 fatalities and 3000 injuries per year in the United States. Tornadoes exceeded hurricanes as the leading cause of storm-related deaths and injuries in the United States, accounting for 1101 fatalities and 11 596 injuries over this window.1 Estimates of the benefits of investment in disaster mitigation and preparedness range from $3.65 (Multihazard Mitigation Council 2005) to $15 (Healy and Malhorta 2009) of savings per $1 spent. However, justifying and promoting investment in disaster mitigation seems to require more than highlighting financial benefits. Identifying the number of lives saved by investments in weather warning systems could provide a tangible example of these benefits. But beyond heartfelt beliefs and anecdotes, there is little robust statistical evidence about whether and to what extent specific weather warning systems save lives. This paper addresses this gap by using the expansion of the National Oceanic and Atmospheric Administration’s (NOAA’s) Weather Radio All Hazards (NWR) network to estimate the benefits associated with investments in this weather warning system. Results show that the introduction of NWR transmitters in a county was associated with an almost 40% reduction in tornado injuries and as much as a 50% reduction in tornado fatalities.

There are a large number of case studies and household surveys examining how individuals responded to severe weather warnings. While this literature is very useful for understanding how individuals responded to their particular situation, the counterfactual is not clear. How would those individuals have fared without a particular warning system? If warnings from a NOAA weather radio were not available, would individuals have received and responded to warnings from a siren, television, or phone? Such questions are difficult to answer because we cannot observe how the same individual would respond to a situation both with and without a given warning system. Instead we need a reliable counterfactual. We want to compare an individual who has access to a particular warning system to a similar individual in a similar situation who does not have access to that warning system.

This paper overcomes the difficulty of obtaining a reliable counterfactual by exploiting variation in the location and timing of the initial broadcast dates of NWR radio transmitters. Suppose transmitters were simultaneously installed in a random selection of U.S. counties. The number of fatalities and injuries caused by tornadoes in areas receiving transmissions could simply be compared to the number of fatalities and injuries caused by tornadoes in areas which do not receive transmissions. Given a large enough number of observed tornadoes, the two groups of areas should have similar average rates of fatalities and injuries per tornado prior to any transmitter installations. This similarity is because, on average, there are no pre-existing differences between the two groups by nature of being randomly selected. After the transmitters begin broadcasting, differences in fatalities and injuries between the two areas could be attributed to an average impact of NWR transmitters. This interpretation, that the identified reduction in fatalities and injuries can be attributed solely to NWR transmissions, would be based on random assignment ensuring that the only average difference between the two groups of areas was the existence of transmitters.

In practice, the location of these installations was not random, and the installations did not occur simultaneously. For example, a tornado that strikes an area with NWR transmitters is most likely striking an area with higher population than a tornado that strikes an area without NWR transmitters. Tornadoes in areas with NWR transmitters kill roughly the same number of people as tornadoes in areas without NWR transmitters, but because areas with NWR have higher populations than areas without NWR we might expect them to also have higher fatalities. Failing to account for population would make NWR transmissions look less effective than they actually are. To correctly estimate the impact of NWR transmissions in this setting, it is necessary to account for any and all factors, like population, which both influence the number of fatalities and injuries caused by a tornado and are correlated with whether the area receives NWR transmissions at the time the tornado occurs. Other factors, such as public education campaigns or the prevalence of appropriate shelter, might also affect fatality rates, but their impact would only need to be controlled for if they are more prevalent or impactful in areas that receive (or do not receive) NWR transmissions and were expanded in each area at the same time as NWR transmissions were introduced.

This paper uses two different strategies to control for these potential sources of bias. The first method is a cross-sectional analysis, which compares fatalities and injuries across a large sample of tornadoes, controlling for potential causes of nonrandom variation in NWR availability such as population and the frequency of tornado occurrence in the area. The second method is a panel method, which examines the average change in fatality and injury rates in areas affected by tornadoes both before and after NWR broadcasts begin. Although the panel method is limited to areas that experience tornadoes more frequently, it controls for any potential bias caused by permanent differences between areas with and without transmitters. Both methods find no evidence that NWR broadcasts reduce property damage. The short time window between receipt of a warning and the event does not permit time for meaningful damage mitigation, so if a difference in property damage had been found that would suggest the method had failed to control for an important difference between areas that receive NWR transmissions and those that do not.

Both methods estimate that NWR transmitter installations are associated with an almost 40% reduction in tornado injuries. Cross-sectional estimates suggest that installations are associated with 50% fewer fatalities, while panel estimates suggest a smaller reduction of 11.4%. These reductions reflect the aggregate reduction in deaths and injuries associated with the availability of NWR broadcasts, and do not reflect the risk reduction for a single individual. Receiving the NWR broadcast requires the individual to own a radio receiver capable of picking up the NWR signal. Additional results suggest that the benefits of further expansion of this warning system may have declined over time, and potential reasons for such a decline are discussed.

Focusing this weather warning system evaluation on NWR is appropriate for several reasons. In addition to the availability of quality data and a clear source of identifying variation, NWR has long been a flagship warning system of the National Weather Service. This research also serves as an example for future efforts to estimate program impacts in the weather community. A strong literature of such studies is essential for enabling policy makers to choose optimal level of investment both within and between warning systems. This paper and others like it can shed objective light on hotly debated questions such as whether new types of warning systems, including warnings delivered directly to cellular phones, have made older systems obsolete.

The rest of the paper proceeds as follows. Section 2 discusses the roll-out of the NWR program and existing knowledge on the effectiveness of weather warning systems. Section 3 discusses the data and methodology used in this study. Section 4 discusses results. Section 5 concludes.

2. Background

a. Relevant historical context

NWR is a network of radio stations that broadcast weather information from the nearest National Weather Service (NWS) office. NWR originally served as a weather station for aviation and later maritime use. The first consumer-targeted radio receiver capable of picking up the NWR frequency were introduced in 1968 (Nelson 2002), but NWR was not truly a public warning system until 1971 when a new “tone alarm” feature enabled the receiver to announce critical messages even when the audio was turned off.

Over time, users have gained more control over which warnings they receive. In 1998, NWR began incorporating Specific Area Message Encoding (SAME), which enables users to select one or more counties or subcounties for which they desired to receive warnings rather than receiving all warnings issued by the transmitter. A 2002 FCC rule change enabled users to opt out of receiving certain types of warnings.2 If individuals opt out of important warnings, this could decrease the effectiveness of the warning system. But if individuals opt out of less relevant warnings, they may pay more attention to those warnings they do receive.

Because the statistical analysis in this paper compares the fatalities and injuries caused by observably similar tornadoes that affected areas with or without NWR transmitters, controls must be included for any factors that affect these outcomes and occur at both the same time and place as NWR installations. To identify potential concerns, it is important to understand the historical factors which influenced the timing and location of NWR installations.

The first NWR installations were unrelated to tornadoes; they reflected the aviation and maritime usage of NWR. A sudden increase in the number of installations followed the “Super Outbreak” of 148 tornadoes on 4 April 1974. In response to this event, a January 1975 White House policy statement designated NOAA Weather Radio as the sole government-operated radio system to provide direct warnings into private homes for natural disasters.3 Figure 1 shows how the distribution of NWR transmitters across the country has changed over time, starting immediately before this policy change in 1975. Figure 2 shows the fraction of tornadoes examined in this study that occurred in areas covered by NWR transmitters; the increase in transmitter installation after 1975 means tornadoes were much more likely to occur in an area covered by NWR transmitters.

Fig. 1.

Counties receiving NWR broadcasts, by date.

Fig. 1.

Counties receiving NWR broadcasts, by date.

Fig. 2.

Percent of tornadoes with transmitter coverage.

Fig. 2.

Percent of tornadoes with transmitter coverage.

A common concern is that new warning systems cause older systems to become unnecessary. To address this unanswered question it is important to know when alternative warning systems came into popular use. Official government tornado forecasts did not begin until 1952,4 and methods for disseminating these warnings to the public have evolved over time. “During the 1950s and 1960s, tornado warnings were disseminated to the public primarily by commercial television and radio stations. The TV and radio stations received these warnings from the [U.S. Weather Bureau] by telephone or teletype” (Coleman et al. 2011, p. 567). Outdoor warning sirens, originally designed as World War II air-raid sirens, were repurposed for weather warnings and have been typically operated by local emergency managers for tornado warnings since about 1970. In late June of 2012, NWS began participation in a system of emergency alerts sent to all phones using commercial cell phone towers.5 This paper is not able to offer definitive evidence that these programs have altered the impact of NWR transmitters on tornado fatalities or injuries. This paper does present suggestive evidence that the benefits associated with expanding NWR coverage have declined over time, and the availability of alternative warning systems is one of several potential explanations.

b. Current literature

There are several seminal papers that examine what factors drive individuals’ responses to warnings in various situations. Balluz et al. (2000) found that roughly 45% of those who responded to a random telephone survey following 1 March 1997 tornadoes in Arkansas reported seeking shelter after learning of the tornado warning. Liu et al. (1996) surveyed two Alabama areas after tornado warnings to learn the type of warning respondents heard first. In an area without tornado sirens, only 28.9% of respondents heard a warning before the first tornado touched down; 73.2% of those who heard a warning first received it via television or radio, and 53.6% of those who heard a warning sought shelter. In an area with tornado sirens, 88.1% of respondents heard a warning before the first tornado touched down; 33.5% of those who heard a warning first received it via television or radio, 61.8% first heard the siren, and 31.2% of those who heard a warning sought shelter. Effective warnings are only part of the equation; Liu et al. (1996) find that most respondents in both areas reported not having access to appropriate shelter, and that between 10% and 15% of respondents did not understand the difference between a tornado watch and a tornado warning. Dow and Cutter (1998) surveyed residents of two South Carolina communities about how they responded to repeated hurricane evacuation orders over the course of a single season. They show that multiple factors influence individuals decision to evacuate in advance of a hurricane, with individual evaluation of risk (based on factors including media reports, weather service statements, and government evacuation orders) and other individual factors (health status, safety of children and pets, work obligations, etc.) all playing a role.

This paper answers a different question than the prior literature, using a different methodology. Specifically, this paper seeks to estimate the difference in outcomes associated with a world where a particular warning system is available, relative to an otherwise identical world where that warning system is not available. An example of similar concurrent work is Ferris and Newburn (2017), who find wireless alert messages for flash flood warnings reduced car accidents by 15.9% relative to the counterfactual with nonwireless alert protocols. Such estimates are critical for making informed policy decisions. However, this study provides the reader with little direct evidence about the factors driving the observed difference in outcomes. To fully understand the effectiveness of weather warning systems, both perspectives are needed.

Beyond this paper and Ferris and Newburn (2017), there has been little empirical research estimating the impact of weather warning systems. However, some papers have attempted to shed light on the benefits of warnings themselves. Doswell et al. (1999) show that shortly after the beginning of public tornado forecasting there was a reduction in deaths relative to inflation-adjusted damage from major tornadoes. Improvements in forecast quality also matter. Simmons and Sutter (2008) find that longer lead times on tornado warnings reduce injuries, and Simmons and Sutter (2005) find that the installation of Doppler radar in the 1990s reduced fatalities and injuries by 45% and 40%, respectively. Sutter and Simmons (2014) find no measurable impact of tornado watches (issued when conditions are favorable for tornadoes) beyond the benefits already granted by warnings, although it should be noted that tornado watches play a vital role in activating the preparations of emergency staff and resources. L. Bakkensen (2016, unpublished manuscript) uses variation is the visibility of tornadoes to examine how individual respond to public warnings versus their own private observations.

3. Data and methodology

a. Data

This paper uses data from the Storm Prediction Center’s national tornado archive, which contains data on all recorded tornadoes between 1950 and 2014.6 Total fatalities and injuries are recorded for each tornado. For tornadoes that cross state lines, each state’s portion of the tornado is able to be treated as a separate observation with independent records of fatalities and injuries. Records for the date, time, counties affected, and properties of the tornado such as start and end coordinates, path length, and tornado width are similarly recorded at the state-by-tornado level.7 The Fujita scale, a categorical measure of wind speeds based on damage assessments, is reported.8 Property damage is reported beginning in 1996.9

Figure 3 shows that small tornadoes are more likely to be recorded in recent years, which is likely due to improvements in radar quality rather than changes in the number and average strength of tornadoes. Small tornadoes are also less likely to cause fatalities or injuries. Including small tornadoes would bias results to appear overly optimistic because more recently recorded tornadoes are both more likely to be in an NWR broadcast area and more likely to involve no injuries or fatalities because they are small. For this reason, analysis excludes all tornadoes with a Fujita scale of 0, and excluded all tornadoes recorded before 1970. This leaves a sample size of over 23 000 recorded tornadoes.

Fig. 3.

Fujita scale by decade.

Fig. 3.

Fujita scale by decade.

Data on NWR locations and the counties which they cover are publicly available online.10 Much gratitude is due to NOAA for providing the date on which each transmitter began broadcasting, as well as the handful of dates that specific transmitters were permanently deactivated. The database NOAA uses to gather initial broadcast dates was not developed until the 1990s. Installation dates prior to that time were gathered by NOAA from old records so some measurement error is possible. Because of data limitations, the entire area of any county to which that transmitter is officially assigned is treated as covered by that transmitter. In many cases, the entire area of that county may not be able to receive transmission from that transmitter due to topography or distance. To the extent that this causes tornadoes that occur in parts of counties that cannot actually receive transmissions to be falsely coded as covered by a transmitter, results would be biased toward finding transmitters have no impact on any outcomes.

Annual county-level population data come from the U.S. Decennial Census and the U.S. Census Bureau’s Intercensal Estimates. Data on state-by-decade housing stocks come from the Historical Census of Housing. Summary statistics for all data can be found in Tables 13.

Table 1.

This table contains summary statistics from both the full sample of tornadoes used for the cross-sectional analysis and the sample of single-county tornadoes used for the unbalanced panel analysis. Hence the unit of analysis is the tornado, with the exception of tornadoes that cross states lines for which each state segment is counted separately. Data come from the NWS’ Storm Prediction Center’s national tornado archive, NOAA records, the Decennial Census, and intercensal population estimates from the U.S. Census Bureau (https://www.census.gov/programs-surveys/popest/data/data-sets.All.html). Additional controls described in section 3b come from the U.S. Census Bureau's Historical Census of Housing (https://www.census.gov/housing/census/about/). Data include all recorded tornadoes since 1970 above a Fujita scale of 0. “Recent” refers to the prior 5 years.

This table contains summary statistics from both the full sample of tornadoes used for the cross-sectional analysis and the sample of single-county tornadoes used for the unbalanced panel analysis. Hence the unit of analysis is the tornado, with the exception of tornadoes that cross states lines for which each state segment is counted separately. Data come from the NWS’ Storm Prediction Center’s national tornado archive, NOAA records, the Decennial Census, and intercensal population estimates from the U.S. Census Bureau (https://www.census.gov/programs-surveys/popest/data/data-sets.All.html). Additional controls described in section 3b come from the U.S. Census Bureau's Historical Census of Housing (https://www.census.gov/housing/census/about/). Data include all recorded tornadoes since 1970 above a Fujita scale of 0. “Recent” refers to the prior 5 years.
This table contains summary statistics from both the full sample of tornadoes used for the cross-sectional analysis and the sample of single-county tornadoes used for the unbalanced panel analysis. Hence the unit of analysis is the tornado, with the exception of tornadoes that cross states lines for which each state segment is counted separately. Data come from the NWS’ Storm Prediction Center’s national tornado archive, NOAA records, the Decennial Census, and intercensal population estimates from the U.S. Census Bureau (https://www.census.gov/programs-surveys/popest/data/data-sets.All.html). Additional controls described in section 3b come from the U.S. Census Bureau's Historical Census of Housing (https://www.census.gov/housing/census/about/). Data include all recorded tornadoes since 1970 above a Fujita scale of 0. “Recent” refers to the prior 5 years.
Table 2.

This table contains data from both the full sample of tornadoes used for the cross-sectional analysis. The unit of analysis is the tornado, with the exception that tornadoes that cross states lines have each state segment counted separately. Data come from the NWS’ Storm Prediction Center’s national tornado archive, NOAA records, the Decennial Census, and intercensal population estimates. Additional controls described in section 3b come from the Historical Census of Housing. Data include all recorded tornadoes since 1970 above a Fujita scale of 0. “Recent” refers to the 5-yr period prior to the date of the observed tornado.

This table contains data from both the full sample of tornadoes used for the cross-sectional analysis. The unit of analysis is the tornado, with the exception that tornadoes that cross states lines have each state segment counted separately. Data come from the NWS’ Storm Prediction Center’s national tornado archive, NOAA records, the Decennial Census, and intercensal population estimates. Additional controls described in section 3b come from the Historical Census of Housing. Data include all recorded tornadoes since 1970 above a Fujita scale of 0. “Recent” refers to the 5-yr period prior to the date of the observed tornado.
This table contains data from both the full sample of tornadoes used for the cross-sectional analysis. The unit of analysis is the tornado, with the exception that tornadoes that cross states lines have each state segment counted separately. Data come from the NWS’ Storm Prediction Center’s national tornado archive, NOAA records, the Decennial Census, and intercensal population estimates. Additional controls described in section 3b come from the Historical Census of Housing. Data include all recorded tornadoes since 1970 above a Fujita scale of 0. “Recent” refers to the 5-yr period prior to the date of the observed tornado.
Table 3.

This table contains data from the sample of single-county tornadoes used for the unbalanced panel analysis. The unit of analysis is the tornado, with the exception that tornadoes that cross states lines have each state segment counted separately. Data come from the NWS’ Storm Prediction Center’s national tornado archive, NOAA records, the Decennial Census, and intercensal population estimates. Additional controls described in section 3b come from the Historical Census of Housing. Data include all recorded tornadoes since 1970 above a Fujita scale of 0. “Recent” refers to the 5-yr period prior to the date of the observed tornado.

This table contains data from the sample of single-county tornadoes used for the unbalanced panel analysis. The unit of analysis is the tornado, with the exception that tornadoes that cross states lines have each state segment counted separately. Data come from the NWS’ Storm Prediction Center’s national tornado archive, NOAA records, the Decennial Census, and intercensal population estimates. Additional controls described in section 3b come from the Historical Census of Housing. Data include all recorded tornadoes since 1970 above a Fujita scale of 0. “Recent” refers to the 5-yr period prior to the date of the observed tornado.
This table contains data from the sample of single-county tornadoes used for the unbalanced panel analysis. The unit of analysis is the tornado, with the exception that tornadoes that cross states lines have each state segment counted separately. Data come from the NWS’ Storm Prediction Center’s national tornado archive, NOAA records, the Decennial Census, and intercensal population estimates. Additional controls described in section 3b come from the Historical Census of Housing. Data include all recorded tornadoes since 1970 above a Fujita scale of 0. “Recent” refers to the 5-yr period prior to the date of the observed tornado.

For NWR transmissions to have a meaningful impact on human behavior, many individuals in areas affected by tornadoes would need to own NWR receivers.11 For this reason it would be ideal to know the prevalence of NWR receivers across space and time, but unfortunately data on the market penetration of NWR receivers are not available. Thus, this study cannot account for variation in the effectiveness of NWR driven by variation in prevalence of receivers; results reflect the average impact of NWR and not how that impact varies with increased receiver ownership.

b. Methodology

This paper uses two complementary strategies to measure the association between NWR transmitters and tornado injuries or fatalities. The first is a cross-sectional analysis that examines a broader set of observations but could plausibly have selection bias concerns. The second is an unbalanced panel fixed effects analysis that is immune to more potential concerns about selection bias, but only estimates the impact associated with a very specific group of tornadoes and hence may be less informative about the impact of NWR transmissions more generally. Both methods use Poisson regression analysis to estimate the causal impact of NWR access on tornado injuries and fatalities.12

The independent variable of interest is coverage, which is a represents the percentage of counties on a tornado’s path that receive broadcasts from at least one NWR transmitter.13 Coverage takes values ranging from 0 to 1, with higher values reflecting cases where a larger percentage of the tornado’s path was in areas receiving NWR broadcasts, meaning coverage is not a binary treatment variable but rather reflects “intensity of treatment.” Differences in coverage are due to the tornado location and whether the tornado occurs before or after transmitter are installed. As discussed in the introduction, because both the location and timing of transmitter installation is nonrandom, it is important to control for any factors which affect the timing and location of transmitter installation as well as the probability of injuries or fatalities. The accuracy of both methodologies rests on the assumption that conditional on these controls, transmitter broadcast areas are only correlated with tornado injuries and fatalities through their transmission of warnings.

First consider the cross-sectional analysis, which examines the correlation between coverage and fatalities or injuries across all tornadoes included in the sample described in section 3a. As discussed above, an impossibly wide variety of controls would be needed to account for all the factors that influence the number of fatalities and injuries caused by a tornado. Fortunately, the cross-sectional analysis only needs to control for factors that affect both fatality or injury rates and are correlated with whether the area receives NWR transmissions at the time the tornado occurs. This is because other factors that affect fatalities or injuries will not influence the correlation between coverage and fatalities or injuries.

Tornadoes with longer paths are mechanically more likely to enter a county with a broadcasting transmitter, so controls include a quadratic measure of path length and indicators for the number of counties affected.14 Several variables control for characteristics of the location of the tornado to account for transmitters being more likely to be built in certain areas such as densely populated areas or areas with certain types of housing infrastructure. These controls include state fixed effects, several controls for population, and decade-by-state measures of the percentage and number of residences of different housing types.15 These flexible population controls are important because Tables 2 and 3 show that transmitters appear to be installed in areas with higher population first, and higher populations should also experience higher counts of fatalities and injuries. Controls also include the areas of the counties impacted by the tornado to account for correlation between a county’s physical area and the probability of that county having a transmitter somewhere in the county.16 Location-specific controls for the number and size of tornadoes a county has historically received account for areas that are more prone to tornadoes being more likely to have NWR transmitters.17 Tables 2 and 3 show evidence that counties with transmitters tend to experience more tornadoes, and that transmitters were first installed in areas with long histories of experiencing more tornadoes.

It is also important to control for unrelated but correlated trends in NWR coverage and outcomes over time. Year fixed effects control for these concerns, such as coverage being higher in later years when fatalities or injuries may have decreased for other reasons.18 While variables such as the month and time of day a tornado occurs are not obviously correlated with NWR coverage, they are important determinants of fatality and injury outcomes and are included to improve model fit.19

In essence, the cross-sectional approach compares outcomes from a tornado in one location and point in time to outcomes from a tornado at a different location and/or point in time. Specific efforts are made to control for nonrandom reasons one tornado may be more or less likely to have both NWR coverage and, for unrelated reasons, higher or lower rates of injuries or fatalities. Although the set of controls described above may be sufficient, this paper also implements a second identification strategy using unbalanced panel fixed effects to resolve any lingering concerns that other factors might still be driving the relation between NWR installations and injuries or fatalities. Panel fixed effects describe a regression methodology that examines repeated observations of the same locations over time, and includes as control variables a binary indicator for each location. These indicators are called “fixed effects” because they control for all temporally constant correlation between that location and the outcome.20 In this case, county fixed effects can control for all temporally constant aspects of the county that might cause fatalities or injuries to tend to be higher or lower. Because fatalities and injuries from each tornado are measured at the state level rather than the county level, this second methodology examines only tornadoes that impacted a single county in order to correctly allocate outcomes. Because the panel fixed effect methodology means the causal impact is estimated based on changes in outcomes, only counties that have been hit by at least two tornadoes and have experienced at least one injury or fatality are included. This leaves 15 539 of the 23 271 observations as the base of the panel fixed effects sample.21 As shown in Tables 1 and 3, these tornadoes are smaller, have shorter paths, and are associated with fewer injuries and fatalities. For these reasons, the cost of resolving potential concerns about spurious correlation is some external validity.

Because year fixed effects and all time-varying controls described above are still included, the estimate of the association between NWR coverage and outcomes for this group is only biased if there are county-specific changes over time that affect both the probability of NWR coverage and the outcome of interest that are not already accounted for by the included time-varying controls such as population and tornado characteristics. Coverage becomes a binary variable representing whether the county was receiving broadcasts from at least one NWR transmitter at the time of the tornado.

4. Results

Table 4 shows results from six different regressions, three using the cross-sectional analysis and three using the unbalanced panel regression. Columns 1 and 2 use injuries as the dependent variable, columns 3 and 4 use fatalities as the dependent variable, and columns 5 and 6 use property loss as the dependent variable. The row labeled “% of counties on tornado path with broadcasts” shows the coefficient on coverage, with standard errors in parentheses. Coefficients on all control variables have been suppressed for readability and because the coefficients on these variables are easily misinterpreted.22 Complete regression results are available as an online appendix.

Table 4.

This table contains the main results of the paper—estimates of the marginal impact of NWR transmitters on injuries, fatalities, and property loss. An asterisk (*) indicates p < 0.05. Property loss only covers years 1996–2012. Data and controls are as described in sections 3a and 3b, with full results available in Table S1 in the online supplemental material. Standard errors are in parentheses. Cross-sectional standard errors are clustered at the tornado level because each state-tornado segment is treated as a separate observation. Unbalanced panel standard errors are robust to heteroskedasticity. The comparison between dependent variable and observed outcomes measures within-sample accuracy; the comparison does not necessarily reflect the model’s out-of-sample predictive capabilities, nor does it indicate whether the reported association between NWR coverage and the dependent variable is unbiased. The units for property loss are millions of dollars.

This table contains the main results of the paper—estimates of the marginal impact of NWR transmitters on injuries, fatalities, and property loss. An asterisk (*) indicates p < 0.05. Property loss only covers years 1996–2012. Data and controls are as described in sections 3a and 3b, with full results available in Table S1 in the online supplemental material. Standard errors are in parentheses. Cross-sectional standard errors are clustered at the tornado level because each state-tornado segment is treated as a separate observation. Unbalanced panel standard errors are robust to heteroskedasticity. The comparison between dependent variable and observed outcomes measures within-sample accuracy; the comparison does not necessarily reflect the model’s out-of-sample predictive capabilities, nor does it indicate whether the reported association between NWR coverage and the dependent variable is unbiased. The units for property loss are millions of dollars.
This table contains the main results of the paper—estimates of the marginal impact of NWR transmitters on injuries, fatalities, and property loss. An asterisk (*) indicates p < 0.05. Property loss only covers years 1996–2012. Data and controls are as described in sections 3a and 3b, with full results available in Table S1 in the online supplemental material. Standard errors are in parentheses. Cross-sectional standard errors are clustered at the tornado level because each state-tornado segment is treated as a separate observation. Unbalanced panel standard errors are robust to heteroskedasticity. The comparison between dependent variable and observed outcomes measures within-sample accuracy; the comparison does not necessarily reflect the model’s out-of-sample predictive capabilities, nor does it indicate whether the reported association between NWR coverage and the dependent variable is unbiased. The units for property loss are millions of dollars.

It is easiest to first interpret results from the unbalanced panel regression, which are reported in columns 2 and 4 of the first row of results in Table 4. The coefficient on coverage estimates the average reduction in fatalities and injuries associated with having at least one NWR transmitter broadcasting over the impacted county. Table 4 reports that having at least one NWR transmitters broadcasting over a county is associated with 38.9% fewer injuries on average, with a 95% confidence interval of [3.4%, 74.4%]. In other words, there is a 95% chance that the true average reduction in injuries is a value within that confidence interval, and the best estimate of the true value is 38.9%. While these ranges are quite broad, it is clear that NWR transmitters are associated with an important and statistically significant reduction in the injuries caused by tornadoes. Having at least one NWR transmitters broadcasting over a county is associated with 11.4% fewer fatalities on average, although the 95% confidence interval of [−45.8%, 68.6%] means that this model fails to reject the null hypothesis that transmitters have no impact on fatalities.

For the cross-sectional analysis, interpretation is similar. However, 13.7% of tornadoes in the cross-sectional sample impact multiple counties, so coverage measures the percentage of counties on a tornado’s path that receive broadcasts from at least one NWR transmitter, rather than being a binary indicator. The coefficient on coverage estimates the percent reduction in death and injuries associated with having at least one NWR transmitter(s) broadcasting over all impacted counties relative to none. Table 4 reports that having at least one NWR transmitter broadcasting over all impacted counties is associated with 34.3% fewer injuries on average, with a 95% confidence interval of [0.2%, 68.4%]. Similarly, having at least one NWR transmitter broadcasting over all impacted counties is associated with 50.1% fewer fatalities on average, with a 95% confidence interval of [8.5%, 91.7%]. While these ranges are again quite broad, they suggest that NWR transmitters cause an important and statistically significant reduction in the injuries and fatalities caused by tornadoes. Applying the cross-sectional analysis to the panel sample yields similar estimates.23

Table 4 also compares the number of fatalities and injuries estimated by each model for each tornado to the number of fatalities and injuries that actually occurred. This gives an indication of the overall within-sample accuracy of these models for predicting the number of fatalities and injuries. The comparison does not necessarily reflect the model’s out-of-sample predictive capabilities, nor does it indicate whether the reported association between NWR coverage and the dependent variable is unbiased. Note that the average difference between estimated and actual fatalities is smaller than the average difference between estimated and actual injuries in part because fatalities have lower variance. The models can correctly predict that the vast majority of tornadoes have zero fatalities.

The average time between a tornado warning and the tornadic event is about 15 min today. This means it is likely difficult for NWR users to move or protect physical capital such as homes after hearing a warning. It should be expected that while NWR may reduce injuries and fatalities, it is unlikely to reduce property damage. Using property damage is examined as the dependent variable provides a convenient opportunity to look for evidence that would suggest results are being driven by something other than the impact of NWR transmitters. If, after using identical controls and regression techniques, NWR transmitters are found to have a significant correlation with damages, that would suggest that the main results might still suffer from nontrivial bias. Fortunately, Table 4 reports no statistically significant correlation between property damage and having at least one NWR transmitters broadcasting over all impacted counties under either specification.24

The coefficient on coverage reflects the relation between the dependent variable and coverage that is not already captured by the control variables. If the control variable perfectly predicts coverage, then there is no remaining variation in coverage to compare to the dependent variable. In extreme cases, this collinearity problem can cause the coefficient on coverage to be biased. Table 4 tests for this issue by regressing coverage on all the same control variables used in each of the six models from Table 4.25 The results suggest that the main results in models 1 through 4 are free from collinearity concerns. For models 5 and 6 there is little variation in coverage that is not already captured by the controls, suggesting that those results are at higher risk of being biased, perhaps explaining the large magnitudes of those coefficients.

Another question of interest is whether the marginal benefits of additional NWR installations have changed over time. In theory, the marginal benefit could move in either direction. If NWR transmitters are being installed in the highest-value locations first, one should expect that the marginal benefit from additional transmitters will naturally decline over time. This would not necessarily imply a decline in the value of existing transmitters. The rise of substitute warning systems might also reduce the benefit of transmitters over time, and would reduce the marginal benefit from preexisting transmitters. On the other hand, advances in warning system technology, such as the introduction of SAME codes discussed in section 2a, could mean new installations provide greater benefits. If warning systems are mutually reinforcing rather than substitutes, new systems might also increase the marginal value of existing systems. Figure 4 provides some empirical insight into this question. The figure presents both the cross-sectional and panel estimates of the association between NWR transmitters and injuries and fatalities over rolling two-decade windows, with the top estimate of each panel representing the corresponding results from Table 4.26 Although standard errors remain quite large, the largest reductions in deaths and injuries appear in earlier time periods. Cross-sectional estimates show slow but steady declines in benefit, although the differences are not statistically significant. Panel estimates are noisier, but they suggest a similar pattern of declining marginal benefit over time. Although not statistically conclusive, these figures suggest that declining marginal benefit stories may be more accurate. Remember that these estimates reflect the marginal benefit of transmitters that are newly installed during each two-decade period, and do not assess the continued value of transmitters that began broadcasting before that time.

Fig. 4.

Changes in injury and fatality prevention over time.

Fig. 4.

Changes in injury and fatality prevention over time.

5. Conclusions

This paper examines how the expansion of the National Oceanic and Atmospheric Administration’s Weather Radio All Hazards (NWR) warning system impacted fatalities and injuries caused by tornadoes. Two different identification strategies find that the presence of NWR transmitters causally reduces tornado injuries by almost 40%. Estimates of the reduction in fatalities associated with NWR coverage range from 50% to just over 10%. While the 95% confidence intervals are quite broad, it is clear that NWR transmitters play an important and statistically significant role in public safety. Although standard errors remain too large to offer statistically significant conclusions about whether the benefits associated with additional installations have changed over time, results are consistent with a slow decline in marginal benefits of new transmitters. This could be caused by new warning systems becoming substitutes for previously existing systems. Alternatively, this could suggest that transmitters were successfully built in the highest-value locations first, resulting in new expansions of the warning system being slightly less valuable than their predecessors. It is important to understand that these results reflect the number of injuries or fatalities prevented by warning system expansion, and do not reflect the total number of injuries and fatalities that have been prevented by the system as a whole. Also, these results reflect the average impact of NWR and not how that impact varies with the prevalence of receiver ownership. It is not surprising that this paper finds the benefit of NWR expansion to unserved counties in recent years is not statistically different from zero, given the near-ubiquitous coverage of the United States this warning system currently provides.

Overall, these results provide quantitative evidence that the NWR warning system has provided significant benefits to public health and safety. In doing so, this paper complements the existing evidence of large financial returns on investment in disaster mitigation. This paper also highlights a robust statistical method for measuring the benefits of expanding warning systems.

Acknowledgments

Much gratitude is due to Jen Burney, Tim Coleman, Julie Cullen, Gordon Dahl, Donald Fletcher, Eve Gruntfest, Jamie Kruse, Craig McIntosh, Paul Niehaus, and three anonymous referees. Thanks are also due to seminar attendees at the American Meteorological Society 95th Annual Meeting’s Third Symposium on Building a Weather Ready Nation, the BREN Occasional Workshop in Environmental and Resource Economics, the GeoValue Data to Decisions Workshop, the Southern Economics Association’s 85th Annual Meeting, the Studies of Precipitation, Flooding, and Rainfall Extremes Across Disciplines (SPREAD) Workshop, the University of California San Diego, and the World Weather Open Science Conference for their insights and assistance. All errors are my own.

I certify that I have no support from or affiliation with any organization or entity with any financial interest in the subject matter discussed in this manuscript.

REFERENCES

REFERENCES
Balluz
,
L.
,
L.
Schieve
,
T.
Holmes
,
S.
Kiezak
, and
J.
Malilay
,
2000
:
Predictors for people’s response to a tornado warning: Arkansas, 1 March 1997
.
Disasters
,
24
,
71
77
, https://doi.org/10.1111/1467-7717.00132.
Blackburn
,
M. L.
,
2014
:
The relative performance of Poisson and negative binomial regression estimators
.
Oxford Bull. Econ. Stat.
,
77
,
605
616
, https://doi.org/10.1111/obes.12074.
Cameron
,
A. C.
, and
P. K.
Trivedi
,
2013
: Regression Analysis of Count Data. 2nd ed. Cambridge University Press, 566 pp.
Coleman
,
T. A.
,
K. R.
Knupp
,
J.
Spann
,
J. B.
Elliott
, and
B. E.
Peters
,
2011
:
The history (and future) of tornado warning dissemination in the United States
.
Bull. Amer. Meteor. Soc.
,
92
,
567
582
, https://doi.org/10.1175/2010BAMS3062.1.
Doswell
,
C. A.
,
A. R.
Moller
, and
H. E.
Brooks
,
1999
:
Storm spotting and public awareness since the first tornado forecasts of 1948
.
Wea. Forecasting
,
14
,
544
557
, https://doi.org/10.1175/1520-0434(1999)014<0544:SSAPAS>2.0.CO;2.
Dow
,
K.
, and
S. L.
Cutter
,
1998
:
Crying wolf: Repeat responses to hurricane evacuation orders
.
Coastal Manage.
,
26
,
237
252
, https://doi.org/10.1080/08920759809362356.
Ferris
,
J. S.
, and
D. A.
Newburn
,
2017
:
Wireless alerts for extreme weather and the impact on hazard mitigating behavior
.
J. Environ. Econ. Manage.
,
82
,
239
255
, https://doi.org/10.1016/j.jeem.2016.11.002.
Finley
,
J. P.
,
1884
:
Tornado predictions
.
Amer. Meteor. J.
,
1
(
3
),
85
88
.
Grice
,
G. K.
, and Coauthors
,
1999
:
The golden anniversary celebration of the first tornado forecast
.
Bull. Amer. Meteor. Soc.
,
80
,
1341
1348
, https://doi.org/10.1175/1520-0477(1999)080<1341:TGACOT>2.0.CO;2.
Healy
,
A.
, and
N.
Malhorta
,
2009
:
Myopic voters and natural disaster policy
.
Amer. Polit. Sci. Rev.
,
103
,
387
406
, https://doi.org/10.1017/S0003055409990104.
Liu
,
S.
,
L. E.
Quenemoen
,
J.
Malilay
,
E.
Noji
,
T.
Sinks
, and
J.
Mendlein
,
1996
:
Assessment of a severe-weather warning system and disaster preparedness, Calhoun County, Alabama, 1994
.
Amer. J. Public Health
,
86
,
87
89
, https://doi.org/10.2105/AJPH.86.1.87.
Multihazard Mitigation Council
,
2005
: Natural hazard mitigation saves: An independent study to assess the future savings from mitigation activities. Multihazard Mitigation Council, 366 pp., http://www.preventionweb.net/publications/view/1087.
Nelson
,
W.
,
2002
: American warning dissemination and NOAA weather radio. M.S. thesis, University of North Carolina Chapel Hill, 142 pp.
Santos Silva
,
J. M. C.
, and
S.
Tenreyro
,
2006
:
The log of gravity
.
Rev. Econ. Stat.
,
88
,
641
648
, https://doi.org/10.1162/rest.88.4.641.
Santos Silva
,
J. M. C.
, and
S.
Tenreyro
,
2010
:
On the existence of the maximum likelihood estimates in Poisson regression
.
Econ. Lett.
,
107
,
310
312
, https://doi.org/10.1016/j.econlet.2010.02.020.
Simmons
,
K. M.
, and
D.
Sutter
,
2005
:
WSR-88D radar, tornado warnings, and tornado casualties
.
Wea. Forecasting
,
20
,
301
310
, https://doi.org/10.1175/WAF857.1.
Simmons
,
K. M.
, and
D.
Sutter
,
2008
:
Tornado warnings, lead times, and tornado casualties: An empirical investigation
.
Wea. Forecasting
,
23
,
246
258
, https://doi.org/10.1175/2007WAF2006027.1.
Sutter
,
D.
, and
K. M.
Simmons
,
2014
:
Preparing for danger: On the impact of tornado watches on tornado casualties
.
Int. J. Mass Emerg. Disasters
,
32
,
1
25
, http://www.ijmed.org/articles/647/.
Winkelmann
,
R.
,
2008
: Econometric Analysis of Count Data. 5th ed. Springer, 320 pp.
Wooldridge
,
J.
,
2010
: Econometrics Analysis of Cross Section and Panel Data. 2nd ed. The MIT Press, 1064 pp.

Footnotes

Supplemental information related to this paper is available at the Journals Online website: https://doi.org/10.1175/WCAS-D-16-0093.s1.

© 2018 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

1

U.S. Natural Hazard Statistics; http://www.nws.noaa.gov/om/hazstats.shtml.

2

The rule change declared that “All existing and new models of EAS equipment manufactured after August 1, 2003 must be capable of selectively displaying and logging messages with state and local event codes” (FCC 02–64). For example, an individual who lives on a high floor of building many miles from the coast might choose to opt out of coastal flood warnings.

3

In some cases collective action by concerned citizens was the impetus for transmitter installation. Following the Super Outbreak in 1974, citizens of Huntsville, Alabama, raised money to buy their own transmitter, then donated the transmitter to NWS. See http:\\www.srh.noaa.gov\hun\?n=stationhistory.

4

Tornado forecasting efforts can be traced back to U.S. Army Sergeant John P. Finley, who twice per day categorized weather conditions across large areas of the United States as favorable or unfavorable for tornadoes. 28% of the favorable conditions in Finley (1884) produced confirmed tornadoes somewhere within their broad geographic area. Despite this promising start, the use of the word “tornado” was banned from official forecasts from 1887 until 1938 due to concerns that incited panic would outweigh any benefits from tornado forecasts (Coleman et al. 2011). Because of this policy, the first successful forecast of a potential tornadic event at a more precise location did not occur until a 1948 forecast at Tinker Air Force Base was made possible by a “fortuitous series of events” (Grice et al. 1999). Following this much-lauded forecast’s success, the Weather Bureau authorized public tornado alerts in 1950 and began issuing tornado forecasts in 1952.

5

This system was founded by the “Warning, Alert, and Response Network (WARN) Act” in 2006.

6

Only data from U.S. states are used in this analysis. Records from Puerto Rico and the Virgin Islands are excluded.

7

Most tornadoes are contained within a single state. In a small number of cases tornadoes exit and then re-enter a state. Such an occurrence generates three separate observations. To account for correlation between observations from the same tornado, standard errors are clustered at the tornado level.

8

Fujita scales are reported through 2006, and Enhanced Fujita scales are reported after January 2007, following a 2007 update to this ranking system. For conciseness this text refers to the ranking in any time period as “Fujita” or “EF Scale.”

9

Categorical damages (≤$50, $50–$500, $500–$5,000, etc.) are available before 1996 but are missing in some cases so are not used.

11

A special receiver is required to pick up the NWR signal because the signal is broadcast in the very high-frequency range of the radio spectrum. These receivers, which are produced by several private companies, are widely available in retail stores, and online. At present receivers cost about $20 or more depending on features.

12

It is well known that log-linear regression models produce biased estimates when the dependent variable has a binding lower bound of zero (Santos Silva and Tenreyro 2006). Both Poisson and negative binomial regressions analysis are commonly used to model count data. Because both the fatality and injury statistics have a large number of zero values, the data suffer from overdispersion. Poisson regression analysis produces consistent estimates regardless of any overdispersion so long as the functional form is correctly specified. Not all negative binomial distributions have this property. For further discussion, see Winkelmann (2008), Wooldridge (2010), Cameron and Trivedi (2013), and Blackburn (2014). Coefficients are estimated using the pseudomaximum likelihood technique of Santos Silva and Tenreyro (2010).

13

Literally, . As discussed in section 3a, transmitter broadcast areas are measured at the county level. Measurement error exists to the extent that a NWR transmitter’s broadcast area does not fully cover a county or covers portions of counties to which the transmitter is not explicitly assigned by NWS.

14

Specifically, the path length (in miles) and squared path length are included as regressors. Like many control variables, these measures have been rescaled (divided by a factor of 10) and centered (had the mean value subtracted so the variable has a mean of 0) in order to improve convergence. Such rescaling and centering does not cause bias. The number of counties affected is controlled for using three indicator variables: one for tornadoes that affected one county, one for tornadoes that affected two counties, and one for tornadoes that affected three or more counties. Three is used as a cut-off for many county-level controls because 99.21% of tornadoes in the sample impact three or fewer counties.

15

State fixed effects are binary indicator variables for each state. Population controls include a cubic polynomial of total population of all affected counties (population, population squared, and population cubed), the log of total population of the affected counties, and three variables containing the population density, , for the three most densely populated affected counties. The population density of the second and third most densely populated counties is set to 0 when the tornado did not affect that many counties. Most tornadoes only impact a single county, and limiting to three such variables indicators avoids collinearity issues, as less than 0.25% of tornadoes strike four or more counties. Controls for housing types include both variables for both the percentage and count of the state’s housing stock that takes the form of detached homes, attached homes, two-to-four unit-per-building homes, five or more unit-per-building homes, and mobile homes.

16

Specifically, controls include three variables that take values corresponding to the area of the three largest counties impacted by the tornado, recentered to a mean of 0. If the tornado only impacts one or two counties, the recentered controls take the mean value of 0 for the additional variables.

17

Specifically, controls for the total number of tornadoes and the number received in the last 5 years for each of the three most-impacted counties on each tornado’s path are included.

18

Year fixed effects, binary indicators for each year, begin in 1971 because prior to that point all areas are untreated. Treatment is considered to begin with the introduction of tone alarm for receivers in 1971. See section 2a.

19

Specifically, the regression include month fixed effects (binary indicators whether the tornado occurred in the given month) and a binary indicator for whether the tornado occurred between 10:00 p.m. and 6:00 a.m.

20

“Balanced” panel data include the same number of observations for each location at the same times. In this case, we only observe outcomes when a tornado occurs; some counties will be observed many times and some will be observed few times, and these observations do not occur at the same time. For this reason, the data are “unbalanced,” although that does not inhibit the ability to use panel fixed effects in a regression.

21

Tables 1 and 3 include summary statistics for all these observations. The individual regression may show fewer observations; for example, the regression on fatalities will exclude counties where tornadoes have never caused fatalities (even if some have caused injuries).

22

Coefficients on the multitude of control variables are difficult to interpret for two reasons. First, many control variables are rescaled and centered to improve convergence, which makes it more difficult to interpret their coefficients. Second and more importantly, the coefficients on these controls should not be interpreted as unbiased estimates of causal impacts due to a variety of potential biases.

23

Results not shown. Applying the cross-sectional analysis to the panel sample yields 26.5% reduction in injuries with a p value of 0.064, and a 40.1% reduction in fatalities with a p value of 0.030. These estimates are not statistically different from the panel estimates.

24

Analysis focuses on data beginning in 1996 because property damages are reported in more detail at this point. See section 3a.

25

The sample is limited to only observations included in that model, and coverage is not regressed on itself.

26

Note that the overall estimates are not simply the average of the estimates from the two-decade windows because the number of observations and availability of uncovered tornadoes to compare to are different in each window. See Fig. 2.

Supplemental Material