1. Introduction
Most people are aware of the grim costs of tornadoes killing dozens of people per year,1 but fewer know about warnings killing hundreds of thousands of hours in sheltering time. Sheltering is costly because it forces people to reduce time spent on work and leisure. These losses can be plausibly measured in monetary terms; Simmons and Sutter (2013) estimate that tornadoes impose roughly $3–4 billion of annual implicit costs2 on the U.S. society, and the opportunity cost of sheltering is one of the largest cost components, amounting to $1.3–2.6 billion.
One proposed way to reduce the societal costs of tornadoes is to provide information on the probability of a tornado happening in a location instead of providing deterministic yes/no prediction (Rothfusz et al. 2018). In theory, probabilistic extreme weather warnings give more detailed information to users and enable them to make better decisions (Murphy 1993; Papastavrou and Lehto 1996). Potential users in the United States also demonstrate preference for receiving probabilistic versus deterministic weather forecasts (Morss et al. 2008, 2010). At the same time, probabilistic warnings might reduce the decision quality for some users, and hence, it is not clear a priori whether their potential societal benefits outweigh the additional cost of development and delivery of more sophisticated forecasts.
The main question of this study is to evaluate whether providing probabilistic tornado warnings instead of deterministic ones would benefit U.S. households. It involves measuring the total societal costs of tornadoes, both with deterministic and probabilistic warnings. If probabilistic warnings indeed significantly reduce the societal costs of tornadoes, then their development and implementation should be supported by the government. The second question of this study is to explore the responses to probabilistic warnings, which can help to improve the design of both deterministic and probabilistic warnings.
This paper uses population surveys to calculate the societal benefits of deterministic and probabilistic tornado warnings. The calculation of societal benefits accounts for their effects on fatalities, injuries, and sheltering time. I assign monetary measures to fatalities and injuries by using the value of statistical life (VSL) approach and price the inconveniences of sheltering time based on the concept of opportunity costs of time.
This work involves three steps. First, I conduct a household survey to learn the population’s protective responses both to current deterministic tornado warnings and to prospective probabilistic ones. These responses account both for probability levels and for housing types. However, extreme weather alerts do not help if protective responses are ineffective in the sense that they have weak effects on casualty rates. So, in the second step, I evaluate the effectiveness of protective responses conditional on housing type by using the data on historic variation in weather information quality and tornado casualties. Finally, we use the current joint distribution of deterministic forecasts and tornado events to estimate the frequency of probabilistic alerts for each probability level. This last step is important, because it allows the forecasting format to change while keeping the quality of forecasting technology constant.
I calculate that probabilistic tornado warnings should create net annual benefits between $76 and $139 million depending on the calculation method used. The lower estimate assumes that the population has identical opportunity costs of time, while the larger estimate assumes that these costs vary across individuals. Varying opportunity costs imply that individuals shelter if and only if their costs of sheltering are below their perceived costs of life or injury. The benefit of probabilistic warnings is calculated relative to deterministic ones, which on their own already create $96–140 million per year of net societal value. This estimate already accounts for imperfect awareness and compliance with warnings and for imperfect protection technology.
Most respondents demonstrate good understanding of probabilistic warnings and good calibration of responses to threat levels. Reported protective responses tend to increase with tornado probabilities. More interestingly, opportunity costs of time implied by their protective responses are consistent with previous estimates of opportunity costs of time in the literature. This supports the idea that potential users correctly deduce their personal risk levels from probabilistic warnings.
In response to probabilistic alerts, more people report being willing to monitor the threat as compared with deterministic warnings, but they expect to shelter when the danger becomes imminent. This leads to more people reacting to probabilistic warnings and eventually more people taking shelter. Hence, probabilistic warnings reduce total casualties, while increasing the total time spent sheltering or monitoring the weather. This increase does not necessarily convert to higher societal costs. If we account for optimal responses to predicted tornado probabilities and deduce opportunity costs from reported protective responses, then the societal value or opportunity cost of sheltering/monitoring time goes down due to a more graduated reaction to probabilistic warnings. Probabilistic warnings deliver this positive effect by enabling users with higher opportunity costs to shelter only if tornado threats are sufficiently high.
This paper contributes to the literature by directly measuring the net benefits of both deterministic and probabilistic tornado warnings for the population. Howard et al. (2021) estimate the value of probabilistic warnings for businesses in the Dallas metropolitan area and find that probabilistic warnings would save an additional $1.3–5.6 billion per year as compared with deterministic warnings. Their calculation does not cover the general population (households), which has very different capabilities to understand and respond to warnings. This paper also significantly improves on their method by using the distribution of probabilistic forecasts, which is more consistent with existing forecasters’ skills. Simmons and Sutter (2013) calculate societal costs of tornadoes for the general population but do not study the value of probabilistic warnings. They estimate that the contemporary costs are roughly $6 billion lower than the hypothetical costs with tornado lethality at the 1925 U.S. level, when warnings were nonexistent. However, this reduction in costs cannot be completely attributed to the effect of deterministic tornado warnings due to other safety improvements happening during this period. This paper takes a more conservative approach to estimate the benefits of both deterministic and probabilistic warnings by accounting for imperfect compliance with warnings and by calculating their effectiveness directly from the variation in casualties between warned and nonwarned populations.
In contrast to my approach, multiple other studies evaluate weather information (Lazo and Chestnut 2002; Lazo et al. 2009; Lazo and Waldman 2011; Wehde et al. 2021) with the contingent valuation method, in which potential users directly report their willingness to pay for the service. The only published valuation study of probabilistic tornado warnings for the population, that by Wehde et al. (2021), falls into this category. It finds that the U.S. population is willing to pay on average $7.5 per person for an app providing probabilistic graphical tornado alerts. This price translates to a one-time aggregate benefit between $900 million and $1.56 billion depending on the aggregation assumptions used. While contingent valuation studies can potentially reflect additional benefits of information, such as peace of mind or the increased safety of others, they suffer from a hypothetical bias emerging due to respondents deliberately overstating their willingness to pay (Blumenschein et al. 2008; Johnston et al. 2017). As a result, contingent valuation studies often provide excessively high and varying estimates of economic benefits. Hence, my direct approach gives an important and more reliable lower bound of the new system’s value.
My study supports the conclusion that the U.S. population can interpret and use probabilistic warnings. Multiple previous studies (Ash et al. 2014; Lindell et al. 2016; Miran et al. 2017) have tested perception and hypothetical responses to the graphical representation of probabilistic severe weather alerts. In general, they have found that people increase protection in response to increasing threat probabilities, even though presentation formats have a strong influence both on average response levels and on the sensitivity of response to presented probabilities. Additionally, LeClerc and Joslyn (2015) find that probabilistic information improves decision-making and reduces the “cry wolf” effect, whereas Krocak et al. (2022) find that probabilistic information allows for better decision-making when compared with categorical verbal descriptions of uncertainty. I find that protective responses are sensitive to projected probabilities, but also that response levels are well calibrated to threat levels and consistent with choices made in other domains (such as speeding; Wolff 2014).
2. Survey design and implementation
a. Data collection
I collect the data from two samples. The mail survey recruited respondents across the whole United States but with an emphasis on tornado-prone regions (see Table 1, along with Table A1 in appendix A). Respondents could choose to respond by mail by using an enclosed envelope or to fill out the survey online. The internet survey recruited participants from the tornado-prone regions only. The use of different sampling methods allowed for a wider representation of different demographic groups. The mail survey reached more older respondents living in rural communities, while the internet survey helped to get answers from younger respondents. I received 718 responses from the mail survey and 403 responses from the internet survey. Questionnaires were practically identical, except for small changes needed to screen respondents in the internet survey.
Mail survey sample.
The mail survey uses stratified probabilistic sampling to get a more representative sample, which allows the use of statistical tests. My initial frame comes from the U.S. Postal Service delivery route database. I stratify the sampling frame by state of residence and by housing type and sample 10 600 addresses, with more addresses from tornado-prone states. I consider the state to be tornado prone if it belongs to one of 20 states with the highest average incidence of significant tornadoes (EF2 and above) per square mile (1 mi ≈ 1.6 km; 1 mi2 ≈ 2.6 km2) within the last 20 years. The selected states include 45% of the U.S. population, but 88% of tornado fatalities. This paper uses only the sample obtained from the tornado-prone states.3
The questionnaire was pretested, first, by using qualitative personal interviews conducted either in person or over Google Meets and Skype. These interviews helped to clarify the questions’ wording and make sure that their interpretation by participants matched my expectations. In the second stage, I conducted quantitative pilots both for the internet sample and for the mail survey.
I pretest our surveys by using, first, qualitative face-to-face4 interviews and, second, quantitative pilot studies. The qualitative interviews helped to clarify the understanding of questions and refine the lists of response options. The interview followed think-aloud protocols (Dillman et al. 2008) in which respondents read all the questions aloud and vocalize their thinking process. Quantitative pilot studies followed the same procedure I intended to use for the main study, but with smaller samples. I conducted two pilot studies for the internet survey and two pilot studies for the mail survey. The pilots helped to adjust my sampling strategies and redesign a few questions that had turned out to be ambiguous to the participants.
b. Representativeness and selection bias
Despite my effort to use different recruiting efforts, both samples had disproportionately more females and more people with a college education or above (see Table 1 along with Table A2 for the internet sample in appendix A). Additionally, the mail survey recruited more older White respondents. Hence, to translate our findings to the U.S. population, I reweight my results to match the U.S. population structure by age and sex.5
3. Use of standard and extended tornado alerts
First, I study how extended tornado warnings would affect protective responses. I ask the respondents to imagine being at home with their family at 7:00 p.m. when a tornado warning is issued. Next, I elicit their protective responses conditional on lead time and on probability of a tornado happening within a given time interval. The internet survey asks the same set of questions for the nighttime warnings (2:00 a.m.).6 Note that individuals can face tornado threats at other times and locations beyond their homes; due to limitations on the number of questions I can include in the study, I focus only on these two scenarios, which I consider to be the most representative. Most respondents choose to respond to a standard tornado warning by taking shelter at home or nearby. The proportion of respondents choosing this action (55%) is surprisingly stable across samples and across times in the same sample (Fig. 1). About 10% of respondents in the internet sample choose to drive to another house or structure or to drive out of the potential tornado path. This proportion is slightly higher for the internet sample (Fig. B1 in appendix B).
Protective response by lead time.
Citation: Weather, Climate, and Society 15, 3; 10.1175/WCAS-D-22-0139.1
Roughly one-third of respondents expect to seek more information without taking shelter. Here, and in the calculation of effectiveness of probabilistic warnings, I consider seeking more information as one of the protective actions, because previous studies have shown that most people take shelter when they know that the danger is imminent. Hammer and Schmidlin (2002) and Klockow (2011) show that most people in a tornado strike zone take shelter, but fewer people do it in a tornado warning zone (Liu et al. 1996; Sherman-Morris 2010). Because tornado strike zones or paths are much smaller than typical warning areas, residents often prefer to collect information before taking protective actions. For example, Hammer and Schmidlin (2002) surveyed residents in the Oklahoma City, Oklahoma, tornado strike zone and found that 55% received tornado warnings from more than one source, and almost 90% of residents eventually either evacuated or took shelter in interior rooms during the tornado. A household survey in the area of the 2011 Alabama tornado outbreak (Klockow 2011) also found that most people monitored media and that many looked at the sky but started sheltering only when a tornado was 1–2 min away from them.
Increasing lead time to 40 min on its own has practically no effect on the total proportion of people taking any protective action (which includes seeking more information without taking shelter), as more than 90% of individuals do it anyway. However, increasing lead time decreases the likelihood of sheltering at home in favor of seeking more information and evacuating. It is very plausible that extended lead time improves the safety of people in vulnerable housing conditions, such as mobile homes, when evacuation is practically the only effective protection option (Schmidlin et al. 2009). However, the safety of people living in more robust homes depends on their ability to interpret additional information they receive while not sheltering and properly responding to it.
Providing probabilistic information is the most crucial aspect of prospective tornado alert systems, but its usefulness relies on users’ ability to understand and react to probabilistic forecasts. The survey indicates that most individuals respond rationally to probabilistic warnings. The proportion of respondents choosing to take protective actions increases with the forecast probability. Almost 100% of respondents expect to take some protective action if they learn that a tornado will happen with 100% probability in the next 40 min in a 10-mi radius from their location. Fewer than 5% of respondents make nonmonotonic choices, meaning that 95% of respondents take protective actions for all probabilities that are higher than their threshold probability.
Most individuals expect to take protective actions when the probability gets to 20%. In the internet survey, 59% of individuals respond when the probability of a tornado within a 10-mi-radius circle is just 10%. In the mail sample, 60% of respondents take protective actions when the probability is 20% (it was the lowest probability in the mail sample). For comparison, my calculations show that the comparable implied probability for the deterministic warning is roughly 35%,7 so the majority of the population expects to take protective actions for much lower probabilities than the tornado probability of the deterministic warning. The range between 0% and 20% probability is also the range of highest sensitivity to risk, in which the largest share of respondents switches from no protection to protection, as can be seen from the slope of the line in Fig. 2. The reaction threshold is higher for nighttime warnings (see Fig. B2 in appendix B). This is consistent with the higher costs of nighttime protective actions for most respondents, as they potentially require interrupting sleep and driving with poor visibility.
Protective response by probability of a tornado.
Citation: Weather, Climate, and Society 15, 3; 10.1175/WCAS-D-22-0139.1
4. Computation of direct societal benefits
a. Overview of the approach
I assign monetary value to injuries in a similar fashion. Most tornado injuries are minor, so following the approach in Simmons and Sutter (2006, 2013), the monetary value of injury is 1/100 of the value of statistical life, which is $111,300 per injury.
The baseline fatality or injury rate per person in a strike area r is the probability that a person in a tornado strike zone is respectively killed or injured in a tornado if the person does not take protective actions. The protective mitigation factor M measures the proportional decrease in risk of injury/death from the expected protective response. It depends both on the expected behavior and on the effectiveness of this behavior in reducing the risk. These two variables strongly depend on housing conditions, so I condition my calculation on living in permanent versus mobile homes and weight by corresponding population proportions. I explain the calculation of the baseline fatality and injury rates and the protective response mitigation factors in the next subsection.
b. Protective response effectiveness
Protective response mitigation factor M measures the proportional effect of protective actions on tornado fatalities and injuries. Because I am not aware of any generalized estimates of protective response effectiveness in the literature, I estimate it indirectly from the casualty effects of tornado warnings and other historical data. This estimation assumes that households take protective actions only in response to warned tornadoes and that the protection response is not universal. I also assume that the protective response has the same proportional effect on reducing both fatalities and injuries.
Simmons and Sutter (2009) find that the warned tornadoes on average have 30%–40% fewer injuries, controlling for tornado strength, strike area, geography, and time. Similarly, Simmons and Sutter (2005) find that when a weather forecast office (WFO) in the United States installs a WSR-88, tornado injuries in covered counties go down by approximately 40%. Based on this evidence, I make a relatively conservative assumption that warnings reduce injuries by 35%. While the paper does not observe an effect of warnings on fatalities, this is likely the result of a much smaller number of fatalities in the sample. Consistent with these observations, I also assume that warnings reduce fatalities by 35%.
Next, I estimate the baseline risk and the mitigation effectiveness for residents of permanent homes. I do this by substituting the risks of mobile home residents into Eq. (4) and solving the resulting system of (1–3) for
To calculate the mitigation effectiveness factor mperm for residents of permanent homes, I need to account for imperfect compliance with issued warnings. Previous studies indicate that while the response rate to warnings Rperm is close to around 30% for warned counties (Liu et al. 1996; Schmidlin et al. 2009), the response rate reaches 70%–90% for population directly in a tornado path and for stronger tornadoes (Klockow 2011; Paul et al. 2015). As only the response of individuals in a path matters for casualties, I assume that 60% of permanent home residents in a tornado path take some protective action (Rperm = 0.6). It follows that taking protective actions mitigates the baseline risk for permanent homes by approximately 65% (mperm = 0.361).
Event studies support my finding of high mitigation effectiveness for permanent homes. For example, Niederkrotenthaler et al. (2013) finds that sheltering in a basement reduced injuries by roughly 80% during the April 2011 Alabama tornadoes, while Daley et al. (2005) find no severe injuries or deaths among people doing so during the Oklahoma City 1999 tornado. The same applies for the 2011 Joplin tornado (Paul et al. 2015). The evidence for using interior rooms as shelters is more mixed. Niederkrotenthaler et al. (2013) find that sheltering in an interior room reduced the risk of injury by about 60%, but Daley et al. (2005) find just a 20%–30% reduction in severe injuries, and Hammer and Schmidlin (2002) find no effect of using an interior room versus any other room in a permanent house.
We apply the same approach to the calculation of injury risks. The calculation assumes an average risk of injury of 0.025 for mobile homes in the strike area and a risk of 0.0224 for permanent homes in the strike area [based on the Simmons and Sutter (2013) calculation]. The baseline risk of injury for permanent homes equals 0.0306, and the baseline risk for mobile homes equals 0.0316. While the predicted injury risk is very similar for both home types, it seems that permanent homes give better protection against death, but not much more protection against nonfatal injuries.15
c. Distribution of probabilistic forecasts
A population’s protective responses depend on perceived probabilities. Hence, we need to know how often each probability is forecast to estimate the costs of probabilistic warnings. This task is nontrivial, because for any probability of a tornado, one can issue different unbiased probabilistic forecasts. For example, one completely unbiased but also completely useless forecast is a forecast that is always equal to the baseline (environmental) probability of a tornado occurring. At the opposite end of the precision spectrum, forecasters can predict a probability of 1 if a tornado is going to happen and 0 otherwise. In practice, dynamic properties of weather systems and imperfect information impose constraints on the maximum precision of tornado forecasts.
I am going to use the signal detection theory to infer the distribution of probabilistic forecasts from the joint distribution of tornado warnings and tornado events.16 The signal detection approach assumes that probabilistic forecasts use the same information as existing standard warnings. If it is indeed true, all the information can be aggregated to one signal equal to the posterior probability of a tornado occurring. In the simplest case, which I use here, this signal has a normal distribution with a dispersion of 117 and a mean depending on the actual state of the world. If the state of the world is indeed the state in which a tornado forms, the signal has a higher mean. The difference between signal means in tornadic and nontornadic state D′ measures the forecaster’s ability to discriminate between two states of the world.
Brooks (2004) demonstrates how to use the historic performance of tornado warnings to estimate the difference in means D′ between the latent signal distribution in tornado and nontornado states. Brooks and Correia (2018) use the same approach and estimate that in recent years, the performance is consistent with 1 < D′ < 1.4, if the baseline probability of a tornado conditional on a storm is 10%.18 I use D′ = 1.35 on the upper end of this range to reflect improvements in warning performance in the early 2000s and potential improvements due to better satellite data and dual polarization radars in more recent years.
The resulting distribution of forecast probabilities (Fig. 3) is concentrated around low-probability events, which follows from both a low baseline probability of a tornado and our relatively modest ability to forecast tornadoes.19 Only 2.5% of forecasts predict probabilities above 50%. However, 30% of forecasts predict that chances are above the baseline 10%, and 14.5% predict that chances of a tornado are above 20%.
Projected distribution of probabilistic tornado forecasts.
Citation: Weather, Climate, and Society 15, 3; 10.1175/WCAS-D-22-0139.1
d. Sheltering costs
The opportunity costs of sheltering reflect the disutility of sheltering instead of continuing normal activities. It is equal to the product of value of time per total annual number of hours spent sheltering in each scenario. Obviously, value of time depends on activities interrupted and their utility versus the utility of sheltering, which can drastically differ both by individual and by time of day. For example, people sleeping in their basements do not have to interrupt this activity for sheltering and hence have exactly zero value of time for sheltering. In contrast, people working at home in an unsafe location might need to stop working, which either reduces their earnings roughly by wage rate per hour or reduces their remaining leisure time.
The first approach assumes that the opportunity cost of time is uniform across the population and sets this parameter based on previous literature. However, I cannot use the same exact numbers because the costs change with inflation, wages, and sample structure. Instead, I rely on studies calculating opportunity costs as the proportion of wage rate and then translate older results for the contemporary state of the economy. As paid work is one of the main activities conducted by working adults, the wage rate provides a natural benchmark for the value of time in this approach. Multiple studies, however, find that even for working adults, the value of time is significantly lower than their wage rate. For example, Larson et al. (2004) find that the value of time varies from 0.5 for adults with a fixed week to 0.8 for adults with a flexible workweek. Wolff (2014) put the value of time as 50% of the wage rate based on an analysis of speeding tickets and gasoline consumption. Consistent with this literature but corrected for the large proportion of individuals out of the labor force in my sample, I use one-third of the average wage rate to estimate the sheltering time. The average civilian nonfarm wage was equal to $29.35 in 2020. This translates to an opportunity cost of sheltering time of $9.80 h−1.
Most values for probabilistic warnings come from the heterogeneity of their users in terms of costs of sheltering versus safety concerns. Probabilistic warnings allow rational sheltering decisions based on an individual cost–benefit analysis with respect to predicted probabilities. For example, a person in a well-protected house might decide against sheltering if the probability is 20% but will shelter when the probability increases to 60%.
The calculated sheltering costs (see Table 2) are comparable to the uniform sheltering costs, which I took as one-third of the median wage rate or $9.80 h−1. Note that the calculation of heterogeneous sheltering costs uses only reported decision and not wage rates. It demonstrates that most individuals neither overreact nor underreact to predicted tornado risks, with protection decisions being highly consistent with other domains used to estimate the value of statistical life. The lowest opportunity cost of sheltering for permanent home residents is just $3.35 h−1 if using the upper-bound approach and $0.80 if using the lower-bound approach. The second group of permanent homes residents, which switches to protection when the risk goes from 20% to 40%, has sheltering costs ranging between $3.35 and $6.71. The average sheltering cost is between $3.20 and $5.90 for permanent home residents and between $126 and $144 for mobile home residents.21 Higher sheltering costs for mobile home residents reflect both limited protection options and their higher effectiveness; the only realistic protection plan involves moving to the closest sturdy shelter or out of the tornado path.
Distribution of opportunity costs.
I assume that everyone taking shelter or evacuating stops normal activities exactly for the duration of the tornado warning. The average warning duration has been decreasing since the early 2000s. For this reason, I use the latest number available from Brooks and Correia (2018). The latest year they cover is 2015, with the corresponding average duration of 37.5 min. I also assume that people choosing to collect more information without sheltering do not bear any time costs. Checking information sources most frequently mentioned in the survey (cell phone apps, internet) requires relatively little time or can be done without interrupting normal activities. While I assume that these individuals would eventually shelter if they happen to be in a strike zone, the average strike zone area is negligible relative to the average warning area.
5. Results
While my calculation does not aim to provide accurate forecasts of total tornado fatalities and injuries in the United States, it is important to match the scale of potential casualties to receive an unbiased estimate of total cost savings, and I do it reasonably well. My predicted tornado casualties with deterministic warnings (around 50 fatalities per year) are similar to historical rates. For comparison, on average, tornadoes were killing 78 people in the United States per year in 1980–2019,22 and this number included people killed outside of their residences.
The calculation presented in Table 3 indicates that deterministic warnings save roughly 19 lives per year, not accounting for victims outside and in places of work. I expect that probabilistic warnings would on average save an additional seven lives per year. This effect comes from many people starting to react to warnings when the forecast probability is still below the threshold required to issue deterministic warnings. The reduction in injuries is proportional to the reduction in fatalities as consistent with my assumptions.
Societal costs by tornado warning approach.
The decrease in fatalities and injuries translates into significant monetary gains from both standard and probabilistic warnings if using the statistical value of life or injury to value casualties. The total casualty (fatalities + injuries) cost of tornadoes without warnings is $871.5 million per year. Deterministic warnings reduce the costs of casualties by approximately $250 ($252.0) million. Probabilistic warnings additionally reduce the costs of casualties by roughly $85 million per year.
Accounting for the opportunity costs of sheltering time obviously decreases the net societal value of deterministic warnings, but it is still fairly large. The net benefit of deterministic warnings is $95.5 million per year under the assumption of uniform opportunity costs and $139.6 million per year under the assumption of heterogeneous opportunity costs. The assumption of the heterogeneity of opportunity costs matters because it implies that only users with lower opportunity costs take shelter in response to warnings if their costs are lower than the average risk implied by the deterministic warning. I find that even for the deterministic warnings, the benefit of reduced casualties outweighs additional opportunity costs of sheltering. This observation is true both for uniform and heterogeneous opportunity costs. However, their net effect on societal costs is relatively modest. In contrast, Simmons and Sutter (2013) find that the societal costs of tornadoes calculated for the constant population and constant value of statistical life and injury go down by around $6 billion between 1925 and 2000. Their approach is very similar to mine, as they also account for the value of statistical lives lost and opportunity costs of time. However, this enormous change in societal costs does not necessarily come from tornado warnings. The calculation also seems to use an inflated fatality baseline due to the most deadly and extremely strong tristate tornado event happening at the beginning of this period. This period also saw improvements in building quality, better health care, and more awareness of tornado protective strategies.
Probabilistic warnings further reduce the societal costs of tornadoes. Most of this effect comes from reducing tornado fatalities and casualties. This safety increase has a downside as more people start sheltering, but as long as decisions to shelter respond rationally to actual opportunity costs, probabilistic warnings would also reduce the societal costs of sheltering. I estimate that probabilistic warnings would provide net benefits23 of $76 million per year if assuming uniform opportunity costs of time and $139.4 million if accounting for costs heterogeneity ($95.1 million if using a lower-bound estimate of sheltering cots). The large discrepancy between the values calculated for uniform and heterogeneous costs shows that a large, if not the largest, value of probabilistic warnings comes from more nuanced sheltering decisions. When forecasters predict a very high chance of a tornado, most individuals expect to take shelter, but when the predicted chance is low, only people with easy access to shelter or no important competing activities do so.
6. Conclusions
I evaluate the benefits of deterministic and probabilistic tornado warnings by asking potential users about their behavioral responses. Based on individual responses, I predict lives saved and hours of sheltering time and convert them into monetary terms. This work requires evaluating the effectiveness of protective responses and the effectiveness of future probabilistic forecasts.
I find that both deterministic and projected probabilistic tornado warnings deliver significant positive net benefits for the United States. Deterministic tornado warnings save around 20 lives per year and create around $96–140 million of net societal benefit. Probabilistic warnings additionally increase this benefit by another $76–139 million per year. I estimate that most probabilistic forecasts will involve low tornado probabilities. Hence, the benefit of probabilistic forecasts emerges mostly because warnings issued for probabilities below deterministic threshold save additional lives. In addition, probabilistic warnings also reduce sheltering by individuals with high sheltering costs when projected probabilities are low, which reduces the total cost of time spent sheltering.
My calculation of the societal benefits of tornado alerts does not account for other potential psychological benefits of tornado warnings. For example, the laboratory experiment conducted by Eliaz and Schotter (2010) demonstrates that people are willing to pay for information not used in decision-making if this information helps them evaluate previously made decisions. In a similar vein, the model of Golman et al. (2021) postulates that people want to get information to fill their information gaps. In addition, many people derive value from public goods only due to their use to others (“nonuse value”). For these reasons, my estimate of societal benefits should be treated as a lower bound, while the real value might be significantly higher. However, it is important that even the calculated benefits seem large enough to justify the costs of developing and implementing probabilistic tornado warnings.
This research has two significant policy implications. First, even though my estimates are on the lower and more conservative side, the projected annual benefits of $76–139 million are still high enough to more than justify the research and development expenses needed to develop and transition to probabilistic warnings. For comparison, in 2023, NOAA expects to spend just $20.9 million on its Tornado Severe Storm Research/Phased Array Radar,24 which includes probabilistic warnings as part of the research agenda. Second, it indicates that probabilistic warnings should be issued for much lower probabilities than the currently existing thresholds for deterministic warnings. I observe that around one-half of the population starts taking some protective measures if the tornado probability is as low as 10%. Hence, the distribution of potential forecasts should include warnings even when the probability of a tornado is estimated to be as low as 10% or potentially 5%. Additional information provided to households will still allow them to make better decisions and shelter only if their perceived risks outweigh the costs of sheltering.
The high calculated benefits of probabilistic warnings point to the need for further research work on their optimal design. While this is already an active research area, it still might benefit from more experimental studies using their actual implementations instead of hypotheticals. Using actual technologies would allow eliciting unbiased users’ preferences between different systems as well as tracking their usage over time, geography, and weather events. This amazing research becomes much easier because of the proliferation of mobile devices and increasing mobile connection speeds.
By implicit costs, I mean those costs that are not paid directly. For example, plumbers stuck in a traffic jam bear implicit costs because they could have earned more if they had worked instead.
I have 100 responses from the other states but decided not to include them because, as evidenced by their open-response comments, many of these respondents have never considered responding to a tornado emergency/heard a warning and, hence, their reported protection plans are unlikely to correlate well with future behavior.
Because of the COVID-19 pandemic, most of the qualitative interviews were conducted online through Zoom, Skype, or Google Meets.
I use the data from the American Community Survey 2018, downloaded from IPUMS (Ruggles et al. 2021).
The mail survey conducted after the internet survey had to drop these questions in an effort to shorten the questionnaire.
I calculate this number by taking the probability of a tornado conditional on a deterministic warning, which is roughly 1 − FAR = 0.3 (Simmons and Sutter 2013), and correcting it upward to reflect a larger area of a 10-mi-radius circle (314.2 mi2) as compared with the average area of a tornado warning (272 mi2).
To save on notation, I use the same variable names to denote both expected injuries and expected fatalities. The formulas are identical.
The formula is derived in the following way. By definition, POD is equal to the proportion of positive events for which the warning is issued: POD = warned tornadoes/total tornadoes. The number of warned tornadoes is equal to the number of warnings multiplied by the proportion of true warnings: warned tornadoes = Nw(1 − FAR). Hence, total tornadoes = warned tornadoes/POD = Nw(1 − FAR)/POD. Then, I calculate the area affected as the product of the total tornadoes multiplied by the average tornado strike area: area affected = A × total tornadoes = ANw(1 − FAR)/POD. Last, I multiply the total area affected by average population density d to get the final formula above. The last step assumes that the tornado strike area is independent of the population density.
Brooks and Correia (2018) find that with storm-based warnings, POD went from 0.7 in 2011 to 0.5 in 2016. Using a POD of 0.5 in my calculation slightly increases my projected benefits of both probabilistic and deterministic warnings. However, as this POD decrease does not reflect a growing frequency of tornadoes or worsening of forecasters’ skills (Brooks and Correia 2018), I choose to keep the same POD of 0.7 both in calculations based on historical data and for future projections.
One can obtain this equation by noting, first, that if there are F forecasts in total, then there are Fi = fiF forecasts predicting probability pi. If the forecast probability matches the true probability of a tornado conditional on forecast, then there are Xi = piFi = pifiF people affected by tornadoes within that predicted probability bin. As the total number of people affected by tornadoes remains constant at PA, I know that
I use only a larger mail sample for calculating protective responses, because responses in the internet sample seem to involve more social desirability bias with more excessive protection. This is evident in a sizable proportion of the population reporting protective actions when the probability of a tornado is zero (see Fig. B2 in appendix B).
Note that in contrast to the mitigation factor M, which combines the propensity of protective actions with their effectiveness, mt measures only the effectiveness of the protective action conditional on acting.
The absence of differences in injury rates between permanent and mobile homes seems counterintuitive and can be a result of measurement issues. Other studies, unfortunately, report varying results due to even smaller sample sizes. For example, Glass et al. (1980) find a much higher injury rate among mobile home residents, but their results are based on just 14 households with mobile homes. Daley et al. (2005) find a higher incidence of severe injuries among mobile home residents than among residents of permanent homes but a lower incidence of minor injuries.
Howard et al. (2021) use a simpler approach by assuming equal forecasting frequency for each probability. However, this approach can easily overestimate the precision of probabilistic forecasts and their value, because it implies a much higher average confidence of the forecaster than allowed by the existing technology.
One can always rescale the signal without the loss of generality to get the dispersion to equal one.
This probability corresponds to the average forecast probability of a tornado weighted by forecast frequency
It is arguably much harder to forecast a tornado 10 min in advance than to forecast rain 1 h in advance.
I count one person multiple times if they receive multiple warnings during the year.
Some individuals do not expect to shelter for any projected risk. The calculation of average sheltering costs assumes that their opportunity costs correspond to 100% probability of a tornado. As this group never takes protective actions, its presence has no effect on total sheltering costs for any type of warning.
My calculation based on the Storm Prediction Center database.
Not accounting for technological costs: research and development and additional training of meteorologists.
https://www.noaa.gov/sites/default/files/2022-05/508_Compliant_Final_FY23_NOAA_Blue_Book_Budget_Summary.pdf for the Green Book NOAA budget request for 2023.
Acknowledgments.
This work was supported by the NOAA/Office of Oceanic and Atmospheric Research under NOAA–University of Oklahoma Cooperative Agreement NA16OAR4320115, U.S. Department of Commerce. The author appreciates the support and comments from other University of Oklahoma CIWRO researchers, but most of all Kimberley Klockow and Harold Brooks.
Data availability statement.
The anonymized survey data and the code used to process them are available online (https://github.com/AlUgarov/Benefits_ProbWarnings).
APPENDIX A
APPENDIX B
Additional Graphs (Internet Sample)
Figure B1 shows protective response by lead time, and Fig. B2 shows protective response by probability of a tornado.
Protective response by lead time (internet sample).
Citation: Weather, Climate, and Society 15, 3; 10.1175/WCAS-D-22-0139.1
Protective response by probability of a tornado (internet sample).
Citation: Weather, Climate, and Society 15, 3; 10.1175/WCAS-D-22-0139.1
REFERENCES
Ash, K. D., R. L. Schumann III, and G. C. Bowser, 2014: Tornado warning trade-offs: Evaluating choices for visually communicating risk. Wea. Climate Soc., 6, 104–118, https://doi.org/10.1175/WCAS-D-13-00021.1.
Blumenschein, K., G. C. Blomquist, M. Johannesson, N. Horn, and P. Freeman, 2008: Eliciting willingness to pay without bias: Evidence from a field experiment. Econ. J., 118, 114–137, https://doi.org/10.1111/j.1468-0297.2007.02106.x.
Brooks, H. E., 2004: Tornado-warning performance in the past and future: A perspective from signal detection theory. Bull. Amer. Meteor. Soc., 85, 837–844, https://doi.org/10.1175/BAMS-85-6-837.
Brooks, H. E., and J. Correia Jr., 2018: Long-term performance metrics for National Weather Service tornado warnings. Wea. Forecasting, 33, 1501–1511, https://doi.org/10.1175/WAF-D-18-0120.1.
Daley, W. R., S. Brown, P. Archer, E. Kruger, F. Jordan, D. Batts, and S. Mallonee, 2005: Risk of tornado-related death and injury in Oklahoma, May 3, 1999. Amer. J. Epidemiol., 161, 1144–1150, https://doi.org/10.1093/aje/kwi142.
Dillman, D. A., J. D. Smyth, and L. M. Christian, 2008: Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method. 3rd ed. John Wiley and Sons, 512 pp.
Eliaz, K., and A. Schotter, 2010: Paying for confidence: An experimental study of the demand for non-instrumental information. Games Econ. Behav., 70, 304–324, https://doi.org/10.1016/j.geb.2010.01.006.
Glass, R. I., R. B. Craven, D. J. Bregman, B. J. Stoll, N. Horowitz, P. Kerndt, and J. Winkle, 1980: Injuries from the Wichita Falls tornado: Implications for prevention. Science, 207, 734–738, https://doi.org/10.1126/science.207.4432.734.
Golman, R., G. Loewenstein, A. Molnar, and S. Saccardo, 2021: The demand for, and avoidance of, information. Social Science Research Network, 2149362, 118 pp., https://doi.org/10.2139/ssrn.2149362.
Hammer, B., and T. W. Schmidlin, 2002: Response to warnings during the 3 May 1999 Oklahoma City tornado: Reasons and relative injury rates. Wea. Forecasting, 17, 577–581, https://doi.org/10.1175/1520-0434(2002)017<0577:RTWDTM>2.0.CO;2.
Howard, S. P., K. E. Klockow-McClain, A. P. Boehmer, and K. M. Simmons, 2021: Firm behavior in the face of severe weather: Economic analysis between probabilistic and deterministic warnings. Wea. Forecasting, 36, 757–767, https://doi.org/10.1175/WAF-D-20-0107.1.
Johnston, R. J., and Coauthors, 2017: Contemporary guidance for stated preference studies. J. Assoc. Environ. Resour. Econ., 4, 319–405, https://doi.org/10.1086/691697.
Klockow, K., 2011: Investigation of individual spatial awareness relating to response during the April 27, 2011 tornado outbreak. Natural Hazards Center Quick Response Rep. 224, 15 pp.
Kniesner, T. J., and W. K. Viscusi, 2019: The value of a statistical life. Social Science Research Network, 3379967, 45 pp., https://doi.org/10.2139/ssrn.3379967.
Krocak, M. J., J. T. Ripberger, S. Ernst, C. L. Silva, and H. C. Jenkins-Smith, 2022: Exploring the differences in SPC convective outlook interpretation using categorical and numeric information. Wea. Forecasting, 37, 303–311, https://doi.org/10.1175/WAF-D-21-0123.1.
Larson, D. M., S. L. Shaikh, and D. F. Layton, 2004: Revealing preferences for leisure time from stated preference data. Amer. J. Agric. Econ., 86, 307–320, https://doi.org/10.1111/j.0092-5853.2004.00580.x.
Lazo, J. K., and L. G. Chestnut, 2002: Economic value of current and improved weather forecasts in the U.S. household sector. NOAA Office of Policy and Planning Rep., 213 pp.
Lazo, J. K., and D. M. Waldman, 2011: Valuing improved hurricane forecasts. Econ. Lett., 111, 43–46, https://doi.org/10.1016/j.econlet.2010.12.012.
Lazo, J. K., R. E. Morss, and J. L. Demuth, 2009: 300 billion served: Sources, perceptions, uses, and values of weather forecasts. Bull. Amer. Meteor. Soc., 90, 785–798, https://doi.org/10.1175/2008BAMS2604.1.
LeClerc, J., and S. Joslyn, 2015: The cry wolf effect and weather-related decision making. Risk Anal., 35, 385–395, https://doi.org/10.1111/risa.12336.
Lindell, M. K., S.-K. Huang, H.-L. Wei, and C. D. Samuelson, 2016: Perceptions and expected immediate reactions to tornado warning polygons. Nat. Hazards, 80, 683–707, https://doi.org/10.1007/s11069-015-1990-5.
Liu, S., L. E. Quenemoen, J. Malilay, E. Noji, T. Sinks, and J. Mendlein, 1996: Assessment of a severe-weather warning system and disaster preparedness, Calhoun County, Alabama, 1994. Amer. J. Public Health, 86, 87–89, https://doi.org/10.2105/AJPH.86.1.87.
Miran, S. M., C. Ling, J. J. James, A. Gerard, and L. Rothfusz, 2017: User perception and interpretation of tornado probabilistic hazard information: Comparison of four graphical designs. Appl. Ergon., 65, 277–285, https://doi.org/10.1016/j.apergo.2017.06.016.
Morss, R. E., J. L. Demuth, and J. K. Lazo, 2008: Communicating uncertainty in weather forecasts: A survey of the U.S. public. Wea. Forecasting, 23, 974–991, https://doi.org/10.1175/2008WAF2007088.1.
Morss, R. E., J. K. Lazo, and J. L. Demuth, 2010: Examining the use of weather forecasts in decision scenarios: Results from a US survey with implications for uncertainty communication. Meteor. Appl., 17, 149–162, https://doi.org/10.1002/met.196.
Murphy, A. H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8, 281–293, https://doi.org/10.1175/1520-0434(1993)008<0281:WIAGFA>2.0.CO;2.
Niederkrotenthaler, T., and Coauthors, 2013: Injuries and post-traumatic stress following historic tornados: Alabama, April 2011. PLOS ONE, 8, e83038, https://doi.org/10.1371/journal.pone.0083038.
Papastavrou, J. D., and M. R. Lehto, 1996: Improving the effectiveness of warnings by increasing the appropriateness of their information content: Some hypotheses about human compliance. Saf. Sci., 21, 175–189, https://doi.org/10.1016/0925-7535(95)00060-7.
Paul, B. K., M. Stimers, and M. Caldas, 2015: Predictors of compliance with tornado warnings issued in Joplin, Missouri, in 2011. Disasters, 39, 108–124, https://doi.org/10.1111/disa.12087.
Rothfusz, L. P., R. Schneider, D. Novak, K. Klockow-McClain, A. E. Gerard, C. Karstens, G. J. Stumpf, and T. M. Smith, 2018: FACETs: A proposed next-generation paradigm for high-impact weather forecasting. Bull. Amer. Meteor. Soc., 99, 2025–2043, https://doi.org/10.1175/BAMS-D-16-0100.1.
Ruggles, S., S. Flood, S. Foster, R. Goeken, J. Pacas, M. Schouweiler, and M. Sobek, 2021: IPUMS USA. IPUMS, accessed 20 April 2021, https://usa.ipums.org/usa/.
Schmidlin, T. W., B. O. Hammer, Y. Ono, and P. S. King, 2009: Tornado shelter-seeking behavior and tornado shelter options among mobile home residents in the United States. Nat. Hazards, 48, 191–201, https://doi.org/10.1007/s11069-008-9257-z.
Sherman-Morris, K., 2010: Tornado warning dissemination and response at a university campus. Nat. Hazards, 52, 623–638, https://doi.org/10.1007/s11069-009-9405-0.
Simmons, K. M., and D. Sutter, 2005: WSR-88D radar, tornado warnings, and tornado casualties. Wea. Forecasting, 20, 301–310, https://doi.org/10.1175/WAF857.1.
Simmons, K. M., and D. Sutter, 2006: Direct estimation of the cost effectiveness of tornado shelters. Risk Anal., 26, 945–954, https://doi.org/10.1111/j.1539-6924.2006.00790.x.
Simmons, K. M., and D. Sutter, 2009: False alarms, tornado warnings, and tornado casualties. Wea. Climate Soc., 1, 38–53, https://doi.org/10.1175/2009WCAS1005.1.
Simmons, K. M., and D. Sutter, 2013: Economic and Societal Impacts of Tornadoes. Springer, 296 pp.
Viscusi, W. K., and J. E. Aldy, 2003: The value of a statistical life: A critical review of market estimates throughout the world. J. Risk Uncertainty, 27, 5–76, https://doi.org/10.1023/A:1025598106257.
Wehde, W., J. T. Ripberger, H. Jenkins-Smith, B. A. Jones, J. N. Allan, and C. L. Silva, 2021: Public willingness to pay for continuous and probabilistic hazard information. Nat. Hazards Rev., 22, 04021004, https://doi.org/10.1061/(ASCE)NH.1527-6996.0000444.
Wolff, H., 2014: Value of time: Speeding behavior and gasoline prices. J. Environ. Econ. Manage., 67, 71–88, https://doi.org/10.1016/j.jeem.2013.11.002.
