Reducing fatalities from tornadoes in the southeastern United States requires considering multiple societal factors, including the risk perceptions that influence how people interpret tornado forecasts and warnings and make protective decisions. This study investigates perceptions of tornado risk in the southeastern United States, operationalized as judgments of tornado likelihood. While it is possible that residents of the Southeast could learn about tornado likelihood in their region from observing the local environment, cognitive-ecological theory from psychology suggests that such judgments of likelihood can be inaccurate, even if other aspects of local knowledge are accurate. This study analyzes data from a survey that elicited different groups’ judgments of tornado likelihood associated with different seasons, times of day, and storm system types. Results are presented from a representative sample of Southeastern residents and are compared with a sample of tornado experts (who have extensive knowledge about the likelihood of Southeastern tornadoes) and a representative sample of Great Plains residents. Overall, the analysis finds that many members of the Southeastern public deviate from the expert sample on tornado likelihood, especially for winter and overnight tornadoes. These deviations from expert opinion mimic the judgments of the Great Plains public. This study demonstrates how psychological theory and a decision science approach can be used to identify potential gaps in public knowledge about hazardous weather risks, and it reveals several such potential gaps. Further research is needed to understand the reasons for deviations between public and expert judgments, evaluate their effects on protective decision-making, and develop strategies to address them.
Tornadoes can strike in most parts of the United States, but there are certain regions and times of day and year where the atmospheric conditions are more favorable for tornado formation. The Great Plains (GP) region of the United States is the source of much of the current scientific knowledge about tornadoes (Rasmussen et al. 1994; Wurman et al. 2012). However, the Southeast (SE) region of the United States experiences high numbers of tornado fatalities relative to some areas of the GP, as analysis of data through 2005 by Ashley (2007) shows. The recurring risks posed by tornadoes in the SE United States are exemplified by the significant loss of life associated with recent tornado outbreaks in the SE, including the 2008 Super Tuesday outbreak that killed over 50 people (NOAA 2009) and the devastating 27 April 2011 outbreak that killed over 300 people in a single day (NOAA 2011).
Developing strategies to reduce loss of life from tornadoes in the SE requires understanding and addressing intersecting meteorological and societal factors (Agee and Taylor 2019; Ashley and Strader 2016; Rasmussen 2015). Recent meteorological research has documented that SE tornadoes are increasing over time (Ashley and Strader 2016; Gensini and Brooks 2018; Moore 2018), deadliest in the spring and winter seasons and around local sunset times (Anderson-Frey et al. 2019), and generated by multiple types of storm systems (Smith et al. 2012). Relative to GP tornadoes, SE tornadoes also are more spread out across different seasons (Long et al. 2018) and occur more frequently at night (Ashley et al. 2008; Childs et al. 2018; Ellis et al. 2019b) and in high-shear, low-CAPE environments (Anderson-Frey et al. 2019; Sherburn et al. 2016).
The societal factors contributing to SE tornado mortality include population density and geographic distribution, housing stock characteristics, and available sheltering options (Ash 2017; Ash et al. 2020; Ashley 2007; Strader et al. 2019), as well as the extent to which members of the public accurately perceive tornado risks, interpret forecast and warning information, and make informed decisions in the face of a storm (Ash 2017; Brotzge and Donner 2013). Here, we focus on psychological factors related to SE residents’ tornado risk perceptions and decisions. While tornado risk can be characterized in different ways (see Ashley and Strader 2016 for a detailed discussion) and risk perceptions can be measured in different ways, in this study we focus on investigating the public’s perceptions of the likelihood of tornadoes. We do so because people’s perceptions of when, where, and how tornadoes occur can influence how they attend to forecasts and warnings, how they make sense of tornado risks, and what actions they take (e.g., Ashley 2007; Demuth et al. 2019; Ellis et al. 2019b; Liu et al. 2019; Mason et al. 2018; NOAA 2009).
We anticipate that knowledge about tornado likelihood is especially important in the SE because of the factors outlined above—that is, the lack of a single, “traditional” tornado season, tornadoes occurring at night, and tornadoes spawning from multiple types of storm systems. Combined with the rapid-onset nature of tornadoes and their limited predictability (Markowski and Richardson 2014), these factors create further challenges for people’s tornado vigilance, risk assessments, and decision-making, potentially amplifying the risks posed by tornadoes in the SE. Although SE residents may have learned about tornado frequency and timing from observing their local environment, accurately judging frequencies of rare events is difficult and prone to biases from small samples due to the low base rates of encountering tornadic storms (Broomell 2020; Fiedler and Juslin 2005; see section 2).
To help to address these issues, we apply a decision science approach for improving decision-making. This approach (i) elicits the descriptive aspects of the decision context (how do members of the public perceive risk?) and (ii) elicits expert judgment of the decision context (how do experts perceive risk?) and compares public and expert perceptions in order to (iii) improve decisions through prescriptive interventions that address key knowledge or communication gaps identified in (ii) (Wong-Parodi et al. 2016). For (i), we investigate what residents of the SE United States currently believe about the likelihood of tornadoes associated with different seasons, times of day, and storm system types, which we refer to as their judgments of tornado likelihood. For (ii), we elicit similar judgments from a sample of meteorologists with tornado expertise and compare these with SE residents’ judgments to investigate whether SE residents perceive (on average) tornado likelihood differently than an expert sample (indicating potential knowledge gaps or failures of communication) and what factors might help explain those differences. One possible influence on SE residents’ judgments of likelihood is misappropriation of information about tornadoes from the more-studied GP region. As an initial step toward (iii), we explore this by eliciting GP residents’ judgments of GP tornado likelihood and comparing those with SE residents’ judgments of SE tornado likelihood.
With these aims in mind, this article addresses four research questions (RQs):
RQ1: How do members of the SE U.S. public judge the likelihood of tornadoes (i) during different seasons of the year, (ii) at different times of day, and (iii) from different types of storm systems?
RQ2: How do the SE U.S. public’s judgments of the likelihood of SE U.S. tornadoes (tornado season, time of day, and storm system type) compare with tornado experts’ judgments of the likelihood of SE U.S. tornadoes?
RQ3: How do the SE U.S. public’s judgments of the likelihood of SE U.S. tornadoes (tornado season, time of day, and storm system type) compare with the GP public’s judgments of the likelihood of GP tornadoes?
RQ4: To what extent are differences between the SE public and expert judgments of the likelihood of SE U.S. tornadoes associated with the public’s sociodemographic characteristics, tornado beliefs, and tornado experiences?
To investigate these questions, we surveyed a sample of meteorologists with expertise in tornadoes in the SE United States, a representative sample of SE residents, and a representative sample of GP residents. The survey elicited judgments of tornado likelihood associated with tornado season, time of day, and storm system types, along with demographic and individual difference variables (see section 3).
In the following, we first review key literature relevant to our study. Second, we describe the survey method and analytic techniques used. We then present the results from the study, organized around the research questions above, and discuss their implications. Overall, we found that members of the SE U.S. public perceive important aspects of SE tornado likelihood to be similar to that of the GP, deviating from expert opinion. This indicates that residents of the SE do have some misperceptions of tornado likelihood, but further research is needed to understand why more definitively and to understand which misperceptions are most important to address. Such understanding can then be used to develop and test interventions for improving decision-making, as indicated above.
2. Literature review
a. Southeastern U.S. tornado season, time of day, and storm system type
The climatology of tornadoes in the SE is an active area of research; here we review key findings to date that are relevant to this study. Tornadoes can occur in the SE at all times of year, and recent work has found that the SE tornado season is bimodal, with a strong peak in activity in the spring (March–May), a secondary peak in the late autumn and winter (November–February), and lowest activity in the late summer and early autumn (July–September; Long et al. 2018; see also Anderson-Frey et al. 2019; Childs et al. 2018). The seasonality of tornado frequency in the Southeast corresponds to the times of year associated with the highest frequency of tornado fatalities (Anderson-Frey et al. 2019; Ashley 2007). This seasonality differs from that in the GP, where tornadoes and tornado fatalities are most frequent in the late spring and early summer (Ashley 2007).
Tornadoes that occur in the SE during the late autumn and winter months have been referred to as cold-season tornadoes (Childs et al. 2018). For a long time, less was known about cold-season tornadoes relative to their warm-season counterparts, but deadly cold-season events, including the 2008 Super Tuesday outbreak (NOAA 2009), have drawn attention to them. These “off season” tornadoes have higher fatality rates overall (Simmons and Sutter 2008) and in the SE compared to those occurring during other months (Anderson-Frey et al. 2019; Ashley 2007). Moreover, recent research has shown that cold-season events are more likely to occur at night compared to warm-season tornadoes (Childs and Schumacher 2018). Nocturnal tornadoes are associated with higher fatality rates than daytime tornadoes, and nocturnal tornado likelihood and fatalities are both especially high in the SE (Ashley et al. 2008; Krocak and Brooks 2020). In the SE, the highest fatality rates occur during winter and spring and around local sunset (Anderson-Frey et al. 2019). Such tornadoes are associated with less accurate tornado watches and warnings (Anderson-Frey et al. 2016; Sherburn et al. 2016), further exacerbating the challenges that nocturnal tornadoes pose for timely receipt of tornado warnings and protective decision-making.
The types of storm systems that generate tornadoes in the SE are also more diverse than the GP. The three most significant types of storm systems that spawn tornadoes in the United States are isolated supercell thunderstorms, quasi-linear convective systems (QLCSs), and tropical cyclones (depicted in Fig. 1). While tornadoes in the GP are primarily associated with isolated convective systems (such as supercell thunderstorms), the SE also has a significant number of tornado events from clustered and linear storm systems (Smith et al. 2012). The prevalence of tornadoes spawned from QLCS in the SE is currently a topic of debate, due to the difficulty of categorizing QLCS systems in radar data, limited radar for observing low-level tornadoes, and low validation rates for nocturnal tornadoes (Ashley et al. 2019; Ellis et al. 2019a; Smith et al. 2012). Tropical cyclones can produce tornadoes in the SE in the summer months (Edwards et al. 2012), but the overall number of SE tornadoes generated by tropical cyclones is low relative to those generated by other types of storm systems (Long et al. 2018).
b. The psychology of likelihood judgment
Several decades ago, pioneering work on judgment and decision-making introduced several heuristic processes that people use to form judgments of likelihood, such as anchoring-and-adjustment, availability, and representativeness (Tversky and Kahneman 1974). This work demonstrated that likelihood judgments may easily diverge from the true generating process, especially when judgment is based on small samples of observations that are locally perceivable and available to an individual. We refer to such observations as local observations, defined by small samples of personal observations held in memory.
Cognitive-ecological theory builds on this work and outlines how a person’s judgment is shaped not only by the sample size of observations but also by the statistical properties of their local observations (Broomell 2020; Fiedler and Juslin 2005). Here, we apply this theory to judgment of tornadoes in the SE to understand how a general population, untrained in meteorology, might use their own and others’ observations of tornadic storms in the environment to develop inaccurate or incomplete judgments about the likelihood of tornado formation at different times. The theory states that the reliability of judgment is derived from understanding the environmental cues available to the public indicating the presence of a tornadic storm. When available cues appear to be highly reliable by correlating with each other but do not reliably reveal the true tornado potential of a storm, this generates an incompatibility between the locally observable cues and the global state of the world (i.e., the true tornadic potential), termed global–local incompatibility (Broomell 2020).
For example, consider a person randomly chosen from the general public on the ground as a decision-maker (DM) attempting to evaluate the likelihood of tornadoes in a given season by observing storms. When storms approach, a DM has available to them many locally perceivable environmental cues such as rain, wind, hail, cloud color, and sky color. However, few of these cues can reliably reveal the presence of a tornado; instead, they primarily indicate the strength of the storm system (Dewitt et al. 2015; Klockow et al. 2014). Therefore, when a storm approaches and all visible cues indicate a low strength storm, this can mask the fact that the conditions for a tornado to form (such as wind shear, lift, and instability) are still present (Broomell 2020). Using the visible cues, this storm will reasonably be judged as nontornadic, even if this storm generated a tornado elsewhere. Such an observation should contribute to the perceived likelihood of tornadoes, but for many, it may actually reduce the perceived likelihood of tornado occurrence. Similarly, when a strong storm approaches without the conditions for a tornado to form, the cues of a strong storm can reasonably appear more associated with a tornado (e.g., stronger ground wind, damaging hail), even though the likelihood of a tornado event is much lower. Therefore, these cues cannot directly reveal the presence of a tornado (or the potential for tornadogenesis), and likelihood judgments of tornadoes based on these environmental cues may not accurately represent the actual likelihood (for a more detailed discussion, see Broomell 2020).
Additional evidence supports our theory that DMs reasonably attempt to judge tornado risk based on local environmental cues. DMs in the path of severe weather often attempt to confirm official tornado warnings (and watches) by observing the local environment, as an attempt to validate whether the warning applies specifically to their location (Ash 2017; Comstock and Mallonee 2005; Demuth et al. 2019; Liu et al. 2019; Nagele and Trainor 2012). DMs also use knowledge about their local environment to form beliefs, some of which may be less scientifically accurate than others, about the likelihood of tornadoes at different locations (Ellis et al. 2018, 2019b; Klockow et al. 2014). For example, researchers have observed residents believing that bodies of water, mountains/hills, or recent construction projects have an effect on their exposure to tornadoes. In some cases, because previous tornadoes have never hit their location, they believe there is some natural barrier that is protecting them (Klockow et al. 2014).
There are several additional issues that contribute to the difficulty of members of the public judging the frequency of tornadoes based on observations of environmental cues. First, to accurately judge the likelihood of tornadoes at different times, a DM would require a large collection of observations in a given region of the United States, because small samples of low base-rate events tend to underrepresent the event under consideration. Second, even when the environment is favorable for tornado formation, all of the necessary ingredients must be present in the right amounts—what Markowski and Richardson (2014) call the “Goldilocks problem” (p. 29). Third, tornadoes generate severe damage in a highly concentrated area. Thus, a typical DM rarely (if ever) observes firsthand a tornado-producing storm, much less a tornado itself. Therefore, many tornadoes that DMs become aware of will be from secondhand accounts, likely through observations by other people and the media. When hearing about tornadoes and storms that are not in their own community, DMs may not be able to differentiate whether a reported event is informative for judging the likelihood of tornadoes in their region. Fourth, the severity of a tornado can shape a DM’s interpretations of their experiences with it (Howe et al. 2014), which can further complicate likelihood judgments. DMs also perceive tornadoes as being unpredictable (Ash 2017; Demuth 2018), which may also influence judgments.
Overall, from a psychological perspective, the above discussion suggests that the environmental characteristics of tornadoes and tornadic storms may affect the public’s judgments of tornado likelihood. This psychological perspective motivates our study of SE residents’ judgments of tornado likelihood, and it underlies our elicitation methodology. Given the potential relevance of local observations and beliefs, we also include in our study an investigation of tornado-related experiences and beliefs and their influence on judgments of tornado likelihood.
The data analyzed in this article were collected from an online survey, which was completed by both experts in SE U.S. tornadoes and the general public as described in sections 3b and 3c. Prior to conducing the survey, we conducted interviews with a smaller group of experts; the interview data were used to help develop the survey, as described in section 3a.
a. Expert interviews: sampling and implementation
We first interviewed several members of the meteorological community with expertise in tornadoes in the SE. To recruit these experts, two authors (R. Morss and J. Demuth) used their knowledge of the SE tornado research and operational community and their membership on the VORTEX-Southeast Science Steering Committee to identify 18 people with expertise in tornadic storms in the SE. They then contacted this group of experts by email and instructed those interested in participating in the interviews to contact S. B. Broomell (SB) and G. Wong-Parodi (GWP) directly, so that participation could remain confidential [as required for institutional review board (IRB) approval]. SB and GWP performed structured interviews with 13 of these experts about the environmental conditions associated with the occurrence of tornadoes in the SE United States. The interviews were conducted from March to May of 2017; each interview lasted approximately 1 h. Audio recordings of the interviews were transcribed and content was analyzed by SB and GWP, with an emphasis on identifying important features of the environment and cues available for assessing SE tornado risk associated with different types of storms as well as nocturnal events. The themes extracted from the interviews were then examined by all authors and used to develop a survey designed to elicit expert and public judgments of tornado likelihood and perceptions of environmental cues and other aspects of tornado risks, described further in sections 3b and 3c.
b. Expert and public surveys: Sampling and implementation
We collected survey data from three groups: 1) a sample of 33 individuals with expertise on tornadoes in the SE, 2) a sample of 1050 respondents in the SE (SE sample), and 3) a sample of 1050 respondents in the GP (GP sample). Public sampling was by state, according to the SE and GP regions depicted in Fig. 2. The expert and public samples received the same survey instrument (see section 3c), except the expert sample was asked to answer the questions with regard to tornadoes in the SE region as shown in the left panel of Fig. 2, and the public samples were asked to answer about tornadoes where they lived. All participants completed the survey online.
The expert data were collected between June and September 2018 with the aim of summarizing expert knowledge (at the time of the study) about SE tornado likelihood and environmental cues in a way that is directly comparable to the data collected from the SE public sample. We chose to use expert data rather than meteorological data for comparison with public perceptions because knowledge about SE tornado climatology was evolving during the time of our project (e.g., the climatology of QLCS tornadoes by Ashley et al. was published in 2019), and because meteorological analyses of tornado likelihood are not readily available in the same form as our public data. To collect expert responses to our survey, we first emailed a request to respond to the survey to approximately 40 meteorologists with expertise in tornadoes in the SE, including those contacted for the interviews and additional experts identified by two of the authors (R. Morss and J. Demuth) and the VORTEX-SE program manager. We then emailed invitations to complete the survey to approximately 25 additional experts and invited the recipients to forward the survey to other experts knowledgeable about tornadoes in the Southeast United States but requested that they not post the survey broadly. To complete the survey, participants in the expert sample had to indicate that they had at least 10 years of professional experience in chasing and studying tornadic storms or have earned a bachelor’s degree, master’s degree, or doctorate in meteorology or a related field that includes an emphasis on severe weather. Responses from 33 experts were collected, and the median time of completion for the expert sample was 26.8 min.
The survey company YouGov collected responses from the public in the SE and GP between 20 September 2018 and 8 October 2018. YouGov uses a sample matching methodology where a random sample is drawn from the target population. For each member of this random sample, YouGov selects one or more matching members from their pool of respondents, generating a representative sample [for details, see YouGov (2018) and section S1 in the online supplemental material]. The sample size was determined ahead of time as the largest sample we could obtain from both regions with an equal number of participants in each state (150 responses per state) constrained by budget. YouGov collected and validated responses from each state within each region. Each of the valid responses passed several predetermined checks for data quality that were implemented by YouGov, who provided us with the final complete dataset of 2100 responses. As shown in the online supplemental material, the sociodemographic characteristics of the SE and GP samples are similar to American Community Survey data for the corresponding regions, with some overrepresentation of female and white respondents. The median time of completion for the public samples was 22.8 min.
c. Survey instrument and measures
The survey consisted of five parts, depicted in Fig. 3. Parts 1 and 2 elicited participant perceptions of tornado timing (time of year and day) and storm system type. Part 3 elicited perceptions of environmental cues that may be associated with tornadoes, with participants randomized into one of three experimental conditions; results from these data are presented in a separate article (S. B. Broomell et al., unpublished manuscript). Parts 4 and 5 collected covariates and demographics that are used to perform the regression analyses discussed in this article. Next, we describe the survey measures analyzed here, categorized as dependent variables, covariates, and demographics.
1) Dependent variables
In Parts 1 and 2 of the survey, each participant provided judgments for our primary dependent variables that capture their judgments of tornado likelihood associated with different times of year, times of day, and storm system types. For each elicitation, participants distributed 100 tokens across bins that represent seasons of a year, times of day, or storm system types, such that the number of tokens placed in each bin represents their judgments of the likelihood of that bin being associated with a tornado. For each elicitation, the number of tokens allocated across the bins were constrained to sum to 100 by the online interface.
After completing a practice trial,1 participants provided their judgments of the likelihood of tornadoes in different seasons of the year by responding to the following prompt:
Suppose that there were a total of 100 tornadoes in a year where you live. How many do you believe would occur in each of the following seasons, given your expectations about when tornadoes happen? The total across the seasons should equal 100.
Four meteorological seasons were defined for the participants as spring (1 March–31 May), summer (1 June–31 August), autumn (1 September–30 November), and winter (1 December–28 February).
Next, participants were asked to provide their judgments of the likelihood of tornadoes at different times of day, within each season:
Suppose that there were a total of 100 tornadoes in the [SEASON] where you live. How many do you believe would occur in each of the following time periods, given your expectations about what time of day tornadoes happen during the [SEASON]? The total across the time periods should equal 100.
Three times of day were defined for the participants: morning [0401–1200 LT (noon)], afternoon (1201–2000 LT), and night (2001–0400 LT). This prompt was repeated four times, for each of the four seasons as defined above.
Last, participants were asked to provide their judgments of the likelihood of tornadoes from three different types of storm systems:
Suppose that there were a total of 100 tornadoes where you live produced by these three storm systems. How many tornadoes do you believe were produced by each of the following types of storm system, given your expectations about tornadoes? The total across the storm systems should equal 100.
The three types of storm systems were defined as isolated supercell thunderstorm, quasi-linear convective system (or squall line), and tropical cyclone (including hurricanes and tropical storms), using the depictions in Fig. 1 accompanied by brief descriptions (see the online supplemental material).
In total, participants provided data for six dependent variables representing their judgments of (variable 1) the likelihood of tornadoes in different seasons of the year (variables 2–5) the likelihood of tornadoes at different times of day for each of the four seasons, and (variable 6) the likelihood of tornadoes being produced by different types of storm systems. Each of these dependent variables consists of numbers allocated to different bins that sum to a constant, making these data compositional (Aitchison 1982). Each bin represents a component of a composition, and each variable is defined by the entire composition. We therefore analyze the dependent variables using compositional analysis techniques, described in section 4.
Part 4 of the survey asked participants about their 1) experiences with tornadoes (experience; Demuth 2015, 2018), 2) beliefs about tornadoes and tornado danger (heuristics), and 3) perceptions of the risks associated with tornadoes of different strength (tornado strength). All items are displayed in Table 1, along with descriptive statistics. The experience scale has been previously tested and validated (Demuth 2015, 2018). The heuristics and tornado strength items were developed for this project based on our interviews with experts (see section 3a), which suggested that these factors might influence public judgments of tornado likelihood. Thus, we analyze their statistical properties as measurement scales to determine their role in our regression analyses (see section 4).
Much of the demographic information used in this article had already been collected by YouGov as a part of panel membership. Part 5 of the survey asked several questions of specific interest for our research topic, including participants’ tenure of residence (“How long have you lived in your current location?”), reported use of weather radar (“When bad weather threatens, approaches, or affects your area, do you look at radar images of your local area?”), and reported knowledge of their location on a radar map (“Can you easily identify where you are on a radar map?”). The demographic variables used in our main regression analyses are discussed further in the next section; the full set of demographic data can be found in section S1 of the online supplemental material.
4. Data analysis method
In this section, we summarize our data analysis method and several data reduction steps performed prior to conducting the main analyses. Even though many of the results reported in this article are descriptive of the dataset, our analyses were guided by a preregistration2 (hosted at https://osf.io/ur65v/) to document our process and minimize the number of analyses performed on the data.
a. Demographic variables
As per our preregistration (and to simplify the analysis), we aggregated two panel measurements of political ideology into a single political affiliation scale ranging from 1 to 5 (Cronbach’s α = 0.81) and three panel measurements of religious practice into a single religiosity scale ranging from 1 to 4 (Cronbach’s α = 0.87). Table 2 displays summary data for the demographic variables used in our analyses; all were included in the preregistration except for the variable tenure, which was accidently omitted from the preregistration but is used in the analyses.
b. Covariates: Experience, heuristic, and tornado strength items and scales
To limit the number of predictors in our model, we preregistered that we would only use the experience and heuristics data in our regression analyses if they generated an acceptable Cronbach’s alpha, and that scales failing to be reliable would be dropped from subsequent analysis. We preregistered that the tornado strength items would be used as individual items if they failed to form a reliable scale. The experience items formed a highly reliable scale (Cronbach’s α = 0.89), and so we averaged the nine items together to form an experience scale for subsequent analyses. The heuristics items were not a reliable scale (Cronbach’s α = 0.17) and so were dropped from analysis. However, Table 1 indicates that for some of the heuristics, SE residents’ beliefs differed from those of the experts, and so for completeness we provide additional analysis in the online supplemental material that also includes the individual items of the heuristics scale. The tornado strength items also did not form a reliable scale (Cronbach’s α = 0.24), so we proceed in accordance with our preregistered plan that we will use the three tornado strength items (labeled fatality, warning, and tendency) individually in the subsequent analyses.
c. Dependent variables: Compositional format and interpretation of likelihood data
Each dependent variable is a distribution of 100 tokens created by the participants to represent judgments of the likelihood of tornadoes across different seasons, times of day, and storm system types. Because these variables follow a compositional data structure (Aitchison 1982), we analyzed them using Aitchison’s compositional approach as implemented in R by van den Boogaart and Tolosana-Delgado (2013). The compositional results look slightly different than normal statistical results, but their interpretation is very similar.
Addressing research question 1 involves interpretations of these compositional data. To illustrate these interpretations, we use the example of our 33 expert judgments of the likelihood of tornadoes from three types of storm systems, depicted on the left of Fig. 4 in a ternary plot. Ternary plots directly display the simplex that defines the compositional structure of our dependent variable. Points are defined by their distance from each leg of the triangle. Each point in the plot represents a single participant’s allocation to each of the three bins. The expert sample is clustered near the middle of the leg farthest from the cyclone vertex. This indicates that as a group, they perceive tropical cyclones to be least likely to generate tornadoes, and they have similar perceptions of likelihood for supercell and QLCS systems. The mean of the expert sample is determined using the geometry of compositions, which we will describe qualitatively to simplify interpretation (for a review, see van den Boogaart and Tolosana-Delgado 2013). Central tendency of a composition is defined by the geometric mean, compositional addition is defined using perturbation (shifting probability mass from one component to another), and compositional scaler multiplication is defined using powering (raising each component to the power defined by the multiplier). The black ellipse represents the 95% standard error for the computation of the mean. To simplify interpretation, we will display these data using the bar graph display on the right side of Fig. 4, which is a projection of the ternary plot.
d. Regression analysis to evaluate differences in likelihood judgments between samples
We addressed research questions 2 and 3 together using a single compositional regression analysis to test for mean differences of each dependent variable as a function of sample.3 To perform this test, we used dummy coding with the SE sample as the reference group. The resulting regression model is
where XEX and XGP are dummy variables for responses from the expert and GP samples, respectively, and is a matrix where each row is an individual and each column is a component. The coefficients bi are vectors where each element is a component of the composition. The coefficient b0 is a vector that represents the compositional mean judgments of the SE sample. For RQ2, we are testing for a significant coefficient b1 indicating that the mean of the expert sample differs from the mean of the SE sample. For RQ3, we are testing for a significant coefficient b2 indicating that the mean of the GP sample differs from the mean of the SE sample.
Since these coefficients are compositions (just like the dependent variables), they define the effect of the predictor variable on the dependent variable in terms of compositional geometry. Similar to the mean likelihood judgments, we will display coefficient estimates using the bar graph display on the right side of Fig. 4. Compositional regression coefficients are interpreted based on their departure from the uniform composition, indicating the degree to which a given predictor perturbs the compositional distribution. For J components, coefficients greater than 1/J increase likelihood, and coefficient components less than 1/J reduce likelihood.
While we preregistered the use of compositional regression analysis, it required several additional steps to data cleaning that were not preregistered. First, zeros allocated to any one component of the composition cannot be handled by the log-ratio transformations required to perform regression. Allocating all tokens to a single component bin generates zeros in all remaining bins, and strongly skews the data. Therefore, for all analyses reported below, we removed the 68 public participants who allocated all 100 tokens to a single component bin for three or more of the six dependent variables. For the compositional regression analyses, we imputed a single tornado token to any individual component that had zero tokens (8% of the data), removing the remaining zeros (Martín-Fernández et al. 2003). Each of these steps was decided on and performed prior to any inferential data analysis (as data analysis using the compositional approach is not possible with zeros).
e. Regression analysis to evaluate factors that explain the distance between likelihood judgments from the SE and expert samples
To address research question 4, we computed differences between SE public and expert judgments of tornado likelihood and analyzed their association with demographics and other covariates. We measured the distance between the mean expert judgments and each individual’s judgments using a chi-square distance metric (Stewart 2017). Given J components in a composition, the difference is computed as
We used this metric because it can handle zeros in the data and approximately satisfies the principles of compositional data analysis (Stewart 2017). We therefore applied this measure to our data prior to imputing any tornado tokens as described above.4 This distance measure maps two compositions, each with J components, into a single scalar value such that the minimal distance between two compositions is 0, and the maximal distance is (4J)1/2.
Descriptive statistics for the difference between the SE sample and the mean expert judgment for each dependent variable are displayed in Table 3. Because these distances are a function of the number of components in the composition (with season having four and the others having three), we display the mean difference in the last column as a percent of the theoretically maximum distance [i.e., mean/(4J)1/2]. After checking the normality of these distance measures (we took their square root to correct for skewness) we subjected them to a hierarchical regression analysis clustered by state, with demographics, the experience scale, and the tornado strength items as predictors (this analysis process was preregistered).
a. Public perception of tornado season, timing, and storm system type in the Southeast United States
Research question 1 asks what residents of the SE believe about the likelihood of tornadoes in different seasons, times of day, and storm system types. The full distribution of responses from the SE sample for the six dependent variables measuring these judgments of likelihood are displayed in Fig. 5. While these boxplots display the full distribution of responses, they do not fully reveal the compositional nature of the variables as described above (i.e., that a single individual’s responses are constrained to sum to 1). Each boxplot represents a single component of the composition, so allocating a large likelihood (e.g., near 1) to a single component bin requires the remaining bins to have very low likelihood (near 0). Therefore, the distributions are skewed toward lower likelihoods because a single high likelihood will constrain all other bins to be near zero. This feature of the data is naturally accounted for by the compositional regression analysis used in the next subsection. (See section S2 in the online supplemental material for summaries of the responses from all samples.)
Overall, the distribution of responses from the SE sample displayed in Fig. 5 shows a large amount of variability, with responses covering the entire probability range for almost any component. The likelihood of tornadoes in winter is a clear exception, with low variance and likelihoods rarely larger than 0.25.
Despite the large variance, several of the variables display clear group-level ordering of the judged likelihoods. For tornado season (top panel of Fig. 5), perceptions of tornado likelihood are clearly ordered as spring, summer, autumn, and winter from most to least likely. SE public perceptions of winter tornado occurrence are concentrated near zero.
For time of day (middle of Fig. 5), SE public perceptions of tornado likelihood are also clearly ordered, as afternoon, night, and morning from most to least likely. This ordering and the distributions are similar across the four seasons; therefore, subsequent analyses are performed on a single time of day variable aggregated across the four seasons.
The perceptions of the likelihood of tornadoes from different storm system types (bottom of Fig. 5) exhibit the widest distribution of responses across components. These distributions may indicate lack of knowledge about the likelihood of tornadoes from any of these different storm systems.
b. Comparison of the SE sample to the expert sample
Research question 2 asks whether the SE sample’s judgments of tornado likelihood differ from those of an expert sample also judging the SE. We tested for group differences using compositional regression analysis (as described in section 4d) separately for each dependent variable of season, time of day (aggregated across seasons), and storm system type.5 The figures discussed in this section present the SE, expert, and GP samples together, but the comparison between the SE and GP samples is not discussed until section 5c.
1) Perceptions of tornado season
The mean perceptions of SE tornado season for the SE and expert samples are displayed in the top-left and top-center panels of Fig. 6. The expert sample perceives that tornadoes in the SE occur most frequently in the spring, followed by autumn and winter, with fewest tornadoes in the summer. This reflects the known tornado season for the SE as summarized in section 2a. Comparing the two panels indicates that the SE public has very different perceptions of tornado season than the experts, especially for summer and winter.
These differences are confirmed using the compositional regression approach, which finds a significant difference between the SE and expert samples [coefficient b1 in Eq. (1); F(3, 2049) = 30.01; p < 0.001; η2 = 0.04]. As shown in the bottom-middle panel of Fig. 6, several components in b1 differ significantly from a uniform distribution (do not contain 0.25 within the confidence interval). For example, the top row of Fig. 6 shows that the experts’ mean perceived likelihood of SE tornadoes in winter is higher than that of the SE sample. The bottom row of Fig. 6 shows that the winter component of coefficient b1 is significantly greater than 0.25, indicating a significant increase in winter likelihood for the expert sample compared to the SE sample. Similarly, the summer component of coefficient b1 is significantly lower than 0.25, indicating a significant decrease in summer perceived likelihood for the expert sample compared to the SE sample.
Overall, the regression model indicates that the SE sample is underestimating tornado likelihood in winter and overestimating likelihood in summer relative to the expert judgments of the tornado season in the SE. These under and over estimations are all in the direction of the GP tornado season, which we will examine further in section 5c.
2) Perceptions of tornado time of day
As discussed in section 5a, we examine sample differences in perceptions of tornado time of day aggregated across seasons (see section S3 of the online supplemental material for the results of timing separately by season). The mean perceptions of time-of-day likelihood for the SE and expert samples are displayed in the top-left and top-center panels of Fig. 7. Both samples perceive the tornado likelihood in the SE to be highest in the afternoon, second highest at night, and lowest in the morning.
Using the compositional regression approach, we find a significant difference between the SE and expert samples [coefficient b1 in Eq. (1); F(2, 2050) = 8.21; p < 0.001; η2 = 0.01]. As displayed in the bottom-middle panel of Fig. 7, the expert sample differs from the SE sample by judging the likelihood of tornadoes in the SE to be lower in the morning and higher at night.
3) Perceptions of tornado storm system type
The mean perceptions of the source of tornadoes for the SE and expert samples are displayed in the top-left and top-center panels of Fig. 8. The SE sample perceived supercells as the primary source of tornadoes in their region, with QLCSs as a secondary source, followed by tropical cyclones. The expert sample judged tornadoes in the SE as most likely to be produced by QLCSs, closely followed by supercells, and least by tropical cyclones.
The compositional regression analysis suggests that there are differences between the SE and expert samples, although these are not as strong as those in sections 5b(1) and 5b(2) [coefficient b1 in Eq. (1); F(2, 2050) = 3.82; p = 0.02; η2 = 0.01]. As displayed in the bottom-middle panel of Fig. 8, the expert sample differs from the SE sample by perceiving that QLCSs are a more likely source of tornadoes in the SE.
Based on the analyses of meteorological data summarized in section 2a, both the SE and expert samples correctly identify TCs as the least likely source of tornadoes in the SE. However, experts, in particular, may be overestimating the likelihood of QLCSs as a source of SE tornadoes compared to supercells. As discussed in section 2a, the prevalence of tornadoes spawned from QLCS in the SE is currently a topic of active research and debate, as are the risks posed by QLCS tornadoes compared to supercell tornadoes in the SE. This is discussed further in section 6.
c. Comparison of the SE sample with the GP sample
Research question 3 asks whether the SE sample judging tornado likelihood in the SE has different perceptions from the GP sample judging the GP. We tested for group differences using the same compositional regression analysis (as described in section 4a and shown in section 5b) separately for each dependent variable of season, time of day (aggregated across seasons), and storm system type.
1) Perceptions of season
The mean perceptions of tornado season for the SE and GP samples are displayed in the top-left and top-right panels of Fig. 6. The ordering of the SE sample’s perceptions is similar to that of the GP sample, with peak tornado season in the spring and summer (reflecting the known tornado season for the GP as summarized in section 2a).
The compositional regression indicates a significant difference between the SE and GP samples [coefficient b2 in Eq. (1); F(3, 2049) = 68.70; p < 0.001; η2 = 0.08]. As displayed in the bottom-right panel of Fig. 6, the GP sample perceives higher likelihood for summer tornadoes and lower likelihood for winter tornadoes relative to the SE sample. However, the SE sample’s perceptions of SE tornado season are closer to those of the GP sample than the expert sample. This suggests that knowledge about GP tornado season may be contributing to SE public misperceptions of tornado season in the SE.
2) Perceptions of time of day
The mean perceptions of tornado time of day for the SE and GP samples are displayed in the top-left and top-right panels of Fig. 7. The ordering of the times of day are similar for the two samples, with tornadoes perceived to be most likely in the afternoon and least likely in the morning. However, the compositional regression finds a significant difference between the SE and GP samples [coefficient b2; F(2, 2050) = 57.82; p < 0.001; η2 = 0.06]. As displayed in the bottom-right panel of Fig. 7, the GP sample perceives the occurrence of tornadoes in the morning as lower than the SE sample, and the occurrence of tornadoes in the afternoon as higher. The SE and GP samples perceive similar likelihood of nighttime tornadoes, even though the research reviewed above indicates that nighttime tornadoes are more likely in the SE than the GP.
3) Perceptions of storm system type
The mean perceptions of tornado storm system type for the SE and GP samples are displayed in the top-left and top-right panels of Fig. 8. The SE sample ordered the three storm system types similar to the GP sample, with supercells as the most likely source of tornadoes and tropical cyclones the least likely source. Using the compositional regression approach, we find a significant difference between the SE and GP samples [coefficient b2 in Eq. (1); F(2, 2050) = 92.53; p < 0.001; η2 = 0.08]. The bottom-right panel of Fig. 8 indicates that, relative to the SE sample, the GP sample perceives supercells as a more likely source of tornadoes and tropical cyclones as a less likely source. This accurately reflects meteorological differences between the GP and the SE reviewed above.
d. Regression analysis of public and expert differences in judgment
Research question 4 asks whether differences between the SE and the expert samples’ judgments are associated with differences in SE participants’ demographic characteristics and other covariates. To investigate this question, we performed regression analysis on the chi-square distance [Eq. (2)] between the SE sample’s judgments of likelihood and the mean expert judgments of likelihood for each dependent variable, as described in section 4e. Results for the three dependent variables—perceptions of tornado season, time of day (aggregated across seasons), and storm system type—are displayed in Table 4; analysis of time-of-day perceptions for each season is provided in section S4 of the online supplemental material. The predictor variables include the three tornado strength items, the experience scale, and the demographics (for completeness, analyses that include each of the heuristics items are in section S4).
Overall, the predictors used in each of the regressions explain only a small proportion of the variance in differences between the expert and SE samples (less than 10%). For tornado season in the SE, individuals in the SE sample with more self-reported experience and education tend to be less distant from the mean expert perception. For aggregate tornado time of day in the SE, individuals who are older and more educated, report looking at the radar when severe weather approaches, and are less religious tend to be less distant from the experts. For storm system type, individuals who are more politically liberal tend to be less distant from the experts, although this variable is a weak predictor. Given these results and the low variance explained, further research is needed to uncover what factors explain the differences between the SE and experts’ judgments.
6. Summary and discussion
Recent research suggests that important aspects of tornado risks in the SE differ from those in the GP, and that these differences contribute to higher mortality from tornadoes in the SE (e.g., Anderson-Frey et al. 2019; Ashley 2007; Ashley et al. 2008; Childs et al. 2018; Krocak and Brooks 2020; Long et al. 2018; Strader and Ashley 2018). One important difference is that the likelihood of tornadoes is more diffuse across seasons, times of day, and storm system types in the SE. Currently, it is an open question as to whether the public in the SE is aware of these aspects of their tornado risk. Cognitive-ecological theory suggests that accurate judgments of frequency require a large collection of observations, and that the statistical properties of the environment may not reveal the true tornado potential across time frames and storm types due to a global–local incompatibility (Broomell 2020). In other words, it can be challenging for people to learn their own tornado climatology by drawing on observations and memory. Here, we directly elicited SE public perceptions about SE tornado season, time of year, and storm system types, and compared them with experts’ perceptions of SE tornadoes and GP public perceptions of GP tornadoes.
Overall, our results suggest that public judgments of several important characteristics of tornado likelihood in the SE deviate from expert judgments. Regarding tornado season, the SE sample overestimates the likelihood of tornadoes in summer and underestimates the likelihood of tornadoes in winter relative to the expert sample. Regarding tornado time of day, the SE sample underestimates the likelihood of tornadoes at night and overestimates the likelihood of tornadoes in the morning relative to the expert sample. Regarding storm system type, on average, the SE sample underestimates the likelihood of tornadoes being generated by QLCSs relative to the expert sample, although comparison with recent meteorological research (Ashley et al. 2019) indicates that the experts may have overestimated the likelihood of tornadoes from QLCSs in the SE.
For all of these differences between the SE and expert samples, the SE sample’s judgments are shifted toward the judgments of GP tornado likelihood provided by the GP residents. This suggests that SE residents may be misappropriating information about tornado likelihood at different times of year and day from the GP region, where tornadoes have been more extensively studied. While this study cannot diagnose the causes of these perceptions, our results suggest that further work on this topic is needed, given the ways in which people’s preexisting perceptions of tornado risk can influence their attention to weather information, their interpretations of tornado forecasts and warnings, and their protective decisions when tornadoes threaten (discussed further below).
Our survey contributes to the generalizability of similar results found by Ellis et al. (2019b), who conducted a phone survey in 2016 of residents in three regions of Tennessee and found that participants did not have accurate perceptions of seasonality and timing of tornadoes. Our study extends these findings by surveying a broader sample of the SE. Further, whereas Ellis et al. (2019b) elicited judgments of which months are most (and least) likely for tornadoes to occur, we used different elicitation procedures to directly capture more complete judgments of likelihood, and our study investigates these judgments in greater detail.
Another contribution of this study is that it is the first to systematically capture the perceptions of a sample of tornado experts with knowledge about the SE. We find that the mean judgments of the expert sample for season and time of day match well with the results of recent meteorological research reviewed in section 2. Judgments of storm system type stood out from the other judgments, as the expert sample was not in agreement about whether supercells or QLCS had a higher tornado likelihood in the SE (as shown in Fig. 4), reflecting currently active debates among experts regarding tornado likelihood and threat from different storm types in the SE. Expert consensus about the relative risk of supercells and QLCS in the SE is evolving at the same time as this project.
There are many meteorological studies that present data on tornado frequency in the SE (e.g., Anderson-Frey et al. 2019; Ashley 2007; Ashley et al. 2008; Childs et al. 2018; Krocak and Brooks 2020; Long et al. 2018; Thompson et al. 2012); however, these data are generated from a different process, and are difficult to compare directly to the public’s judgments in a consistent way. We therefore relied on the expert sample to provide judgments of the same quantities in the same context as the public samples, based on their meteorological knowledge about tornadoes in the SE. As such, the expert sample provides an important benchmark, because their perceptions of tornado season, timing, and storm system type are more likely to be based on a global understanding of the atmosphere and records of tornado occurrences. However, even expert knowledge of these distributions is limited by the size and accuracy of the datasets that are used to estimate them.
We also investigated the extent to which differences between the public’s and experts’ perceptions can be explained by differences among members of the public in sociodemographic characteristics and tornado-related experiences and beliefs. As discussed in section 5a, the public judgments (relative to the expert judgments) were highly variable, and the analysis of these differences indicated that the collection of demographics and covariates measured in this survey accounted for very little of this variance. Nevertheless, we did find a few significant predictors of the distance between individual SE respondents’ judgments and the mean expert judgments. Members of the SE public with more tornado experience and higher educational attainment generated distributions of tornado season that were less distant from the experts. Members of the SE public who are older, more educated, and reported looking at the radar when severe weather approaches generated distributions of tornado timing that were less distant from the experts. These results suggest that exposure to meteorological information and education may be helping people understand current meteorological knowledge about SE tornado likelihood or motivating them to build such understanding. In our analyses, none of these variables predicted the distance between experts’ and the public’s perceptions of SE tornado storm system type. This likely reflects current scientific debates and uncertainty with regard to this topic.
In the online supplemental material we present analyses with extended individual difference measures, including the heuristic items that were dropped from the main analyses based on our preregistered analysis plan. These analyses indicate that several of these items may be significant predictors of public judgments of tornado likelihood. In future work we hope to refine these items based on the results reported here and to investigate the extent to which local knowledge and heuristics that are formed by the public in tornado-prone regions influence tornado risk perceptions.
While our survey is the first to collect a representative sample of judgments of tornado likelihood in the SE, the study has several limitations that could be improved for future work. First, it would be valuable to refine the measurement instruments for eliciting judgments of likelihood, and to develop methods for diagnosing causation of the SE public’s judgments. Questions of interest include the following: Are judgments based on knowledge generalized from the GP? Do recent experiences affect judgments? Second, our public survey data are cross-sectional data, representing a snapshot of judgments in time (September–October 2018). Although notable SE winter tornado events occurred in 2008 and 2011, such devastating, large-scale events had not occurred in the SE for several years. Further, in 2017 and 2018, four hurricanes generated substantial damage and loss of life in the SE, including hazards such as flooding and tornados; these recent experiences may have affected the tornado judgments we collected in 2018. Thus, it would be beneficial for future work to take a longitudinal approach to evaluate whether and how significant tornado events affect tornado risk perceptions, and how long such effects last.
Another limitation is that the expert sample was significantly smaller than the public samples, which generates challenges for comparing the two samples. The number of people with scientific expertise in SE tornadoes is unavoidably small. While we would have used a larger expert sample had it been possible, the experts’ responses exhibited much less variability than the public sample, resulting in similar standard errors around the means of both samples.
As discussed in the introduction, the ultimate goal of this research is to improve tornado risk communication and public decision-making in ways that reduce harm from tornadoes in the SE. Building on previous work by Childs and Schumacher (2018) and Ellis et al. (2019b), we find that members of the SE public may lack knowledge about the likelihood of tornadoes in different situations. These knowledge gaps may lead people to be less attentive to weather information and to be less prepared to respond to tornado threats, for example, in the winter or at night. This may be particularly problematic when combined with other barriers to effective protective action in the SE, such as the higher levels of mobile and manufactured housing stock and the limited safe sheltering options that many residents of such housing have even if they are warned (Ash et al. 2020; Ashley et al. 2008; Strader et al. 2019).
Misunderstanding the likelihood of tornado threats may also serve as a miscalibrated prior for evaluating official tornado watches and warnings (Ellis et al. 2019a). SE residents who believe tornadoes are not likely in the winter may discount official warnings because they perceive an extremely low probability that a hazardous event will occur (especially given that the chances of a tornado at any given location and time are low to begin with). If some members of the SE public maintain vigilance only during the summer and spring (when they believe tornadoes are most likely), then they may be less responsive to tornado warnings in the late autumn and winter, leading to more fatalities. Therefore, further research is needed to explore the potential effects of these types of misunderstandings on decision-making about protective action, especially in conjunction with the other factors contributing to tornado vulnerabilities in the SE.
In sum, the above discussion suggests that improved knowledge of tornado likelihood may help members of the SE public respond more effectively and more quickly to tornado threats. This is especially important given the difficulties in predicting and observing tornadoes in the SE United States, which mean that in some situations people will need to take shelter without personally observable signs of danger in order to avoid being caught in a deadly situation. An additional challenge, however, is that tornadoes in the SE will require more vigilance at all times of year and day, with regard to more types of storm systems relative to the GP. This type of vigilance is hard to maintain for low base-rate hazards, and it presents additional psychological challenges that will need to be addressed to improve protective decision-making for tornadoes in the SE.
This material is based upon work supported by the VORTEX-SE Program within the NOAA/OAR Office of Weather and Air Quality under Grant NA160AR4590218. The National Center for Atmospheric Research is sponsored by the National Science Foundation.
Prior to performing these elicitations, participants engaged in a practice trial to familiarize them with the task. In this trial, participants allocated 100 individual purchases of ice cream across four seasons of the year, representing how many ice cream purchases they believe would occur in each time of year.
Preregistration is the practice of writing out specific details of data collection, analysis plans, and data exclusion to document the research process. Preregistration has been generally adopted for empirical research as a step toward reducing research practices that can undermine objective statistical inference.
We preregistered using a nonparametric Kolmogorov–Smirnov (KS) test to evaluate differences between mean elicited distributions of each dependent variable (DV). This test is conservative when applied to discrete distributions (as the type we have elicited here) and was very sensitive to the limited sample size of the expert group (Conover 1999). For all compositional DVs, the KS test found significant differences between the SE and GP samples, all other tests were nonsignificant. We therefore do not report on the KS tests further. We also preregistered the compositional analysis approach that matches the distributional assumptions of our DVs.
The analysis of the distance measure did exclude the 68 participants who placed all tokens into a single bin for three or more of our dependent variables.
As a robustness check, we also performed these regressions using the demographics and other covariates, as outlined in our preregistration. The results (shown in section S3 of the online supplemental material) are identical, and the effects that were largest and easiest to interpret were for differences across samples, and so we present the simpler analyses in the main text.