1. Introduction

Consider the following scenario: You are at home, and the sky has looked ominous for the past couple of hours. Suddenly, you receive a Wireless Emergency Alert (WEA) issued by the U.S. National Weather Service (NWS) on your “smartphone” indicating that a tornado warning was just issued for your location. What would you do? Of course, the intent behind the issuance of WEAs is that they will provide immediate, life-saving information about a natural or human-made disaster, and (implicitly) prompt individuals to take protective action (FEMA 2022). Indeed, fostering a better connection between knowledge and good decisions in a severe weather situation is a core principle of the 2019–22 NWS strategic plan (NOAA 2020). Given the desire of the NWS to promote good decision-making in severe weather situations, it might seem obvious that sheltering in place is the smart approach in the scenario above, and those especially knowledgeable about severe weather should be the ones most prone to make good sheltering decisions. Conversely, those least knowledgeable about severe weather should be those most prone to making poor sheltering decisions. But is the relationship between having the relevant knowledge and making good decisions this straightforward?

One factor that may well influence the role of knowledge on the decisions people make in a severe weather situation is something in the psychological literature known as the Dunning–Kruger effect (DKE; Kruger and Dunning 1999) or sometimes the “unskilled and unaware” effect (Serra and DeMarree 2016). The DKE reflects the common finding that individuals with low levels of knowledge about a subject routinely overestimate their actual knowledge, while individuals with high levels of knowledge underestimate their knowledge. The DKE is an interesting phenomenon and the overconfidence exhibited by those with low knowledge has been interpreted as reflecting poor metacognitive skills; that is, individuals unskilled in a particular domain lack the metacognitive knowledge to realize it (Kruger and Dunning 1999; Serra and DeMarree 2016). Conversely, the reason why those who possess high levels of knowledge often underestimate their performance has been attributed to causes other than metacognitive ones. Rather than overestimate their own performance, they tend to overestimate the performance of others on a test, and therefore underestimate their own performance (Kruger and Dunning 1999; Dunning et al. 2003; Ehrlinger et al. 2008). Over the past 20 years, the pervasiveness of the DKE, especially the overconfidence of the unskilled, has been demonstrated in a variety of different knowledge domains, both academic and in the real world: humor, logical reasoning, and English grammar (Kruger and Dunning 1999); performance on psychology exams (Ehrlinger et al. 2008; Miller and Geraci 2011; Serra and DeMarree 2016; Schlösser et al. 2013); debate tournament performance and gun safety knowledge (Ehrlinger et al. 2008); anesthesiologists overconfidence in their perceived knowledge of a neuromuscular blocking drug (Naguib et al. 2019); health literacy (Canady and Larzo 2023); and endorsement of antivaccine policy (Motta et al. 2018). It has even been argued the DKE might account for some of the rise in anti-intellectualism in the United States (Welshon 2022).

Given the ubiquity of the DKE, it is interesting that it has only been applied to weather in a handful of studies. Nunley and Sherman-Morris (2020) did an excellent job of summarizing the state of knowledge as of 2020, so the research they reviewed will only be briefly mentioned here. Siems (2016), using qualitative data collected from survivors of Hurricane Katrina (who did not evacuate), noted just less than three-fourths of the interviewees mentioned DKE “indicators” in stating why they did not evacuate. In the context of climate change, both Sharp and Høj (2008) and Stoutenborough and Vedlitz (2014) found that individuals who perceived themselves as fairly knowledgeable performed poorly on objective tests of climate change. In master’s thesis research published after that of Nunley and Sherman-Morris (2020), Braat (2020) had participants take a European Union–funded quiz on climate change literacy after first estimating their knowledge of climate change. Again, Braat (2020) found the classic DKE effect, such that participants with the lowest 50% of objective knowledge significantly overestimated their scores on the quiz.

The paucity of research on the relationship between perceived and actual weather knowledge therefore led Nunley and Sherman-Morris (2020) to investigate it firsthand, to see if the DKE would also be found. In their study, they first gave participants a test of their perceived weather knowledge, followed immediately by an objective test of their actual weather knowledge. Their sample included two groups: one was recruited from a Facebook specialty weather page, and the other was recruited via Mechanical Turk. The two samples were then divided into quartiles based on their objective knowledge scores (as it typically done in DKE research). Although the sample drawn from the specialty weather page answered more objective questions correctly, both samples exhibited the DKE. More specifically, the participants in quartile 1 (for both samples) exhibited the largest and statistically significant overestimation of their knowledge, while the participants in quartile 4 (again, for both samples) exhibited the largest underestimation of their knowledge. The finding that the DKE was demonstrated in the context of weather knowledge reveals the extent to which the finding of overconfidence among individuals who objectively know the least applies.

Another factor to consider when examining the decisions made to severe weather, and one related to the DKE, is the role of knowledge itself in terms of influencing said decisions. The assumption that possessing more knowledge will lead to better decisions (especially in a potentially risky situation such as a tornado) is known as the knowledge deficit model (Stoutenborough and Vedlitz 2014). In other words, the basic idea is that the more one knows about a topic area (i.e., the lack of a deficit) the better decisions one makes. Interestingly, however, research on the role of knowledge on making good decisions is decidedly mixed. Numerous studies have investigated the role of prior experience on the sheltering decisions people make. In the case of tornadoes, some studies have found prior experience increases preparatory or protective sheltering decisions (e.g., Comstock and Mallonee 2005; Ripberger et al. 2019) while others have found no relationship (e.g., Armstrong and Usery 2022; Miran et al. 2018; Schmidlin et al. 2009). Allan et al. (2017) also found that increased knowledge of tornadoes was associated with a decreased belief in tornado myths. However, as noted by Demuth (2018), the inconsistency in the literature likely arises due to how “prior experience” is defined. As Demuth (2018) noted, if a single item is used to define experience, inconsistent findings are common. If, however, a more broadly defined measure is used, including both indirect and direct experience, a positive relationship between experience and risk perception is generally found.

In research investigating other weather phenomena, inconsistent results are again the rule. Morss et al. (2016), for instance, found that prior hurricane experience was associated with an increased intention to evacuate in a hurricane scenario. Conversely, Ramasubramanian et al. (2019) found that increased prior flood knowledge was associated with a decrease in weather risk perceptions. Bradford et al. (2012) also found a complex relationship involving prior flood experience. Although they found no relationship between awareness of residing in a flood risk area and higher levels of preparedness, they did find that those with previous flood experience exhibited higher preparedness levels. In the realm of climate change, possessing more knowledge of the science underlying climate change is not necessarily associated with an increased concern of a warming planet. Kellstedt et al. (2008) found no relationship between climate knowledge and concern, while others have found that factors other than knowledge may be more important to predicting increased concern (Hart and Nisbet 2012; Malka et al. 2009). Clearly, the relationship between one’s knowledge and comprehension and decision-making in weather-related scenarios is complex.

The DKE findings of Nunley and Sherman-Morris (2020), when coupled with the mixed results of the role of knowledge itself on informing severe weather decisions, pose an interesting question first asked in the opening paragraph: are those individuals who are objectively least knowledgeable about severe weather (but highly confident of their knowledge) those who are least likely to act in a situation requiring effective protective action. Stated differently, it may be that individuals unable to recognize their own incompetence may be precisely those individuals most prone to make poor decisions in a dangerous situation. Indeed, a recent study by Canady and Larzo (2023) provides preliminary evidence that such a finding may occur. Canady and Larzo (2023) investigated the role of the DKE in terms of one’s health literacy on medical decision-making. They found that those with low functional health literacy engaged in a variety of poorer health decisions, when compared with those with high functional health literacy, including making fewer medical appointments, taking more medications, taking fewer supplements, and engaging in more tanning. As noted by Canady and Larzo (2023), those individuals with low health literacy are those who need education the most yet may not recognize their own need for more information.

Does a similar connection between perceived knowledge and making protective action decisions also exist for those individuals facing severe weather? As noted by Dunning et al. (2004), “Much future work could profitably focus more on the consequences of mistaken self-assessments, to provide a more comprehensive and systematic account of when mistaken judgment is likely to produce its greatest costs, as well as when it might provide valuable benefits.” The exploratory study, discussed below, was therefore designed to address the potential consequences (if any) of overconfidence in one’s weather knowledge.

2. Study rationale

The experiment discussed below investigates the possibility that those individuals who objectively know the least about severe weather (but think they know the most) might be less likely to indicate they would engage in protective action when faced with a severe weather scenario. Participants in the experiment first took an assessment of their perceived severe weather knowledge and estimated their perceived percentile ranking of their knowledge. The participants then took an objective severe weather knowledge test and estimated the number they got correct. Last, the participants saw four wireless emergency tornado warning alert graphics on a simulated smartphone screen that accompanied a tornado scenario. Two scenarios involved being at home, while two scenarios involved being located in a grocery store. After each scenario, the participants made two protective action decisions: one about immediately sheltering in place and the other the likelihood they would leave their current location. These two decisions were used based on NOAA’s recommendations of what to do during a tornado: sheltering in place and seeking safety in a sturdy building (NOAA 2022ba). The two home-based scenarios were used based on the assumption that sheltering in place at home (in one’s safest location) would arguably be considered the wisest choice. The two grocery store scenarios were used because shopping for groceries is a common routine for many and it would be plausible to encounter a tornado warning while shopping. In these shopping situations, it was reasoned that a grocery store would likely be considered a “sturdy building” so driving away would arguably be a relatively unwise decision.

Two hypotheses (H1 and H2) were tested:

• H1: Individuals with less knowledge of severe weather will overestimate their knowledge while individuals with more knowledge will underestimate their knowledge, thereby illustrating the DKE.

• H2: Those individuals with the lowest levels of objective weather knowledge will be less likely to choose appropriate protective action decisions following severe weather scenarios, while individuals with the highest levels of objective weather knowledge will be most likely to choose appropriate protective action decisions.

3. Method

b. Materials

A 21-question quiz tapping into each participant’s perceived knowledge of severe weather was constructed based on questions used in previously published research. Eight questions were taken from Nunley and Sherman-Morris (2020), six from Casteel (2016), two from Ripberger et al. (2019), two from an online local weather story (Bianchi 2022) and fact checked by the author and one question each from Ripberger et al. (2015), Sherman-Morris and Antonelli (2018), and the National Severe Storms Laboratory Severe Weather 101 web page (NSSL 2022). Thirteen questions assessed knowledge of tornadoes, five assessed knowledge of hurricanes, and one question each assessed knowledge of severe thunderstorms and types of precipitation. Using the method of Nunley and Sherman-Morris (2020), the first 20 questions all began with the stem “I know the . . .” followed by the specific weather question. The first question, for instance, was “I know the hazards associated with severe thunderstorms.” Participants answered each question using a seven-choice Likert scale, ranging from strongly disagree to strongly agree. Question 21 asked the participants to provide a percentile ranking of their weather knowledge, on a 1–99 scale.

A companion 21-question quiz of objective weather knowledge was also constructed, and assessed the information queried in the quiz of perceived weather knowledge. The first 20 questions were multiple-choice questions and the last question asked participants to estimate the number of questions they got correct (of 20). All of the questions from the perceived and objective quiz of severe weather knowledge are shown in the appendix.

Tornado warnings were used, given the relatively short-fused nature of a tornadic event and the need to take immediate protective action, as compared with other, longer events such as a hurricane or blizzard (Drost et al. 2016; Erickson and Brooks 2006). Using tornado warnings also aligns with the findings of Gelino and Reed (2020), who found that a simulated WEA tornado warning with a lead time of either 1 or 5 min produced much higher estimates of seeking shelter than lead times of 15 min or longer. Weyrich et al. (2020b) also found that shorter intervals between a severe weather warning and the threatened event prompted a greater behavioral response. Similarly, Gutter et al. (2018) found that a “particularly dangerous situation” tornado watch prompted the highest likelihood of participants who stated they would likely discontinue an activity. Four tornado scenarios were therefore created to assess each participant’s willingness to follow the instructions from a simulated WEA message as it would appear on an Android smartphone screen. Two of the warnings asked the participants to imagine themselves at home while the other two utilized shopping at a grocery store scenario. As an example, the instructions [adapted from Gelino and Reed (2020)] for the home scenario were as follows:

Imagine yourself at home during typical daylight hours on a day off from work. Your county is under a tornado watch and thunderstorms have been impacting your location for the duration of the day. You’ve therefore engaged yourself in a preferred hobby/activity. Imagine that, during this time, you receive the Wireless Emergency Alert notification shown below on your cellular device alerting you of extreme weather in your area. You look out your window and see dark and scary clouds in the sky.

Immediately following the instructions was a simulated WEA graphic with the headline “Emergency alert: Extreme” followed by a date and time. The months of April, May, and June were used (as they have the highest average number of tornadoes) and the time stamps were late afternoon and evening. Two of the warnings were for tornadoes, while the other two were for tornado emergencies. The text of the warnings was taken from the sample 360-character warnings found on the NWS’s Wireless Emergency Alert web page (NOAA 2022dc). Participants were not asked if they knew the difference between tornado warnings and emergencies, although previous research has found that most individuals have little understanding of what tornado emergencies actually are (Mason and Senkbeil 2015). Tornado emergencies were included, however, to provide more message variety so that all four WEAs did not use the same wording. All four warning versions are shown in Fig. 1.

View Full Size

The four possible WEA warnings associated with the two decision scenarios. Examples of the two scenarios include the following: 1) Home-based decision scenario—“Imagine yourself at home during typical daylight hours on a day off from work. Your county is under a tornado watch and thunderstorms have been impacting your location for the duration of the day. You’ve therefore engaged yourself in a preferred hobby/activity. Imagine that, during this time, you receive the Wireless Emergency Alert notification shown below on your cellular device alerting you of extreme weather in your area. You look out your window and see dark and scary clouds in the sky.” 2) Grocery store decision scenario—“Imagine yourself at the grocery store shopping for food on a day off from work. Your county is under a tornado watch and thunderstorms have been impacting your location for the duration of the day. While pushing your shopping cart, you receive the Wireless Emergency Alert notification shown below on your cellular device alerting you of extreme weather in your area. You look out the store’s windows and see dark and scary clouds in the sky.”

Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-22-0115.1

• Download Figure
• Download figure as PowerPoint slide

4. Results

The average correct score on the objective quiz of severe weather knowledge was 51.2% correct (10.23 of 20). Scores ranged from 3 to 20 (only one individual of the entire sample received a perfect score). The average predicted score after taking the test was 45.8% (9.16 of 20). Both scores indicate that the test was challenging to many of the participants. Notably, the scores on the objective test did not differ between those who identified as either male or female, t(589) = 0.79, p = 0.43. The summed predicted knowledge scores also did not differ by gender, t(589) = 0.05, p = 0.96. Interestingly, when the predicted percentile rank of one’s knowledge was analyzed by gender, those identifying as female rated their perceived knowledge as lower (51.9th percentile) than those identifying as male (56.4th percentile), t(589) = 2.48, p < 0.05.

In the analyses that follow, all tests for nonhomogeneity of variance were nonsignificant (p > 0.09), validating the use of analyses of variance (ANOVA) to analyze the data. Effect sizes are reported as partial eta squared ${\eta }_p^2$, as suggested for repeated measures designs (Lakens 2013; Richardson 2011). Additionally, magnitudes for effect sizes are interpreted following Cohen’s (1988) suggested thresholds: 0.01 = small, 0.06 = medium, and 0.14 = large. Table 1 provides a summary of all of the ANOVAs discussed below.

Table 1

ANOVA test statistics and effect sizes for all five analyses. Here, df indicates degrees of freedom, SS indicates sum of squares, and MS indicates mean sum of squares. Note that “effect” and “Wilks’s lambda value” apply to the first two columns in the multivariate test rows in analyses 2 and 5 and that “variable” and “type-III SS” apply to the first two columns in all other rows and analyses.

View Table

To assess whether the participants exhibited the DKE, they were divided into quartiles based on their score on the objective test of severe knowledge, as is common in DKE research. For the first analysis, difference scores were created for each participant by subtracting the actual percentile score obtained from the predicted percentile rank score. As such, overconfidence is indicated by positive numbers whereas underconfidence is indicated by negative numbers. These data were then entered into a gender × quartile ANOVA. Only those participants who identified as either “female” or “male” were included in this analysis. The main effect of gender was significant, F(1, 583) = 6.26, p < 0.02, and ${\eta }_p^2=0.01$ (small effect size). Females’ estimates of their ability were more calibrated (i.e., the mean difference between predicted and actual percentile was smaller; 5.18) than those of males (9.59). The main effect of quartile was also significant, F(3, 583) = 165.7, p < 0.001, and ${\eta }_p^2=0.46$ (large effect size). Bonferroni comparisons revealed that participants in all four quartiles differed from one another (all p < 0.001). Participants in quartile 1 were the most confident, followed by quartile 2, quartile 3, and then quartile 4. These results are shown in Fig. 2. Note that individuals in both quartiles 1 and 2 were overconfident, and their predictions severely overestimated their true knowledge. In contrast, individuals in quartile 3 were highly calibrated (i.e., the difference score was essentially zero at −0.9) while those in quartile 4 underestimated their performance. As an alternative method of visualizing the data, Fig. 3 shows the perceived percentile rank and actual percentile rank separately, as a function of quartile. The gender × quartile interaction was not significant (p = 0.23), indicating that performance did not differ as a function of gender. Gender was therefore not included in any of the additional analyses below.

View Full Size

Difference scores as a function of quartile. The vertical bars denote 0.95 confidence intervals.

Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-22-0115.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Predicted vs actual knowledge percentile rank. The vertical bars denote 0.95 confidence intervals.

Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-22-0115.1

• Download Figure
• Download figure as PowerPoint slide

To provide a direct comparison with the findings of Nunley and Sherman-Morris (2020), who used standardized scores on both quizzes of perceived and objective knowledge, the same standardized scores were used in this second analysis. The 20 Likert questions of perceived knowledge were summed and standardized (unlike the analysis above, where predicted percentile rankings were used), as were the objective quiz scores. The data were entered into a repeated measures ANOVA, with DKE (perceived score/actual score) the repeated measure. As suggested by others for analyzing repeated measures research (O’Brien and Kaiser 1985; Perreault et al. 2014), Wilks’s lambda multivariate test was used on the repeated measures to avoid potential problems associated with violations of compound symmetry and sphericity. The critical DKE × quartile interaction was again significant, F(3, 597) = 70.66, p < 0.001, and ${\eta }_p^2=0.26$ (large effect size). As seen in Fig. 4, individuals in quartile 1 with the lowest levels of objective knowledge of severe weather significantly overestimated their knowledge, while individuals in quartiles 3 and 4 significantly underestimated their knowledge. Bonferroni comparisons confirm that individuals in quartile 1 significantly overestimated their knowledge, while individuals in quartiles 3 and 4 significantly underestimated their knowledge (all p < 0.001). For individuals in quartile 2, their perceived and actual scores did not differ.

View Full Size

Predicted vs actual weather knowledge (standardized scores). The vertical bars denote 0.95 confidence intervals.

Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-22-0115.1

• Download Figure
• Download figure as PowerPoint slide

To examine whether individuals in the lowest quartile of objective weather knowledge were more likely to make poor sheltering decisions, the scores of all four sheltering decisions were summed to create a composite total sheltering score, with 400 the highest possible score. These data were then entered into a one-way ANOVA with quartile the categorical variable. This analysis revealed that sheltering decisions did vary as a function of quartile, F(3, 597) = 5.41, p < 0.002, and ${\eta }_p^2=0.02$ (small effect size). Bonferroni comparisons showed that individuals in both quartiles 1 and 2 were significantly less likely to shelter than those in quartile 4. Quartiles 1, 2, and 3 did not differ among each other. These results are shown in Fig. 5.

View Full Size

Decision to “seek shelter now” as a function of quartile group. The vertical bars denote 0.95 confidence intervals.

Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-22-0115.1

• Download Figure
• Download figure as PowerPoint slide

The scores of all four decisions to leave one’s residence or the grocery store were also summed to create a composite total leaving score, with 400 again the highest possible score. These data were also entered into a one-way ANOVA. This analysis also showed that leaving decisions varied as a function of quartile, F(3, 597) = 6.55, p < 0.001, and ${\eta }_p^2=0.03$ (small effect size). Bonferroni comparisons showed that individuals in quartile 1 were significantly more likely to leave than those in quartiles 2, 3, and 4, who did not differ. These results are shown in Fig. 6.

View Full Size

Decision to leave as a function of quartile group. The vertical bars denote 0.95 confidence intervals.

Citation: Weather, Climate, and Society 15, 2; 10.1175/WCAS-D-22-0115.1

• Download Figure
• Download figure as PowerPoint slide

One final analysis was conducted, to analyze for the DKE in a manner similar to that of Kruger and Dunning (1999). Instead of examining perceptions of severe weather knowledge before taking the objective test, this analysis examines the participants’ predicted number correct (vs. the actual number correct) immediately after they took the objective test. The participants were again divided into quartiles based on their actual knowledge, and the data were analyzed using a repeated-measures ANOVA, similar to the analyses above. The DKE × quartile interaction was significant, F(3, 597) = 11.8, p < 0.001, and ${\eta }_p^2=0.06$ (medium effect size). Interestingly, this analysis showed that individuals in quartiles 3 and 4 indeed underestimated their knowledge, as their actual scores were significantly higher than their predicted scores (both p < 0.001). However, individuals in quartiles 1 and 2 did not differ in their predicted versus actual scores. In this regard, only one aspect of the DKE was shown, where those more knowledgeable were underconfident. This issue will be briefly addressed in section 5.

5. Discussion and conclusions

There are two main findings from the study reported here. The first finding is that the DKE was found—individuals with the least objective knowledge of severe weather overestimated their knowledge while those with the most objective knowledge underestimated their knowledge, illustrating the DKE. Indeed, the effect size was large, regardless of whether difference scores or standardized scores were used. This finding supports H1 and equally importantly, replicates the findings of Nunley and Sherman-Morris (2020). This finding adds to the significant literature already in existence showing the wide variety of knowledge domains to which the DKE applies, from overestimating test scores, debate performance, and gun safety (e.g., Ehrlinger et al. 2008) to endorsement of antivaccine policy (Motta et al. 2018), among others. Note, too, that the DKE occurred in the present study even though the participants, on average, should have been fairly knowledgeable about tornadoes, as the sample was limited to individuals living in the most tornado-prone states. This evidence suggests that the lower knowledge participants indeed had a more difficult time monitoring their own understanding of severe weather, unlike the higher knowledge participants. This finding is congruent with the metacognitive failure explanation of poor performance by the lowest knowledge individuals proposed by Kruger and Dunning (1999).

The second, and more relevant finding to those in the weather enterprise, is that in a simulated tornado warning scenario, participants with low knowledge were less likely to state they would “take shelter now” and more likely to say they would try to leave. Individuals in both quartiles 1 and 2 were less like to indicate that they would “take shelter now,” relative to those in quartile 4. Similarly, individuals in quartile 1 were more likely to indicate they would leave their residence (or the grocery) store, relative to individuals in quartiles 2, 3, and 4. For low knowledge individuals, both decisions go against the suggested courses of action by NOAA, where the advice is to shelter in your current location and to specifically not be out driving in one’s car (NOAA 2022b). This finding also supports H2. Although both effect sizes were small, as noted by Sutton et al. (2021), even small effect sizes can have large, real-world implications in terms of the numbers of individuals impacted by protective sheltering decisions.

The current findings would appear to be the first to show that individuals with low knowledge of severe weather (and who tend to overestimate their knowledge) may be more prone to make unwise decisions when faced with a potentially dangerous and immediate severe weather situation. They also support the findings of Canady and Larzo (2023), who found a similar effect such that those with lower levels of medical literacy made poorer medical decisions in real-life situations.

Unfortunately, the current results do not say anything about the underlying reasons that low knowledge individuals made “poorer” decisions. At least two possibilities present themselves. First, research has shown that many individuals confuse a tornado watch with a tornado warning (Donner et al. 2022; Walters et al. 2019). While a majority of the population does understand the difference between a watch and a warning (Jauernic and Van Den Broeke 2016; Ripberger et al. 2015), those who do confuse the two (i.e., the low knowledge individuals) might be less likely to perceive the urgency of the warning and therefore less likely to engage in appropriate sheltering behavior. A second possibility is suggested by the research of Klockow et al. (2014). Using qualitative interviews from survivors of the Alabama and Mississippi tornado outbreak in April 2011, they found that many of the interviewees were guided by folk science and past, personal experience. It is possible that those with less knowledge were precisely the ones most guided by folk science. As noted by both Donner et al. (2022) and Allan et al. (2017), myths or misconceptions about severe weather may lead to dangerous decisions. That may be precisely what occurred here. Future research is needed to assess this idea.

It is also important to note that there were only two analyses for which the results differed by gender. First, females rated their perceived knowledge of severe weather as lower than males’. This is consistent with the finding of Ehrlinger and Dunning (2003) who found that females rated their knowledge of science lower than that of males. Interestingly, the objective knowledge scores did not differ as a function of gender, which is also consistent with the findings of Ehrlinger and Dunning (2003). Women were also more calibrated in their knowledge scores; the difference between their predicted and actual scores on the two weather quizzes was less than that for males. Most important, the gender × quartile interaction was not significant. Participants identifying as female were just as likely to be either over- or underconfident as were males, showing that (at least in this study) the DKE was equally likely to occur in females and males.

Interestingly, the analysis comparing the predictions of how many questions were answered correctly on the weather knowledge quiz with the actual scores (as is done in most research studying the DKE) did not produce the traditional DKE, where the individuals in quartile 1 should have overestimated their scores. Instead, they appeared to have been fairly well calibrated (and realized their low levels of knowledge), while the high knowledge individuals (quartiles 3 and 4) continued to underestimate their performance. As noted above, the quiz of severe weather was challenging, as evidenced by the overall low average score correct (51.2%). It might be that once the individuals in quartile 1 took the test and encountered the specificity (and difficulty) of the questions, they were forced to confront their true lack of understanding. Inspection of the mean number of objective questions the individuals in quartiles 1 (6.7/20) and 2 (8.8/20) thought they got correct after taking the knowledge quiz supports this idea. This may be one of those boundary conditions originally mentioned by Kruger and Dunning (1999) where the DKE may not occur, similar to the finding of Sanchez and Dunning (2018) that “rank beginners” (i.e., those individuals first learning about a topic) did not exhibit a DKE. It is possible the low knowledge individuals in the current study were similar to Sanchez and Dunning’s (2018) rank beginners. Given the normal ubiquitous nature of the DKE in the research literature, this would be a fruitful avenue of future research.

The real-world implications of this research are straightforward, for both scholars and those involved in the weather enterprise. As mentioned previously, the intent behind the issuance of WEAs is to prompt all warning recipients to take immediate protective action (FEMA 2022). WEAs are mass distributed, via cell towers in the tornado warning area, to all smartphones in that area, assuming that the telephone’s owner has not opted out of the WEA service. The present results, however, suggest that individuals with lower knowledge of severe weather may be less likely to heed the warning. Additionally, individuals with poor understanding of severe weather warnings have been shown to be more likely to believe many myths about tornadoes (Allan et al. 2017), such as that sheltering under an underpass is a good decision. Consequently, those same individuals may be more likely to choose to ignore the warning, or if they do heed it, make decisions that may place themselves in more danger. And if they continue to believe that they are as knowledgeable (or more) than those individuals with objectively higher levels of knowledge, they may not recognize their own need for more information, or to correct their misunderstandings, consistent with the metacognitive explanation of the DKE (Canady and Larzo 2023).

Clearly, there is a need to promote better weather literacy in the general public and this goal likely requires a multipronged approach. Fleischhut et al. (2020) suggest that one place to start would be to incorporate weather literacy into the school curricula using real-world examples of weather and climate risks. Senkbeil et al. (2021) also noted the need for additional education, especially concerning being in a car during a tornado. Additional education alone, however, is likely not going to be a panacea promoting better decision-making (Demuth et al. 2022). Also, as noted above by Klockow et al. (2014), many members of the public operate on folk science beliefs, which is fundamentally different from the perspective adopted by meteorologists. As they note, “This difference of epistemology is unlikely to be resolved by universally identifying and dispelling local tornado ‘myths’ through re-education.” If true, this does not bode well for those with low knowledge.

Additionally, the notion that possessing more knowledge will lead to better decisions (i.e., the knowledge deficit model) has come under criticism in the science communication literature lately as being much too simplistic (e.g., Gustafson and Rice 2016; Simis et al. 2016; Sturgis and Allum 2004). Gustafson and Rice (2016) posit a variety of person-level variables (e.g., interest, ideology, partisanship, socioeconomic status) that influence what they term the “knowledge–attitude–practice” gap in sustainability communication. Similarly, important individual difference variables have been found to influence the sheltering decisions people make for severe weather (e.g., Morss et al. 2016). Simis et al. (2016) also found that the views that scientists hold about the “public” influences whether they fall prey to the simplistic thinking behind the knowledge deficit model. Fully half of the biological, physical, and social scientists they surveyed from a large midwestern university viewed the public as nonscientific, and another quarter viewed themselves as not part of the “public,” creating an artificial “us–them” dichotomy. Clearly, the attitudes that scientists hold about the public are likely just as important as the attitudes that the public holds about severe weather, adding further complications to the ideas of improving weather literacy.

While the results of the current study add to the body of research highlighting the influence of a variety of psychological factors on weather-based decisions, potential limitations must also be considered. First, although the sample was designed to be broadly representative of those individuals most likely to encounter tornado warnings, it was still a nonprobability sample. Results therefore may not generalize to other populations. For instance, individuals such as tourists, who may be visiting a tornado-prone state from one where few tornadoes occur, may make different sheltering decisions, regardless of their knowledge of severe weather. Future research is needed to address this question.

Another limitation is that the two protective action decisions were dichotomous—stay or leave. While others have expanded the decision-making process to include preparatory actions as well (e.g., Armstrong et al. 2020; Kang et al. 2007), the yes/no behavioral questions were used here because, as mentioned above, the intent behind the use of WEAs distributed by the NWS is to prompt immediate and safe sheltering decisions (NOAA 2022a). A related point is that although NOAA urges sheltering in place in a sturdy building, the decision to stay or leave is more nuanced. If one lives in a mobile home, the WEAs used in the current study specifically suggested moving to the closest substantial shelter, which likely would require driving away. As defined in the present research, that would be considered a poor decision, but for the individual living in a mobile home, it could be a quite wise decision indeed. Future research would benefit from trying to tease apart these leaving/staying nuances to ensure that a “good” or “poor” decision is appropriate to the context in which an individual finds themselves. Providing more than two decisions options should be explored as well, possibly utilizing options that fall on a range from wise to unwise decisions.

Another potential limitation is that the procedure used in this study relied on behavioral intentions rather than assessing actual behavior. There are at least three arguments, however, against this potential criticism. First, research assessing how participants respond to simulated severe weather scenarios is quite common and has been used in a variety of studies (e.g., Gutter et al. 2018; Schultz et al. 2010; Weyrich et al. 2018, 2019, 2020b). Second, work by Ripberger et al. (2015) has shown that behavioral intentions are a valid proxy for actual behavior. A recent and most convincing study by Weyrich et al. (2020a) examined the issue of behavior intentions versus actual behavior directly. They collected responses in real time to notifications on a weather app, and participants could click on a link to indicate their actual behavioral response. In the second phase of the study, the researchers conducted a scenario-based experiment using the same warnings and assessed behavioral intentions. Interestingly, and directly relevant to the question at hand, Weyrich et al. (2020a) found no differences in responses as a function of type of study (field vs simulation).

Despite these potential limitations, the present results add to the growing body of literature examining the role of knowledge on decision-making in severe weather contexts. The results are also the first to demonstrate that overconfidence in one’s knowledge coupled with poor objective knowledge may lead to poor decision-making in a tornado situation. Although the need for additional education is clear, the metacognitive challenges facing those with low knowledge of severe weather suggest a worrisome conundrum: those who know the least about severe weather, thinking they know a lot, are probably those individuals least likely to seek out additional education on the topic.

Data availability statement.

Anonymized data may be made available upon request.