Rapid technological changes have contributed not only to improved weather forecasts but also the availability of weather information. This is especially true regarding mobile technology for which thousands of weather apps are available (Zabini 2016). While recent social science research has aided in a better understanding of weather risk communication and decision-making, less is understood about the public’s actual weather knowledge, how they assess their knowledge about the weather, and how knowledge may relate to weather forecast information use. Psychology literature shows people with lower levels of knowledge about a topic are prone to overestimating their knowledge (Kruger and Dunning 1999). This Dunning–Kruger effect (DKE) is a cognitive bias where people are unable to recognize their own incompetence (Kruger and Dunning 1999). Former president of the American Meteorological Society Dr. Marshall Shepherd has recently brought the DKE to public attention in an interview on National Public Radio, and on Forbes.com, as it pertains to climate change (Shepherd 2016). However, the effect has yet to be studied as it relates to meteorology. Weather forecasts can provide life-saving information during an extreme weather event, which makes it important to know how knowledgeable the public is about weather information, including whether individuals overestimate their weather knowledge. It is possible that one who overestimates his or her weather knowledge may be overconfident in his or her ability to make important decisions during an extreme weather event. This could prove to be a costly decision. If we know where the public overestimates their knowledge, and in which areas weather knowledge is lacking, information can be provided to help improve decision-making.
Literature review
Weather knowledge.
Much of the existing literature on weather knowledge focuses on high-impact weather such as hurricanes or tornadoes. For example, Drake (2012) analyzed multiple studies to generalize understanding of tropical cyclone forecast information. After Hurricane Charley, the National Weather Service concluded education needed to be improved surrounding forecast tracks of tropical cyclones and other tropical products because of the public’s confusion surrounding the products due to their lack of scientific knowledge (NWS 2006). Research shows the public pays close attention to tropical graphics, such as the cone of uncertainty, but may also misunderstand it (Broad et al. 2007). The public confusion surrounding the tropical products extends outwards. Even emergency managers, who have had specialized training, are susceptible to the same misinterpretations (Drake 2012). A recent survey attempted to quantify some of the gaps of knowledge in hurricane information. This study reported that a majority of participants could correctly answer questions relating to the dates associated with hurricane season and the Saffir–Simpson scale, but only one-third could identify the ingredients needed for hurricane development (Sherman-Morris and Antonelli 2018).
Similarly, research has examined individuals’ understanding of tornado and severe weather information. Many studies have questioned whether participants know the difference between a “watch” and a “warning.” Some of these studies required that participants provide a definition (e.g., Silver 2015; Sherman-Morris 2010) while others used multiple choice format with single (e.g., Mitchem 2003) or multiple questions (e.g., Ripberger et al. 2019). The results from these studies are quite varied, possibly due to the different question format. Powell and O’Hair (2008) reported that 58% could successfully describe the difference between a watch and a warning, while 80% could correctly define a tornado warning in a study conducted by Sherman-Morris (2010). More recent research has attempted to create a baseline measure of knowledge about tornado warnings across the United States (Ripberger et al. 2019). This research demonstrated that exposure to warnings was associated with higher objective warning comprehension. It also indicated a relationship with age where comprehension of warning information increases in older age groups. Similar results were also presented by Powell and O’Hair (2008), who noted that older participants responded more correctly about warning information, while participants from California were less accurate. Similarly, correctly defining tornado watch has been positively associated with a participant’s age and frequency of weather information use (Sherman-Morris 2010). A connection was also found between paying attention to weather information and knowledge about watches and warnings, at least for men (Stewart 2009). Research is more limited on knowledge of other meteorological hazards such as winter weather. Because weather affects regions of the United States differently, location could frame the type and amount of exposure to weather information, which may in turn influence knowledge.
Overestimation of knowledge.
A vast amount of literature has explored the concept of people overestimating their knowledge in other fields. People tend to hold positive beliefs about their competence to a logically impossible degree (Dunning et al. 2004; Alicke and Govorun 2005), and frequently overestimate their knowledge and underestimate their limitations. Dunning et al. (2004, p. 83) showed people are not adept at spotting the limits of their knowledge and expertise and are “blissfully unaware of their incompetence.” This is an example of the DKE. This cognitive bias has been evaluated in many settings, ranging from the workforce to academia in which DKE has been identified in university students’ logical reasoning ability (Ehrlinger et al. 2008), specialist physicians’ clinical practice (Violato and Lockyer 2006), and salespersons’ ability to sell (Jaramillo et al. 2003).
Low-performing individuals tend to overestimate their abilities, while high-performing individuals are more readily able to estimate their abilities. A study conducted by Bell and Volckmann (2011) evaluated perceived and assessed knowledge of undergraduate students enrolled in general chemistry. Students who had a high level of performance in the chemistry course, estimated their knowledge with great accuracy; however, students who had a low performance in the course overestimated their knowledge. This was supported by Lindsey and Nagel (2015), who showed low-performing students tended to overestimate their abilities and high-performing students estimated their abilities much better. Lindsey and Nagel (2015) measured this utilizing knowledge surveys and final exams from students enrolled in physics and chemistry courses. These results add to a body of research that shows people do, indeed, lack megacognitive awareness. Similar to Lindsey and Nagel (2015), Nuhfer and Knipp (2003) determined students exhibit overconfidence of their abilities at the start of a class. This lack of metacognitive skills is also apparent in students outside of chemistry and physics. A study of medical students showed students with higher grades tended to underestimate their performance and students who received lower grades overstated their performance (Edwards et al. 2003).
Jaramillo et al. (2003) showed there were significant differences between a salesperson’s self-evaluation and their rating from a supervisor. The salespersons that performed poorly overestimated their job performance and the top salespersons underestimated their job performance (Jaramillo et al. 2003). The salespersons that performed poorly also were found to be significantly more inaccurate (Jaramillo et al. 2003). In the medical field, 304 professionals were evaluated for DKE. Violato and Lockyer (2006) examined the discrepancy between self and peer assessments of specialist physicians in internal medicine, pediatrics, and psychiatry clinical domains. The results show that these practicing medical physicians tended to rate themselves higher than their assessment from their peers. Thus, these data suggest practicing medical physicians are also inaccurate in assessing their own performance.
The concept of knowledge overestimation has been used to a limited extent in a hazards context. Siems (2016) aimed to identify indications of the DKE in individuals who encountered natural or human caused disasters by evaluating the decision-making of individuals who survived disasters. Siems (2016) derived 12 indicators from the works of Dr. Justin Kruger and Dr. David Dunning, the authors who coined DKE, to determine if the DKE existed in transcripts of interviews from survivors of disasters. The work of Siems (2016) suggested not all indicators need be present for the DKE to exist and confirmed the presence of the DKE in both natural and human-caused disasters. Almost three-fourths (73%) of survivors of natural disasters showed DKE indicators while 47% of participants met the criteria for the DKE (Siems 2016). Interestingly, people with advanced warning of a natural disaster, showed increased DKE signs (Siems 2016) and participants who reported to have an authoritative role also displayed indicators of the DKE. While not explaining their results using the DKE, Paton et al. (2008) found evidence of participants overestimating their knowledge of the local volcano hazard. When they further studied the influence of a public information campaign, they found that the exposure to hazard information led to participants feeling safer, even though they had not actually adopted any additional protective behaviors (Paton et al. 2008).
DKE tendencies were shown by hospital administrators during Hurricane Katrina. As the dangerous hurricane approached the coast of Louisiana, the hospital administrators at Pendleton Methodist Hospital decided to neither evacuate [Supreme Court of Louisiana, Lacoste v. Pendleton Methodist Hospital, 948 So. 2d 184 (La. 2007)] nor remove the backup power generator from the basement of the hospital (which flooded frequently). The flooding caused power failure, which led to a patient dying. The court records showed the administration was skilled in the hospital operations but lacked the knowledge necessary to properly plan and safeguard for the patients during a natural disaster (Insurance Journal 2010). This case shows a clear lack of skill awareness by hospital administrators in which the administrators were overly confident in their ability to prepare and respond to natural disasters—an example of DKE.
Another area of research which has relevance to estimation of weather knowledge is the perceived versus actual understanding of climate change. In past work, the public demonstrated a limited understanding of climate change (Bulkeley 2000a). Similar to literature in psychology and communications, a study focused on climate change showed people who cite being knowledgeable on an issue were not as knowledgeable as they perceived. Stoutenborough and Vedlitz (2014) conducted a study on climate change and found those with higher perceived knowledge had lower assessed knowledge. An Australian study also analyzed self-evaluated versus actual public knowledge about climate change (Sharp and Hoj 2010). Participants in the study believed they were knowledgeable on climate change; 51% claimed to be well informed about climate change and 9% claimed they knew a lot about climate change. After the self-assessment, a test was given, and the results showed low levels of knowledge in those who claimed to be informed on climate change (Sharp and Hoj 2010).
Perceptions people have of their knowledge complicates climate change educational efforts (Selm et al. 2019) due to perceptions of intelligence determining how people engage with issues (Waylen et al. 2004). An implication of DKE is those who are confident in their knowledge level are unlikely to be motivated to acquire new knowledge. As described by Gross and Latham (2012, p. 574), “[i]ndividuals with below proficient skills are unlikely to seek remediation for skills they believe they have…because they have a high level of confidence in their ability.” While understanding of climate change can be attributed to a host of societal factors, the lack of knowledge about climate change is partly due to its complexity (Bulkeley 2000b). As climate change became mainstream, climate change discussion has appeared on television in political debates, and on local and national news. However, the medium requires communicating this complex issue at a middle-school level. The watering down of the explanation of climate change and its risks, by experts who are utilized by the media, created an oversimplified idea, which some argued gave viewers a false sense of knowledge about climate change and its impacts (Stoutenborough and Vedlitz 2014). Additionally, those who trust the media are more likely to believe they are knowledgeable on climate change (Stoutenborough and Vedlitz 2014). The oversimplified explanation can cause harm if the public believes they are informed but are lacking fundamental knowledge of climate change and other hard sciences. This effect was termed the “easiness effect of science popularization” by Scharrer et al. (2016). This effect states that oversimplification of popularized science depictions hinders individuals’ recognition of their inability to make accurate judgments on scientific issues. Laypeople exhibited greater confidence in their knowledge, had a higher trust in their own judgment, and showed a weaker desire for advice from more knowledgeable sources after reading popularized science articles (Scharrer et al. 2016).
Finally, other factors have been shown to influence perceived knowledge. Gender and race have played a role in perceived science knowledge. Research shows not only women but also minorities have low confidence in science, technology, engineering, and math (STEM) fields (Aronson and Inzlicht 2004). The low confidence can stem from negative cultural stereotypes (Steele and Aronson 1995). Negative stereotypes have a strong presence in today’s culture primarily for women and minorities (Selm et al. 2019), which was demonstrated in Grunspan et al. (2016) in which men are more likely to be named by peers as being knowledgeable about STEM course content than women. Women believe they are less knowledgeable in science because of low confidence due to cultural stereotypes and men perceive women to be less knowledgeable in science (Grunspan et al. 2016). McCright (2010) and Stoutenborough and Vedlitz (2014) showed women exhibit more knowledge on climate change than men but underestimate their knowledge. Selm et al. (2019) found that women with a higher level of educational obtainment self-reported lower climate change knowledge than men with a higher level of educational obtainment, which suggests the stereotype threat is possibly exacerbated by the educational system. There is also evidence that women perceive they have less weather knowledge about tornado warnings than men, despite results showing comparable levels of objective knowledge (Ripberger et al. 2019). Self-perceived knowledge of weather information has not been explored in depth outside of the severe weather context, including differences between men and women, which makes this area an important area to explore.
Hypotheses
The objective of this study was to gain a better understanding of individual’s perceived and actual (assessed) weather knowledge and to compare the two for evidence of DKE. Additionally, both perceived and assessed knowledge were tested for differences that might be explained by gender, frequency of weather forecast use, and especially use of specialty weather websites. Surveys were the selected method to evaluate these variables. Based on previous DKE research, the authors hypothesized that individuals with lower assessed knowledge will perceive themselves to be more weather knowledgeable than they are. Hypotheses tested by the survey analysis include the following:
- H1: Participants with lower assessed weather knowledge will exhibit higher self-perceived weather knowledge.
- H2: Users of specialty weather websites will have a have higher self-perceived weather knowledge.
- H3: Users of specialty weather websites will score higher on assessed weather knowledge.
- H4: More frequent consumers of weather information will have 1) higher self-perceived weather knowledge and 2) higher assessed weather knowledge.
- H5: Women will have lower perceived weather knowledge than men.
Methodology
The survey.
A survey was constructed and administered to two samples using Qualtrics to test the hypotheses. The survey was pretested by several colleagues before it was distributed to the two groups. This survey contained Likert-type questions to measure self-perceived weather knowledge and multiple-choice questions to measure assessed weather knowledge. Self-perceived weather knowledge was the first variable of interest measured followed by assessed weather knowledge. Self-perceived weather knowledge was measured using 12 Likert-type questions. The Likert-type questions consisted of seven categories from strongly disagree to strongly agree (Table 1). Assessed weather knowledge was measured using 12 multiple-choice questions that relate to the self-perceived weather knowledge questions. The order in which self-perceived knowledge and assessed knowledge varies from previous studies of DKE. Although this order varies from previous DKE studies, we believed it would be more important to have a measure of self-perceived weather knowledge that would not be biased by the specific questions used in the assessed weather knowledge section. In fact, Dunning (2011, p. 276) noted that “making a topic or its elements more familiar by exposing participants to them . . . leads people to be more confident that they can provide correct answers.” The wording of self-perceived and assessed weather knowledge questions was matched as closely as possible so that comparisons could be made about individual responses. Unfortunately, we did not include an opportunity for participants to assess the accuracy of their individual responses, which would have strengthened the relationship between our findings and previous work on DKE.
Self-perceived and assessed weather knowledge questions.
To overcome some of the limitations with question variability discussed above, the literature was surveyed for questions about tornadoes and hurricanes that were utilized in previous work. At the time the survey was developed, a set of previously validated questions addressing the specific goals of this study did not exist. Therefore, questions were adapted from Gutter et al. (2018) and Sherman-Morris and Antonelli (2018), and additional assessed weather knowledge survey items were designed using information on the National Weather Service and National Hurricane Center websites. The multiple-choice format allowed for more reliable comparisons with past and future studies. Additional research has since been published regarding the validation of a tornado knowledge question set (Ripberger et al. 2019). Our question set had many similarities to that of Ripberger et al. (2019), including a very similar severe thunderstorm hazards question, and tornado questions with similar language about watch/warning size and the possibility of tornadoes in each. To our knowledge, no such question set existed at the time of this study for tropical or winter weather. We purposefully excluded questions about weather safety in favor of more physical weather knowledge questions. Participants were also asked demographic questions (age, gender, race, education, and income), and indicated (from a list of choices) sources of weather information and the frequency with which they obtained a weather forecast.
Descriptive statistics for all variables were calculated utilizing IBM SPSS Statistics. The Shapiro–Wilk test was utilized to determine whether data were normally distributed. Nonparametric tests were used to determine statistical significance of differences and associations in the data since the Shapiro–Wilk test indicated the data were not normally distributed. To test for significant differences between groups, the Mann–Whitney U test was used to test was used for two groups and the Kruskal–Wallis H test was used for data with more than two groups. Cronbach’s alpha was used to determine whether it was justifiable to interpret scores that were aggregated together for the response variables that were averaged or summed. Cronbach’s alpha is an estimate of internal consistency associated with the scores that can be derived from a composite score. If the Cronbach’s alpha was greater than 0.60 the data could be combined.
Samples.
Two samples were selected in order to compare followers of specialty weather pages to the general public to determine any differences in self-perceived and actual weather knowledge. The goal in choosing a sample from a specialty weather site was to examine how this population might be different based on their use of more specialized weather media. The specialty weather page selected was Firsthand Weather (FHW). FHW has over 100,000 users on Facebook. One of the authors (who is a co-owner of FHW) shared a link to the survey on the FHW social media account. The sample size was filtered to 479 responses from this site by eliminating responses that took less than 3 min and responses that were less than 70% complete. The sample was predominantly White (95.2%) and predominantly female (70.3%). Age ranged from 18 to 78, with the average age 53.3 years (SD = 12.3).
A second sample was collected using Amazon’s Mechanical Turk (MT) to get an idea of the response more representative of the general public. MT is an online marketplace where people are able to complete short tasks and surveys for compensation. MT tends to yield a more representative sample when compared to convenience sampling (Levay et al. 2016). Participants had to live in the United States as a condition of eligibility. These participants were paid $0.50 for completing the survey. The sample size was filtered to 428 responses from the MT site. Race was much closer to the U.S. population than the FHW sample. Individuals identifying as White comprised 77.1% of the sample (76.7% U.S.). This was followed by 8.2% identifying as African American or Black (13.4% U.S.), 7.2% Asian (5.8% U.S.), 4.9% Hispanic or Latino (18.1% U.S.), and 2.6% other (U.S. Census Bureau 2018). The sample was more evenly divided by gender (58.9% female) and closer to the U.S. population (50.8% female, 49.2% male). Age ranged from 18 to 71, with the average age 37.8 years (SD = 12.8).
Results
Weather sources and obtainment frequency.
Before examining the hypotheses for this study, it is useful to examine where the participants obtained their weather information and how frequently they obtained weather information. First, participants indicated their frequency of obtaining weather forecasts with a question previously utilized in surveys by Sherman-Morris and Brown (2012) and Sherman-Morris et al. (2020). The participants were able to select from the following choices to indicate how often they typically watch, hear or read a weather forecast: multiple times per day, once per day, multiple times per week but not daily, multiple times per month or never. The majority of FHW participants (73.9%) cited seeking weather forecasts multiple times per day followed by 14.7% seeking weather forecasts once per day. The MT participants did not cite seeking weather forecasts as frequently (see Fig. 1). Over one-third (37.9%) of the MT participants cited receiving weather information once per day followed by multiple times per day (24.6%) and multiple times per week but not daily (23%). The difference between the two samples was statistically significant (p ≤ 0.001) using a chi-square test.
Weather forecast sources (MT and FHW).
Citation: Bulletin of the American Meteorological Society 101, 7; 10.1175/BAMS-D-19-0081.1
Participants were next asked to identify their primary weather sources with a multiple-choice question in which they could select more than one response. The sources on the multiple-choice question included the Weather Channel TV, the Weather Channel app, the Weather Channel website, the National Weather Service, AccuWeather app, AccuWeather website, local television news, Facebook, Twitter, NOAA Weather Radio, Internet search (smartphone/tablet), Internet search (desktop/laptop), smartphone’s default weather app, and/or other. The participants had the ability to select multiple sources. Many participants relied on traditional outlets for weather information. Of the 907 responses, close to half (45.9%) of the participants cited local television news as their primary source of weather information (Fig. 2). The growing presence of social media was evident in the responses. After local television news, the next two most cited sources of weather information were Facebook (37%) and the Weather Channel mobile application (35.3%). There were some significant differences between the two samples, however, as determined by a chi-square test. Participants in the MT sample were more likely to report using their smartphone’s default weather app, an Internet search using their desktop/laptop, the Weather Channel website (all significant at p ≤ 0.001), and the Weather Channel TV (p = 0.004). Participants in the FHW sample were more likely to use the National Weather Service website, local television news, Facebook, and NOAA Weather Radio (all significant at p ≤ 0.001).
Frequency in obtaining weather forecasts (MT and FHW).
Citation: Bulletin of the American Meteorological Society 101, 7; 10.1175/BAMS-D-19-0081.1
Self-perceived weather knowledge.
Average self-perceived knowledge was highest for questions about the tornado watch, tornado warning, and the difference between wintry precipitation types (see Table 2). Over half (58.4%) strongly agreed they knew the meaning of a tornado watch; 60.9% strongly agreed they knew the meaning of a tornado warning, and 61% strongly agreed they knew the difference between wintry precipitation types. Cronbach’s alpha (0.912) showed it was justifiable to analyze “self-perceived weather knowledge scores” as a single variable. The responses were averaged in SPSS, which generated a “self-perceived knowledge score” variable that had a possible point value ranging from 1 to 7, where 1 indicated low weather knowledge and 7 indicated high weather knowledge. The mean score for self-perceived knowledge was 5.95 for the FHW sample and 4.82 for MT.
Average self-perceived weather knowledge. Significance determined with Mann–Whitney U and Kruskal–Wallis H tests.
A Mann–Whitney U test was used to test whether significant differences existed based on sample (FHW or MT) and gender. Whether the participant was from the MT or FHW sample (p ≤ 0.001) and gender for the FHW sample (p ≤ 0.001) were significant. FHW participants believed they had higher self-perceived weather knowledge overall as well as for each individual topic. Males in the FHW sample believed they had higher self-perceived weather knowledge than females in the FHW sample. No significance (p = 0.211) was detected in the MT sample for gender.
The Kruskal–Wallis H test was used to test for differences based on age and frequency of obtaining a weather forecast. This test was selected because these variables consisted of more than two groups. Age did not lead to significant differences in either sample (p = 0.377 MT; 0.771 FHW). Figure 3 shows the variation in perceived weather knowledge based on how frequently one obtains a weather forecast. The Kruskal–Wallis H test showed significant differences in self-perceived knowledge based on forecast obtainment frequency (p ≤ 0.001) for both the MT and FHW samples. A Mann–Whitney U test was utilized to examine the significance of differences between pairs of forecast obtainment categories. In both samples, participants who cited checking the forecast more frequently had higher self-perceived weather knowledge scores. Checking once per day or more led to significant differences among the MT sample, but the greatest level of difference among the FHW sample was found between those who obtained a forecast multiple times per day versus those obtaining a forecast less frequently.
Self-perceived weather knowledge vs assessed weather knowledge.
Citation: Bulletin of the American Meteorological Society 101, 7; 10.1175/BAMS-D-19-0081.1
Assessed weather knowledge.
Following the perceived knowledge questions, participants were asked to complete questions that measured their actual weather knowledge. In SPSS, the responses were graded and analyzed. If the participant answered the question correctly, the participant received a 1. If the participants did not answer the question correctly, the participant received a 0. Participants could receive a total of 12 points for answering all questions correctly. The questions answered correctly by the most participants in both samples were the Saffir–Simpson scale question and the hurricane season question (Fig. 4). The participant’s scores were summed, which generated a “assessed knowledge score” that had a possible point value ranging from 0 to 12, where 0 indicated no weather questions were answered correctly and 12 indicated all weather questions were answered correctly. Cronbach’s alpha (0.859) showed it was justifiable to treat “assessed weather knowledge scores” as one variable. The mean score for assessed weather knowledge for the FHW sample was 6.35 and 3.96 for MT.
Assessed weather knowledge compared to self-perceived weather knowledge.
Citation: Bulletin of the American Meteorological Society 101, 7; 10.1175/BAMS-D-19-0081.1
Whether the participant was grouped in the FHW or MT sample, gender, age, and weather forecast obtainment frequency were tested for significant differences. FHW participants had higher assessed weather knowledge than MT participants (p ≤ 0.001). Gender was significant (p ≤ 0.001) in the FHW sample with males performing better, and older age groups in the MT sample scored higher (p ≤ 0.001). Gender was not significant in the MT sample (p = 0.558), nor was age in the FHW sample (p = 0.238).
Figure 5 shows how assessed knowledge varies with frequency of obtaining a weather forecast. A Kruskal–Wallis H test showed significant differences in assessed knowledge based on forecast obtainment frequency (p ≤ 0.001) for both samples. Participants who cited checking the forecast more frequently had higher assessed weather knowledge scores. As with perceived knowledge, a Mann–Whitney U test indicated significant differences in assessed knowledge between pairs of forecast obtainment categories. For the MT sample, significant differences were detected when one obtained a forecast at least multiple times per week versus less frequently. For the FHW sample, checking the forecast multiple times per day led to significant differences over those who obtained a forecast in each less frequent category. However, assessed knowledge among participants who obtained a forecast less often such as once per day or multiple times per week was only significantly different from those who never obtained the forecast.
Forecast obtainment frequency vs assessed knowledge means plot.
Citation: Bulletin of the American Meteorological Society 101, 7; 10.1175/BAMS-D-19-0081.1
Self-perceived versus assessed scores.
Spearman’s rho showed a strong positive correlation (rs = 0.522, p ≤ 0.001) between self-perceived weather knowledge and assessed weather knowledge. As self-perceived weather knowledge increased, assessed weather knowledge increased. To evaluate for evidence of knowledge overestimation, the self-perceived weather knowledge and assessed weather knowledge scores were standardized so that they could be compared directly. This was necessary because they were measured using different scales. As is common in DKE studies (e.g., Dunning 2011), the scores were examined in terms of the quartile of participants’ performance on the objective assessed knowledge questions (Fig. 6). Participants’ standardized assessed knowledge scores for each sample were ranked separately and then divided among four equal categories (bottom, third, second, and top). A mean score was calculated for each quartile grouping. The same procedure was followed for the standardized perceived knowledge scores. This method differs from many DKE studies, which plot perceived performance on the specific set of knowledge questions by the quartile of the objective performance. In the current study, perceived knowledge was more like a participant’s prediction of how well they would expect to do rather than how well participants thought they did or how well they thought they did compared to others. We believe this can be a strength in the applicability of our results, because our perceived knowledge is more general, and not based on specific test questions.
Self-perceived and assessed standardized scores by performance quartile on the assessed measure.
Citation: Bulletin of the American Meteorological Society 101, 7; 10.1175/BAMS-D-19-0081.1
As can be seen in Fig. 6, the average standardized perceived knowledge scores are higher than the average standardized assessed knowledge score for participants in the bottom quartile. That is, participants with assessed scores in the bottom quartiles of both samples showed the highest level of knowledge overestimation. On the other end, participants in the top quartile in terms of assessed knowledge underestimated their knowledge. A Kruskal–Wallis H test on the MT sample (p ≤ 0.001) showed that perceived knowledge scores were significantly different across quartiles with differences found between all but quartile 1 to 2 and quartile 3 to 4. Assessed scores were different among all quartile pairs as well as overall (p ≤ 0.001). The difference between perceived and assessed knowledge was significant overall (p ≤ 0.001) as well as between all but the second and third quartiles. Kruskal–Wallis H tests on the same variables in the FHW sample also showed significant differences among the quartiles (p ≤ 0.001), though the specific pairwise comparisons yielded some different results. Perceived knowledge scores were significantly different across quartiles with differences found between all but quartile 2 to 3. Assessed scores and the difference in scores between perceived and assessed were different overall (p ≤ 0.001), and among all quartile pairs. Significance levels were adjusted by the Bonferroni correction due to the number of pairs.
Discussion and conclusions
The results of this study indicated support for each of the hypotheses with only partial support for a gender influence on self-perceived weather knowledge (Table 3). The key takeaway from this study is that participants who scored low in assessed weather knowledge weather tend to overestimate their weather knowledge the most, which is strong evidence of the DKE as suggested in H1. Our results identified patterns of DKE similar to those found in previous studies despite the differences in methodology described above. Participants in the lowest quartile overestimated their knowledge. The results of this research are supported by previous literature, which showed people who have the least objective knowledge about a subject tend to overestimate their knowledge and underestimate their limitations (Dunning et al. 2004; Alicke and Govorun 2005). We also found that perceived and assessed weather knowledge were fairly close among those participants in the middle two quartiles. At least among the FHW sample, perceived weather knowledge was also fairly consistent between the second and third quartiles. DKE studies are not consistent with respect to how these middle groups rate their knowledge (e.g., Kruger and Dunning 1999), although the second quartile in our sample seemed to have less difference between perceived and actual scores than some of these other DKE studies (Kruger and Dunning 1999; Dunning 2011). It is not clear whether this difference might be based on methodological decisions, or if the ubiquity of weather information might play a role. Because of the significance of obtaining a weather forecast multiple times per day on differences in both assessed and perceived weather knowledge, it is likely that exposure plays some role. However, a future study is necessary to further explain that role.
Hypotheses.
Perceived weather knowledge was high overall with only two topics yielding average scores below the midpoint value of 4 (and only in the MT sample). These two topics were statements about the Saffir–Simpson scale and enhanced Fujita scale. A majority of the FHW and MT participants cited high self-perceived knowledge of a tornado watch, a tornado warning, and the difference between wintry precipitation types. More than three-quarters of FHW participants strongly agreed they knew the meaning of a tornado watch and a tornado warning. More than three-quarters of FHW participants also strongly agreed they knew the difference between snow, sleet, and freezing rain. Only about one-third of MT participants strongly agreed they knew the meaning of a tornado watch and 42% of MT participants strongly agreed they knew the meaning of a tornado warning. Just under half of MT participants strongly agreed they knew the difference between snow, sleet, and freezing rain.
It was hypothesized that users of specialty weather websites (FHW participants) would test higher on the self-perceived weather knowledge (H2) and assessed weather knowledge (H3) sections of the survey. Hypotheses 2 and 3 were supported in that the FHW participants cited higher self-perceived weather knowledge and received higher assessed weather knowledge scores. The reason behind these results is not known and should be a focus in future studies. It is possible that people who are weather savvy utilize weather specialty websites for more technical weather discussions, but it is also possible that people become more weather knowledgeable over time through use of weather specialty websites. This study also showed participants who cited checking the weather forecast more frequently had higher assessed weather knowledge scores as hypothesized (H4).
The final hypothesis (H5) suggested women would have lower self-perceived weather knowledge. This hypothesis was only partially supported in this study. Females in the FHW sample showed lower self-perceived weather knowledge when compared to males. Females in the MT sample did rate their self-perceived weather knowledge lower than males, but the difference was not significant. In other research, males expressed greater confidence in their knowledge of tornado warnings (Ripberger et al. 2019). A potential explanation for this may be due to the fact that males also performed better on the assessed knowledge questions in our samples, where typically gender does not have a significant influence. Additionally, male participants scored significantly higher in the FHW sample for assessed weather knowledge.
This study also showed that, while television is still an important source of weather information, it is continuing to decrease in popularity as new sources of information are introduced. This study showed around half of participants used local television as a weather source while less than a decade ago more than two-thirds of participants in a Lazo et al. (2009) study used local television as a weather source. Future work should examine how much of a role new sources of weather information play in one’s perceived weather knowledge. Smartphones make weather forecasts instantly accessible, but many smartphone applications do not provide the same level of context and explanation as traditional weather media such as the local weathercast. Thus, the “easiness effect of science popularization” mentioned earlier (Scharrer et al. 2016) could be exaggerated by the delivery of weather information via smartphone, depending on the nature of the information obtained.
A limitation of this study was utilizing only FHW as a weather specialty page. Ideally, multiple specialty weather pages would have been utilized to evaluate if the audience of each page has more weather knowledge. Also, interviews would have been helpful to gain additional insight on the reasons why FHW users are more knowledgeable. It is not known if FHW users are more knowledgeable because 1) the content on the FHW website adds to their knowledge base or 2) if the FHW website attracts more weather literate or weather-salient people. Weather salience has previously been linked with weather knowledge through increased attention to weather and weather information (Stewart 2009). Future research could incorporate an audience from another specialty weather page to see if that audience is more knowledgeable and try to determine why that audience is more or less weather literate or weather salient.
Other studies showed a wide range in comprehension of weather information. Powell and O’Hair (2008) showed 58% of participants correctly identified the difference between a watch and a warning and 36% were completely incorrect. The number was higher in other studies. For example, 80% could correctly define a tornado warning in Sherman-Morris (2010). The success rate of this current study is lower than previous studies and it is possible the low success rates of properly identifying a tornado watch and tornado warning was due to the way the question was asked and scored. If participants selected one of the criteria wrong, their response was coded as incorrect. This study was also open to the whole United States, so some questions may have been less relevant to locations that do not see certain natural hazards. There is some evidence that participants from the West region may not have scored as well on the questions as other regions, which is consistent with other studies (Ripberger et al. 2019; Powell and O’Hair 2008). Future studies on weather knowledge should consider the weather experienced by the participants. It should also be noted that participants were encouraged to not utilize any resources other than their knowledge to answer the weather questions on the assessed knowledge part of the survey, but it is possible some participants tried to identify answers utilizing outside resources.
The key takeaway in this study, the overestimation of weather knowledge, could have big implications during a natural disaster if their overestimation of weather knowledge creates a false sense of safety. Siems (2016) showed DKE tendencies were evident when analyzing evacuation response patterns during Hurricane Katrina with about three-quarters of non-evacuees showing signs of overestimating their knowledge. An important consideration for a future study is whether one who overestimates his or her weather knowledge may also be overconfident in his or her ability to make important decisions during an extreme weather event. Having found evidence of overestimation of weather knowledge by those with the lowest assessed knowledge, the authors plan to examine whether this overestimation has any influence on decision-making in a weather context in a future paper. Also, this research identified some areas where the public is less knowledgeable. For example, both MT and FHW samples scored low on questions about when tornadoes are likely and what conditions cause them. While not understanding meteorological conditions responsible for a tornado may not influence safety decisions, a lack of awareness of tornado season could cause inaccurate risk perceptions. More understanding of this could help improve protective decision-making. However, as demonstrated by Paton et al. (2008), simply providing information will not necessarily improve understanding or lead to safer decisions; other efforts may be needed to help individuals recognize their gaps in knowledge. Weather communicators should also consider existing gaps when designing weather messages.
References
Alicke, M. D., and O. Govorun, 2005: The better-than-average effect. The Self in Social Judgment, M. D. Alicke, D. A. Dunning, and J. I. Krueger, Eds., Studies in Self and Identity Series, Psychology Press, 85–106.
Aronson, J., and M. Inzlicht, 2004: The ups and downs of attributional ambiguity: Stereotype vulnerability and the academic self-knowledge of African American college students. Psychol. Sci., 15, 829–836, https://doi.org/10.1111/j.0956-7976.2004.00763.x.
Bell, P., and D. Volckmann, 2011: Knowledge surveys in general chemistry: Confidence, overconfidence, and performance. J. Chem. Educ., 88, 1469–1476, https://doi.org/10.1021/ed100328c.
Broad, K., A. Leiserowitz, J. Weinkle, and M. Steketee, 2007: Misinterpretations of the “cone of uncertainty” in Florida during the 2004 hurricane season. Bull. Amer. Meteor. Soc., 88, 651–668, https://doi.org/10.1175/BAMS-88-5-651.
Bulkeley, H., 2000a: Discourse coalitions and the Australian climate change policy network. Environ. Plann., 18C, 727–748, https://doi.org/10.1068/c9905j.
Bulkeley, H., 2000b: Common knowledge? Public understanding of climate change in Newcastle, Australia. Public Understanding Sci ., 9, 313–333, https://doi.org/10.1177/096366250000900301.
Drake, L., 2012: Scientific prerequisites to comprehension of the tropical cyclone forecast: Intensity, track, and size. Wea. Forecasting, 27, 462–472, https://doi.org/10.1175/WAF-D-11-00041.1.
Dunning, D., 2011: The Dunning–Kruger effect: On being ignorant of one’s own ignorance. Advances in Experimental Social Psychology, Vol. 44, J. M. Olson and M. P. Zanna, Eds., Academic Press, 247–296, https://doi.org/10.1016/B978-0-12-385522-0.00005-6.
Dunning, D., C. Heath, and J. M. Suls, 2004: Flawed self-assessment: Implications for health, education, and the workplace. Psychol. Sci. Public Interest, 5, 69–106, https://doi.org/10.1111/j.1529-1006.2004.00018.x.
Edwards, R. K., K. R. Kellner, C. L. Sistrom, and E. J. Magyari, 2003: Medical students self-assessment of performance on an obstetrics and gynecology clerkship. Amer. J. Obstet. Gynecol., 188, 1078–1082, https://doi.org/10.1067/mob.2003.249.
Ehrlinger, J., K. Johnson, M. Banner, D. Dunning, and J. Kruger, 2008: Why the unskilled are unaware: Further explorations of (absent) self-insight among the incompetent. Organ. Behav. Hum. Decis. Processes, 105, 98–121, https://doi.org/10.1016/j.obhdp.2007.05.002.
Gross, M., and D. Latham, 2012: What’s skill got to do with it? Information literacy skills and self-views of ability among first-year college students. J. Amer. Soc. Inf. Sci. Technol., 63, 574–583, https://doi.org/10.1002/ASI.21681.
Grunspan, D. Z., S. L. Eddy, S. E. Brownell, B. L. Wiggins, A. J. Crowe, and S. M. Goodreau, 2016: Males under-estimate academic performance of their female peers in undergraduate biology classrooms. PLOS ONE, 11, e0148405, https://doi.org/10.1371/journal.pone.0148405.
Gutter, B .F., K. Sherman-Morris, and M. E. Brown, 2018: Severe weather watches and risk perception in a hypothetical decision experiment. Wea. Climate Soc., 10, 613–623, https://doi.org/10.1175/WCAS-D-18-0001.1.
Insurance Journal, 2010: New Orleans hospital settles post-Katrina wrongful death lawsuit. 28 January, accessed 1 May 2018, www.insurancejournal.com/news/%20southcentral/2010/01/28/106964.htm.
Jaramillo, F., F. A. Carrillat, and W. B. Locander, 2003: Starting to solve the method puzzle in salesperson self-report evaluations. J. Pers. Sell. Sales Manage., 23, 369–377.
Kruger, J., and D. Dunning, 1999: Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J. Pers. Soc. Psychol., 77, 1121–1134, https://doi.org/10.1037/0022-3514.77.6.1121.
Lazo, J. K., R. E. Morss, and J. L. Demuth, 2009: 300 billion served: Sources, perceptions, uses, and values of weather forecasts. Bull. Amer. Meteor. Soc., 90, 785–798, https://doi.org/10.1175/2008BAMS2604.1.
Levay, K. E., J. Freese, and J. N. Druckman, 2016: The demographic and political composition of Mechanical Turk samples. SAGE Open, 6, 1–17, https://doi.org/10.1177/2158244016636433.
Lindell, M. K., C. S. Prater, R. Perry, and C. N. William, 2006: Fundamentals of Emergency Management .FEMA, 485 pp.
Lindsey, B. A., and M. L. Nagel, 2015: Do students know what they know? Exploring the accuracy of students’ self-assessments. Phys. Rev. Spec. Top. Phys. Educ. Res., 11, 1–11, https://doi.org/10.1103/PHYSREVSTPER.11.020103.
McCright, A., 2010: The effects of gender on climate change knowledge and concern in the American public. Popul. Environ., 32, 66–87, https://doi.org/10.1007/s11111-010-0113-1.
Mitchem, J. D., 2003: An analysis of the September 20, 2002, Indianapolis Tornado: Public response to a tornado warning and damage assessment difficulties. Quick Response Research Rep. 161, 55 pp.
Nuhfer, E., and Knipp, D., 2003: The knowledge survey: A tool for all reasons. Improve Acad ., 21, 59–78, https://doi.org/10.1002/j.2334-4822.2003.tb00381.x.
NWS, 2006: Service assessment for Hurricane Charley, August 9-15, 2004. NOAA/NWS Service Assessment, 29 pp., www.weather.gov/media/publications/assessments/Charley06.pdf.
Paton, D., L. Smith, M. Daly, and D. Johnston, 2008: Risk perception and volcanic hazard mitigation: Individual and social perspectives. J. Volcanol. Geotherm. Res., 172, 179–188, https://doi.org/10.1016/j.jvolgeores.2007.12.026.
Powell, S. W., and H. D. O’Hair, 2008: Communicating weather information to the public: People’s reactions and understandings of weather information and terminology. Third Symp. on Policy and Socio-Economic Research, New Orleans, LA, Amer. Meteor. Soc., P1.3, https://ams.confex.com/ams/88Annual/webprogram/Paper132939.html.
Ripberger, J. T., M. J. Krocak, W. W. Wehde, J. N. Allan, C. Silva, and H. Jenkins-Smith, 2019: Measuring tornado warning reception, comprehension, and response in the United States. Wea. Climate Soc., 11, 863–880, https://doi.org/10.1175/WCAS-D-19-0015.1.
Scharrer, L., Y. Rupieper, M. Stadtler, and R. Bromme, 2016: When science becomes too easy: Science popularization inclines laypeople to underrate their dependence on experts. Public Underst. Sci., 26, 1003–1018, https://doi.org/10.1177/0963662516680311.
Selm, K. R., M. N. Peterson, G. Hess, S. M. Beck, and M. R. McHale, 2019: Educational attainment predicts negative perceptions women have of their own climate change knowledge. PLOS ONE, 14, e0210149, https://doi.org/10.1371/journal.pone.0210149.
Sharp, A., and S. Hoj, 2010: Assessing the public’s real knowledge of global warming. Australian and New Zealand Marketing Academy Conf ., Sydney, Australia, University of Western Australia, 7 pp., www.researchgate.net/profile/Anne_Sharp/publication/267246374_Assessing_the_public’s_real_knowledge_of_global_warming/links/54618f100cf2c1a63bff937f.pdf.
Shepherd, M., 2016: When does skepticism become bias in science? Forbes.com, 1 November, accessed 20 April 2018, www.forbes.com/sites/marshallshepherd/2016/11/01/when-does-skepticism-become-bias-in-science/#30a7e74e34ef.
Sherman-Morris, K., 2010: Tornado warning dissemination and response at a university campus. Nat. Hazards, 52, 623–638, https://doi.org/10.1007/s11069-009-9405-0.
Sherman-Morris, K., and M. E. Brown, 2012: Experiences of Smithville, Mississippi residents with the 27 April 2011 Tornado. Natl. Wea. Dig., 36, 93–101.
Sherman-Morris, K., and K. B. Antonelli, 2018: Hurricane knowledge and interpretation of forecasted error cone and wind potential graphics. J. Emerg. Manag., 16, 137–148, https://doi.org/10.5055/jem.2018.0363.
Sherman-Morris, K., P. S. Poe, C. Nunley, and J. A. Morris, 2020: Perceived risk, protective actions and the parasocial relationship with the local weathercaster: A case study of Hurricane Irma. Southeast. Geogr., 60, 27–47, https://doi.org/10.1353/sgo.2020.0003.
Siems, J. W., 2016: Disaster threat and the Dunning-Kruger effect. M.A. thesis, Center for Homeland Defense and Security, Naval Postgraduate School, 90 pp., www.hsdl.org/?view&did=798825.
Silver, A., 2015: Watch or warning? Perceptions, preferences, and usage of forecast information by members of the Canadian public. Meteor. Appl., 22, 248–255, https://doi.org/10.1002/met.1452.
Steele, C. M., and J. Aronson, 1995: Stereotype threat and the intellectual test performance of African Americans. J. Pers. Soc. Psychol., 69, 797–811, https://doi.org/10.1037/0022-3514.69.5.797.
Stewart, A. E., 2009: Minding the weather: The measurement of weather salience. Bull. Amer. Meteor. Soc., 90, 1833–1842, https://doi.org/10.1175/2009BAMS2794.1.
Stoutenborough, J. W., X. Liu, and A. Vedlitz, 2014: Trends in public attitudes toward climate change: The influence of the economy and climategare on risk, information, and public policy. Risk Hazards Crisis Public Policy, 5, 22–37, https://doi.org/10.1002/RHC3.12045.
U.S. Census Bureau, 2018: QuickFacts demographics table. Accessed 1 November 2018, www.census.gov/quickfacts/fact/table/US/PST045218.
Violato, C., and J. Lockyer, 2006: Self and peer assessment of pediatricians, psychiatrists and medicine specialists: Implications for self-directed learning. Adv. Health Sci. Educ., 11, 235–244, https://doi.org/10.1007/s10459-005-5639-0.
Waylen, A. E., M. S. Horswill, J. L. Alexander, and F. P. McKenna, 2004: Do expert drivers have a reduced illusion of superiority? Transp. Res. Part F, 7, 323–331, https://doi.org/10.1016/J.TRF.2004.09.009.
Zabini, F., 2016: Mobile weather apps or the illusion of certainty. Meteor. Appl., 23, 663–670, https://doi.org/10.1002/met.1589.
