Workshop measures.

We assessed our achievement of project goals both during and after the workshops. At the start of the workshop, the participants were asked to complete a preworkshop assessment packet, which included questions regarding their attitudes and beliefs about the teaching of the curriculum [the Weather Safety Education questionnaire (WSEQ)] and also a set of items intended to assess the teachers’ orientation to the workshop training goals. The participants completed a goal-attainment scaling procedure in which they articulated up to three of their own goals for the workshop and then rated the degree of importance for achieving each goal (Kiresuk et al. 1994). The next pair of measures assessed the teacher’s efficacy-related beliefs about providing instruction in science generally. Self-efficacy is an essential element in teachers both believing that they can provide instruction and that the instruction will lead to successful student learning outcomes (Bandura 1977, 1997). The teachers completed the Science Teaching Efficacy Belief Inventory (STEBI; 25 items; Riggs and Enochs 1990) and an adaptation of the STEBI that was specific to providing instruction in weather science and safety, Weather Science and Safety Teaching Efficacy Inventory (WSSTEI; 20 items) that was created by the first author. The teachers also completed a Weather Science and Safety Content Test (WSSCT) to assess their amount of available knowledge before beginning the workshop week. The WSSCT consisted of 22 multiple choice, true–false, and short-answer items. Approximately one-half of the measure was devoted to weather science, while the other half assessed content pertaining to weather safety practices. The workshop faculty developed an initial pool of items (nearly 50 in all) and then placed half of the items on the WSSCT as a pretest. We used the remaining half of the items as a posttest, which was a parallel form of the WSSCT pretest measure. We assessed the effects of the workshop in a pre- and posttest manner, with project measures administered on the first and last days of the workshop. Additional details on the development of these measures, their psychometric properties, and their use with teachers in earlier workshops are available in Stewart et al. (2015).

Postworkshop teaching and learning measures.

We also monitored teachers’ use of the curriculum and what students learned from it, as well as the parents’ awareness of the lessons during the subsequent school year. We observed a subsample of nine teachers (14% of all participants) deliver different MoD lessons to their classes. In addition, a subset of seven teachers administered postlesson content tests to 291 students. The teachers who volunteered for in-class observations and for the postlesson content tests were those who had been using the MoD lessons and who had the scheduling availability to accommodate the authors when we made the request. To assess student learning about weather science and safety, we developed 12 content tests: one for each of the MoD lesson areas of lightning, tornadoes, hurricanes, and floods and one for each of the three grade levels (K–2, 3–5, and 6–8). The content tests contained a mix of weather science and safety concepts operationalized in 10–19 objective and open-ended questions that were drawn directly from the MoD curriculum. The tests were designed to be general and comprehensive with respect to their coverage of MoD curriculum content.

Follow-up with teachers and parents.

Next, we followed up with the workshop teachers through an online Use and Dissemination questionnaire; 32 teachers (48%) responded to this instrument. Finally, we used our Home Connection survey with a subsample of 133 parents whose children had been taught from the MoD curriculum to assess their awareness and knowledge of weather science and safety for lightning, tornadoes, hurricanes, and floods. The design of the project and the use of all project measures were reviewed and approved by the University of Georgia Institutional Review Board. We report on the attainment of the project goals as revealed by the use of these measures below.

Goal 1: Enhance teacher attitudes and build interest in the MoD curriculum.

The teachers in our workshops, via the Weather Safety Education questionnaire, estimated that a little over 25% of the students that they taught were adequately prepared to deal with hazardous weather. The teachers reported spending an average of about 15.5 h yr⁻¹ providing instruction about general weather concepts. Of this time, slightly over one-third of the instructional time was devoted to topics pertinent to weather safety. Although 11 (16.7%) of the teachers believed it was important to instruct students about the weather hazards unique to their geographic area, a majority (n = 54, 81.8%) believed it was important to provide both instruction about weather hazards in general and also about those unique to their locale. None of the workshop participants had either known about or used the MoD curriculum previously. Admittedly, because teachers could self-select to attend the workshop, interest in the training and the MoD curriculum was high to begin with.

Goal 2: Increase teacher knowledge about the MoD weather science and safety curriculum.

Although the teachers brought a great deal of interest in weather to the workshop, they did not generally possess much knowledge about the science or safety of lightning, tornadoes, hurricanes, or floods. Their preworkshop mean score on the WSSCT was 8.00 out of 22 items [standard deviation (SD) = 2.62]. Their scores nearly doubled (mean M = 14.35, SD = 2.63) on a parallel form of this test that we gave at the end of the workshop week. The teachers answered significantly more items correctly at the end of the workshop (using a parallel form of the WSSCT) compared to their performance at the beginning of the workshop: F (1, 65) = 20.57, p < 0.0001. Although attending the workshop was associated with answering an additional six questions correctly, the mean posttest score (65% correct) indicated that there was still additional science and safety content that the teachers had not fully understood or mastered. A content analysis of the teachers’ posttest responses revealed that the knowledge about tornado science and safety was significantly greater (M = 75% correct; 95% confidence interval: 71%–80%) than their knowledge in the areas of lightning, floods, and hurricanes (with means ranging from 62% to 65% correct). These results were consistent with our observations during the workshops that the teachers showed much interest in tornado science and safety. Further, one of the counties (Bibb) had recently been affected by tornadoes, which may have built extra interest in the topic.

Goal 3: Increase and evaluate the usage of the MoD curriculum by teachers in the participating districts.

Use of the curriculum.

Regarding the actual use of the MoD curriculum during the following school year, 32 teachers delivered 178 MoD lessons to a total of 2,465 students in K–8. In Table 1, we summarize the number and type of MoD lessons that teachers used with their students along with their degree of usage of the MoD curriculum. Overall, the teachers reported using the MoD curriculum “somewhat” to “much” in educating their students about weather science and safety. Table 2 shows that with the exception of the disaster preparation (Be Disaster Safe; M = 4.40) lesson for grade 6, teachers’ reported that their use of the MoD generally does not rise above a rating of 4.0 (corresponding to much use).

Table 1.

Amount of MoD lesson use by grade K–2, 3–5, and 6–8 teachers who responded to the Dissemination and Use questionnaire. The survey respondents indicated their extent of MoD lesson use via a five-point rating scale: 1 = not at all, 2 = a little, 3 = somewhat, 4 = much, and 5 = completely. Values with an asterisk represent the mean of means for the extent of MoD lesson use. Values in parentheses are percentages of the total numbers.

View Table

Table 2.

Summary of students’ performances on postlesson MoD content tests.

View Table

Goal 4: Disseminate the MoD curriculum to teachers who did not participate in training.

The 32 teachers who responded to the Use and Dissemination questionnaire indicated that they discussed the curriculum with a total of 203 fellow teachers. The number of teachers with whom the workshop participants discussed the curriculum ranged from 2 to 20 teachers: M = 6.55, SD = 4.22. There were 25 (78.1% of those who responded) teachers who gave all of their extra MoD kits to other teachers. Two teachers indicated that they could use additional kits to give to other teachers. We asked the teachers to indicate the frequency with which they discussed MoD with their colleagues. The outbreak of severe weather at or near the teachers’ schools represented the occasion where teachers most often reported discussing the curriculum (n = 20, 62.5%). Learning about severe weather outbreaks that occurred somewhere else (beyond their locale) was the second most frequent occasion for discussing the MoD curriculum (n = 16, 50.0%). A weather safety or awareness week (e.g., tornado safety awareness week) was the third most frequent occasion for discussing the MoD (n = 13, 40.6%). The teachers reported that their colleagues taught the MoD to an estimated 3,096 students (M = 123.8 students, median = 102 students).

A final item in this section of the questionnaire asked the teachers about the likelihood of recommending the Weather Science and Safety workshop to a teacher colleague if it were to be offered in the future. The teachers used a five-point rating scale (from 1 = very likely to recommend to 5 = very unlikely to recommend). Overall, the teachers were favorably disposed to recommending the workshop to their colleagues: M = 1.47, SD = 0.56. The mean value fell between the “very likely to recommend” and “likely to recommend” anchors of the rating scale.

Goal 5: Evaluate students’ weather science and safety knowledge.

Seven of the 66 workshop teachers agreed to administer the postlesson tests to their classes following the coverage of one or more MoD lessons for their grade level. Five of the seven teachers were the same ones who allowed us to observe them during the MoD lesson delivery. The four MoD curriculum areas of lightning, tornadoes, hurricanes, and floods were all covered although generally by differing grade levels. That is, all three grade categories (K–2, 3–5, and 6–8) did not take the content test from the four curricular areas because we did not have a sufficient number of teachers from the different grades to deliver lessons from the four areas. We provide summary information about the lessons, class sizes, and the outcomes of the postlesson tests in Table 2.

Overall, on the 291 postlesson assessments that students completed, the mean proportion of correct responses was approximately 60%. Although this percentage ostensibly appears low in magnitude, several considerations can put the students’ achievement on the postlesson tests into perspective. First, teachers often taught one science and one safety lesson from the respective content areas (lightning, tornadoes, hurricanes, and floods). As we indicated previously, however, we designed the tests to be general and comprehensive. Therefore, when students completed the postlesson measure, they may not have received some of the instruction that would have been necessary to provide correct responses to the measure.

We examined the students’ performances on the postlesson tests for lightning and tornadoes, given that the majority of teachers delivered instruction on these MoD topics. In the K–2 test for lightning, the students most frequently missed an item covered in the first lesson that involved separating electrical charge by shuffling their shoes over carpet and then discharging the accumulated charge by touching an object (69% correct responses). The second item that K–2 students missed frequently pertained to lightning safety (also 69% correct). This item involved making an estimate of the distance of lightning based upon an elapsed amount of time between the lightning flash and the thunder. This content was associated with the second MoD K–2 lightning lesson. In the grades 3–5 postlesson lightning test, the students most frequently missed a question about how a thunderstorm cloud is like a capacitor (i.e., it stores electrical charge) that appeared in the first lesson (36% correct). They also frequently missed (45% correct) all or parts of a second-lesson question that asked them to label a diagram to accurately indicate 1) typical locations of positive charges and 2) locations of negative charges, a stepped leader, a return stroke, and cloud-to-ground lightning. Qualitatively, students generally were accurate in locating negative charges at the base of the cloud and positive charges at the ground; they made more errors in identifying the stepped leader. Finally, the students frequently missed a safety-related item from the fourth lesson that asked (true or false) whether it was safe to touch a person who had been struck by lightning (27% correct).

With respect to the tornado postlesson test, students regularly missed a first-lesson item that asked them to name the rotation inside of a thunderstorm that can lead to a tornado (11% correct). Rather than selecting the correct answer (i.e., mesocyclone), students tended to respond with “funnel cloud” or “tornado.” The students also frequently missed a question from the second MoD tornado lesson that involved the sources of moisture that can contribute to favorable conditions for tornadoes (17% correct). They were asked to indicate why the Atlantic Ocean was not a good source for moisture in the Midwest “Tornado Alley.” The remaining item, also from the second lesson, asked the students to name the collision of advancing cold air with a warm air mass (i.e., a cold front, 16% correct).

Beyond these item-level patterns, the students’ responses generally revealed that they understood and were able to reflect some of the principal safety facts related to the type of weather lesson that they received. Consequently, we were encouraged that the students had achieved some degree of learning and that the specific safety content embodied in the MoD curriculum was reflected in their postlesson test performances. Although this weather safety specificity part of the postlesson tests supported that students had learned from what their teachers delivered from the MoD curriculum, it was not possible to determine further how much of the students’ weather science–related performances was due to their exposure to the MoD curriculum and what they might have learned from other sources or in earlier grades.

Goal 6: Evaluate the weather safety behavior of students and families.

We evaluated this goal from the perspective of the students’ teachers and from a survey of parents whose children were exposed to the MoD curriculum. Regarding the teachers’ perspectives, the final section of the Use and Dissemination questionnaire asked the teachers to provide an estimate of the extent to which they believed that their students’ parents were aware of the lessons that they taught from the MoD curriculum. The teachers provided their estimates using a four-point rating scale (from 1 = not at all to 4 = always). For teachers in the grades K–2 and 3–5 categories, the responses ranged from 1 to 3. The mean for K–2 was between the “occasionally” and “frequently” anchors (M = 2.36, SD = 0.50). We observed similar results for students in grades 3–5 (M = 2.20, SD = 0.79), which again was between occasionally and frequently with respect to parental awareness. Teachers in grades 6–8 produced a mean estimate of 2.00 (occasionally; SD = 0.50).

One of the primary goals of this project involved making the parents of the students who were exposed to the MoD curriculum at school more aware of weather safety issues and practices. The presumed mechanism for reaching the parents with the MoD curriculum involved two related features: 1) the teachers’ use of the MoD curriculum with their students and, more specifically, 2) using the Home Connection feature that was incorporated in many of the MoD weather science and safety lessons. The Home Connection aspect of MoD typically involved the completion of homework assignments that involved the student’s parents and family. To assess the effects of such homework activities and the parents’ involvement, we developed the Home Connection questionnaire. The items of this questionnaire appear in Table 3. The instrument contained seven yes–no questions. If the parents responded “yes” to the last item (“As a result of my child’s learning about weather safety at school, we have taken steps at home to prepare for severe weather—like tornadoes, lightning, hurricanes, and floods”), we asked them to briefly describe the steps that they or the family had taken at home to prepare for severe weather. The teachers sent the questionnaires home with their students to the parents who would then complete them and return them via the U.S. Postal Service. As an inducement to increase participation, we paid parents $10 for completing and returning the form. It was difficult to control the time interval between the teacher’s MoD lesson delivery and the completion of the form by the students’ parents at home. In general, we estimate that the parents typically completed the forms within one month of their students’ involvement in a weather science or safety lesson. Based upon the teacher’s class size, we estimate that of the Home Connection questionnaires sent home, approximately 35% of them were returned.

Table 3.

Parents’ responses to the Home Connection questionnaire (n = 133). Values in parentheses are percentages of the number total.

View Table

The responses to the Home Connection questionnaire are summarized in Table 3. The data were encouraging in that, across the seven questions that we asked, a wide preponderance of the responses indicated that the parents had some degree of contact with the child’s learning experiences concerning weather science and safety. Approximately 70% of the parents indicated that they had known of or had been involved with some aspect of their child’s learning about weather science and safety. It is not likely that the parents were responding to non-MoD-related schoolwork or homework because we designed the Home Connection questionnaire to be especially responsive to the curriculum features that the students and parents might experience, especially as this involved safety (items 2, 4, 6, and 7). Because the MoD curriculum is both unique and specific in emphasizing safety around tornadoes, hurricanes, lightning, and floods, it is unlikely that the parents’ responses in Table 3 reflected non-MoD curriculum coverage. In addition, the teachers who distributed the Home Connection questionnaire responded to our Use and Dissemination survey as having relied principally upon the MoD curriculum with their students.

There were 94 parents who indicated that as a result of their child’s learning about weather safety at school, steps had been taken at home to prepare for severe weather (item 7). This represented nearly 71% of the parents who responded to the survey. Ninety-three parents provided comments to explain the nature of their family’s preparations. The parents’ comments ranged from statements that indicated a general readiness for severe weather—“We have gone thru examples of what to do and where to go during severe weather,” “We have emergency kits prepared, also a generator for power plus evacuation information on hand at all times”—to more specific plans for particular kinds of weather hazards: “We will go to our downstairs bathroom if there is a tornado. Stay indoors if there is lightning,” “We have a household plan for tornadoes like meet in the bathroom. Stay inside for lightning. Go to safe place in case of hurricane (out of town).” The parents most frequently mentioned their preparations for tornadoes (27 comments), followed by lightning (13 comments). Some parents mentioned severe weather or storms in general (15 comments each). Finally, there were 14 comments that mentioned safety or a safe place to shelter during severe weather. Overall, the parents’ comments were encouraging and conveyed a degree of readiness for severe weather.

Limitations.

We begin our discussion of the results by noting some of the project’s limitations. The first limitation involved the sizes of our samples as the project progressed in time. From our prior experience in conducting the workshops from 2008 to 2010, we observed that not all of the teachers who attended the workshop chose or were able to implement the MoD curriculum during the subsequent school year (Stewart et al. 2015). Such was also the case in the present project despite our being more stringent in emphasizing, both during the workshop application and training phases, the importance of actually delivering MoD lessons when possible. Between school years, teachers may move to different grades or to different schools, and this can affect their ability to implement the curriculum. This effectively limited the numbers of teachers who were subsequently taught the lessons, who allowed us to observe their instruction, who administered postlesson content tests, and who volunteered to send the Home Connection questionnaire home to their students’ parents. Given the longitudinal nature of this project, some degree of attrition is to be expected (Shadish et al. 2002).

Another and related limitation involves selection bias. We deliberately recruited teachers who expressed an interest in weather science and safety and a desire to learn how to deliver the MoD curriculum to their students. Our workshops were limited further by selecting teachers from within the state of Georgia. From our Weather Safety Education questionnaire, many of our teachers reported being dissatisfied with current curriculum resources for weather science and safety and routinely conveyed anecdotal experiences of severe weather affecting their locations in the past. These previous experiences made the weather salient for many teachers and were responsible, in part, for their interest in our workshop (Stewart 2009). Our results thus generalize only to teachers who have had similar kinds of weather-related experiences as teachers and who possess an above-average level of interest to provide instruction about weather science and safety.

A third limitation of our study is that we did not utilize any control or comparison groups for teachers, their students, or the students’ parents. Because we recruited teachers from Georgia counties that were socioeconomically vulnerable to the effects of severe and extreme weather and because many of these counties possessed a large number of students from underserved groups, we enrolled virtually all of the interested teachers in the workshops. That is, we did not have a sufficiently large pool of interested and willing teachers for some of them to serve as a no-workshop control/comparison group. Similarly, it was difficult to recruit teachers who were not in the workshop to administer postlesson content tests to their students. Anecdotally, once teachers in a grade had learned about the MoD curriculum from a teacher who participated in the workshop, they were interested in using the curriculum. They were somewhat self-conscious, however, about allowing us to assess their students out of concerns that their instruction may not be on par with the teachers who did attend the workshop. These are all significant limitations of our workshop evaluation. Nonetheless, the results seem sufficiently compelling such that our future efforts in providing teacher training can be designed (and funded) so that control and comparison groups can be used.

Weather science and safety education was effective.

This project extends our earlier work that demonstrated that Georgia teachers could increase their knowledge of weather science and safety through our workshops and also disseminate the results of their professional learning to both their teaching colleagues and to their students (Stewart et al. 2015). In the present project, not only did the teachers build their weather-related knowledge and sense of teaching self-efficacy, they also were able to convey this instruction to their students. Further, by involving their parents in the MoD-related Home Connection activities, it appears that parents became both more knowledgeable and better prepared for severe and extreme weather. To our knowledge, this project is the first of its kind to demonstrate systematically the effectiveness of weather science and safety education for teachers, their students, and the students’ parents. That is, our workshops were effective in the sense that by training and equipping teachers with the MoD curriculum, we were able to reach students and their parents in a way that made them more aware of and perhaps better prepared for severe and extreme weather.

Considerations in working with teachers and school districts.

Based upon our experiences in this project, several considerations are salient in moving forward to diffuse the MoD curriculum to educators in a broader manner. First, although ARC worked at the national level with agencies such as the Storm Prediction Center, the National Hurricane Center, and the National Weather Service to develop a broad curriculum that encompassed most of the weather hazards that people might experience, teachers at the local level must work within the parameters of their local school districts to use resources like the MoD. The systematic inclusion of the Masters of Disaster or other weather or climate curricula must align with the educational performance standards in the state where the teaching occurs—otherwise, teachers cannot use it in more than a minor way. In Georgia, we had to demonstrate how the various MoD lessons satisfied Georgia performance standards for science at each grade level so that teachers could use the curriculum along with their existing instructional materials and texts. Showing teachers how the MoD and other weather-related curriculum materials satisfy performance standards will greatly facilitate its regular use in classroom instruction. In this regard, we have aligned the MoD with the Next Generation Science Standards, which, as of 2016, had been adopted by 18 states and the District of Columbia (Heitin 2016).

Second, we have learned that teachers derive much benefit from professional learning workshops at both the content and process levels. In our workshop, we helped teachers become more familiar with the science and safety content associated with various forms of severe weather. Beyond this, we helped them with the processes of thinking about and practicing the implementation of the curriculum within their schools. Providing this kind of professional development is important because although teachers may possess much interest in weather, they may not have had much science education training within their degree programs. For example, the prototypical fourth-grade teacher in our workshop earned a degree in elementary education and had completed one or two science education courses. Although this was sufficient for work as a teacher in a self-contained classroom, many teachers viewed weather as a specialty science that required them to have more detailed knowledge of the physical and Earth sciences before they felt confident in using the MoD. We conclude from this that teachers would benefit from greater exposure to the atmospheric and Earth sciences in their preparation programs and that current teachers would continue to benefit from professional development such as we provided in our workshops.

The MoD curriculum complements existing resources.

The Masters of Disaster curriculum joins an impressive collection of resources for K–12 educators that exist in Project Atmosphere (Ginger et al. 1996), DataStreme, GLOBE (Butler and MacGregor 2003), and S’COOL (Chambers et al. 2003), among others. Although these resources provide excellent coverage of weather science, they are less focused upon the specific topics of weather safety for lightning, tornadoes, hurricanes, and floods. Thus, the MoD curriculum can make a unique contribution because of its focus upon weather-related hazards and upon the themes of disaster preparation and recovery. Along with the existing AMS, NASA, and National Weather Service online resources (www.weather.gov/owlie/ and www.weather.gov/safety), the use of the MoD curriculum can help achieve a “Weather-Ready Nation” (National Weather Service 2013). The authors hope that using the MoD curriculum will enable children and their families to better prepare and respond to severe weather.