Many high school students with a passion for meteorology visit college campuses each year, inquiring about program details and learning of the rigorous math requirements and somewhat daunting attrition risk. The natural question for many is, “Can I do it?” To address this question, the study herein has investigated the sensitivity of meteorology major outcomes at Valparaiso University to incoming test scores (based on 2008–14 first-year cohorts). These former students are classified as meteorology alumni (MET Degree), graduates of other programs (Other VU Degree), and those who left the University (Left VU). There is an additional performance classification, which indicates students who completed the program in good standing (On Track), students that completed the program with greater difficulty (Below Track), those who left the program in good academic standing (Left On Track), and those who left the program after not doing well (Left Below Track). Results indicate that incoming test scores are indeed indicative of student outcomes, with a substantially greater likelihood of being On Track and graduating with the MET Degree among students with higher incoming ACT scores. Among students who Left Below Track, signs of academic challenges typically appear within first semester calculus and/or the introductory meteorology course. Higher admission standards would increase retention and graduation rates, but at a cost of barring admission for some students who have been successful in the program.
Standardized test scores (such as the ACT and SAT) and high school grade point average (GPA) are predictive of academic performance, retention, and associated degree completion from undergraduate programs (Bridgeman et al. 2008; Allen and Robbins 2010; Westrick et al. 2015); this relationship has been found for colleges of all selectivity levels, types of classes (grouped disciplines), and demographic factors such as gender and race/ethnicity (Bridgeman et al. 2008). The sensitivity to test scores is stronger for selective institutions, while those with lower admission standards have higher first-to-second year attrition rates (Westrick et al. 2015). Although test scores and high school GPA are correlated, each provides distinctive insight into the likelihood of success in college coursework (Bridgeman et al. 2008; Westrick et al. 2015).
Test scores are also predictive when applied to areas of study, with benchmark ACT scores available for typical freshman coursework and corresponding to a 50% probability of a B or better grade in the associated courses (Mattern et al. 2015; Allen et al. 2017). Performance within science, technology, engineering, and math (STEM) disciplines is dependent on math ability (Bridgeman et al. 2008), and the ACT benchmark for STEM courses (including Calculus I) is higher than for the other assessed content areas (Radunzel et al. 2015; Allen et al. 2017). The STEM benchmark of 26 represents the rounded average of math and science scores, while an ACT Math score of 27 indicates a 50% probability of a B or better grade in first semester calculus (Radunzel et al. 2015).
Pre-college preparation is critical for academic performance in the first year, which subsequently impacts student outcomes in the following years (Allen et al. 2008). Second-year enrollment is affected by first-year GPA and social engagement (Pascarella and Terenzini 2005), while retention through year three is dependent on academic performance, pre-college educational development, college commitment, and social connectedness (Allen et al. 2008). The role of social factors is consistent with Tinto (1975) and Bean (1980) models of student persistence in college, both of which assert that persistence is influenced by the match between the institution and student, as well as preparation for college (Cabrera et al. 1993). Ultimately, the fit of major and student interest affects the likelihood of success within a degree program (Allen and Robbins 2010); as an example within STEM, Besterfield-Sacre et al. (1997) found that attitudes toward engineering influenced the likelihood of students staying within the major, while low SAT Math scores increased the likelihood of being dismissed from engineering due to poor performance.
Testing critics have argued that test scores are primarily an indicator of socioeconomic status (SES), with little additional value regarding the likelihood of student success (e.g., Kohn 2001; Crosby et al. 2003; Atkinson 2005). Assessing these claims quantitatively, Sackett et al. (2009) found that while SES is correlated with test scores, most of the sensitivity of academic performance to test metrics is independent of SES. Westrick et al. (2015) affirmed the importance of test scores on college performance and found that SES is a weak predictor of academic performance and retention.
The efficacy of incoming test scores and high school GPA in predicting success within meteorology (or atmospheric science) undergraduate programs has not been previously evaluated in the literature; however, it is generally understood that attrition is substantial (∼50%) in this math-intensive course of study (Roebber 2005). Undergraduate success in the discipline has been evaluated in many other ways, including the differing learning styles of students and faculty (Roebber 2005), the role of spatial thinking (McNeal et al. 2018), and numerous examples of experiential learning through case studies, field work, and undergraduate research (e.g., Hindman 1993; Gallus et al. 2000; Roebber 2005; Morss and Zhang 2008; Quardokus et al. 2012; Croft and Ha 2014; Mullendore and Tilley 2014; Milrad and Herbster 2017). Other studies have been concerned with employment challenges for meteorology students (Knox 2008), as well as the additional value of mentoring in expanding opportunities for students (Windham et al. 2004; Sloan et al. 2017).
Given the clear costs and benefits of successfully obtaining a bachelor’s degree, coupled with the quantitative demands of undergraduate degrees in meteorology (atmospheric science) and the generally understood high attrition within these programs, it is beneficial to quantify the attrition risk as it relates to incoming test scores and associated math skill. To address this issue, the study herein was undertaken to evaluate the following questions:
What percentage of incoming meteorology majors graduate from the program, complete a different major, or leave the university?
How many struggle through coursework, or leave the major as a result?
How sensitive are student outcomes to incoming test scores?
How early in their academic program do students experience trouble?
Would higher admission standards improve retention and what are the ethical ramifications of improving retention this way?
Data and methods
Student outcomes have been assessed for seven years of incoming meteorology majors (2008–14 first-year cohorts) at Valparaiso University (VU). Along with an evaluation of reported incoming test scores, these former students were given classifications related to program success and graduation, as follows.
These are program graduates who did not need to retake any required mathematics, physics, or meteorology courses to stay above the meteorology/math/physics GPA requirement of 2.3. (The required calculus-based physics courses are a two-semester sequence taken in the second year of study, while math courses include a three-semester calculus sequence, differential equations, and a one-credit linear algebra course. The introductory meteorology course only requires math placement beyond college algebra, while first semester calculus must be completed prior to the second year in the program and third semester calculus before the third year.)
These are program graduates who had to retake at least one required mathematics, physics, or meteorology course in order to stay above the required GPA, and/or were below this standard for at least one of the semester-end evaluations.
Left on track.
These are students that were doing fine in the program (B− or better grades in meteorology, math and physics coursework), but decided to change majors or leave Valparaiso University.
Left below track.
These are students who changed majors after struggling (with a C+ or lower) in required meteorology, math or physics coursework.
These students graduated from VU with a major in meteorology.
Other VU degree.
These students graduated from VU with a different major.
These students did not graduate from VU.
Data visualization was used to examine the sensitivity of student outcomes to incoming ACT scores, with the Welch’s t test and a 95% confidence threshold used for assessing the null hypothesis of equal population means. The Welch approach is a two-sample test of equal population means which does not assume equal group variances (Crawley 2005; Derrick et al. 2016). SAT scores were considered as well, but are not evaluated herein since more than twice as many students report ACT scores.
Results and discussion
During this seven-cohort period, 273 students entered the meteorology program at VU, with a similar percentage of male (52%) and female (48%) pupils. Most students started as freshman, while 28 transferred from another institution. Regarding student performance, a little under half of the students completed the program On Track, 11% completed the program Below Track, 9% Left On Track, and roughly a third Left Below Track (Fig. 1a). Roughly 58% of the students graduated with a MET Degree, while the rest graduated as Other VU Degree or Left VU (Fig. 1b). The combined graduation rate was 78%, with none of the students still working toward their VU undergraduate degree after the completion of the 2017/18 academic year.
Incoming student test scores varied substantially, with a mean ACT Composite of 26.4 (with a range from 18 to 35) and ACT Math average of 26.3 (with a range of 16 to 36). Student outcomes were sensitive to incoming scores, with significantly higher average ACT Composite and ACT Math scores for On Track students compared with both Below Track and Left Below Track groups (Table 1; Figs. 2a,c). This sensitivity to incoming test scores affected the graduation classification as well, with significantly higher ACT Composite and ACT Math scores for MET Degree graduates than Other VU Degree graduates and those who Left VU (Table 1; Figs. 2b,d).
Summing across test scores allows an evaluation of student outcomes for different segments of the distribution (Table 2). For example, more than 60% students with ACT Math scores of 23 or less Left Below Track. Prospects were vastly improved for students with ACT Math scores of 25 and above, with roughly two-thirds On Track in the major. These impacts carried over to graduation, with a MET Degree completion rate of 72% for ACT Math scores of 25 or greater; in contrast, the MET Degree was completed by only 35% of students with ACT Math scores of 23 or less. The associated likelihood of leaving the institution without graduating (Left VU) was much higher for the students with lower scores.
To better understand the circumstances of students with the Left Below Track designation, it is useful to consider their performance in first semester calculus and the introductory meteorology course (Table 3). Within first semester calculus, slightly more than half of these students earned a C+ or lower, while a quarter did not complete the course. Within the introductory meteorology course, roughly half earned a C+ or lower, but most completed the course. Less than 20% of the Left Below Track students earned a B− or better in each of these courses, yet changed majors after difficulty in subsequent coursework.
While it is clear from the results of the study that increased admission standards (at the program or university level) would increase retention within the meteorology program, there are ethical considerations surrounding any decision to modify admissions standards. On the one hand, is it ethical to admit students with a low ACT Math score, given a strong chance that they will leave the program after not doing well? On the other hand, is it ethical to reject these students from trying, when some have completed the program and plenty of others graduated from the university with another major? There may not be clear answers to these concerns, but perhaps a better balance can be achieved. Thankfully, better knowledge of academic risk can inform recruitment and advising of students. Academic difficulty typically appears within early coursework, such that most students with good performance in the first year are highly likely to succeed in the program, while those with difficulty can reassess approaches to learning, seek more help, or change their course of study in a timely fashion.
Given differences among colleges and universities, it is fair to consider whether the results herein are generalizable to other institutions. Although Valparaiso University is not among the most selective universities, being a smaller private institution may nonetheless influence the student body (when compared with larger, public universities). However, it seems highly likely that the sensitivity to test scores is a common feature of meteorology programs, since all such programs require rigorous mathematics, large public universities typically report a wide range of incoming test scores, and previous research has found that the sensitivity to test scores is present across a wide range of institutions and demographic factors (Bridgeman et al. 2008).
This study does not include other student factors which may be important predictors, such as high school grade point average, class rank, and high school math courses. High school GPA is available for most of these students, but a scaling change associated with advanced coursework and the lack of math course information makes an evaluation difficult. Demographic factors would also of interest for the fine-tuning of test score indicators, but the sample size is too small for such an evaluation.
The author thanks Valparaiso University for funding this work and Rusta Ault for valuable data acquisition. I also thank colleagues within Valparaiso University for their valuable insight and context, especially Bart Wolf, Teresa Bals-Elsholz, Kevin Goebbert, Adam Stepanek, and Andrea Welch.