Abstract

Many high school students with a passion for meteorology visit college campuses each year, inquiring about program details and learning of the rigorous math requirements and somewhat daunting attrition risk. The natural question for many is, “Can I do it?” To address this question, the study herein has investigated the sensitivity of meteorology major outcomes at Valparaiso University to incoming test scores (based on 2008–14 first-year cohorts). These former students are classified as meteorology alumni (MET Degree), graduates of other programs (Other VU Degree), and those who left the University (Left VU). There is an additional performance classification, which indicates students who completed the program in good standing (On Track), students that completed the program with greater difficulty (Below Track), those who left the program in good academic standing (Left On Track), and those who left the program after not doing well (Left Below Track). Results indicate that incoming test scores are indeed indicative of student outcomes, with a substantially greater likelihood of being On Track and graduating with the MET Degree among students with higher incoming ACT scores. Among students who Left Below Track, signs of academic challenges typically appear within first semester calculus and/or the introductory meteorology course. Higher admission standards would increase retention and graduation rates, but at a cost of barring admission for some students who have been successful in the program.

Standardized test scores (such as the ACT and SAT) and high school grade point average (GPA) are predictive of academic performance, retention, and associated degree completion from undergraduate programs (Bridgeman et al. 2008; Allen and Robbins 2010; Westrick et al. 2015); this relationship has been found for colleges of all selectivity levels, types of classes (grouped disciplines), and demographic factors such as gender and race/ethnicity (Bridgeman et al. 2008). The sensitivity to test scores is stronger for selective institutions, while those with lower admission standards have higher first-to-second year attrition rates (Westrick et al. 2015). Although test scores and high school GPA are correlated, each provides distinctive insight into the likelihood of success in college coursework (Bridgeman et al. 2008; Westrick et al. 2015).

Test scores are also predictive when applied to areas of study, with benchmark ACT scores available for typical freshman coursework and corresponding to a 50% probability of a B or better grade in the associated courses (Mattern et al. 2015; Allen et al. 2017). Performance within science, technology, engineering, and math (STEM) disciplines is dependent on math ability (Bridgeman et al. 2008), and the ACT benchmark for STEM courses (including Calculus I) is higher than for the other assessed content areas (Radunzel et al. 2015; Allen et al. 2017). The STEM benchmark of 26 represents the rounded average of math and science scores, while an ACT Math score of 27 indicates a 50% probability of a B or better grade in first semester calculus (Radunzel et al. 2015).

Pre-college preparation is critical for academic performance in the first year, which subsequently impacts student outcomes in the following years (Allen et al. 2008). Second-year enrollment is affected by first-year GPA and social engagement (Pascarella and Terenzini 2005), while retention through year three is dependent on academic performance, pre-college educational development, college commitment, and social connectedness (Allen et al. 2008). The role of social factors is consistent with Tinto (1975) and Bean (1980) models of student persistence in college, both of which assert that persistence is influenced by the match between the institution and student, as well as preparation for college (Cabrera et al. 1993). Ultimately, the fit of major and student interest affects the likelihood of success within a degree program (Allen and Robbins 2010); as an example within STEM, Besterfield-Sacre et al. (1997) found that attitudes toward engineering influenced the likelihood of students staying within the major, while low SAT Math scores increased the likelihood of being dismissed from engineering due to poor performance.

Testing critics have argued that test scores are primarily an indicator of socioeconomic status (SES), with little additional value regarding the likelihood of student success (e.g., Kohn 2001; Crosby et al. 2003; Atkinson 2005). Assessing these claims quantitatively, Sackett et al. (2009) found that while SES is correlated with test scores, most of the sensitivity of academic performance to test metrics is independent of SES. Westrick et al. (2015) affirmed the importance of test scores on college performance and found that SES is a weak predictor of academic performance and retention.

The efficacy of incoming test scores and high school GPA in predicting success within meteorology (or atmospheric science) undergraduate programs has not been previously evaluated in the literature; however, it is generally understood that attrition is substantial (∼50%) in this math-intensive course of study (Roebber 2005). Undergraduate success in the discipline has been evaluated in many other ways, including the differing learning styles of students and faculty (Roebber 2005), the role of spatial thinking (McNeal et al. 2018), and numerous examples of experiential learning through case studies, field work, and undergraduate research (e.g., Hindman 1993; Gallus et al. 2000; Roebber 2005; Morss and Zhang 2008; Quardokus et al. 2012; Croft and Ha 2014; Mullendore and Tilley 2014; Milrad and Herbster 2017). Other studies have been concerned with employment challenges for meteorology students (Knox 2008), as well as the additional value of mentoring in expanding opportunities for students (Windham et al. 2004; Sloan et al. 2017).

Given the clear costs and benefits of successfully obtaining a bachelor’s degree, coupled with the quantitative demands of undergraduate degrees in meteorology (atmospheric science) and the generally understood high attrition within these programs, it is beneficial to quantify the attrition risk as it relates to incoming test scores and associated math skill. To address this issue, the study herein was undertaken to evaluate the following questions:

  • What percentage of incoming meteorology majors graduate from the program, complete a different major, or leave the university?

  • How many struggle through coursework, or leave the major as a result?

  • How sensitive are student outcomes to incoming test scores?

  • How early in their academic program do students experience trouble?

  • Would higher admission standards improve retention and what are the ethical ramifications of improving retention this way?

Data and methods

Student outcomes have been assessed for seven years of incoming meteorology majors (2008–14 first-year cohorts) at Valparaiso University (VU). Along with an evaluation of reported incoming test scores, these former students were given classifications related to program success and graduation, as follows.

Program success.

On track.

These are program graduates who did not need to retake any required mathematics, physics, or meteorology courses to stay above the meteorology/math/physics GPA requirement of 2.3. (The required calculus-based physics courses are a two-semester sequence taken in the second year of study, while math courses include a three-semester calculus sequence, differential equations, and a one-credit linear algebra course. The introductory meteorology course only requires math placement beyond college algebra, while first semester calculus must be completed prior to the second year in the program and third semester calculus before the third year.)

Below track.

These are program graduates who had to retake at least one required mathematics, physics, or meteorology course in order to stay above the required GPA, and/or were below this standard for at least one of the semester-end evaluations.

Left on track.

These are students that were doing fine in the program (B− or better grades in meteorology, math and physics coursework), but decided to change majors or leave Valparaiso University.

Left below track.

These are students who changed majors after struggling (with a C+ or lower) in required meteorology, math or physics coursework.

Graduation.

MET degree.

These students graduated from VU with a major in meteorology.

Other VU degree.

These students graduated from VU with a different major.

Left VU.

These students did not graduate from VU.

Data visualization was used to examine the sensitivity of student outcomes to incoming ACT scores, with the Welch’s t test and a 95% confidence threshold used for assessing the null hypothesis of equal population means. The Welch approach is a two-sample test of equal population means which does not assume equal group variances (Crawley 2005; Derrick et al. 2016). SAT scores were considered as well, but are not evaluated herein since more than twice as many students report ACT scores.

Results and discussion

During this seven-cohort period, 273 students entered the meteorology program at VU, with a similar percentage of male (52%) and female (48%) pupils. Most students started as freshman, while 28 transferred from another institution. Regarding student performance, a little under half of the students completed the program On Track, 11% completed the program Below Track, 9% Left On Track, and roughly a third Left Below Track (Fig. 1a). Roughly 58% of the students graduated with a MET Degree, while the rest graduated as Other VU Degree or Left VU (Fig. 1b). The combined graduation rate was 78%, with none of the students still working toward their VU undergraduate degree after the completion of the 2017/18 academic year.

Fig. 1.

The number of students in (left) performance and (right) graduation classifications. The associated percentages of students within each group are also provided, rounded to the nearest percentage point. The x-axis labels for Left On Track and Left Below Track performance classifications have been abbreviated.

Fig. 1.

The number of students in (left) performance and (right) graduation classifications. The associated percentages of students within each group are also provided, rounded to the nearest percentage point. The x-axis labels for Left On Track and Left Below Track performance classifications have been abbreviated.

Incoming student test scores varied substantially, with a mean ACT Composite of 26.4 (with a range from 18 to 35) and ACT Math average of 26.3 (with a range of 16 to 36). Student outcomes were sensitive to incoming scores, with significantly higher average ACT Composite and ACT Math scores for On Track students compared with both Below Track and Left Below Track groups (Table 1; Figs. 2a,c). This sensitivity to incoming test scores affected the graduation classification as well, with significantly higher ACT Composite and ACT Math scores for MET Degree graduates than Other VU Degree graduates and those who Left VU (Table 1; Figs. 2b,d).

Table 1.

Welch’s t-test comparisons of ACT scores among student performance and graduation groups, using On Track and MET Degree as the baseline categories. For each test, * indicates that we can reject the null hypothesis of equal ACT population means, using a 95% confidence interval. P values are provided in parentheses for all insignificant Welch’s t-test results.

Welch’s t-test comparisons of ACT scores among student performance and graduation groups, using On Track and MET Degree as the baseline categories. For each test, * indicates that we can reject the null hypothesis of equal ACT population means, using a 95% confidence interval. P values are provided in parentheses for all insignificant Welch’s t-test results.
Welch’s t-test comparisons of ACT scores among student performance and graduation groups, using On Track and MET Degree as the baseline categories. For each test, * indicates that we can reject the null hypothesis of equal ACT population means, using a 95% confidence interval. P values are provided in parentheses for all insignificant Welch’s t-test results.
Fig. 2.

Student (top) ACT Composite and (bottom) ACT Math scores among incoming meteorology students, classified by (left) program success and (right) graduation. The differences between the ACT Composite scores among the On Track group (compared with Left Below Track and Below Track groups) are statistically significant; this is also the case for the ACT Math scores. (Strictly speaking, using a confidence threshold of 95%, Welch’s t-test results allow us to reject the null hypothesis of equal population mean ACT scores.) The differences in mean scores between the MET Degree graduates and the other graduation categories are also significant.

Fig. 2.

Student (top) ACT Composite and (bottom) ACT Math scores among incoming meteorology students, classified by (left) program success and (right) graduation. The differences between the ACT Composite scores among the On Track group (compared with Left Below Track and Below Track groups) are statistically significant; this is also the case for the ACT Math scores. (Strictly speaking, using a confidence threshold of 95%, Welch’s t-test results allow us to reject the null hypothesis of equal population mean ACT scores.) The differences in mean scores between the MET Degree graduates and the other graduation categories are also significant.

Summing across test scores allows an evaluation of student outcomes for different segments of the distribution (Table 2). For example, more than 60% students with ACT Math scores of 23 or less Left Below Track. Prospects were vastly improved for students with ACT Math scores of 25 and above, with roughly two-thirds On Track in the major. These impacts carried over to graduation, with a MET Degree completion rate of 72% for ACT Math scores of 25 or greater; in contrast, the MET Degree was completed by only 35% of students with ACT Math scores of 23 or less. The associated likelihood of leaving the institution without graduating (Left VU) was much higher for the students with lower scores.

Table 2.

Evaluation of student percentages within performance and graduation groups, given ranges of ACT Math scores. (For example, for students with ACT Math scores between 27 and 36, 72.2% were On Track and 75.9% earned the MET degree.) The percentages have been rounded to the nearest 0.1%.

Evaluation of student percentages within performance and graduation groups, given ranges of ACT Math scores. (For example, for students with ACT Math scores between 27 and 36, 72.2% were On Track and 75.9% earned the MET degree.) The percentages have been rounded to the nearest 0.1%.
Evaluation of student percentages within performance and graduation groups, given ranges of ACT Math scores. (For example, for students with ACT Math scores between 27 and 36, 72.2% were On Track and 75.9% earned the MET degree.) The percentages have been rounded to the nearest 0.1%.

To better understand the circumstances of students with the Left Below Track designation, it is useful to consider their performance in first semester calculus and the introductory meteorology course (Table 3). Within first semester calculus, slightly more than half of these students earned a C+ or lower, while a quarter did not complete the course. Within the introductory meteorology course, roughly half earned a C+ or lower, but most completed the course. Less than 20% of the Left Below Track students earned a B− or better in each of these courses, yet changed majors after difficulty in subsequent coursework.

Table 3.

Early course indicators for the 90 students with the Left Below Track classification. The grade below B− designation includes students with first semester calculus and introductory meteorology course grades of C+ or below (including course failure). Noncompletion indicates that the student did not take the course or withdrew from the course. The percentages of the 90 students are provided in parentheses, rounded to the nearest percentage point.

Early course indicators for the 90 students with the Left Below Track classification. The grade below B− designation includes students with first semester calculus and introductory meteorology course grades of C+ or below (including course failure). Noncompletion indicates that the student did not take the course or withdrew from the course. The percentages of the 90 students are provided in parentheses, rounded to the nearest percentage point.
Early course indicators for the 90 students with the Left Below Track classification. The grade below B− designation includes students with first semester calculus and introductory meteorology course grades of C+ or below (including course failure). Noncompletion indicates that the student did not take the course or withdrew from the course. The percentages of the 90 students are provided in parentheses, rounded to the nearest percentage point.

Conclusions

While it is clear from the results of the study that increased admission standards (at the program or university level) would increase retention within the meteorology program, there are ethical considerations surrounding any decision to modify admissions standards. On the one hand, is it ethical to admit students with a low ACT Math score, given a strong chance that they will leave the program after not doing well? On the other hand, is it ethical to reject these students from trying, when some have completed the program and plenty of others graduated from the university with another major? There may not be clear answers to these concerns, but perhaps a better balance can be achieved. Thankfully, better knowledge of academic risk can inform recruitment and advising of students. Academic difficulty typically appears within early coursework, such that most students with good performance in the first year are highly likely to succeed in the program, while those with difficulty can reassess approaches to learning, seek more help, or change their course of study in a timely fashion.

Given differences among colleges and universities, it is fair to consider whether the results herein are generalizable to other institutions. Although Valparaiso University is not among the most selective universities, being a smaller private institution may nonetheless influence the student body (when compared with larger, public universities). However, it seems highly likely that the sensitivity to test scores is a common feature of meteorology programs, since all such programs require rigorous mathematics, large public universities typically report a wide range of incoming test scores, and previous research has found that the sensitivity to test scores is present across a wide range of institutions and demographic factors (Bridgeman et al. 2008).

This study does not include other student factors which may be important predictors, such as high school grade point average, class rank, and high school math courses. High school GPA is available for most of these students, but a scaling change associated with advanced coursework and the lack of math course information makes an evaluation difficult. Demographic factors would also of interest for the fine-tuning of test score indicators, but the sample size is too small for such an evaluation.

ACKNOWLEDGMENTS

The author thanks Valparaiso University for funding this work and Rusta Ault for valuable data acquisition. I also thank colleagues within Valparaiso University for their valuable insight and context, especially Bart Wolf, Teresa Bals-Elsholz, Kevin Goebbert, Adam Stepanek, and Andrea Welch.

REFERENCES

REFERENCES
Allen
,
J.
, and
S.
Robbins
,
2010
:
Effects of interest-major confluence, motivation, and academic performance on timely degree attainment
.
J. Couns. Psychol.
,
57
,
23
35
, https://doi.org/10.1037/a0017267.
Allen
,
J.
,
S.
Robbins
,
A.
Casillas
, and
I.
Oh
,
2008
:
Third year college retention and transfer: Effects of academic performance, motivation and social connectedness
.
Res. High. Educ.
,
49
,
647
664
, https://doi.org/10.1007/s11162-008-9098-3.
Allen
,
J.
,
J.
Radunzel
, and
J.
Moore
,
2017
: Evidence for standard setting: Probabilities of success in “benchmark” college courses, by ACT test scores. ACT Tech. Brief, 12 pp., www.act.org/content/dam/act/unsecured/documents/R1648-evidence-for-standard-setting-2017-05.pdf.
Atkinson
,
R. C.
,
2005
:
College admissions and the SAT: A personal perspective
.
APS Obs
.,
8
,
15
22
.
Bean
,
J. P.
,
1980
:
Dropouts and turnover: The synthesis and test of a causal model of student attrition
.
Res. High. Educ.
,
12
,
155
187
, https://doi.org/10.1007/BF00976194.
Besterfield-Sacre
,
M.
,
C. J.
Atman
, and
L. J.
Shuman
,
1997
:
Characteristics of freshman engineering students: Models for determining student attrition in engineering
.
J. Eng. Educ.
,
86
,
139
149
, https://doi.org/10.1002/j.2168-9830.1997.tb00277.x.
Bridgeman
,
B.
,
J.
Pollack
, and
N.
Burton
,
2008
: Predicting grades in different types of college courses. College Board Research Rep. 2008-1, ETS RR-08-06, 27 pp., https://files.eric.ed.gov/fulltext/ED562859.pdf.
Cabrera
,
A.
,
A.
Nora
, and
M.
Castaneda
,
1993
:
College persistence: Structural equations modeling test of an integrated model of student retention
.
J. Higher Educ.
,
62
,
123
139
, https://doi.org/10.2307/2960026.
Crawley
,
M. J.
,
2005
: Statistics: An Introduction Using R. Wiley, 327 pp.
Croft
,
O. J.
, and
J.
Ha
,
2014
:
The undergraduate “consulting classroom”: Field, research, and practicum experiences
.
Bull. Amer. Meteor. Soc.
,
95
,
1603
1612
, https://doi.org/10.1175/BAMS-D-13-00045.1.
Crosby
,
F. J.
,
A.
Iyer
,
S.
Clayton
, and
R. A.
Downing
,
2003
:
Affirmative action: Psychological data and the policy debates
.
Amer. Psychol.
,
58
,
93
115
, https://doi.org/10.1037/0003-066X.58.2.93.
Derrick
,
B.
,
D.
Toher
, and
P.
White
,
2016
:
Why Welch’s test is Type I error robust
.
Quant. Methods Psychol
.,
12
,
30
38
, https://doi.org/10.20982/tqmp.12.1.p030.
Gallus
,
W. A.
, Jr.
,
D. N.
Yarger
, and
D. E.
Herzmann
,
2000
:
An interactive severe weather activity to motivate student learning
.
Bull. Amer. Meteor. Soc.
,
81
,
2205
2212
, https://doi.org/10.1175/1520-0477(2000)081<2205:AISWAT>.3.CO;2.
Hindman
,
E. E.
,
1993
:
An undergraduate field course in meteorology and atmospheric chemistry
.
Bull. Amer. Meteor. Soc.
,
74
,
661
668
, https://doi.org/10.1175/1520-0477-74.4.661.
Knox
,
J. A.
,
2008
:
Recent and future trends in U.S. undergraduate meteorology enrollments, degree recipients, and employment opportunities
.
Bull. Amer. Meteor. Soc.
,
89
,
873
884
, https://doi.org/10.1175/2008BAMS2375.1.
Kohn
,
A.
,
2001
:
Two cheers for an end to the SAT
.
Chron. Higher Educ.
,
9
,
B–12
.
Mattern
,
K.
,
J.
Radunzel
, and
P.
Westrick
,
2015
:
Development of STEM readiness benchmarks to assist educational and career decision making
. ACT Research Rep. Series
2015
(
3
),
34
pp., www.act.org/content/dam/act/unsecured/documents/ACT_RR2015-3.pdf.
McNeal
,
P.
,
T.
Ellis
, and
H.
Petcovic
,
2018
:
Investigating the foundations of spatial thinking in meteorology
.
J. Geosci. Educ.
,
3
,
246
257
, https://doi.org/10.1080/10899995.2018.1483119.
Milrad
,
S. M.
, and
C. G.
Herbster
,
2017
:
Mobile radar as an undergraduate education and research tool: The ERAU C-BREESE field experience with the Doppler on Wheels
.
Bull. Amer. Meteor. Soc.
,
98
,
1931
1948
, https://doi.org/10.1175/BAMS-D-15-00281.1.
Morss
,
R. E.
, and
F.
Zhang
,
2008
:
Linking meteorological education to reality: A prototype undergraduate research study of public response to Hurricane Rita forecasts
.
Bull. Amer. Meteor. Soc.
,
89
,
497
504
, https://doi.org/10.1175/BAMS-89-4-497.
Mullendore
,
G. L.
, and
J. S.
Tilley
,
2014
:
Integration of undergraduate education and field campaigns: A case study from deep convective clouds and chemistry
.
Bull. Amer. Meteor. Soc.
,
95
,
1596
1601
, https://doi.org/10.1175/BAMS-D-13-00209.1.
Pascarella
,
E. T.
, and
P. T.
Terenzini
,
2005
: How College Affects Students: A Third Decade of Research. Jossey-Bass, 848 pp.
Quardokus
,
K.
,
S.
Lasher-Trapp
, and
E. M.
Riggs
,
2012
:
A successful introduction of research early in an undergraduate atmospheric science program
.
Bull. Amer. Meteor. Soc.
,
93
,
1641
1649
, https://doi.org/10.1175/BAMS-D-11-00061.1.
Radunzel
,
J.
,
K.
Mattern
,
J.
Crouse
, and
P.
Westrick
,
2015
: Development and validation of a STEM benchmark based on the ACT STEM score. ACT Tech. Brief, 7 pp., www.act.org/content/dam/act/unsecured/documents/2015-Tech-Brief-Development-and-Validation.pdf.
Roebber
,
P. J.
,
2005
:
Bridging the gap between theory and applications
.
Bull. Amer. Meteor. Soc.
,
86
,
1641
1649
, https://doi.org/10.1175/BAMS-86-4-507.
Sackett
,
P. S.
,
N. R.
Kuncel
,
J. J.
Arneson
,
S. R.
Cooper
, and
S. D.
Waters
,
2009
:
Does socioeconomic status explain the relationship between admission tests and post-secondary academic performance?
Psychol. Bull.
,
135
,
1
22
, https://doi.org/10.1037/a0013978.
Sloan
,
V.
,
R.
Haacker
,
T.
Barnes
, and
C.
Brinkworth
,
2017
:
Long-term impacts of a career development workshop for undergraduates
.
Bull. Amer. Meteor. Soc.
,
98
,
1961
1967
, https://doi.org/10.1175/BAMS-D-15-00214.1.
Tinto
,
A.
,
1975
:
Dropout from higher education: A theoretical synthesis of recent research
.
Rev. Educ. Res.
,
45
,
89
125
, https://doi.org/10.3102/00346543045001089.
Westrick
,
P. A.
,
L.
Huy
,
S. B.
Robbins
,
J. M. R.
Radunzel
, and
F. L.
Schmidt
,
2015
:
College performance and retention: A meta-analysis of predictive validities of ACT scores, high school grades, and SES
.
Educ. Assess.
,
20
,
23
45
, https://doi.org/10.1080/10627197.2015.997614.
Windham
,
T. L.
,
A. J.
Stevermer
, and
R. A.
Anthes
,
2004
:
SOARS: An overview of the program and its first 8 years
.
Bull. Amer. Meteor. Soc.
,
85
,
42
47
, https://doi.org/10.1175/BAMS-85-1-37.
For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).