Abstract

Numerical weather prediction models have become widespread tools that are accessible to a variety of communities, ranging from academia and the national meteorological services to commercial weather providers, wind and solar energy industries, and air quality modelers. Mesoscale meteorological models that are used to refine relatively coarse global weather forecasts to finer atmospheric scales have become mainstream. The wide use of mesoscale meteorological models also generates new requirements in undergraduate education concerning the knowledge and application of these models. In this paper, we present teaching strategies, course outcomes, student activities, impacts, and reflections on the possible future direction of the graduate-level atmospheric modeling course using the Weather Research and Forecasting (WRF) Model at Wageningen University, the Netherlands. This information is based on 15 years of experience in teaching the course and the continuous implementation of new educational techniques to adapt to students’ needs and improve their chances in their academic careers and the atmospheric sciences job market.

An active learning approach is a successful method in teaching atmospheric modelling of the atmospheric environment at the master’s level.

Numerical weather prediction (NWP) has rapidly developed from basic single-layer barotropic models in the 1950s to very advanced high-resolution Earth system models. Bauer et al. (2015) explained in detail why weather forecasting has undergone a key silent revolution in society, where the current-day global models show skill for lead times up to 7 days.

Despite the rapid developments in computing power, physical understanding, and data-assimilation techniques, global models are still unable to resolve finescale atmospheric mesoscale flows. For example, models cannot resolve the meso-β (20–200 km) scale of phenomena like sea breezes and lake-effect snow storms or the meso-γ (2–20 km) scale of phenomena like deep convection, complex orographically or thermally driven flows, urban heat islands, or coastal jets. Despite rapid grid refinement in global models, there remains an obvious need for spatial refinement as well in order to understand and forecast small-scale spatiotemporal meteorological features. Mesoscale meteorological models have fulfilled this task and have undergone a rapid proliferation at universities, operational weather centers, and in the commercial sector. For example, several consortia in Europe, such as the HIRLAM-ALADIN and COSMO communities, have developed a high-resolution limited area model for forecasting purposes (Baldauf et al. 2011; Bengtsson et al. 2017). In the United States, the fifth-generation Pennsylvania State University–NCAR Mesoscale Model (MM5; Dudhia and Bresch 2002), the Weather Research and Forecasting (WRF) Model (Powers et al. 2017), the Advanced Regional Prediction System (Xue et al. 2003), and the Regional Atmospheric Modeling System (Pielke et al. 1992) have been developed and are still in use. Wide user communities of these models have emerged in recent years (Dudhia 2014). For instance, the WRF community consists of about 30,000 users today (Powers et al. 2017).

The wide application and the power of these models have changed the way meteorology is taught. On one hand, students have to be prepared to understand and apply these model infrastructures. On the other hand, mesoscale models are very good teaching tools to integrate and advance students’ understanding of atmospheric dynamics and physics. A decade ago, Orf et al. (2007) presented the results of a workshop where educators discussed the role of models as educational tools, since these models had already been established as research tools in meteorology, climatology, oceanography, and other Earth sciences (Hoskins 1983; Keyser and Uccellini 1987). At the time, traditional curricula in geosciences had not yet taken advantage of the availability of models as educational tools. Broader integration of models into science curricula were considered as beneficial for the future. The workshop underlined the benefit of sharing resources, software methodology, and lecture presentations so that others may use them in their own classes. Considering this, one can say that the Linked Environments for Atmospheric Discovery (LEAD) project was pioneering. It was a long-term project which involved compiling and running WRF models for classroom experiments at the undergraduate level. The project was largely developed by Millersville University (Meyers et al. 2007; Clark et al. 2008). The recent initiative of the Big Weather web makes big data infrastructure affordable and adequate for universities involved in numerical weather prediction by combining recent technologies of virtualization, cloud computing and storage, and big data management (Goebbert et al. 2019; http://bigweatherweb.org/Big_Weather_Web/Home/Home.html). Today, several universities offer courses in meteorological modeling, applying an active learning strategy (see the “Student activities and grading” section below): for example, Rob Fovell (www.atmos.albany.edu/facstaff/rfovell/ATM562/index.html), Gary Lackmann (www4.ncsu.edu/∼gary/mea716/), and van den Heever (2018).

In this paper, we present an overview of the graduate-level course in atmospheric modeling that we have offered for the past 15 years at Wageningen University (the Netherlands). We share the course learning objectives and outcomes, examples of student projects, and retrospective reflections of alumni on the impact the course has had on them and their careers. Three samples of student work are presented to demonstrate the skills students acquire through this course by formulating and performing their own research projects on modeling a mesoscale meteorological phenomenon. Here, we intend to offer the meteorological community an accessible vehicle to teach atmospheric modeling in a dynamic and activating format.

COURSE DESIGN.

Teaching philosophy, programmatic learning outcomes, and course outcomes.

Several courses for MSc (master of science) programs in Earth and Environment, Climate Studies, and Environmental Sciences offer a broad palette of courses within the domain of soil science, hydrology, and meteorology. The students admitted to these programs usually represent a diverse range of nationalities and cultures and have a strong educational background in environmental sciences, though not necessarily in physics. Students start their MSc with survey courses that introduce the programs, teach data collection and analysis skills using basic Python or R. As students progress in their MSc program, they specialize according to their preferences. Students within these MSc programs have the opportunity to follow courses on boundary layer processes, atmospheric dynamics, atmospheric chemistry and composition, as well as numerical representations of atmospheric and water flows. The atmospheric modeling course integrates knowledge and skills gained from these specialized courses and puts into practice how the physical and dynamical processes interact within the context of a model. The course endeavors not only to prepare students to model studies using a mesoscale meteorological model from a technical point of view, but also to prepare them to perform fundamental and applied research, for example, a MSc capstone thesis research project. Research skills are emphasized such as systematic and critical thinking, developing research questions based on scientific arguments, applying innovative model experiments or analysis strategies, and placing results within the current literature. Students are also expected to be able to hold discussions with instructors and fellow students as well as present their results in oral and written formats. Course outcomes are listed below. Upon successful completion of this course, students are able to

  • use and modify (for advanced students) atmospheric models currently used in research and meteorological and air quality institutes and consultancies;

  • address how modeling can assist in understanding socially relevant environmental problems such as daily weather forecasts, extreme weather, wind, energy, air quality, and flash floods;

  • design numerical experiments (sensitivity analysis) related to specific research questions and explain and discuss the principles and theory of atmospheric models from local to regional scales;

  • systematically integrate the knowledge of atmospheric processes obtained in previous courses and other disciplines, such as atmospheric dynamics, boundary layer processes, atmospheric chemistry, hydrology, or land–atmosphere interaction, and test physical and biochemical parameterizations and their impact on weather and air quality forecasts;

  • assess the potential applications of these models as well as their limitations and synthesize external input in their own research;

  • apply these models in real-world scenarios in order to understand and interpret meteorological and air quality phenomena;

  • evaluate model performance by comparing model results with field observations or other models; and

  • present and defend model results and their analysis that are related to concrete research questions.

INGEBORG SMEDING (ALUMNA) METEOGROUP, WAGENINGEN, THE NETHERLANDS

“After high school, it was clear to me that I wanted to study meteorology, but I was not sure about the university. In the Netherlands, there are two options: Utrecht and Wageningen. I selected Wageningen, as it was recognized to be the more applied university. In my view, the atmospheric modeling course is a perfect example of applied education. After my graduation project, where we used MM5 (predecessor of WRF) to model the sea breeze event in the Netherlands, I assisted in setting up the new atmospheric modeling course together with Jordi Vila. It was a perfect way to apply my recent experience with MM5, get some education experience and to work with students.

In 2005, I started working as a researcher at MeteoGroup (formerly known as Meteo Consult) in Wageningen. MeteoGroup is one of the world’s leading providers of full-service B2B weather solutions, operating wherever weather impacts business decision-making. As MeteoGroup was (and is) closely linked to Wageningen University, I was still involved in the course as tutor, advisor, and reviewer of reports and presentations. Later, we had several students at MeteoGroup for internships, which is still the case today. The big advantage of having students who attended the atmospheric modeling course at MeteoGroup is that they already have experience with running MM5/WRF and they can do research independently. This means that they can define a research question, make a research plan on how to execute the study, write a report, and present their results. Some of these students even became my colleagues. The impression we got from these students told us so much more than a normal job interview would have done!

The WRF knowledge in our company has a big competitive advantage. At conferences, the audience is frequently surprised that a commercial company has so much knowledge on mesoscale modeling. In 2017 the cooperation between Wageningen University and MeteoGroup resulted in the organization of a joint WRF training workshop at the yearly EMS conference. MeteoGroup has spent a lot of research effort to come up with the optimal settings for WRF (e.g., parameterizations, vertical level distribution) for different weather scenarios that typically occur in western Europe throughout the year. As a result, NCAR allowed MeteoGroup to use the name ‘MG incorporated WRF’ for its own model version. Even more, some model improvements developed by MeteoGroup have been integrated into the operational version of WRF. Some examples of what MeteoGroup provided to the WRF community: ECMWF model level initialization, land/sea/lake initialization, and derived parameters like maximum wind speed.

Over the years, MeteoGroup has performed several consultancy studies by means of WRF. The flexibility to tailor the model exactly to the customer’s needs is still a great unique selling point. For example, we did historical reruns for insurance companies in case of severe storms, wind climate studies, and air quality modeling. Today, MeteoGroup runs WRF twice a day at 3 km resolution for a big part of Europe to provide our forecasters with an additional high-resolution model.”

Student activities and grading.

The student activities in this course are built upon the idea of an active learning environment where, in principle, the students drive their own research projects. Zayapragassarazan and Kumar (2012) underlined that effective learning involves providing students with a sense of progress and control over their own learning. This requires a situation where students have the opportunity to test and develop their own ideas. This is achieved by connecting students’ ideas to concrete experiences and thus, these experiences become an “active” part of the learning process. Active learning offers students meaningful opportunities to discuss, listen, write, read, and reflect on the content, ideas, issues, and concerns related to an academic subject. Prince (2004) concluded that there is clear support for active, collaborative, and problem-based learning. The course uses a student-centered pedagogy in which students learn about a meteorological topic and to perform academic research through the experience of solving an open-ended problem. The learning process does not focus on obtaining a defined solution, but aims for developing desirable skills and attributes, that is, modeling skills, knowledge acquisition, and communication, and intends to enhance critical appraisal and literature retrieval. In terms of teaching strategy, the course builds upon the seven principles of good practice in undergraduate education (Chickering and Gamson 1987):

  • Encouraging contacts between students and faculty, which helps students to keep on working and overcome drawbacks. We achieve this by the presence of staff in the classroom during each class in order to discuss model results and strategy. A teaching assistant is also present to help with technical issues such as Linux problems, model compilation, and bug fixing.

  • Developing reciprocity and cooperation among students, where students are involved, share codes, and discuss each other’s work. This makes the study a collaborative and social activity. In our course, students researching related topics prepare questions together in order to interact with guest lecturers (see below). Hence, together they create awareness of the research topics of their peers and their joint modeling challenges, which often results in code sharing. Students also provide feedback on the final presentations of their peers.

  • Using active learning techniques, where students discuss and write about their learning activities and connect it to their experience and to daily life, thus making the material part of themselves. In our course, students write a research proposal for their project. Herein, students often choose topics that are either triggered by their fascination with a certain phenomenon or weather event that impacts their hometown or country (e.g., hurricanes and flash floods in vulnerable countries) or from field experience (e.g., tornado chasers and mountaineers). A student’s final presentation should state societal relevance of the studied topic, for which items from newspapers or social media feeds are often used.

  • Giving prompt feedback, where students receive timely and appropriate feedback on their performance and suggestions for the next steps in their activities. To facilitate this, a student’s modeling results are discussed daily. The results are compared with observations and synthesized. Once these steps are done, students and instructors decide what the next logical steps are in the model experiment, which are discussed the next day.

  • Emphasizing time on task, where students learn time management. This aspect is introduced by a number of deadlines for handing in a research proposal, an overview of model results, a presentation, and a report. During the daily discussion about results, students are asked to reflect on whether the planned simulations and analysis of results are feasible and available or whether the proposed research should be adjusted.

  • Communication of high expectations of the student project work. Students get motivated by ambitions set by the lecturers. Moreover, clarity about the expected level makes students aware of the efforts they are expected to put forth. During intake interviews a few weeks before the course starts, the lecturers communicate their expectations to the student and let the students know that they can expect frequent high-level input from the lecturers. We stimulate some students to write their reports in the form of a journal article or inspire them to publish their results in the magazine of the national meteorological society. This helps students to make sure their level of scientific writing and research approaches a certain maturity.

  • Respecting and promoting diverse talents and ways of learning and, as such, opening the opportunity for students to show their talents and work in their own personal way. The course facilitates different levels of complexity. Students use modern scientific computing languages for plotting and analyzing model outcomes, although simple spreadsheet tools are also offered to students with less developed computer skills. Moreover, for students who find a mesoscale model too complex to deal with, they can use a more simple bulk model for the atmospheric boundary layer instead.

More concretely, the course starts with 16 h of refresher and integrative lectures with the aim of reactivating the knowledge and skills students obtained in earlier courses in meteorology (see previous section and Fig. 1) and numerical methods (Introduction to Python and Water and Airflow Numerical Techniques). This is achieved through conducting practical sessions focused on land–atmosphere interactions, mass-flux parameterizations, and chemical mechanisms. More details on the 8-week course timeline are shown in Fig. 1. Given the inflow of a rather heterogeneous student population, the students’ knowledge and skills are brought to a similar level in the first week. In week 2, students spend 16 h being introduced to the principles of atmospheric modeling (Stensrud 2007; Warner 2011) and they learn how to work with state-of-the art atmospheric models, in this case, a compiled version of the WRF mesoscale model (MM5 until 2011, Fig. 1). Students are guided through the processes needed to run the model, that is, setting up model domains, model grid spacing, and geographic databases, reading and applying forcing files from external sources like GFS or ECMWF, and selecting physical parameterizations. To put into practice the pre- and postprocessing, the students perform a canned experiment on a sea breeze circulation as observed at the Dutch North Sea coast. Students are then guided through the available model output by a series of assignments that ask them to plot typical relevant meteorological parameters, such as 2-m temperature, 10-m wind speed, and direction. Different plotting formats are shown: temporal evolution, cross sections, and profiling. Students are asked to then compare the model results with WMO surface observations as well as the rich Cabauw research facility dataset (Beljaars and Bosveld 1997), particularly the surface energy and radiation balance, 213-m tower data, and ceilometer data.

Fig. 1.

Atmospheric modeling course schedule.

Fig. 1.

Atmospheric modeling course schedule.

GERT-JAN DUINE (ALUMNUS) UNIVERSITY OF CALIFORNIA, SANTA BARBARA, UNITED STATES

“By the time I was following the atmospheric modeling course, it was focused on the theory behind and use of the WRF Model. The course reflects, to a great extent, some of the strengths of the Wageningen University: putting theory into practice while being personally guided by the professors. I enjoyed the freedom you get as a student to follow your own interests and the course appeared to be very much the guidance for my academic career afterward.

The setup of the course is pretty straightforward, but flexible enough to challenge the students to show their creativity. Pretty quickly after a few lectures about common practices in mesoscale modeling, you get to choose your own case study. By setting up, running, and evaluating the model on its performance, the student gets a first glance on what it actually means to simulate real cases using numerical models. In a conference-like setting, you get to present and defend the findings in front of your fellow students but also the researchers affiliated with the Meteorology and Air Quality (MAQ) group. This is the scary part, but as it shapes you on your way to the master’s thesis, and even the PhD defense, it is perhaps also the most valuable part of the course. Together with the feedback given during the oral presentation, the course culminates with the writing of a report, from which some of these reports even turn into scientific papers.

As I was always intrigued by the development of nocturnal flows in mountainous terrain, I chose to simulate a nighttime drainage flow over the Balearic Islands. Retrospectively, my selected case study appeared to be very much related to the career path I followed, as my PhD subject was focused on the characteristics of down-valley winds in a similar kind of environment. Until now, I always stayed connected to the mountain meteorology community. So, personally, I consider the atmospheric modeling course as the starting point of my career.

By providing this course, the MAQ group has recognized in an early stage its importance for its future alumni. Although the course periods at WUR are relatively short (only 8 weeks), it is long enough to get yourself acquainted with the basics of numerical modeling on a mesoscale and to develop skills related to this. Whether it were to deal with technical issues, modifying parameters within the model, or just using the Linux environment; all proved to be valuable in my career. And, yes, even now, I sometimes open the course manual to freshen up my mind a little.”

After setup and testing WRF, students prepare a short (approximately 4 pages, workload 12 h) research proposal focusing on a mesoscale weather phenomenon of their choice. Normally, students formulate research projects inspired by a paper from the literature, which offers students a reference for expected model outcomes. Students also collect observational data either from routine observations as stored by NCEI, satellite products, soundings, or from special campaigns for model validation. The research proposal lists a number of proposed sensitivity studies, for example, variation of the land use maps, sea surface temperature, or utilized physical parameterizations.

After 4 weeks (note students work on the project only 4 hours per day) of student engagement by active learning in classroom when they perform their research, students give an oral presentation about their results (Fig. 1). To actively engage students throughout the course, we encourage students to formulate questions and discuss the work of their peers. The subsequent day, students obtain detailed peer feedback in a session where we discuss the successful aspects of all of the presentations. By doing so, fruitful elements of scientific presentations are collected and can be carried to subsequent courses. This also enables us to teach students to provide feedback in a constructive way. Finally, the students spend 20 h on completing their written report. In summary, the course provides all crucial elements needed for academic research, that is, formulating research questions, designing experiments, comparing model results against observations, discussing results within the context of existing literature, drawing conclusions, and finally, presenting the scientific results.

The course also informs students about their future prospects in the labor market. Three guest lecturers present how they use the mesoscale NWP models in their jobs. The first guest lecturer addresses how the Dutch national weather service (KNMI) utilizes their limited area model in refining global ECMWF forecasts to local conditions and how the model plays a role in the issuing of weather warnings, as an example of the use of mesoscale models in meteorological institutes. Over time, guest lectures have evolved from treating model initialization and verification to illustration of demanding computing infrastructure and 3D visualization of model output. The second guest lecturer focuses on model application for the interaction of the PBL with the coupled soil–vegetation system on cloud formation, as an example of the relevance of the physical process on weather forecasting. Initially, the lecture focused on the Netherlands though later on, a more international context, especially on West Africa, was added. The last guest lecturer presents a survey on the concepts of geostrophic balance and thermal wind balance in the context of the baroclinic life cycle of low pressure systems. This lecturer also gives a historic overview of the representation of these low-pressure systems within NWP models in the post-war era as an example to integrate theoretical aspects of dynamic meteorology in the analysis of weather forecasting. This lecture initially dealt with stability indices for convection and has evolved to a model illustration of cyclogenesis, frontogenesis, and Q vectors.

To better embed the guest lectures and activate student participation, students are separated into subgroups and assigned to each guest lecturer to discuss the presented material. The subgroups are composed such that the student’s research topic relates well to the theme of the guest lecturer’s subject. The subgroup meets in advance of the lecture to prepare the questions which are then discussed during or directly after the guest lecture. This setup appears to be very effective since the students actively acquire knowledge that they can directly apply to their research project.

The students are graded in three areas, using a rubric (see supplemental material at https://doi.org/10.1175/BAMS-D-17-0166.2):

  • Research competency (40%). We evaluate whether students are able to formulate, direct, and execute their research project and can relate their results to the peer-reviewed literature.

  • Oral presentation (20%) that consists of a 10-min presentation followed by a 5-min question-and-answer discussion with their peers and professors.

  • A written report (40%) of a maximum of 20 pages with eight figures. Herein, we intend to train students on how to write an academic paper with limited space, allowing the students learn how to carefully choose which findings to report and select their figures and tables accordingly.

Here, we discuss the validity and reliability of the assessment in order to ensure that all students achieve the learning outcomes. Table 1 presents a consistency check of the formulated learning outcomes and the tasks the students have to perform and which are assessed. Clearly, all learning outcomes are achieved. Also, we ensure reliability of the assessment since the supervising lecturers independently grade the three aspects and come to a consensus on the final grade while exchanging arguments from each other’s assessment report.

Table 1.

Consistency table showing the course learning outcomes and student tasks for the atmospheric modeling course.

Consistency table showing the course learning outcomes and student tasks for the atmospheric modeling course.
Consistency table showing the course learning outcomes and student tasks for the atmospheric modeling course.

COURSE RESULTS: A RETROSPECTIVE LOOK.

General results and statistics.

Since this course was first offered as Atmospheric Modeling in 2003, 182 students from a wide variety of educational backgrounds, interests, and nationalities have learned how to develop and carry through a research project using a mesoscale model. Ideally, the group size would be 15 or less in order to enable a high-level of interaction and discussion. On average, the group size is 12, with a minimum of 4 students and maximum of 18 students. Figure 2 shows the variety of meteorological phenomena studied by the students. About 21% of the students explore a certain mesoscale wind pattern governed by contrasts in land/sea mask, land use and/or orography. Typically, these include land and sea breezes but they also include very locally known flows such as the Santa Ana winds in the United States, the Meltemi winds over the Mediterranean Sea, and chinook flows. Boundary layer studies are also very frequent, which aligns with the research focus of boundary layer phenomena at our university (e.g., Steeneveld et al. 2015). Thus, several students examine the “golden days” at the Cabauw research tower, the GABLS3 case study (Bosveld et al. 2014), or other observational campaigns the meteorology department has taken part in, for example, Boundary-Layer Late Afternoon and Sunset Turbulence (BLLAST) in the Pyrenees (Lothon et al. 2014). Students show a strong interest in understanding and forecasting extreme weather, especially squall lines, derechos, blizzards, fog, and hurricanes/typhoons.

Fig. 2.

Pie chart of atmospheric themes and mesoscale flows simulated by students, totaling 182 case studies over the 15 years that the atmospheric modeling course has been offered.

Fig. 2.

Pie chart of atmospheric themes and mesoscale flows simulated by students, totaling 182 case studies over the 15 years that the atmospheric modeling course has been offered.

Figure 3 shows the geographical distribution of the weather phenomena that students have chosen to study over the past 15 years. About 1/3 of these projects focus on the Netherlands, which can be explained by the students connection with the local weather and recent severe weather events but also by local developments in infrastructure planning concerning the urban environment and wind farms. This illustrates the fact that the course often connects well to a student’s home environment. Moreover, the department’s main research lines focus on atmospheric boundary layers and the richness of the long-term observations from the Cabauw research facility (atmospheric thermodynamic profiles, radar, surface radiation and energy balance components, soil temperatures and moisture, and special cloud campaigns) and how it supports research projects in the home country. About 10% of the studies are performed in study areas in the United States. Typically, students perform research projects focused on severe weather such as derechos or reintensification of hurricanes or the lake effect snow over the Great Lakes or the Great Salt Lake. Research projects about weather phenomena in China are increasing, that is, research projects about fog over the Yellow Sea and typhoons have become more popular. Both weather phenomena are critical for society.

Fig. 3.

Total counts for location of atmospheric phenomena, according to country, for student-designed WRF simulations over the 15 years that the atmospheric modeling course has been offered.

Fig. 3.

Total counts for location of atmospheric phenomena, according to country, for student-designed WRF simulations over the 15 years that the atmospheric modeling course has been offered.

Examples of student work.

Below, we present three examples of executed research projects that illustrate the typical sensitivity studies on parameterizations, orography, and sea surface temperature, respectively. The examples are from the top 10% of the projects and are summaries of the original student reports.

Example 1: Finding Doris: From a classic split cold front to the gray zone problem (by student Lars van Galen).

Doris was a severe extratropical storm that hit the United Kingdom, the Benelux, and Denmark on 23 February 2017 and caused severe wind damage with gusts up to 155 km h–1 (Budnitz et al. 2018). This study investigated two research questions associated with storm Doris. The first question was whether WRF was able to reproduce the split cold front that characterized this storm and how the WRF forecast compared to the ECMWF forecast. The second question addressed the “gray zone problem” (Wyngaard 2004) and was to assess whether application of the cumulus parameterization at 3-km model grid spacing improved the simulation of the showers and stratiform precipitation. Although the gray zone problem has been researched extensively before (Honnert et al. 2011; Boutle et al. 2014), such a study has never been done for the synoptic environment described above.

To start with, Lars van Galen set up a model domain and resolution based on the scale of the phenomenon and current literature, followed by the selection of physical parameterizations. First, it was demonstrated that WRF was able to simulate the track of Doris well (Fig. 4), despite a few inconsistencies. The WRF simulation produced a delayed deepening of the cyclone, although the core pressure was modeled well. Furthermore, WRF simulated the storm somewhat too far to the south at the later stages of the simulation. Finally, the storm passage in WRF was ∼1 h too late compared to observations.

Fig. 4.

Modeled and observed (ECMWF/KNMI observational analysis) cyclone track of Doris from 0600 UTC 22 Feb 2017 to 1800 UTC 23 Feb 2017.

Fig. 4.

Modeled and observed (ECMWF/KNMI observational analysis) cyclone track of Doris from 0600 UTC 22 Feb 2017 to 1800 UTC 23 Feb 2017.

From the split cold front analysis it appeared that WRF outperformed the ECMWF model in terms of cloud top height. However, the timing of the split cold front was 1 h delayed in WRF, while the vertical structure of the split front was also well represented. This included a forward tilt of the θe profile with height, convergence associated with the upper front ahead of the surface front, and consequently, also upper-level cloudiness ahead of the surface cold front (Fig. 5).

Fig. 5.

Comparison of (a) observed infrared satellite imagery (source: www.sat24.com) and (b) infrared satellite imagery as simulated by WRF, at 0300 UTC 23 Feb 2017.

Fig. 5.

Comparison of (a) observed infrared satellite imagery (source: www.sat24.com) and (b) infrared satellite imagery as simulated by WRF, at 0300 UTC 23 Feb 2017.

WRF appears able to reproduce the spatial structure and location of the showers behind the surface cold front over the Netherlands, as well as the precipitation associated with the occlusion tip of the storm. Still, there were some notable differences between the runs with and without cumulus parameterization (CP). The simulation with CP underestimated the extent of showers over the Netherlands. The numerical experiment without CP captured the areal extent of the showers best. However, this simulation developed convective bands from the stratiform precipitation, while satellite observations indicate that these convective bands were absent. The numerical experiment with CP performed better by keeping the precipitation of stratiform nature. Thus, although applying a CP mattered in this study, it depended on the type of precipitation as to whether applying a CP affects the model results.

Example 2: Warm winds of change hit the Antarctic Peninsula (by student Marit van Tiel).

The Antarctic Peninsula, located in West Antarctica, has been one of the fastest warming regions on Earth over the past 50 years. The peninsula consists of a 1,500-km-long, 2-km-high mountain ridge oriented from north to south (Grosvenor et al. 2014). These mountains act as a climatic barrier between the warmer oceanic air of the west and the cold continental air of the east. As a result of this climatic barrier, a much stronger warming trend is seen in the seasons of austral summer and autumn on the east side than on the west side of this barrier. The summer warming, which causes more melt, is thought to be the main factor affecting the breakup of several ice shelves. The student formulated three potential mechanisms to explain the föhn (Elvidge 2013):

  1. Latent heat mechanism: On the windward side of the mountain, clouds and precipitation are generated, the precipitation rains out, and warm air descends on the lee side of the mountain.

  2. Isentropic drawdown mechanism: Low-level flow will be blocked by the mountain and not able to pass over the mountain. Air with a relatively high potential temperature from higher levels upwind will then descend on the lee side of the mountain.

  3. Warm source region mechanism: Warm air is already flowing over the mountain and mixes potentially warmer and drier air when passing the mountain top, resulting in warm air descending on the lee side.

To predict or to prevent the loss of the last part of the Larsen ice shelf, it is important to investigate what this flow looks like, what the conditions are when this föhn occurs, and what effect the topography has on this airflow. Therefore, Marit van Tiel formulated the following two research questions:

  • Can WRF reproduce the föhn wind of 6 January 2006?

  • What is the influence of topography on the föhn wind?

Marit van Tiel’s assessment based on a statistical model evaluation (Willmott 1982) indicates that WRF reproduces the föhn wind of 6 January 2006 rather well, though only for a few parameterization settings. The role of microphysics scheme is critical and determines the maximum wind speed area in the jet. WRF showed that the föhn effect is present as gap flows (Fig. 6). Three jets are formed with the strongest jet at 1000 UTC. The flow is blocked and the mechanism causing the föhn is isentropically drawn down (Fig. 7). The temperature distribution shows high warming near the lee side of the mountain but not farther downwind. A temperature tendency during the model run in the middle of the jet also showed this; no heating effect at the surface where the jet occurs at a higher level. The hydraulic jump could be the reason behind this rapidly diminishing effect or WRF is simply not able to simulate the high temperatures on the Larsen ice shelf. The simulated föhn wind is substantially reduced for lowered orography, while a uniform orography does not show föhn jets anymore (Fig. 7).

Fig. 6.

Modeled spatial pattern of wind speed (m s–1) and direction for the Antarctic Peninsula for 0900 UTC 6 Jan 2006. Contour lines indicate the terrain height (m).

Fig. 6.

Modeled spatial pattern of wind speed (m s–1) and direction for the Antarctic Peninsula for 0900 UTC 6 Jan 2006. Contour lines indicate the terrain height (m).

Fig. 7.

Modeled wind speed and potential temperature pattern for (a) reference orography, (b) 50% reduced orography, and (c) orography cut off to 1 km for 0600 UTC 6 Jan 2006.

Fig. 7.

Modeled wind speed and potential temperature pattern for (a) reference orography, (b) 50% reduced orography, and (c) orography cut off to 1 km for 0600 UTC 6 Jan 2006.

Example 3: Tropical-like cyclones in the Mediterranean: An approach with the WRF Model (by student Aristofanis Tsiringakis).

Being Greek from origin, Aristofanis Tsiringakis was interested in studying medicanes, that is, warm-core cyclones that appear in the Mediterranean Sea only about once per year. The WRF Model was used to simulate the event and to study the sensitivity to sea surface temperature, orography, and boundary layer scheme. The final goal was to observe the changes that occur in the meteorological variable and connect them with physical effects and mechanisms. Comparing the reference run with observations indicates that WRF was able to represent the intensity and trajectory of the medicane reasonably well. The reference run showed characteristics of the medicane such as the warm core presence, the axisymmetrical cloud formation, the rotational winds around the eyewall, and a cloud-free eye for a period of 6 hours. The medicane prove to be very sensitive to changes of SST, mostly in intensity, but also in trajectory. A run with an SST raised by 5 K shows a much lower minimum pressure (Fig. 8) during the mature phase of the medicane. It also shows a slightly higher wind speed and almost triple the latent heat flux compared to the reference run, resulting in a much higher precipitation (Fig. 9).

Fig. 8.

Observed (diamonds) and modeled core pressure of (a) the medicane and (b) surface wind speed at a location south of Sicilia as simulated by the WRF Model.

Fig. 8.

Observed (diamonds) and modeled core pressure of (a) the medicane and (b) surface wind speed at a location south of Sicilia as simulated by the WRF Model.

Fig. 9.

Modeled accumulated precipitation from 0000 UTC 7 Nov until 0000 UTC 9 Nov in mm. (left) Reference run, (middle) no orography run, and (right) SST+5 run.

Fig. 9.

Modeled accumulated precipitation from 0000 UTC 7 Nov until 0000 UTC 9 Nov in mm. (left) Reference run, (middle) no orography run, and (right) SST+5 run.

In a second sensitivity experiment, Aristofanis Tsiringakis designed a new experiment in which the orography was removed over the land around the Mediterranean Sea. The changes in orography primarily affect medicane track (more northward) and changes the spatial distribution of rainfall. Finally, the simulated medicane appears very sensitive to the presence of a boundary layer scheme (not shown). Without a boundary layer scheme, WRF forecasts a lower core pressure, though the higher surface winds prevent cloud and precipitation formation and no medicane formed in this run.

Feedback from alumni.

Ultimately, the overarching goal of our MSc program is to prepare our students for a career in academia, consultancy or environmental/weather/climate decision-making in a government agency. In the sidebars throughout this article, three alumni now working in different institutes or companies reflect on the impacts of the atmospheric modeling course in their MSc program and on their career. In general, the alumni evaluated the course as very positive and influential for either the company or university where they are currently employed. They mention that the course is intensive and prepared them well for their later career. They are also of the opinion that the research and computing skills as well as the understanding of the model parameterizations acquired in the course have been useful for completing the MSc and PhD study. Overall, we conclude that the course meets a demand and is successful in the formulated course outcomes.

Figure 10a shows the distribution of grades obtained by the students on a one to ten scale, since 2008. The mean grade is 7.6, only three students failed the course while seven students scored a grade of 9 or higher. As such, the course success rate of 98% exceeds the typical score of 70% in other graduate courses in meteorology. This small failure rate is in part due to the requirements that we ask prior to taking the course. From independent student evaluations taken after the course, students rate the question “I am satisfied with this course” a 4.1 on a Likert scale of 1–5, with 1 indicating strongly disagree and 5 strongly agree (Fig. 10b), while the question “I learned a lot from this course” was rated 4.4 on the same scale. These results underline the students’ satisfaction with the course setup and learning outcomes.

Fig. 10.

(a) Histogram of student grades for the atmospheric modeling course (0–10 scale), and (b) course evaluation results (1–5 scale).

Fig. 10.

(a) Histogram of student grades for the atmospheric modeling course (0–10 scale), and (b) course evaluation results (1–5 scale).

Reflections on 15 years of teaching atmospheric modeling.

In this section, we reflect on the evolution of the course during the last 15 years. At the very beginning of the course, about five students (mainly Dutch) were enrolled in the course yearly. With the introduction of the BSc/MSc system in Europe, the course attracted a wider audience and currently runs for typically 15–20 students. At the same time, the course has evolved to into a more international classroom where about 30% of the students are from abroad. Obviously, this has widened the number of topics studied. In addition, the computational resources have grown over time, which now allows students to perform more demanding runs than in the past. For instance, at the start of the course, nearly all students ran a case with four nested domains of 30 × 30 grid points and 28 levels for two days ahead. Now, runs at larger domains with 100 × 100 points and 50 levels are not uncommon. Moreover, the model infrastructure has evolved from MM5 up to 2007 and the subsequent introduction of WRF. Within the last two years, the WRF-Chem and the WRF single-column model have also been offered as research tools to the students. The broadness of the equipment available also led to student projects that more often are related to recent weather events such as forest fires, heat waves, etc. As such, a better connection with societal relevance has been achieved. In addition, in the MM5 era, students mostly visualized their model output (time series and vertical profiles) using spreadsheets and NCAR’s RIP (Read/Interpolate/Plot) software. Nowadays, mostly the NCAR Command Language (NCL) and Python are used, which has advanced the visualization options and has simplified the plotting procedures for the students. Finally, the computer infrastructure used for the course has evolved from an in-house built network of desktop machines with a serially compiled model version, to a modern high-performance computer cluster allowing for parallel computing. Hence, students now also learn to estimate computer resources with respect to their mesoscale model settings.

ROGIER FLOORS (ALUMNUS) DTU, RISO LABORATORY, DENMARK

“From 2004 to 2007 I did my bachelor’s degree in Soil, Water and Atmosphere at Wageningen University & Research. I knew that I wanted to specialize in meteorology, so I took the mesoscale modeling course as a ‘free choice’ course during my BSc. The course was using the MM5 model, and for me it was the first ‘hands-on’ experience with numerical weather prediction, a large FORTRAN codebase, and a computer cluster.

These are still things I work with today, because I am working as a researcher at the Danish Technical University, where I do research to improve wind resource assessments. We frequently use the successor of the MM5 model, the WRF Model, to predict the wind on kilometer scale and combine it with microscale models that can take into account local speed-up effects due to surface roughness and elevation.

Traditionally, wind resource assessments are done by performing measurements at the site of interest and extrapolating these to the turbine locations, which requires a detailed description of, e.g., topography and wake effects. Over the years the wind energy industry has started to adopt mesoscale modeling more and more due to the increasing computer power, where the model grid spacing and the wind turbine rotor diameter are slowly approaching each other. Every time the grid spacing is changed, it is useful to be aware how processes are parameterized and which part of that process is going to be resolved by the model.

Furthermore, wind turbines are always exposed to meteorological phenomena, so issues like icing, corrosion, gusts, and terrain-induced extreme conditions come up frequently and require knowledge about meteorology. So it was useful that I was exposed to theoretical and practical work related to mesoscale modeling already 13 years ago.”

From the didactic point of view, in the last four years, we have had to interview registered students. Since students from a wide range of backgrounds are interested in taking the course, we have needed to put more effort into guiding students in whether or not they should take the course. This means that some students decide to quit the course or attend a later edition when they are better prepared and armed with the knowledge from other MSc courses.

Compared to other atmospheric modeling courses (see the first section), all the courses implemented active learning and taught students to take responsibility in their research projects. Differences appear in the research themes. Van den Heever (2018) mostly focused on modeling severe storms, though combined the modeling with teaching of observational methods and strategies. On the other hand, modeling in the Fovell (2018) course focused more on theoretical aspects than modeling real world cases. Concerning the presentation of model results, students recorded presentations in the course by Lackmann (2018), which have now been shared online.

Concerning future developments, we aim to offer the WRF version of the course with data assimilation (WRF-DA) within a few years. WRF-DA is getting more and more popular for preparing initial conditions as well as for regional reanalysis projects (Knox 2018), and therefore it will be a valuable instrument for our course as well.

In addition, so-called scale-aware parameterizations are becoming more and more available in WRF (e.g., Shin and Hong 2013). These parameterizations have been designed such that their relative impact on the tendencies reduces for finer grid cells. With the increasing computing power, and model resolution, we foresee that scale-aware parameterizations will become more mainstream and as such, students should become familiar with this new philosophy.

Moreover, Hacker et al. (2017) have shown the value of software container technology for NWP research and education. Container technology has profound implications for education and research in numerical weather prediction. A container is a software-based packaging and distribution tool that collects all elements and dependencies of a Linux-based application. Containers store the run-time environment together with one or more applications for ease of transportation and installation.

With potentially growing student populations, we may need additional infrastructure that supports a more easy model interface and management. Recently, the GIS4WRF plugin for QGIS was released (Meyer and Riechert 2018), which allows for the complete steps for WRF simulation in a more visual manner. With GIS4WRF, everything from downloading input data, preprocessing input data, running simulations, visualizing, and finally postprocessing results can be done.

CONCLUDING REMARKS.

Fifteen years of experience in forming mesoscale model users is presented in detail. We want to form model users and future developers with a critical and constructive attitude that can help to improve model development, performance and evaluation. Creating an active learning environment and applying hands-on methods, MSc students with diverse geoscience backgrounds learn to execute all aspects of a research project using a mesoscale model in only 8 weeks. They follow a step-by-step interactive approach that encompasses the following aspects: formulating a research question(s) or hypothesis, performing model simulations, comparing model outcomes with observations, synthesizing the results with respect to the literature, confirming or refuting the hypothesis, and finally, reporting the results in an oral presentation and a written report. Here, we have presented the course outline, teaching strategy, and outcomes of a course in atmospheric modeling that anticipates the wide introduction of mesoscale meteorological models in the field of academic research and applications. We show how student knowledge and research skills can be successfully activated by performing a research project using a mesoscale model, as illustrated by the three examples. Finally, alumni indicate that this course was very important for their careers after graduating from their MSc program. In the supplementary material, we share lecture slides, samples of student work (research proposal, reports, and presentation), the course manual, and the rubric used for a evaluating student’s work.

ACKNOWLEDGMENTS

We thank Bert Holtslag (Wageningen University), Ben Wichers Schreur (KNMI), and Aarnout van Delden (IMAU, University Utrecht), who act as guest lecturers for this course. Also, we are grateful to Kees van den Dries (Wageningen University) for his technical support for this course since the beginning and to former MSc and PhD students Marlous Jonker, Folmer Krikken, Marina Sterk, Natalie Theeuwes, Michał Kłeczek, Peter Kalverla, and Lars van Galen, who acted as teaching assistants in one or more editions of the course. We thank Wim van den Berg, Ingeborg Smeding (both MeteoGroup), and Michiel van der Molen (Wageningen University) for their efforts in setting up the course and for providing research case studies. Finally, we thank our colleague Ellen Torfs (Education Support Centre, Wageningen University) for her input on the manuscript.

REFERENCES

REFERENCES
Baldauf
,
M.
,
A.
Seifert
,
J.
Förstner
,
D.
Majewski
,
M.
Raschendorfer
, and
T.
Reinhardt
,
2011
:
Operational convective-scale numerical weather prediction with the COSMO model: Description and sensitivities
.
Mon. Wea. Rev.
,
139
,
3887
3905
, https://doi.org/10.1175/MWR-D-10-05013.1.
Bauer
,
P.
,
A.
Thorpe
, and
G.
Brunet
,
2015
:
The quiet revolution of numerical weather prediction
.
Nature
,
525
,
47
51
, https://doi.org/10.1038/nature14956.
Beljaars
,
A. C.
, and
F. C.
Bosveld
,
1997
:
Cabauw data for the validation of land surface parameterization schemes
.
J. Climate
,
10
,
1172
1193
, https://doi.org/10.1175/1520-0442(1997)010<1172:CDFTVO>2.0.CO;2.
Bengtsson
,
L.
, and
Coauthors
,
2017
:
The HARMONIE–AROME model configuration in the ALADIN–HIRLAM NWP system
.
Mon. Wea. Rev.
,
145
,
1919
1935
, https://doi.org/10.1175/MWR-D-16-0417.1.
Bosveld
,
F. C.
, and
Coauthors
,
2014
:
The GABLS third intercomparison case for model evaluation, Part B: SCM model intercomparison and evaluation
.
Bound.-Layer Meteor.
,
152
,
157
187
, https://doi.org/10.1007/s10546-014-9919-1.
Boutle
,
I. A.
,
J. E.
Eyre
, and
A. P.
Lock
,
2014
:
Seamless stratocumulus simulation across the turbulent gray zone
.
Mon. Wea. Rev.
,
142
,
1655
1668
, https://doi.org/10.1175/MWR-D-13-00229.1.
Budnitz
,
H.
,
L.
Chapman
, and
E.
Tranos
,
2018
:
Better by bus? Insights into public transport travel behaviour during Storm Doris in Reading, UK
.
Weather
,
73
,
54
60
, https://doi.org/10.1002/wea.3058.
Chickering
,
A. W.
, and
Z. F.
Gamson
,
1987
:
Seven principles for good practice in undergraduate education
.
AAHE Bulletin
,
39
(
3
),
3
7
. https://files.eric.ed.gov/fulltext/ED282491.pdf.
Clark
,
R. D.
,
S.
Marru
,
M.
Christie
,
D.
Gannon
,
B.
Illston
,
K. K.
Droegemeier
, and
T.
Baltzer
,
2008
:
The LEAD-WxChallenge Pilot Project: Enabling the Community. 24th Conf. on IIPS, New Orleans, LA
,
Amer. Meteor. Soc., 5B.6
, https://ams.confex.com/ams/88Annual/techprogram/paper_134261.htm.
Dudhia
,
J.
,
2014
:
A history of mesoscale model development
.
Asia-Pac. J. Atmos. Sci.
,
50
,
121
131
, https://doi.org/10.1007/s13143-014-0031-8.
Dudhia
,
J.
, and
J. F.
Bresch
,
2002
:
A global version of the PSU–NCAR mesoscale model
.
Mon. Wea. Rev.
,
130
,
2989
3007
, https://doi.org/10.1175/1520-0493(2002)130<2989:AGVOTP>2.0.CO;2.
Elvidge
,
A. D.
,
2013
:
Polar föhn winds and warming over the Larsen C Ice Shelf, Antarctica
.
Ph.D. thesis, University of East Anglia
,
248
pp., https://core.ac.uk/download/pdf/29111919.pdf.
Goebbert
,
K. H.
,
J. T.
Allen
,
V. A.
Gensini
, and
M.
Ramamurthy
,
2019
:
Data-driven scientific workflows: A summary of new technologies and datasets explored at the Unidata 2018 Workshop
.
Bull. Amer. Meteor. Soc.
,
100
,
ES97
ES99
, https://doi.org/10.1175/BAMS-D-18-0265.1.
Grosvenor
,
D. P.
,
J. C.
King
,
T. W.
Choularton
, and
T.
Lachlon-Cope
,
2014
:
Downslope föhn winds over the Antarctic Peninsula and their effect on the Larsen ice shelves
.
Atmos. Chem. Phys.
,
14
,
9481
9509
, https://doi.org/10.5194/acp-14-9481-2014.
Hacker
,
J. P.
,
J.
Exby
,
D.
Gill
,
I.
Jimenez
,
C.
Maltzahn
,
T.
See
,
G.
Mullendore
, and
K.
Fossell
,
2017
:
A containerized mesoscale model and analysis toolkit to accelerate classroom learning, collaborative research, and uncertainty quantification
.
Bull. Amer. Meteor. Soc.
,
98
,
1129
1138
, https://doi.org/10.1175/BAMS-D-15-00255.1.
Honnert
,
R.
,
V.
Masson
, and
F.
Couvreux
,
2011
:
A diagnostic for evaluating the representation of turbulence in atmospheric models at the kilometric scale
.
J. Atmos. Sci.
,
68
,
3112
3131
, https://doi.org/10.1175/JAS-D-11-061.1.
Hoskins
,
B. J.
,
1983
:
Dynamical processes in the atmosphere and the use of models
.
Quart. J. Roy. Meteor. Soc.
,
109
,
1
21
, https://doi.org/10.1002/qj.49710945902.
Keyser
,
D.
, and
L. W.
Uccellini
,
1987
:
Regional models: Emerging research tools for synoptic meteorologists
.
Bull. Amer. Meteor. Soc.
,
68
,
306
320
, https://doi.org/10.1175/1520-0477(1987)068<0306:RMERTF>2.0.CO;2.
Knox
,
J. A.
,
2018
:
15 Years of teaching atmospheric dynamics to undergraduates via an introduction to data assimilation course. 27th Symp. on Education
,
Austin, TX
,
Amer. Meteor. Soc.
,
8.3
, https://ams.confex.com/ams/98Annual/webprogram/Paper336503.html.
Lothon
,
M.
, and
Coauthors
,
2014
:
The BLLAST field experiment: Boundary-Layer Late Afternoon and Sunset Turbulence
.
Atmos. Chem. Phys.
,
14
,
10 931
10 960
, https://doi.org/10.5194/acp-14-10931-2014.
Meyer
,
D.
, and
M.
Riechert
,
2019
:
Open source QGIS toolkit for the Advanced Research WRF modelling system
.
Environ. Modell. Software
,
112
,
166
178
, https://doi.org/10.1016/j.envsoft.2018.10.018.
Meyers
,
E.
,
H.
Gadde
,
J.
Kurdzo
,
T.
Daley
,
J.
Vogt
,
R.
Junod
,
E. N.
Wiziecki
,
S.
Yalda
, and
R. D.
Clark
,
2007
:
Integrating LEAD research in education
.
TeraGrid 2007
,
Madison, WI, NSF
, www.researchgate.net/publication/228367038_Integrating_LEAD_Research_in_Education.
Orf
,
L.
,
G.
Lackmann
,
C.
Herbster
,
A.
Krueger
,
E.
Cutrim
,
T.
Whitaker
,
J.
Steenburgh
, and
M.
Voss
,
2007
:
Models as educational tools
.
Bull. Amer. Meteor. Soc.
,
88
,
1101
1104
, https://doi.org/10.1175/BAMS-88-7-1101.
Pielke
,
R. A.
, and
Coauthors
,
1992
:
A comprehensive meteorological modeling system—RAMS
.
Meteor. Atmos. Phys.
,
49
,
69
91
, https://doi.org/10.1007/BF01025401.
Powers
,
J. G.
, and
Coauthors
,
2017
:
The Weather Research and Forecasting Model: Overview, system efforts, and future directions
.
Bull. Amer. Meteor. Soc.
,
98
,
1717
1737
, https://doi.org/10.1175/BAMS-D-15-00308.1.
Prince
,
M.
,
2004
:
Does active learning work? A review of the research
.
J. Eng. Educ.
,
93
,
223
231
, https://doi.org/10.1002/j.2168-9830.2004.tb00809.x.
Shin
,
H. H.
, and
S.
Hong
,
2013
:
Analysis of resolved and parameterized vertical transports in convective boundary layers at gray-zone resolutions
.
J. Atmos. Sci.
,
70
,
3248
3261
, https://doi.org/10.1175/JAS-D-12-0290.1.
Steeneveld
,
G. J.
,
R. J.
Ronda
, and
A. A. M.
Holtslag
,
2015
:
The challenge of forecasting the onset and development of radiation fog using mesoscale atmospheric models
.
Bound.-Layer Meteor.
,
154
,
265
289
, https://doi.org/10.1007/s10546-014-9973-8.
Stensrud
,
D. J.
,
2007
: Parameterization Schemes: Keys to Understanding Numerical Weather Prediction Models. Cambridge University Press,
480
pp.
van den Heever
,
S. C.
,
2018
.
Understanding severe storm dynamics through the active development and use of modeling and observational tools
.
27th Symp. on Education
, Austin, TX, Amer. Meteor. Soc., 8.1, https://ams.confex.com/ams/98Annual/webprogram/Paper335692.html.
Warner
,
T.
,
2011
:
Quality assurance in atmospheric modeling
.
Bull. Amer. Meteor. Soc.
,
92
,
1601
1610
, https://doi.org/10.1175/BAMS-D-11-00054.1.
Willmott
,
C. J.
,
1982
:
Some comments on the evaluation of model performance
.
Bull. Amer. Meteor. Soc.
,
63
,
1309
1313
, https://doi.org/10.1175/1520-0477(1982)063<1309:SCOTEO>2.0.CO;2.
Wyngaard
,
J. C.
,
2004
:
Toward numerical modeling in the “terra incognita
.”
J. Atmos. Sci.
,
61
,
1816
1826
, https://doi.org/10.1175/1520-0469(2004)061<1816:TNMITT>2.0.CO;2.
Xue
,
M.
,
D.-H.
Wang
,
J.-D.
Gao
,
K.
Brewster
, and
K. K.
Droegemeier
,
2003
:
The Advanced Regional Prediction System (ARPS), storm-scale numerical weather prediction and data assimilation
.
Meteor. Atmos. Phys.
,
82
,
139
170
, https://doi.org/10.1007/s00703-001-0595-6.
Zayapragassarazan
,
Z.
, and
S.
Kumar
,
2012
:
Active learning methods
. NTTC Bulletin, Vol.
19
(
1
), National Teacher Training Centre, Puducherry, India, 3–5, http://jipmer.edu.in/sites/default/files/2581_2012_NTTC_Bulletin_2012_Mar.pdf.

Footnotes

A supplement to this article is available online (10.1175/BAMS-D-17-0166.2)

© 2019 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Supplemental Material