Integrating Cloud and Container Technologies into University Numerical Weather Prediction (NWP) Curriculum
| What: | University faculty joined the Developmental Testbed Center (DTC) for a live, virtual 3-day workshop to learn about the capabilities of the DTC’s containerized end-to-end NWP system on the cloud and to discuss challenges of incorporating NWP into their curricula. Attendees participated in hands-on practice and breakout discussions sessions to assess the merits of using the system, including consideration of improvements or changes needed to make the use of containers and cloud technologies feasible for teaching NWP in their respective classrooms. |
| When: | 7–9 June 2021 |
| Where: | Held virtually (https://dtcenter.org/events/2021/integrating-cloud-container-technologies-university-numerical-weather-prediction-nwp-curriculum) |
According to the American Meteorological Society (AMS), principles of numerical weather prediction (NWP) is a topic that should be addressed within required courses in order to obtain an undergraduate degree in atmospheric sciences (www.ametsoc.org/ams/index.cfm/about-ams/ams-statements/statements-of-the-ams-in-force/bachelor-s-degree-in-atmospheric-science/). The context by which NWP (which is further defined by the AMS to include data assimilation, forecast, statistical postprocessing, and dissemination) is presented may take on a variety of forms such as gaining theoretical knowledge through textbooks or experiential learning through development of 1D or idealized models. In the end, undergraduate exposure to NWP varies significantly between university programs for a number of reasons, in large part because NWP is resource- and knowledge-intensive, and specific expertise necessary to develop and maintain these systems is limited. However, integrating NWP into upper-level course work for undergraduates or throughout graduate studies is important as we consider educating our future workforce in meteorology. Until recently, integrating a full 3D numerical model into an undergraduate setting that can be used for critically evaluating the output and understanding the full atmospheric system may have seemed like an unrealistic goal. With the advent of containerization software and cloud computing, this option has become achievable.
The Developmental Testbed Center (DTC) leveraged and built upon the work initially undertaken by the Big Weather web project (http://bigweatherweb.org/) and detailed in Hacker et al. (2017) to establish a full, end-to-end containerized NWP system that can be run on a local workstation or on the cloud. The DTC system (https://dtcenter.org/community-code/numerical-weather-prediction-nwp-containers) is based on the Weather Research and Forecasting (WRF) Model, and includes a full set of components for running an NWP system from start to finish: WRF Pre-Processing System (WPS), Advanced Research WRF (ARW; Skamarock et al. 2019) model, Gridpoint Statistical Interpolation (GSI; Shao et al. 2016) data assimilation, Unified Post-Processor (UPP), Python plotting, Model Evaluation Tools (MET; Brown et al. 2020), and METviewer. Container technology allows for the complete software system (including the operating system, libraries, code, etc.) to be bundled and shipped to users in order to skip the arduous software compilation and installation process, reducing spinup time and leading to a more efficient setup process. Cloud computing can provide compute resources to resource-limited institutions that might otherwise be unable to use these technologies. These combined tools of software containers and cloud computing represent an innovative teaching tool to integrate hands-on learning directly into NWP course curricula in order to foster an interest and excitement in NWP while educating the next-generation workforce.
Motivated by recent successful collaborations with university faculty to incorporate the DTC end-to-end containerized system into their classrooms, the DTC held a virtual workshop in June 2021. The workshop was tailored to the academic community, especially university faculty interested in incorporating hands-on learning opportunities into new or existing NWP curricula. To that end, a total of 10 universities from across the United States and three international institutions were represented at the workshop. The workshop had three main objectives: 1) inform participants about the current capabilities the DTC has established, 2) discuss how these tools/technologies may be integrated into NWP curriculum, and 3) consider what improvements could be made to enhance the full system for academic needs in the future. To facilitate the accomplishment of these objectives, the workshop content began with a focus on describing the DTC’s end-to-end containerized NWP system and provided hands-on practical experience by allowing participants to run the system on a cloud instance prepared by the DTC staff. Subsequent breakout session discussions allowed for initial impressions of the tools to be shared while seeding thoughts on classroom accessibility. To set the stage, a couple of participants offered an overview of what their NWP curriculum currently includes, providing some examples on current approaches. Testimonials from two other professors who have integrated the DTC tools in their classroom were provided to the participants as well, in an effort to share experiences using these tools. These presentations led to breakouts directed at explicitly discussing what the utility of the DTC tools in a classroom might be and understanding how they could best be integrated into existing NWP curriculum. As the workshop progressed, open dialogue continued with specifics related to containerization software, cloud compute resources, and customization aspects that were of interest to the participants. Numerous ideas for future enhancements were also gathered as a general familiarity with the current system was gained.
Several high-level themes came out of the presentations and breakout sessions during the workshop and are briefly summarized below.
Benefits of containerization and cloud computing
Using containerized components allows the system to easily be ported to any platform (physical or cloud) and be executed in a similar manner. This was recognized as highly advantageous despite the need to learn another software syntax, and use of containerization was encouraged. While some universities have on-premises hardware that could be utilized for executing the end-to-end system, providing information on how to launch these tools on the cloud was very well received and opens the doors for small programs that do not have substantial local compute resources. Of course, the financial cost of cloud computing cannot be dismissed, and workshop participants were provided with specific information related to Amazon Web Services (AWS) opportunities through the AWS Academy (https://aws.amazon.com/training/awsacademy/) in that regard. As an added bonus, it was highlighted that using new tools/technologies such as containers and cloud computing provides the opportunity for students to learn additional technical skills that make them more marketable when looking for job opportunities.
Need for training the trainers
In an effort to simplify the learning curve for students to more quickly and easily run a complete NWP simulation, the DTC end-to-end containerized NWP system uses a number of tools that, on their own, also need to be understood. For professors to be comfortable with teaching the material in their classroom, they first need to develop a level of expertise with the system as a whole and the technologies it is based on, and spinup time to learn these technologies must be factored into the course work preparation. Given this, a strong desire was expressed during the workshop for a variety of training materials to be prepared and offered that provide background knowledge of each tool and component. This material could be in the form of in-person tutorials, prerecorded instructional videos, and online documentation. While these materials may eliminate the need for in-depth content presented to each class by DTC staff, some collaboration and interaction between the DTC and the students is still thought to be advantageous.
Desire for customization
At the present time, three case studies are provided within the DTC system with the option to customize the system to run any case of interest. It was noted that certain university courses may have a particular application of interest (hurricanes, fire weather, severe storms, etc.) and enhancing the tools and documentation to allow for students to easily adapt the system to different topic areas is highly desirable and may increase enthusiasm and engagement. By doing this, it will naturally encourage contribution of new cases back into the DTC system, which is made possible through the open and collaborative nature of the GitHub repository for this project (https://github.com/NCAR/container-dtc-nwp). Another point raised for consideration is the need to cater the material covered in the class to the appropriate education level. What is covered at various levels of undergraduate courses may be substantially different from each other and, further, from that of a graduate course. Having the ability to customize the content is critical to making it applicable and usable in a variety of courses.
Areas for further consideration
A few aspects were identified as still needing attention so that they do not cause major hurdles for integration into the classroom. These include providing detailed instructions for a variety of platforms, as small differences in behavior between an operating system (e.g., MacOS, Linux, Windows) can lead to frustrating challenges for students to overcome. Additionally, by default, elements of the DTC end-to-end containerized NWP system can remain as a “black box.” In some cases (at the undergraduate level) this can be a benefit, while in others (at the graduate level) this may hinder a firm understanding and further investigation into a specific process. Ensuring students have clear documentation to be able to go into the code base, make a change, and recompile is an important feature that should be highlighted to facilitate a deeper understanding, as needed. Finally, as with any course in the atmospheric sciences curricula, prerequisites must be carefully considered in an effort to maximize understanding and ultimate success when using the system.
Future opportunities to connect
The workshop ended with participants expressing interest in future collaborations both in the classroom and beyond by holding short courses at AMS or special sessions in the education conference at AMS or American Geophysical Union (AGU) meetings. The workshop organizers and authors of this meeting summary encourage anyone interested in pursuing a collaboration in this area to please reach out, as we would be excited to build new connections and evolve the DTC end-to-end containerized NWP system in a way that is meaningful and applicable to the academic community.
Acknowledgments.
The co-authors would like to thank Dr. Mike Baldwin and Dr. Julie Lundquist for participating in the workshop and reading this summary to verify its accuracy. This workshop was also summarized in the August 2021 issue of the Developmental Testbed Center newsletter, Transitions.
References
Brown, B. , and Coauthors, 2020: The Model Evaluation Tools (MET): More than a decade of community-supported forecast verification. Bull. Amer. Meteor. Soc., 102 (4), E782–E807, https://doi.org/10.1175/BAMS-D-19-0093.1.
Hacker, J. P. , J. Exby , D. Gill , I. Jimenez , C. Maltzahn , T. See , G. Mullendore , and K. Fossell , 2017: A containerized mesoscale model and analysis toolkit to accelerate classroom learning, collaborative research, and uncertainty quantification. Bull. Amer. Meteor. Soc., 98, 1129–1138, https://doi.org/10.1175/BAMS-D-15-00255.1.
Shao, H. , and Coauthors, 2016: Bridging research to operations transitions: Status and plans of community GSI. Bull. Amer. Meteor. Soc., 97, 1427–1440, https://doi.org/10.1175/BAMS-D-13-00245.1.
Skamarock, W. C. , and Coauthors, 2019: A description of the advanced research WRF Model version 4. NCAR Tech. Note NCAR/TN-556+STR, 145 pp., https://doi.org/10.5065/1dfh-6p97.
