2021 Energy–Climate Hackathon
| What: | Approximately 40 participants—with expertise spanning energy, computer science, and weather and climate research—joined a week-long Energy–Climate data “hackathon” in a run-up to the 2021 UN COP26 conference. Six projects to promote and grow data and science understanding were initiated and developed by teams over the course of the week, supported by access to state-of-the-art computational resources and stimulated by keynote speakers from industry and academia. The hackathon concluded with teams presenting their output to a panel of invited experts. The conference was initiated by University of Oxford Dr. Sarah Sparrow, Professor David Wallom, Professor Tim Woollings, and University of Reading Professor David Brayshaw, Dr. Hannah Bloomfield, in partnership with the Met Office, the United Kingdom’s national meteorological service, and with support from the United Kingdom’s CEDA-JASMIN service and Gurobi optimization software. |
| When: | 18 May (half-day “scoping” event) and 21–25 June 2021 (main hackathon) |
| Where: | Online |
Decarbonization is driving rapid change in the production, transmission, and consumption of energy. Nonetheless, current pandemic recovery plans are projected to bring emissions to a new maximum in 2023 from which they will continue to rise with fossil fuels’ continued growth (IEA 2021). Investment into clean energy, especially in emerging and developing economies, is therefore required to keep emissions on a Paris-compliant trajectory.
Growing installations of renewable generation and the electrification of heating and transport are increasing the energy sectors’ exposure to weather and climate (Deakin et al. 2021) and the impact of growing weather sensitivity is further compounded by an uncertain and changing climate (Bloomfield et al. 2021a; Jaroszweski et al. 2021).
Accurate quantification of weather and climate risk on scales relevant to energy networks is therefore essential in supporting a rapid transition to low carbon and renewable energy systems, a key COP26 priority theme, though there remain many technical and scientific challenges (Bloomfield et al. 2021b).
The Energy–Climate Data Hackathon, initiated by the Universities of Oxford and Reading and supported by the Met Office, therefore sought to address three aims: investigate the risks and challenges facing future energy systems, to develop novel solutions to support exchange of both data and science understanding, and to foster the growth of an integrated community working at the interface of energy and climate research. The hackathon was open to everyone working in the fields of energy or climate, including from industry, policy, or academia. Participants’ experience spanned energy system science and modeling, climate science and modeling, software development, data analytics, and other mathematical and physical modeling backgrounds. Projects within the hackathon’s scope included developing decision support aids, producing improved solutions for data exchange between energy and climate science, model development, and creating novel end-user applications. Project initiators brought their ideas forward at an initial scoping meeting (18 May), with the aim of building a team to develop the projects over the course of the week-long hackathon (21–25 June).
Event structure
The hackathon was advertised from April 2021, with the choice to register as a participant or to put forward a project idea as a potential “team lead” (several potential team leads were also identified and approached directly by the organizing team). Team leaders were encouraged to discuss their ideas with the organizers in advance of the scoping meeting, and benefited from support in developing their initial ideas into suitable hackathon projects.
On the scoping day, participants were given an overview of the event, alongside technical information regarding the United Kingdom’s CEDA-JASMIN computing platform (Lawrence et al. 2013), which was provided for their use, and details regarding common datasets (e.g., meteorological data from the United Kingdom’s CEDA archive). A series of short talks showcasing existing tools and datasets available to use during the hackathon such as the Copernicus Climate Data Store (Copernicus 2021) or the Earth System Model Evaluation Tool (Eyring et al. 2020) were presented by external speakers. After a short break, all the team leads seeking to put forward a hackathon project were invited to give 3-min lightning talks to introduce their proposal.
Following the lightning talks, participants joined breakout rooms to further discuss the topic(s) in which they were interested. The scoping day concluded with a final plenary discussion, and details about how to prepare for the next steps of the hackathon. This included the team leads preparing a one-page “project proposal”: this detailed their anticipated software/data requirements as well as an expanded outline of the project concept.
Seven projects were presented on the scoping day. However, between then and the start of the hackathon, some groups working on similar ideas chose to merge, while other new groups formed based on discussions between participants. The one-page project proposals were also circulated to all participants in advance of the event enabling them to indicate their preferences for which project they wished to be involved in. The distribution of participants across projects was sufficiently well spread and balanced that all participants could join their first-choice team.
In the run-up to the start of the main hackathon event, participants joined a dedicated communications platform on Slack, which catalyzed the further development and discussion of project ideas. Technical assistance was given on preparing workstations with relevant tools (including SSH connection to the High Performance Computing resources hosted on the U.K. CEDA-JASMIN platform and use of Jupyter Notebooks; Kluyver et al. 2016).
Hackathon time was primarily taken up by dedicated “group hack time” (see Fig. 1 for the event timetable), during which participants worked in team groups or subgroups communicating via Slack channels and Zoom breakout rooms. Keynote talks were held on the first four days. These talks were as follows:
Climate risks in the U.K.—Prof. Nigel Arnell (University of Reading)
Contributions to global warming from individual greenhouse gases—Prof. Myles Allen (University of Oxford)
How weather and climate data are used at National Grid ESO—Dr. Dan Drew (National Grid)
The role of the financial sector in the energy–climate transition—Dr. Nicola Ranger (University of Oxford)
Hackathon timetable (BST time zone). The timetable was distributed to participants, who could each flexibly plan their own schedule to suit time-zone differences and other needs (e.g., work and/or caring responsibilities).
Citation: Bulletin of the American Meteorological Society 103, 5; 10.1175/BAMS-D-21-0305.1
Participants also heard from a representative of Meteomatics (a private weather service provider) on the first day, with an introduction to their weather data API.
Troubleshooting sessions were offered to participants on the second and fourth evenings (but were in practice little used). Brief update presentations held before lunch on days 2–4 gave teams the opportunity to share their progress, and exchange feedback and ideas across groups. Social events took place on Gather.Town, meeting on a remote “island” located in the digital ether.
The event concluded with final presentations on the afternoon of the day 5, recordings of which have been made available online (Oxford e-Research Centre 2021). A panel of invited experts commented on the merits and successes of the different projects, giving feedback on how they might be taken forward and feeding into the Met Office’s lead-up to COP26 (Met Office 2021). Support for open-access and open-source science was taken as a guiding principle for the hackathon with groups uploading code and data to repositories on GitHub (Energy–Climate Hackathon 2021). Teams additionally shared resources such as presentation slide PDF and tutorial documents through Google Drive, which have now been included in the main GitHub repository. A brief summary of each project is given below.
Projects and their outcomes
PV output forecasting using smart meter data of nearby PVs.
The premise of this project was that with the recent uptake of rooftop solar photovoltaics, more accurate very short-term forecasting techniques are needed to assist in load balancing within networks. The aim was to develop a forecasting technique that uses the output data of nearby PV systems to yield high-quality PV output forecasts of a center station. Improving forecasts in this way could create new business models for prosumers and enhance services of energy suppliers. This hackathon project used data from University of Reading sustainability services (Sustainability Services 2021) as well as crowd sourced data from local residents. Wind speed, wind direction, and incoming solar radiation were used as meteorological inputs. During the hackathon, a novel feature selection algorithm was proposed and combined with machine learning algorithms to produce the forecasts. The developed technique is currently being extended and validated with data from 202 rooftop PV systems from the “PV-sensor field” collected by Utrecht University (Elsinga and van Sark 2017). This project highlighted the ongoing challenges of needing high-quality open energy data and very high-resolution (sub–half hourly) meteorological data for the optimal operation of these algorithms.
Identifying key features of meteorological data relevant for power system studies.
Power system models perform complex calculations on a set of input parameters, to achieve a specific goal (e.g., minimizing the total cost of electricity, which includes both building and operating the generation equipment). Due to computational constraints, power system studies commonly only use singular representations of the parameters. The results obtained from the calculations are, however, found to be strongly sensitive to different choices of the input parameters (Schyska et al. 2021). The meteorological input, for instance, can have a substantial impact on the power system model results (Bloomfield et al. 2016; Schlott et al. 2018; Kies et al. 2021). This project compared the outputs from power system simulations using three EURO-CORDEX climate power model simulations spanning the whole twenty-first century (Jacob et al. 2014) to identify the key features of meteorological input data with relevance for the output of power system simulations.
The hackathon team developed advanced statistical comparisons of model simulations, including a very large covariance correlation matrix on many key input/output parameters. This highlighted some important similarities and marked differences between climate models. For example, a stark difference (sign flip) was seen in the temporally correlation of onshore wind capacity factor and solar capacity factor between models. Based on feature maps, links could be made between the simulation outcomes and different characteristics of the input data. The group is continuing with this work after the project, including more climate models and further analysis metrics to increase the robustness of the results.
Tutorials for using subannual data within pyam.
The open-source Python package pyam provides a suite of tools for analysis, processing, and visualization of output from integrated assessment models (Huppmann et al. 2021). The source code is available on GitHub (pyam 2021). The package already supports working with subannual time series data as used by several other projects in this hackathon. Alas, there are to date no tutorials for these use cases, which may restrict new users from benefiting from the full capabilities of pyam. As part of this hackathon project, a hands-on tutorial was held for interested participants, and several functions were implemented to improve performance (run times and memory usage) and improve the plotting module.
Developing methods to estimate subdaily energy relevant climate variables from daily data.
Power system model simulations are commonly performed using half-hourly or hourly input time series (such as regional electricity demand or renewable generation). The high temporal resolution is important for accurate estimates of power system properties like peak load or ramps in renewable power generation. As the weather dependence of power systems continues to increase, many power system modeling inputs can now be constructed using outputs from numerical weather prediction models and climate models. Unfortunately, most climate model datasets are only produced at temporal resolutions coarser than the hourly data required for power system models [e.g., the U.K. Climate Projections (UKCP) global and regional datasets provide output at daily resolution (Murphy et al. 2018) and the Subseasonal to Seasonal (S2S) Prediction project database (Vitart and Mladek 2020) produces a few outputs at 6-hourly resolution and the rest at daily resolution]. There are a limited number of models that produce some variables at hourly resolution but these are available for either limited areas or for limited periods in the future (Kendon et al. 2021).
This project investigated machine learning methodologies to estimate hourly data from daily data, using the ERA5 dataset as this provides hourly data against which methods can be verified. The validation focused around three high-impact events where hourly data were crucial to balancing energy supply to energy demand. The fields of interest were 2-m temperature, as this is a key input for energy demand, and 10-m wind speed and surface solar irradiance, as these are key inputs for wind and solar power energy generation, respectively. The hackathon team compared multiple machine learning methodologies, finding that simpler methods of linear interpolation for downscaling temperature and solar irradiance proved hard to beat (compared to, e.g., linear regression, random forests, and neural networks). For wind speed, a linear regression based approach modeled the data best. It was much better at capturing the hour-to-hour variability in wind speed than previous approaches, but still had issues accurately capturing wind speed magnitude. This project is part of the U.K. Climate Resilience Programme, and feeds into ongoing work at the Met Office, which will support the codevelopment of a climate service prototype for the energy sector.
Extreme weather for electricity systems.
This project explored a dataset of gridded meteorological “adverse weather scenarios” produced by the Met Office in collaboration with the National Infrastructure Commission and Climate Change Committee (Dawkins et al. 2021). The team had several approaches to exploring this dataset: data visualizations to study the extreme events, comparison to existing multidecade climatological datasets (Bloomfield et al. 2020a,b), and using as input to a power system model Calliope (Pfenninger and Pickering 2018) to study impact on energy system reliability, successfully linking events with stress indicators. The project showed that this dataset can aid testing future energy system resilience to weather and climate extremes, and results are being condensed into material for a publication, as well as training material to be available with the dataset.
Quantifying uncertainty in power system design using distributed computing.
Power system models are a widely used tool in the design and simulation of potential future energy systems. Such models use optimization-based techniques to solve an algebraic program to seek a future power system that minimizes the combined cost of building and running the associated infrastructure. This optimization problem typically involves numerous constraints to ensure, e.g., that the supply of electricity matches the demand for electricity at an hourly level. Recent research has clearly demonstrated the importance of solving these models over “climatologically robust” weather samples (i.e., weather records spanning many decades; e.g., Hilbers et al. 2019; Bloomfield et al. 2016), but the complexity of the optimization problem has rendered a thorough investigation of current and future climate uncertainty computationally intractable. This hackathon project sought to overcome this computational-complexity barrier by prototyping a distributed computing experiment using the citizen science climateprediction.net framework (Stainforth et al. 2004).
Climateprediction.net is the world’s largest climate modeling experiment, running climate model experiments on users’ home computers. The hackathon team developed a prototype version of a climateprediction.net workflow, taking raw climate data from a GCM (which can be run on home computers), bias adjusting and converting the output to produce hourly demand and renewable generation time series, and then running through a simple “U.K.–Ireland” power system model implemented in a package known as Calliope (Pfenninger and Pickering 2018). The project forms the groundwork for future project proposals using the framework to do a full sensitivity analysis.
Feedback and lessons learned
In response to requests for participant feedback the event was described as a “very collaborative and insightful experience” where participants enjoyed the networking opportunities with others from around the globe across the energy and climate disciplines. The technical support was reviewed as very prompt and supportive from the organizers and the JASMIN team with participants reflecting, “It is simply hard to fault the organization and execution of the hackathon and the enthusiasm and skills of participants” and “A lot of work was put into the event by the hackathon organisers and this went a long way.” Despite the overall success of the hackathon, there remain several areas that could be improved in subsequent events.
In terms of “open research”—which was a guiding principle for the hackathon—time constraints proved a challenge. In particular, although project code and notebooks were uploaded to GitHub, without dedicated time to write good documentation many of the projects contain only a brief description of their intended purpose and operational instructions on GitHub. This is unfortunate, as the final presentations each team prepared were highly informative, including many helpful graphs and figures. Future hackathons might improve the availability of documentation by dedicating additional time for documenting projects and improving code readability.
Although a significant number of participants dropped out between registration and participation (from 90 registrations to 40 participants), there was strong commitment during the main week-long event. Also, although the event ran very smoothly on a technical level (with the Zoom, Slack, and JASMIN providing an excellent platform), the virtual hackathon format was challenging for participants joining from very different time zones. The virtual format may have benefited the ability of group members to drop in and out enabling participation in parallel meetings more efficiently than an in-person event may have allowed, increasing participation, with the downside that the proportion of participants’ time during the event may have been reduced compared to a face-to-face event.
Overall, however, a core message that emerges from the hackathon is the importance of preparation and engagement from a core team before the event (particularly the organizers, the project team leaders, and technical support). Identifying the projects early on and engaging with team leaders to develop the ideas and understand their needs were key factors that allowed participants to work effectively. Having the means to prepare compute and data resources and tutorials before the event, for a common computing platform, helped.
Summary
Connecting researchers virtually with video conferencing and communications software is low cost, has a small carbon footprint, and made the hackathon accessible to a wide audience of researchers (particularly early career researchers with limited travel budgets, and researchers with other work or family responsibilities). Participants benefited from the sharing of knowledge relating to the use of tools key to the energy–climate community, including application of power system models, optimizers, and processing tools for working with meteorological data in Python.
Rapid progress was aided through the hackathon format, which enabled a deep level of practical engagement between researchers from different disciplines. Climate modelers were able to provide assistance in reading large volumes of meteorological data while energy modelers could provide details on how their various models worked and the spatial and temporal resolutions of meteorological data that would be most useful as inputs. Industry experts also contributed useful context to how the projects could provide most useful outputs for them.
The projects tackled very different problems in the quantification of climate impacts to energy systems. Some groups had focused on small time scale operation of future energy systems, while other groups were interested in multi-decade-long planning and risk assessment. The pace of progress was impressive across the board, with each team broadly achieving their aims. Teams developing new tools, including the solar PV prediction and power system distributed computing, were able to rapidly develop prototypes, identifying where difficulties arise and further work is needed. While none of the projects would be considered finalized after only one week, they have exceeded the organizers’ expectations in providing prototypes and scoping work that can form the basis of novel research and development work. Indeed, it is considered a very positive result that several groups are looking at future collaboration and turning their research into papers or grant proposals.
Acknowledgments.
The workshop was initiated through the Met Office Academic Partnership and benefited from funding from the Met Office and Reading University. Thanks are given to the project team leaders and participants for their hard work and contributions throughout the event. Special mention to Bryn Pickering for helping multiple groups to setup their own Calliope power system modeling environments. Thanks are given to the members of staff at the University of Reading who responded to Hannah’s request for rooftop solar PV data for one of the projects.
Thanks are given to the presenters of the scoping day inspiration session: Eduardo Damasio-Da-Costa (ECMWF), Alberto Troccoli (WEMC), Ranjini Swaminathan (U. Reading), Lion Hirth (Neon Energy), and Masao Ashtine (U. Oxford).
We are very grateful to the United Kingdom’s CEDA-JASMIN service for providing JASMIN access and support before and during the event and to Gurobi Optimization for generously providing gratis use of their commercial solver for the duration of the workshop.
The work reported in the “Developing methods to estimate subdaily energy relevant climate variables from daily data” section is part of the U.K. Climate Resilience programme, which is supported by the UKRI and codelivered by the Met Office and NERC on behalf of UKRI partners AHRC, EPSRC, and ESRC.
Data availability statement.
The full output of projects is available at https://github.com/2021-Energy-Climate-Hackathon.
References
Bloomfield, H. C. , D. J. Brayshaw , L. C. Shaffrey , P. J. Coker , and H. E. Thornton , 2016: Quantifying the increasing sensitivity of power systems to climate variability. Environ. Res. Lett. , 11, 124025, https://doi.org/10.1088/1748-9326/11/12/124025.
Bloomfield, H. C. , D. J. Brayshaw , and A. Charlton-Perez , 2020a: ERA5 derived time series of European country-aggregate electricity demand, wind power generation and solar power generation. University of Reading, https://researchdata.reading.ac.uk/273/.
Bloomfield, H. C. , D. J. Brayshaw , and A. Charlton-Perez 2020b: MERRA2 derived time series of European country-aggregate electricity demand, wind power generation and solar power generation. University of Reading, https://researchdata.reading.ac.uk/239/.
Bloomfield, H. C. , D. J. Brayshaw , A. Troccoli , C. Goodess , M. De Felice , L. Dubus , P. Bett , and Y.-M. Saint-Drenan , 2021a: Quantifying the sensitivity of European power systems to energy scenarios and climate change projections. Renewable Energy , 164, 1062–1075, https://doi.org/10.1016/j.renene.2020.09.125.
Bloomfield, H. C. , and Coauthors , 2021b: The importance of weather and climate to energy systems: A workshop on next generation challenges in energy–climate modeling. Bull. Amer. Meteor. Soc. , 102, E159–E167, https://doi.org/10.1175/BAMS-D-20-0256.1.
Copernicus, 2021: Climate Data Store. ECMWF, https://cds.climate.copernicus.eu/cdsapp#!/home.
Dawkins, L. , I. Rushby , M. Pearce , E. Wallace , and T. Butcher , 2021: Adverse weather scenarios for future electricity systems. NERC EDS Centre for Environmental Data Analysis, accessed 21 June 2021, https://catalogue.ceda.ac.uk/uuid/7beeed0bc7fa41feb10be22ee9d10f00.
Deakin, M. , H. Bloomfield , D. Greenwood , S. Sheehy , S. Walker , and P. C. Taylor , 2021: Impacts of heat decarbonization on system adequacy considering increased meteorological sensitivity. Appl. Energy , 298, 117261, https://doi.org/10.1016/j.apenergy.2021.117261.
Elsinga, B. , and W. G. van Sark , 2017: Short-term peer-to-peer solar forecasting in a network of photovoltaic systems. Appl. Energy , 206, 1464–1483, https://doi.org/10.1016/j.apenergy.2017.09.115.
Energy–Climate Hackathon, 2021b: 2021-Energy-Climate-Hackathon. GitHub, https://github.com/2021-Energy-Climate-Hackathon.
Eyring, V. , and Coauthors , 2020: Earth System Model Evaluation Tool (ESMValTool) v2.0—An extended set of large-scale diagnostics for quasi-operational and comprehensive evaluation of Earth system models in CMIP . Geosci. Model Dev. , 13, 3383–3438, https://doi.org/10.5194/gmd-13-3383-2020.
Hilbers, A. P. , D. J. Brayshaw , and A. Gandy , 2019: Importance subsampling: Improving power system planning under climate-based uncertainty. Appl. Energy , 251, 113114, https://doi.org/10.1016/j.apenergy.2019.04.110.
Huppmann, D. , and Coauthors , 2021: pyam: Analysis and visualisation of integrated assessment and macro-energy scenarios [version 2; peer review: 2 approved, 1 approved with reservations]. Open Res. Europe , 1, 74, https://doi.org/10.12688/openreseurope.13633.2.
IEA, 2021: Sustainable recovery tracker. IEA, www.iea.org/reports/sustainable-recovery-tracker.
Jacob, D. , and Coauthors , 2014: EURO-CORDEX: New high-resolution climate change projections for European impact research. Reg. Environ. Change , 14, 563–578, https://doi.org/10.1007/s10113-013-0499-2.
Jaroszweski, D. , R. Wood , and L. Chapman , 2021: Infrastructure. Technical Report of the Third UK Climate Change Risk Assessment, Met Office Tech. Rep., 224 pp.
Kendon, E. , C. Short , J. Pope , S. Chan , J. Wilkinson , S. Tucker , P. Bett , and G. Harris , 2021: Update to UKCP local (2.2 km) projections. Met Office Rep., 114 pp., www.metoffice.gov.uk/pub/data/weather/uk/ukcp18/science-reports/ukcp18_local_update_report_2021.pdf.
Kies, A. , B. U. Schyska , M. Bilousova , O. El Sayed , J. Jurasz , and H. Stoecker , 2021: Critical review of renewable generation datasets and their implications for European power system models.Renewable Sustainable Energy Rev. , 152, 111614, https://doi.org/10.1016/j.rser.2021.111614.
Kluyver, T. , and Coauthors , 2016: Jupyter notebooks—A publishing format for reproducible computational workflows. Positioning and Power in Academic Publishing: Players, Agents and Agendas , F. Loizides , and B. Scmidt , Eds., IOS Press, 87–90.
Lawrence, B. N. , and Coauthors , 2013: Storing and manipulating environmental big data with JASMIN. 2013 IEEE Int. Conf. on Big Data, Silicon Valley, CA, IEEE, 68–75, https://doi.org/10.1109/BigData.2013.6691556.
Met Office, 2021: Climate data challenge—Working in partnership for innovation. Met Office News Team, https://blog.metoffice.gov.uk/2021/04/02/climate-data-challenge-working-in-partnership-for-innovation/.
Murphy, J. M. , and Coauthors , 2018: UKCP18 land projections: Science report. Met Office Rep., 191 pp., www.metoffice.gov.uk/pub/data/weather/uk/ukcp18/science-reports/UKCP18-Land-report.pdf.
Oxford e-Research Centre, 2021: Energy–Climate Data Hackathon June 2021. YouTube, www.youtube.com/playlist?list5PLbg_hLS4uA7SMz7HTtRX-plD7k-wl0pzc.
Pfenninger, S. , and B. Pickering , 2018: Calliope: A multi-scale energy systems modelling framework. J. Open Source Software , 3, 825, https://doi.org/10.21105/joss.00825.
pyam, 2021: pyam: Analysis and visualization of integrated-assessment and macro-energy scenarios. GitHub, https://github.com/IAMconsortium/pyam.
Schlott, M. , A. Kies , T. Brown , S. Schramm , and M. Greiner , 2018: The impact of climate change on a cost-optimal highly renewable European electricity network. Appl. Energy , 230, 1645–1659, https://doi.org/10.1016/j.apenergy.2018.09.084.
Schyska, B. U. , A. Kies , M. Schlott , L. von Bremen , and W. Medjroubi , 2021: The sensitivity of power system expansion models. Joule , 5, 2606–2624, https://doi.org/10.1016/j.joule.2021.07.017.
Stainforth, D. A. , M. R. Allen , D. Frame , J. Kettleborough , C. Christensen , T. Aina , and M. Collins , 2004: Climateprediction.net: A global community for research in climate physics. Environmental Online Communication , Springer, 101–112.
Sustainability Services, 2021: Download energy data. University of Reading, accessed 21 June 2021, https://sites.reading.ac.uk/sustainability/energy/data/download-energy-data.
Vitart, F. , and R. Mladek , 2020: Parameters. ECMWF S2S Archive, https://confluence.ecmwf.int/display/S2S/Parameters.
