Recent hurricanes have demonstrated the need for real-time flood forecasting at street scale in coastal urban areas. Here, we describe the high-impact high-resolution (HIHR) system that operationally forecasts flooding at very high resolution in the New York–New Jersey metropolitan region. HIHR is the latest upgrade of the Stevens Flood Advisory System (SFAS), a highly detailed operational coastal ocean modeling system. SFAS, based on the Hydrologic–Hydraulic–Hydrodynamic Ensemble (H3E) modeling framework, consists of four sets of nested coastal and inland flood models that provide ensemble flood forecasts with a horizon of at least 96 h from regional to street scales based on forcing from 100 different meteorological output fields. HIHR includes nine model domains with horizontal resolution ranging from 3 to 10 m around critical infrastructure sites in the region. HIHR models are based on an advanced hydrodynamic code [the Stevens Estuarine and Coastal Ocean Model (sECOM), a derivative of the Princeton Ocean Model] and nested into the H3E models. HIHR was retrospectively evaluated by forecasting the coastal flooding caused by Superstorm Sandy in 2012 using water-level sensors, high-water marks, and flood maps. The forecasts for the 95th percentile show a good agreement with these observations even three days before the peak flood, while the 50th percentile is negatively biased because of the lack of resolution on the meteorological forcing. Forecasts became more accurate and less uncertain as the forecasts were issued closer to the peak flooding.
A coastal ocean operational system suitable for probabilistic flood forecast at street scale in the New York–New Jersey metropolitan region is introduced and retrospectively evaluated during Superstorm Sandy.
Coastal flooding is one of the main natural hazards in terms of economic damages and losses of human life (Jonkman 2005), as demonstrated by recent tropical cyclones such as Katrina, Ike, Irene, Sandy, Harvey, Irma, and Maria along the U.S. coasts. Floods have become the second-leading cause of weather-related deaths behind heat-related incidents (Hoss and Fischbeck 2016; Weaver et al. 2014). Average economic flood losses in 2005 were estimated to be approximately $6 billion (U.S. dollars) per year in the largest coastal cities of the world (Hallegatte et al. 2013) because of the combined effect of rainfall and storm surge from storm activity. Storm surge, defined here as the rise of seawaters above the local astronomical tide level, occurs because of forcing from winds and low pressure systems (Glickman 2000). Wind is usually the primary source of the storm surge and atmospheric pressure the secondary. Hereafter, we refer to storm surge as coastal flooding.
Hurricanes Harvey, Irma, and Maria in 2017 were harsh reminders of the damage caused by coastal flooding as record recovery costs are expected in the United States. The long-term increase in damage is the result of several climate factors, such as rising sea levels and intensified storms, and several societal factors, such as increased population and urban development (Changnon et al. 2001; Walsh et al. 2014). Sea level rise is expected to accelerate over the twenty-first century, primarily because of increasing expansion of warming seawater and accelerated melting of land-based ice sheets (Church et al. 2013). In addition, over 50% of the world population now lives in coastal areas, and this percentage is increasing as the inland rural population is moving to the coast (Creel 2003). To accommodate this population growth, coastal urban development will continue, reducing the soil infiltration capacities (Hsu et al. 2000).
It becomes critical then to accurately forecast the location and timing of coastal flooding, as well as the peak magnitude, likelihood of occurrence, and inland extent. Without adequate forecasts, it is difficult for emergency managers and decision-makers to devise evacuation plans and to take other preparedness actions necessary to protect life and property (Young 2002; Wheater 2002). Forecast lead time, the period of time between the issuance of a forecast and the occurrence of the phenomenon that was predicted, is key to preparedness. A combination of lead time, forecast accuracy, and communication (among many other things) will lead to more lives potentially being saved and fewer properties damaged. Producing accurate coastal flood forecasts requires numerical models that properly consider the important coastal ocean processes that affect storm surges as well as precipitation and freshwater discharges (Orton et al. 2012; Saleh et al. 2016; Fewtrell et al. 2008).
Modeling flooding from storm surges in coastal urban settings requires very high horizontal resolution. Traditionally, coastal flood modeling has been addressed by means of two methodologies: bathtubbing (or static mapping) and hydrodynamic modeling (or dynamic mapping). The bathtubbing considers as flooded all land areas with elevation equal to or lower than the sea level. This methodology has been extensively used because of its simplicity and low computational costs (e.g., Hinkel et al. 2014). Although it can give a first approximation of possible flood heights, it generally results in substantial overestimation of the flood extent because it is not a physics-based model of flow dynamics (Ramirez et al. 2016; Vousdoukas 2016). Coastal flood propagation in urban areas depends on the complex interaction between the flow and the urban infrastructures along with other factors such as wind, which are not considered in bathtubbing.
Physics-based hydrodynamic models solve the nonlinear Reynolds-averaged equations of motion to simulate flow dynamics. If applied to a coastal region, the models can capture the dynamics arising from the interaction between surge flows and the urban infrastructure. As the coastal floodwater inundates a previously dry area, these models must adapt to include these now wet areas into the computations. Similarly, the models must then simulate the receding flood by drying these areas and removing them from the computations. These models typically handle the flooding and drying process in a phenomenological fashion requiring wet–dry checks and minimums in the depth allowed (Zheng et al. 2003). Although hydrodynamic models yield more accurate results than static mapping, they are much more complex and more computationally demanding.
Despite the better accuracy of hydrodynamic models, forecasts are not perfect because of a variety of factors. Small differences in atmospheric forcing and offshore ocean boundary conditions can lead to very different forecasts (Lorenz 1963). To deal with imperfect forecasts, a common approach is to appeal to the probabilistic or ensemble forecasting. An ensemble of forecasts is generated by prescribing different initial conditions or forcing fields (Leith 1974). Ensembles are used to identify the most likely estimate of the future flooding and its associated uncertainty from the ensemble spread (Buizza 2008). Our ability to operationally provide flood forecasts with ensembles and hydrodynamic models has been limited by the extremely high computational cost that is required. However, the computing power available today is starting to become sufficient to model flows around these urban topographic features at street scales (Blumberg et al. 2015).
In this paper, we describe an operational high-impact high-resolution (HIHR) system that includes ensemble flood forecasting at street scales in the New York–New Jersey metropolitan region using hydrodynamic modeling. This system is the latest extension of the Hydrologic–Hydraulic–Hydrodynamic Ensemble (H3E) modeling framework, a highly detailed, well-validated operational flood forecast system (Georgas et al. 2016). The first version of the system, Storm Surge Warning System (SSWS), was built in 2002 and has been operational since then (Bruno et al. 2006). Over the years, the system has been updated, extended, and evaluated (Di Liberto et al. 2011; Georgas et al. 2016; Orton et al. 2012; Georgas and Blumberg 2010). It was highlighted during Superstorm Sandy as being preferred by the New York City (NYC) Office of Emergency Management for its ease of use (Sullivan and Uccellini 2013).
THE STEVENS FLOOD ADVISORY SYSTEM.
H3E uses four sets of linked coastal and inland flood models (Fig. 1) with a forecast horizon of at least 4 days (96 h) reinitialized every 6 h (Georgas et al. 2016). At regional scale, water-level forecasts are provided by the Stevens Northwest Atlantic Prediction (SNAP) model (Blumberg et al. 2015). River stages are forecast by a hydrologic (HYDRO) model based on the Hydrologic Engineering Center Hydrologic Modeling System (HEC-HMS) code (Saleh et al. 2016). At local scale, the New York Harbor Observing and Prediction System (NYHOPS) model is one-way nested into the SNAP domain and coupled into the river discharges from HYDRO (Georgas and Blumberg 2010). SNAP and NYHOPS are based on the parallel version of the Stevens Estuarine and Coastal Ocean Model (sECOM) code (Blumberg et al. 1999; Georgas et al. 2007; Jordi et al. 2017). The sECOM is a three-dimensional, finite-difference, free-surface, wetting and drying, hydrodynamic, and surface wave model. The three models (SNAP, HYDRO, and NYHOPS) provide ensemble forecasts based on a 100-member set of different meteorological output fields, including the European Centre for Medium-Range Weather Forecasts (ECMWF) ensemble and high-resolution deterministic (Buizza et al. 2007; ECMWF 2018), Global Ensemble Forecast System (GEFS; Cui et al. 2012), Global Forecast System (GFS) and GFS-experimental (GFS-e; Han and Pan 2011), and Canadian Meteorological Centre (CMC; Charron et al. 2010) models, as well as regional outputs from the North American Mesoscale Forecast System (NAM; Skamarock et al. 2005) model and Rutgers Weather Research and Forecasting (Rutgers-WRF; Seroka et al. 2012) Model. Details on these meteorological models and the techniques used to interpolate them into the H3E model grids are provided in the online supplemental material (https://doi.org/10.1175/BAMS-D-17-0309.2).
The latest extension of H3E includes the HIHR system. The system consists of nine model domains (Fig. 2) focusing on critical infrastructure sites such as airports and marine terminals. The horizontal resolution ranges from 3 to 10 m to resolve buildings at street scale, as the region is highly urbanized. In total, HIHR has up to 5 × 106 grid cells covering an area of 3.3 × 108 m2. Topography and bathymetry in HIHR models were derived from a 1-m-resolution unified digital elevation model (DEM) representing bare earth and covering northern New Jersey, New York City, New York, and areas along western Long Island Sound (see supplemental material). Building heights were added to the unified DEM on those grid cells that corresponded to the building’s footprint. Obstructions and walls with scales smaller than the grid resolution were implemented by disallowing transport and mixing across gridcell interfaces up to the overtopping condition. No other urban structures were incorporated unless they were present in the unified DEM. HIHR models are one-way nested into the NYHOPS model domain. Whereas NYHOPS is based on the three-dimensional sECOM code, HIHR only uses the two-dimensional depth-integrated part of the sECOM solver to reduce computational requirements. The bottom drag is represented by the Manning’s roughness coefficient that depends on the land use. Flooded land areas are allowed to drain by using a drainage coefficient that also depends on the land use.
Every 6 h, H3E models are run from the previous-cycle forecast using the in-house Pharos Hyperscale Supercomputing Facility, which includes two head nodes (control computers) and 64 compute nodes (processing computers) based on dual Intel E5-2680, version 2, CPUs with 10 cores per CPU (20 cores per node) with between 128 and 256 GB of RAM memory per node. There are in total 1,280 available computational cores and 9.2 TB of RAM memory. All H3E models, including HIHR, have been fully operational since August 2016. Using a new parallel version of the sECOM code based on Jordi and Wang (2012) and Jordi et al. (2017), the model runs very efficiently. The 100 members of SNAP, HYDRO, and NYHOPS models take less than 3 h to complete using 1,000 computational cores. A single member of the nine HIHR models takes about 15 min to complete also using 1,000 cores. This would imply 25 h to run 100 HIHR members, which is not practical for 6-h operational cycles since we require that the system must finish before the next cycle begins.
Consequently, only three HIHR members are run every 6 h to provide the best forecast estimate and its uncertainty. The water level from the 100 NYHOPS ensemble members are used to compute the 5th, 50th, and 95th percentiles using a phase-aware statistics method (Schulte and Georgas 2018) in the HIHR boundaries. When the ensemble members predict the peak flooding at different times, traditional ensemble statistics result in a large negative bias of percentile peaks as well as smoothing of the peak distribution. To remedy this time difference problem, one needs to compute timing statistics separately from statistics related to flood magnitude (Liu et al. 2011). Such a separation can be achieved by means of a wavelet spectral decomposition of the ensemble members (Schulte and Georgas 2018). The best estimate is provided by the 50th percentile and the forecast uncertainty by the spread between the 5th and 95th percentiles. A cycle of the nine HIHR models for the three percentiles takes about 1 h to complete using 1,300 cores. Since Pharos has only 1,280 cores, we divided HIHR forecasts into two separate runs using approximately 650 cores each and doubling the computing time to 2 h. HIHR models are run after SNAP, HYDRO, and NYHOPS models. A whole operational cycle (100 ensemble members for SNAP, HYDRO, and NYHOPS, and 3 percentiles for HIHR) takes less than 5 h, including pre- and postprocessing. We are aware that different entities requiring forecast information may have needs for quicker turnaround requirements for their operational needs. Fortunately, computational capabilities are improving such that 100 HIHR ensembles can be run in the near future in a shorter time.
Real-time results from the whole H3E models are accessible through the Stevens Flood Advisory System (SFAS) website (http://stevens.edu/SFAS). Figure 3 shows the opening SFAS page, which includes the station locations where present and forecast flooding conditions are provided. Present conditions are based on real-time water-level observations. We have installed and maintain 22 real-time environmental monitoring stations [Aquatrack 5002 water level and Greenspan Analytical EC250 conductivity/temperature (C/T) sensors and power cells]. Real-time water-level data from partner institutions and agencies are retrieved through the Internet (Georgas et al. 2016). Forecast flood conditions are based on the H3E simulations. To provide common context for the severity of occurring or anticipated flooding, we use the terminology of the National Weather Service (NWS) for the critical flood levels (NWS 2012).
CASE STUDY: SUPERSTORM SANDY’S FLOODING.
Superstorm Sandy serves as an ideal case study for evaluating HIHR forecasts because of the large coastal flooding it produced and the large quantity of available observations for model validation. On October 2012, Sandy made landfall in the New York–New Jersey metropolitan region, resulting in an enormous impact on life and property damage with the cost exceeding $50 billion (Schubert et al. 2015). HIHR retrospective forecasts were run using the NYHOPS forecasts issued at 0000 UTC 27, 28, 29, and 30 October 2012. Since the maximum water level in the region was registered around 0120 UTC 30 October 2012 (Fig. 4), these forecasts correspond to a lead time of approximately 73, 49, 25, and 1 h (73-, 49-, 25-, and 1-h forecasts). We replicated the current operational procedure to provide forecasts in real time, although there are a few differences. First, only the topographic and bathymetric datasets taken before Sandy were used. Second, the number of NYHOPS ensemble members was reduced to 21 because only the 21 members of GEFS model forecasts were available. Third, the forecast period was reduced to 84 h and only a cycle per day was run because of the available GEFS forecasts.
Prior to Sandy, water-level sensors were deployed in the region by the U.S. Geological Survey (USGS) to continuously measure water levels. Together with the permanent sensors, five water-level time series were available within the HIHR domains during Superstorm Sandy (see Fig. 2 for locations). Figure 4 compares the observed peak water level during Sandy with the 5th, 50th, and 95th percentiles for the 73-, 49-, 25-, and 1-h forecasts. We focus on peak water level since it is one of the most significant attributes for flood risk assessment. The time evolution of the observed and forecast water level for the five sensors is shown in online supplemental Figs. ES1–ES5. We note that all water levels used hereafter are referenced to the North American Vertical Datum of 1988 (NAVD88). In general, the peak water level was underestimated for the 50th percentile, although it converged toward the observed one as the lead time was close to the event. The 95th percentile was much closer to the observed water level for the 73-, 49-, and 25-h forecast. The exception was Flushing Bay, where the 50th percentile for 49- and 25-h forecasts overestimated the peak in water level. The tidal dynamics at Flushing Bay are complicated as they depend on the tidal wave entering from Long Island Sound, which is out of phase from the one entering through the New York Bight apex and affects the other sensors in the New York–New Jersey harbor.
To understand why the forecasts were best at the 95th percentiles for the 73-, 49-, and 25-h forecasts, we analyzed the forecast wind fields based on the GEFS model. We computed the 5th, 50th, and 95th percentiles of the wind speed for the 21 members again using the phase-aware method. Figure 5 compares the wind speed measured at 10-m height at the National Oceanic and Atmospheric Administration (NOAA) National Data Buoy Center (NDBC) buoy 44065 in the New York Bight apex (Fig. 1) with the three wind percentiles for the 73-, 49-, 25-, and 1-h forecasts at the same location during Sandy. The wind speed in the 50th percentile was underestimated, and the 95th percentile was closer to the observed one, which can explain why the 95th-percentile water-level forecast was better than the 50th-perentile one. In this regard, Leonardo and Colle (2017) showed that GEFS has had inferior skill when forecasting the intensity of tropical cyclones in the North Atlantic. Other ensemble meteorological forecasts, such as ECMWF, have been shown to have better skill than GEFS. However, intensity forecasts of tropical cyclones have improved much less compared to track and landfall forecasts in the last years (Rappaport et al. 2009). The major problem is that most global meteorological models have an inadequate model resolution to resolve the inner core of the tropical cyclones and associated winds (Gall et al. 2013).
Immediately following Sandy, ground-based high-water marks (HWMs) were identified from debris lines and marks on structures (McCallum et al. 2013). The vertical accuracy of the HWMs ranges from 0.02 to 0.20 m depending on the quality of the mark, whereas the horizontal accuracy is approximately 3 m (Koenig et al. 2016). To account for errors associated with the horizontal accuracy, we used the 1-m-resolution unified DEM to add to the vertical accuracy an error equivalent to the topographic differences in a 3-m radius around the HWM location. A total of 59 HWMs were measured in the HIHR domains. Figure 6a shows their location as well as the difference between the observed HWM elevations and those for the 95th-percentile forecast at 73 h before the forecast. The corresponding figures for other percentiles and lead times can be found in supplemental Figs. ES6–ES17. As a summary, comparisons between the observed HWMs and the 73-, 49-, 25-, and 1-h forecast ones are shown in Figs. 6b–e, respectively. Consistent with the sensor results, the 95th percentiles were closer to the observed HWMs (the 1:1 diagonal line) for the 73-, 49-, and 25-h forecasts, and the 50th percentile was better for the 1-h forecast. The root-mean-square errors between the observations and the 50th percentile for the 1-h forecast was 0.23 m. However, there are three HWMs included in these statistics that persistently presented large deviations. The mark quality for two of them is poor, and thus they are not very reliable. The other one may be affected by some urban obstacle smaller than the corresponding HIHR model resolution (10 m for this HWM). The root-mean-square error without them is reduced to 0.10 m.
Along with peak magnitude of the water level, the maximum horizontal flood extent is another important attribute for flood risk assessment. To evaluate the performance of HIHR models for forecasting coastal flood extents, we used an estimate of spatial coastal flood boundaries produced by the Federal Emergency Management Agency (FEMA) Modeling Task Force that were made by extrapolating observations from water-level sensors and HWMs with bathtubbing (FEMA 2015). The FEMA flood extents were first interpolated to the grids used in the HIHR models. Then we defined the number of overland grid cells that were flooded in both FEMA and HIHR as “hits,” the number of inland grid cells that were flooded in FEMA but not in HIHR as “misses,” and the number of inland grid cells that were flooded in HIHR but not in FEMA as “false alarms.” An example of the hits, misses, and false alarms for the 95th percentile of the 73-h forecast is shown in Fig. 7a. The hits dominated overall, but there were some misses, mainly concentrated in the Newark Liberty International Airport and John F. Kennedy International Airport. False alarms were scarcer and mostly around John F. Kennedy International Airport and LaGuardia Airport. Airports are typically almost flat surfaces, and a few centimeters difference in the water level can cause a significant difference in the flood extent, especially if bathtubbing is used, as in the FEMA estimate. Similar figures for the other percentiles and forecasts can be found in supplemental Figs. ES18–ES29.
We also used three categorical verification scores to quantify the agreement between the estimated FEMA flood extents and the forecast ones following Saleh et al. (2017): the probability of detection (POD) is a measure of HIHR forecast flooding that overlays the FEMA estimated flooding (percentage of hits with respect to the sum of hits and misses), the false alarm ratio (FAR) provides a metric of HIHR forecast flood extent that did not occur according to FEMA (percentage of false alarms with respect to the sum of hits and false alarms), and the critical success index (CSI) gives the overall performance of the HIHR forecast flooding (percentage of hits with respect to the sum of hits, misses, and false alarms). Perfect values for these scores are POD = 100%, FAR = 0%, and CSI = 100%. They are shown in Figs. 7b–d for the 73-, 49-, 25-, and 1-h forecasts. The behavior of POD and CSI were similar, increasing from a value of about 50%–90% for the 50th-percentile forecast. The 95th-percentile forecast was closer to the FEMA’s estimate than the 50th percentile, even for the 1-h forecast. Since the 50th percentile for the 1-h forecast shows a better agreement with the water-level time series and the HWMs than the 95th percentile, results suggest that FEMA’s approach overestimated the flood extent.
A few other studies have also evaluated the performance of flood models at street scale in the New York–New Jersey metropolitan region during Sandy. Wang et al. (2014) found a 30-m mean absolute difference between the FEMA estimated flood extent for New York City and the simulated one using intermediate approaches between static mapping and hydrodynamic modeling. As discussed, our results based on fully hydrodynamic modeling suggest that FEMA somewhat overestimated the flood extent. Blumberg et al. (2015) showed that a sECOM hydrodynamic flood model was capable of hindcasting overland water elevation from HWMs and crowdsourced photographs, videos, and stories with an error of 0.07 m in Hoboken, New Jersey. Our result of 0.10 m for the 50th percentile and 1-h forecast throughout the HIHR model domains is similar. Saleh et al. (2017) evaluated flood forecasts issued 3, 2, and 1 days before Sandy in the confluence of the Passaic and Hackensack Rivers with Newark Bay in New Jersey. They found that the flooding caused by Sandy was closer to the 95th percentile than the 50th one, which agrees with our results for the 73-, 49-, and 25-h forecasts.
In this study, we described the HIHR system that includes coastal flood forecasting at street scale in the New York–New Jersey metropolitan region. HIHR is the latest upgrade of the H3E modeling framework, which consists of four sets of nested coastal and inland flood models with a forecast horizon of at least 96 h that are reinitialized every 6 h based on different atmospheric model predictions of surface meteorological factors. The inner set of models corresponds to HIHR, which has nine very high-resolution (from 3 to 10 m) model domains focusing on critical infrastructure sites. Based on the two-dimensional version of sECOM code along with wetting and drying, and variable drainage as a function of land use, HIHR models provide coastal flood forecasts for the expected flooding (50th percentile) and its uncertainty as the spread between the 5th and 95th percentiles.
The agreement between the observations and forecasts for the 95th percentiles during Sandy represents a good starting point in the use of HIHR for flood forecasting at street scale. Nevertheless, a continuing challenge for operational modeling is to demonstrate that forecasts are reliable and certain. Based on a single Sandy case study, it is premature to imply that the HIHR models can make accurate-enough forecasts of hurricane-induced coastal flooding several days out that automatically trigger street-scale evacuation plans. In addition, the uncertainty defined as the spread between the 5th and 95th percentiles at the 73-h lead time was too high. This is likely around the time that emergency managers need to start making evacuation decisions, especially for a metropolitan area where so many people may have to be moved. Currently, evacuation zones are at a much coarser resolution than the street scale. For example, New York City has six evacuation zones (http://maps.nyc.gov/hurricane/). Therefore, flood forecasting at street scale is a step forward and may provide some benefits in the current paradigm.
There are several aspects of our system that can be improved to provide more accurate, reliable, and useful forecasts. First, most of the meteorological forcing used have an inadequate resolution to resolve the inner core of the tropical cyclones. Although dynamic downscaling through high-resolution [O(1) km] meteorological forecasts embedded into the global meteorological models would be highly desirable, these high-resolution models are currently quite expensive to run in terms of computational resources. We are investigating the use of statistical downscaling to compensate for the lack of details not captured in the global meteorological models and that cannot be captured by simply interpolating the boundary condition data to the ocean model grid. Second, in the case of other tropical storms such as Hurricane Katrina, water-related destruction was largely by waves (Wang and Oey 2008). Whereas our forecasting system is primarily concerned with resolving coastal flooding at inland regions away from wave effects, the use of advance wave models that take into account wave–current interactions improves water-level forecasts even at regions away from direct wave effects (Marsooli et al. 2016).
A third aspect is the improvement of the ensemble techniques. Figure 4d shows that the 50th percentile for the 73-h peak-water-level forecast at Flushing Bay appeared to be most accurate, while subsequent forecasts were poorer. However, the correspondent water-level time series in Fig. ES4 indicates that the maximum peak in water level at Flushing Bay for the 73-h forecast was centrally (50th percentile) predicted to occur at the previous tidal cycle from when the peak was in fact observed (the forecast was off phase). Subsequent cycles did not have the same issue, and the observations were closer to the 95th percentile, as with other stations. We are developing ensemble clustering methods for flood forecasting to deal with this issue (Schulte 2017). Other aspects such as enabling three-dimensional dynamics or inclusion of a more detailed representation of the sewer system will be the focus of future improvements when more computing power will be available. Finally, we are actively working on the implementation of data assimilation as the New York–New Jersey metropolitan region is relatively well equipped with real-time water-level observations. Nevertheless, it is critical that the federal government and agencies in charge of public safety continue to be leaders in investing in more observational data and making that data widely available.
This work was partially funded by a research task agreement entered between the trustees of the Stevens Institute of Technology and the Port Authority of New York and New Jersey, effective 19 August 2014. The authors would like to thank Kevin Ying for his assistance with the Pharos Hyperscale Supercomputing Facility.
A supplement to this article is available online (10.1175/BAMS-D-17-0309.2)