A Technical Overview of the New York State Mesonet Standard Network
1. Introduction
Natural disasters are common to New York. The state experiences hurricanes, tornadoes, severe thunderstorms and winds, coastal, urban and inland flooding, blizzards, heavy snows, snow squalls, ice storms, ice jams, and extreme heat and cold. Indeed, since 1996 the Federal Emergency Management Agency (FEMA) has issued 40 weather-related major disaster declarations for the state (FEMA 2019). These high-impact weather events have had a significant impact on life and property; over the last decade (2009–18), New York has suffered an annual average of 18 fatalities, 56 injuries, and over $180 million in losses due to weather [National Oceanic and Atmospheric Administration (NOAA); NOAA 2019]. According to Lazo et al. (2011), of all of the state economies in the country, New York’s is the most vulnerable to weather variability and extremes.
Despite New York’s vulnerability to weather events, only 27 Automated Surface Observing System (ASOS) stations are deployed across all of New York State, which is an insufficient number of ground stations to adequately cover the complex terrain and diverse climates of the state. In addition, ASOS sites are not equipped with sensors to measure solar radiation, soil moisture, and snow depth—critical information for monitoring severe storms, flooding, and winter weather. Tropical Storm Lee and Hurricane Irene in 2011 and Hurricane Sandy in 2012 revealed the deficiencies of the sparse ASOS network as the greatest impacts from those events occurred largely within the gaps between sites. Warnings were less effective because of the lack of available ground truth across many of the most vulnerable watersheds of the state.
Following Hurricane Sandy in 2012, the federal government granted the state $17.4 billion in aid through the 2013 Hurricane Sandy Relief Package. Of that total, approximately $30.5 million was used to fund the design and deployment of the New York State Mesonet (NYSM) Early Warning Weather Detection System. The basic design proposed for the standard network was based on similar mesonetworks that were already deployed in other states and regions (Mahmood et al. 2017), such as in Oklahoma (McPherson et al. 2007), western Texas (Schroeder et al. 2005), Kentucky (Mahmood and Foster 2008; Mahmood et al. 2019), southern Alabama (Kimball et al. 2010), Delaware, New Jersey, and Nebraska (Shulski et al. 2018). Nevertheless, a number of innovations were needed to accommodate the state’s challenging geography of mountainous terrain, forests, and large water bodies; its very rural and densely populated areas; its wide variety of extremes (e.g., hurricanes, ice storms, and lake-effect snows); and long, harsh winter conditions.
Initially funded in April of 2014, the now-completed NYSM is composed of a total of 181 observing platforms: 126 standard sites, 20 “snow” sites, 18 “flux” sites, and 17 “profiler” sites. Standard sites measure air temperature, relative humidity (RH), atmospheric pressure, total solar radiation, precipitation, wind speed and direction, snow depth, and soil temperature and moisture. Snow sites measure snow water equivalent. Flux stations measure incoming/outgoing shortwave and longwave radiation, ground heat fluxes, and eddy covariance fluxes of momentum, sensible heat, latent heat, and carbon dioxide. Profiler stations measure atmospheric profiles of wind (using lidar), and temperature, moisture, and liquid water (using microwave radiometers). The first station was installed near Schuylerville (SCHU; Fig. 1) in August of 2015, and the last station was installed near Stony Brook (STON; Fig. 1) in February of 2018. Each station collects averages of measurements every 5 min, and data are relayed to the University at Albany via real-time communications networks. All data are quality controlled, archived, and disseminated from the University at Albany.
A map of the 126 standard stations of the New York State Mesonet showing topography and county boundaries. The four-letter station identifier is commonly the first four letters of the nearest town. The first and last stations installed, Schuylerville (SCHU) and Stony Brook (STON), respectively, are circled in red.
Citation: Journal of Atmospheric and Oceanic Technology 37, 10; 10.1175/JTECH-D-19-0220.1
This paper focuses on the standard network only and is organized as follows. The network design is discussed briefly in section 2. The site configuration, sensors, power, and communications are described in detail in section 3. Network operations and maintenance are reviewed in section 4. Data quality control and metadata and data dissemination are discussed in sections 5 and 6, respectively. A few results are shown in section 7 that highlight the benefits of the network. Section 8 provides a summary and a look ahead.
2. Network design
New York’s complex topography, significant tracts of forest, and large bodies of water limit site selection options. Furthermore, very remote regions with few roads limit site accessibility, whereas densely populated urban areas offered little available ground space for new stations. To overcome these challenges, the NYSM adopted a formal, rigorous process for siting so as to achieve site locations that provided the most accurate, representative data possible for long-term monitoring. Furthermore, station locations were identified that were as protected as possible from high-impact weather disruptions.
The standard network of 126 weather stations provides the backbone of the mesonet infrastructure (Fig. 1). An average station spacing of 27.2 km (16.9 mi) was optimized to sample mesoscale phenomena. At least one station was sited in each of New York’s 62 counties, with additional sites in larger counties. For those counties with diverse landscapes (e.g., complex terrain and large bodies of water), sites were placed in a manner to best sample each region. The National Weather Service (NWS) provided additional input highlighting specific areas and watersheds that are especially sensitive to high-impact weather and observational gaps.
Station locations were selected on the basis of a number of criteria (Brotzge et al. 2016). Those criteria included relatively equal distribution and spacing for mesoscale monitoring, general weather station guidelines set forth by the World Meteorological Organization (WMO; WMO 2012), FEMA recommendations, state and local regulations, and site host preferences. WMO guidelines specify the ideal terrain, vegetation height, and distance from obstacles needed to minimize sensor error. FEMA provided additional guidelines to ensure that mesonet stations continued to operate during and after high-impact events. FEMA requested that stations not be sited within 100-yr flood zones or within designated wetlands. The New York State Historic Preservation Office (SHPO) required that no station be located within a viewshed including a historically designated property. Some regional regulations applied as well. The Adirondack Park Agency (APA) required that no site be seen from a public road so as to preserve natural viewsheds; during the siting process the NYSM team met with APA officials in the field and negotiated exact site locations that were shielded from public view. Many site hosts also had specific locations in mind, and these were used when other criteria were satisfied. Once a site selection was done, an environmental assessment was completed by FEMA, followed by an archaeological excavation at the site to confirm that the grounds contained no historically sensitive artifacts.
Station site hosts were solicited through advertising, organizational periodicals, word of mouth, and cold calls. A 30-yr site lease was signed with site hosts to minimize site moves and disruption of the long-term record. Potential site locations were rejected if site hosts could not guarantee that the land would remain available for 30 years. Site hosts are not paid but are given free access to data from their particular station. About one-half of stations are on public land, and one-half are on private land. Private land is a mix of dairy, crop, and undeveloped land use. Public land is owned by a wide variety of site hosts, including town and county municipalities, colleges and universities, kindergarten–grade 12 (K–12) schools, prisons, and state parks.
Final site locations were a compromise of many competing factors. To determine site representativeness, WMO criteria were evaluated separately for temperature, wind, solar radiation, and precipitation for each station location on a scale from 1 to 5. Those sites at which a particular variable, such as temperature, is measured in ideal conditions with minimal interference or obstructions are categorized as “class 1”; “class 5” is assigned to a site at which significant obstructions or issues impact the quality and/or representativeness of the measurement. An analysis review of the network after installation was complete found that a majority of sites satisfy WMO class-1 criteria for most variables. However, many sites across the heavily forested Adirondack and Catskill Mountains had to be located within 100 m of trees, and so a significant number of these stations do not satisfy class-1 criteria for wind. Likewise, no ground space could be found for placement of stations in New York City (NYC); the five stations in NYC were placed on rooftops hosted by City University of New York (CUNY) colleges. Four of these stations (Brooklyn, Bronx, Queens, and Staten Island) are located on 4–5-story buildings, and the station in Manhattan is located on a 20-story building. In the end, each site location is a compromise among competing requirements, but the implementation of siting innovations such as the strict adherence to a formal siting process; adoption of WMO, FEMA, and local siting standards and regulations; and the use of long-term lease agreements facilitate a robust network designed for high-quality, long-term monitoring across a wide variety of climate zones. All site metadata, including WMO classifications and soil textures, are available on the NYSM website [e.g., see the Voorheesville (VOOR) site metadata at http://nysmesonet.org/about/sites#network=nysm&stid=voor].
3. Site design
Each site was designed to provide the most accurate data possible in real time with minimal interruption. System innovations such as hardened infrastructure, redundant communications, and redundant sensors ensure continuous site operations. Particular sensor selections were made with the goal of optimizing winter weather measurements.
a. Site configuration
Each standard station is configured within a 10 m × 10 m square area to provide adequate space for sensor installation (Fig. 2). At the center of most sites is a steel 9.1-m fold-over tower manufactured by Rohn (55 G). The tower uses a hand crank winch system for lowering the upper portion of the tower. The fold-over mechanism eliminates the need for climbing, which is a critical issue during the winter months when snow and ice can make climbing hazardous. For areas that are susceptible to hurricanes (i.e., Long Island, NYC, and areas in the lower Hudson River Valley), the fold-over towers are not able to withstand the additional wind loading; for the 10 stations in those areas, a fixed, climbable tower was used instead. Chain link or cattle guard fencing is used at most sites. Fenced sites have a gate centered on the north side, which allows the tower to fold over through the open gate without hitting the fencing.
Photograph of a standard station located near Philadelphia, New York (PHIL). Each site is fenced within a 10 m × 10 m square area. Solar panels are placed on the southeast corner of the site to minimize shadows on the panels. A gate is placed on the north center side of the site; the fold-over tower folds down to the north through the open gate. Soil sensors are buried in the northwest corner of the station.
Citation: Journal of Atmospheric and Oceanic Technology 37, 10; 10.1175/JTECH-D-19-0220.1
b. Sensors
As mentioned, NYSM standard stations measure a variety of near-surface and subsurface variables, including air temperature at two levels, relative humidity, wind speed and direction, atmospheric pressure, precipitation, solar radiation, snow depth, and soil temperature and moisture at three depths, and visible images are typically facing northward. Table 1 presents a complete listing of variables measured; additional monitoring and engineering data are also collected (not listed). For example, six additional monitoring variables are collected from the weighing gauge alone, including the input voltage; load cell, electronics, and orifice ring temperatures; self-reported quality control flags; and heat status. These data are used to monitor sensor behavior, quality control, network communications, and system health.
List of variables collected every 5 min from all NYSM standard stations. An asterisk indicates, for the wind “_merge” columns, that prior to 1 Mar 2018 “_prop”is used if data are available and otherwise “_sonic” is used and that starting 1 Mar 2018 “_sonic” is used if data are available and otherwise “_prop” is used.
All data are processed to 5-min intervals by averaging, selecting maximum values, or integrating the raw data in native resolution (3, 12, or 60 s). Five-minute data and images are transmitted to a central facility located at the NYSM headquarters at the University at Albany for postprocessing and dissemination. All raw data are also archived. A typical standard station is shown in Fig. 2.
Specific sensors were selected on the basis of the network requirements for accuracy, measurement range, robustness, power usage, ease of maintenance, and cost. A variety of international vendors competed through an open procurement process for selection. The final list of sensors used for the standard sites and their specifications are summarized in Table 2 and described below. Altogether, the NYSM has 1512 sensors assigned to the standard sites, with an additional 2500 pieces of supporting equipment such as mounts, modems, and antennas. All equipment is tracked using an in-house online inventory database dubbed the “meta db.”
List of standard site sensors and their specifications.
Air temperature is measured at 2 and 9 m AGL using a fast-response thermistor (R. M. Young Co. 41342). The temperature sensors are mounted in aspirated radiation shields (R. M. Young Co. 43502), which prevent radiative heating or cooling of the sensor and protect it from the elements. Collecting temperature measurements at both 2 and 9 m enables monitoring of low-level temperature inversions that provide practical, useful information to the agriculture community. Relative humidity is measured at 2 m AGL. A slow-response temperature sensor and a humidity sensor are combined in the Vaisala, Inc., Humicap HMP155; the additional temperature sensor allows for a more accurate dewpoint measurement. The HMP155 is mounted within an R. M. Young nonaspirated, multiplate radiation shield. The redundant temperature measurements (provided by the two temperature sensors and humidity sensor) are useful for data quality control to identify issues with any of the three thermistors.
Redundant measurements of wind speed and direction are collected using two independent sensors mounted at 10 m. The G. Lufft Mess- und Regeltechnik GmbH (“Lufft”) two-dimensional ultrasonic (sonic) anemometer (v200A) uses sound pulses to extract the wind speed and direction. A top cover minimizes disruption from precipitation and interference from birds landing on top of the sensor. The sensor body contains a heater, which is judiciously used because of its significant power draw. The sonic anemometer is used in parallel with the R. M. Young wind monitor (Model 05103). Vector averaging is computed for both wind speed and direction. The data from the two independent sensors provide much-needed redundancy for real-time quality control and fill data gaps when one sensor fails. The comparisons can identify erroneously slower or zero wind speeds reported by the propeller during snow and freezing-rain events. In turn, such instances can be used as a signal for detecting freezing rain. This is an area of ongoing research and development.
Solar radiation is measured using a Li-Cor, Inc., pyranometer (LI-200SA) mounted at 3 m AGL. The pyranometer is mounted on the end of a horizontal boom oriented toward the southeast to avoid shadows cast from the tower or other sensors and is set within a leveling mount.
A camera is mounted at about 2.5 m AGL. Visible images are taken every 5 min during daylight hours. The camera’s infrared capability allows for nighttime photos, but to reduce bandwidth usage the images are sampled less frequently (hourly) during the overnight hours. Most cameras are mounted facing north approximately level with the horizon, such that the image includes the gate entrance, and equal portions of the ground and sky. The photographs are used to monitor visibility, cloud cover and type, precipitation type, snow depth, and vegetation cover and height. The camera also provides an additional level of security to the site, as well as real-time viewing of unique weather events, interesting atmospheric phenomena, and all kinds of wildlife, and it is quite useful for data quality control and assurance.
The site datalogger, barometer, communications equipment, and power strip are housed within a site enclosure, which is mounted on the tower at about 2 m AGL for easy access. Each enclosure is equipped with a door switch, which is triggered whenever the enclosure door is opened. This switch is used to automatically flag data whenever the enclosure door is open and ensures that all data collected are appropriately flagged while a field technician is on site. Each site has either a Campbell Scientific, Inc., CR3000 or CR6 datalogger. All sensors have cabling connected to the datalogger, and the datalogger is plugged into a power strip. All sites use a similar datalogger program, which directs the sampling rates and averaging intervals as well as the timing of various heating and reporting elements. Some datalogger programs differ slightly among sites to accommodate collocated, nonstandard sensors associated with the flux and snow networks. The datalogger also provides some backup storage capacity where up to 6 months of data are stored locally; whenever the station loses its external communications, the datalogger will continue to archive data locally (excluding camera images). Once communications are reestablished, the missed data are automatically retrieved from this local storage unit.
Barometric pressure is measured using a Vaisala BAROCAP digital barometer PTB330. The barometer is mounted within the site enclosure, which is mounted at about 2 m AGL. A pressure tube extends from the bottom of the pressure sensor through one of the enclosure holes to the outside to reduce errors associated with dynamic changes in pressure caused by external temperature and wind.
Several additional sensors are mounted a few meters from the tower. Precipitation is measured using a Pluvio2 200 manufactured by OTT Hydromet GmbH. The gauge uses a weighing mechanism (a “hermetically sealed load cell”) that records the change in weight between samples to estimate precipitation and precipitation intensity (OTT 2015). A primary advantage of this gauge is that all precipitation types are included in the measurement. The NYSM is the only statewide network using a weighing gauge. Although more expensive than a tipping-bucket gauge, it is essential for measuring frozen precipitation accurately. The gauge has an internal fan to minimize radiative heating errors and a proprietary mathematical algorithm to correct for wind and temperature effects. The rim of the gauge is heated to prevent snow and ice buildup across the gauge orifice, a problem known as capping and dumping (e.g., Rasmussen et al. 2012). The gauge holds up to 1500 mm and must be emptied periodically (at least once per year). Antifreeze is added to each gauge in the autumn to prevent freezing of the liquid and then is emptied in the spring; no additional oils are added to prevent evaporation. An evaporation rate can be deduced from the drop in daily water level in the bucket; however, the accuracy of this estimate has not yet been tested. If bucket evaporation estimates prove comparable to known standards such as pan evaporation, these data could be valuable for some agricultural applications. The precipitation gauge is mounted on a pole at a height of 1.0, 1.5, or 2.0 m AGL, depending on the average maximum snow depth at each site.
A double Alter shield from the Belfort Instrument Co. is placed around the precipitation gauge to minimize wind turbulence effects that can lead to substantial undercatch of frozen precipitation (Rasmussen et al. 2012; WMO 2018). The double shield, with diameters of 1.2 and 2.4 m, is especially important in northern climates such as the Northeast where a significant percentage of precipitation falls as snow. The tops of the 1.2- and 2.4-m-diameter rings are mounted 0.1 and 0.15 m above the top of the gauge, respectively.
Snow depth is measured using an acoustic-based sensor, the Campbell Scientific SR50A-L. Measurements can be compared visually with fixed metal snow sticks placed against the station fencing and within camera view. To maximize data quality, a 1.2 m × 1.2 m white snow board is placed under the SR50A-L to provide a level surface that is clean of vegetation. Sometimes data collection is limited during snowfall because of hydrometeors interfering with the acoustic beam. Drifting of the snow, particularly in high-wind conditions, can limit the accuracy and representativeness of the measurement.
Soil temperature and moisture are measured using the Stevens HydraProbe. Three soil sensors are used at each site, installed at depths of 5, 25, and 50 cm below the surface. During installation, a soil pit was dug at each site, and then each probe was installed horizontally into undisturbed soil at the appropriate depth. The soil pit was then refilled and allowed to settle for 2 weeks prior to commencement of regular observations. The sensor reports soil temperature, soil moisture, soil conductivity, and dielectric permittivity. Soil sensors are buried about 4.5 m to the northwest of the tower. Detailed soil analysis (type and texture) was compiled for all sites at all depths, and this information is stored as part of the site metadata. Note that the five rooftop sites in NYC have no equivalent measurements for soil temperature or moisture.
c. Power and communications
NYSM standard stations are designed to operate with minimal interruption, even during high-impact weather events. Several innovations, such as the use of solar power and redundant communications, improve system readiness. For example, standard ground stations utilize solar power at all but the five NYC sites and one research center site (located at Whiteface) because most sites are located far from utility power sources. However, New York’s high latitude and snowy and cloudy climate necessitated the need for a relatively large and sophisticated solar power system. Each station requires a minimum of 42 W (maximum of 150 W with heaters in use). Heating is recommended by the manufacturers of the precipitation gauge and sonic anemometer, and these have a significant power draw. The power requirements are largely driven by the precipitation gauge (53-W maximum with heater), sonic anemometer (20-W maximum with heater), camera (7 W), aspirated shields (3.5 W each), relative humidity (4 W), and the soil sensors (0.9 W each). To save power, we are evaluating leaving the sonic anemometer heater off; as best as we can determine, the only negative impact this has had on data quality is during icing events because the ice is no longer melted off of the transducers. Fortunately, conditions that lead to ice buildup on the transducers (i.e., wind-driven snow or freezing rain) are rare. Intelligent heating is applied to the precipitation gauge such that the heater is only turned on during precipitation events when the temperature is below 4°C.
The power supply system is composed of four solar panels, six batteries, and a MorningStar Control Charger. The solar panels are relatively large, with each panel measuring ~1.2 m × ~2.1 m because of the overall power needs of the site. The power wattage of the solar panels varies across the network, ranging from 325 to 390 W each. Sites with additional snow or flux measurements and two sites that utilize satellite communications have an additional two solar panels.
Each tower is protected with a lightning rod and grounding network. An aluminum lightning rod is installed at the top of the tower. The rod is connected to the grounding system with an aluminum conductor cable, which is kept isolated from the tower. The lightning-conducting cable is connected to a network of underground cabling connected to two grounding rods buried 1.2 m deep on either side of the tower, as well as the site fencing, precipitation gauge, and power system.
The NYC rooftop locations support direct communications, utility power, and improved security. Each of these sites is connected to a generator or uninterruptible power source so as to maintain data collection for an additional 24+ hours during extended power outages.
Each standard site is equipped with at least two real-time communication options. Most sites use cellular communications as the primary communications option. This permits two-way communications with each station, which allows mesonet operators to monitor and edit software command instructions remotely. The NYSM has a state contract with Verizon, Inc., such that the mesonet pays a monthly rate per site as dependent upon our data use requirements. Each station transmits roughly 170 kilobytes of data every 5 min, corresponding to a station total of ~50 megabytes per day. Each site establishes a virtual private network with the mesonet processing system, which keeps the communications internal and inaccessible to hackers. Each cellular site is equipped with a cellular modem connected to two antennas: a unidirectional antenna, pointing at the nearest cell phone tower, and an omnidirectional antenna, which allows for alternative, secondary cell phone towers to be obtained should the primary tower be disabled. Two dedicated routers at the University at Albany data center provide redundant paths to Verizon’s cellular network. Software is used to automatically switch between the paths should the primary communication network fail to respond in a timely manner.
Two sites across the network have no cellular network access because of their remote locations. As an alternative, these two sites, located near Claryville (CLAR) and Croghan (CROG), use a satellite communications system operated by ViaSat. At these sites, a satellite antenna is placed to the south of the tower and is aimed in the direction of the satellite.
The NYSM operates a backup communications network via Geostationary Operational Environmental Satellite (GOES). GOES communications are operated by NOAA, and NOAA has provided the mesonet with free use of the network. The backup system sends a limited, one-way directional transmission by each station once per hour. Although short, the window allotted allows for most of the 5-min data collected during the previous hour to be transmitted. Every 5 min, observations are read from the primary cellular link, and anything that is missing is filled in with data from the GOES link, when available. This occurs locally somewhat regularly because of short-lived cellular interruptions at individual sites, cellular modem failures, or low-voltage cutoffs that take down the cellular modem. When primary communications are reestablished, full-resolution data are collected from the station to overwrite what was received via GOES during the outage. As an example of how this redundancy has been successful, one instance occurred when both regional Verizon data centers were rendered offline by a networking error by their Internet service provider. The outage lasted about 6 h, during which the NYSM was unable to communicate with 85% of its stations via the cellular link, but GOES transmissions continued uninterrupted.
4. Operations and maintenance
NYSM operations and maintenance (O&M) are designed to minimize outages while providing users with confidence in the quality of the data received. NYSM O&M requires a staff of 12, including a director; program, quality control, data, and calibration managers; software and network engineers; a lead field technician; and four field technicians. These core staff oversee all aspects of the NYSM including the standard, flux, snow, and profiler subnetworks.
Data are collected and transmitted via Verizon, ViaSat, and GOES to a server cluster located at the University at Albany. LoggerNet software developed by Campbell Scientific is used to monitor communications from each station. Additional processing of the raw data is done within the cluster. Quality control is applied to the data automatically and manually, and quality control flags are created for each data value. The data quality assurance and control are described in greater detail in section 5. The quality-controlled data are then combined with fixed metadata, such as site information (e.g., latitude, longitude, and elevation) and sensor calibration coefficients to create processed netCDF files. Raw and processed (netCDF) files are archived, and data derived from the processed netCDF files are disseminated to users and made available for display on the NYSM website.
Processed data and associated quality flags are manually reviewed daily, and “trouble tickets” (TTs) are issued when a sensor or infrastructure problem is detected. The NYSM employs five field technicians to maintain the standard and specialty networks. Each technician is assigned to one of four regions (west, north, central and southeast) across the state with each region consisting of 20–40 stations. Technicians visit a site whenever a TT is issued, usually within 3–7 working days. In addition, each site has two mandatory visits per year, once every spring and autumn, for regular seasonal passes. During each seasonal pass, sensors are cleaned, leveled, and inspected, desiccant is replaced in the site enclosure and SR50A, the precipitation gauge is emptied and, in the fall months, antifreeze is added, select sensors are rotated out for recalibration, and station surroundings are documented with photographs. Each station is visited several more times during the summer to cut vegetation in and around the site perimeter. The technicians collectively drive about 160 000 km (~100 000 mi) annually. The NYSM owns and operates a fleet of six vehicles to service the network.
Select sensors require regular calibration, and those sensors that require calibration are rotated out of operations on a scheduled basis (Table 3). In general, the NYSM returns sensors to their associated vendor for recalibration. Sensors in need of repair are fixed by the NYSM team when possible or are returned to the vendor when necessary. The “meta db” inventory database is used to track all equipment and associated sensor metadata.
List of sensors and their approximate rotation periods.
The NYSM runs an Operations Center in order to monitor site communications, manually review the sensor data quality and camera images, and facilitate coordination with field technicians for repairs (Fig. 3). The Operations Center is staffed 0830 to 1700 LT from Monday through Friday. In general, student interns are used to staff the center, either as part-time employees or as part of an internship for research credit, and one or two students are usually working at any given time. Over the last four years (2016–20), the Operations Center has hosted 35 students from the University at Albany Department of Atmospheric and Environmental Sciences, providing them with a unique opportunity for training, learning, and pursuing research.
The NYSM Operations Center is used for monitoring system networking and data and communicating with field technicians.
Citation: Journal of Atmospheric and Oceanic Technology 37, 10; 10.1175/JTECH-D-19-0220.1
5. Data quality assurance and control
To assure that the data records are of high quality, one needs to 1) properly site the measurement locations, 2) select high-quality sensors, 3) follow good practices on operations and maintenance, and 4) apply rigorous data quality assurance (QA) and quality control (QC) procedures in a timely fashion. In sections 2–4, we discussed what NYSM protocols are followed for the first three steps. However, for any meteorological network, sensors will occasionally fail, and unexpected or rare events can cause data issues. For example, freezing rain, heavy wet snow, lightning (about one strike per year across the entire network), mice, birds, insects, and human activity can impact data quality (Fig. 4). To adequately identify these errors, NYSM QA/QC includes automated algorithms applied in real time and manual review on daily, monthly, and annual time scales.
A number of external factors can lead to sensor failure and/or erroneous data, such as (a) cows at the Clifton Springs (CLIF) site, (b) rodents in the enclosure, (c) snow and ice on the solar panels, (d) a hornet nest on the enclosure, (e) tall grass and weeds, and (f) lightning [at Wantagh (WANT)].
Citation: Journal of Atmospheric and Oceanic Technology 37, 10; 10.1175/JTECH-D-19-0220.1
a. Automated QA
Automated QA routines were adopted from the Oklahoma Mesonet (Fiebrich et al. 2010) and have since been revised to accommodate NYSM sensors and New York’s climate. They include a variety of filters and tests, such as range filters, step tests, and similarity tests (Fig. 5). The automated QA applies a sequence of filters and tests to each variable at a given time, assigns flag values when issues are found, and records log files. Flag values from all tests are input into the “decider.” The “decider” considers individual flag values and any manual overrides in the meta database when assigning a final flag value. Every datum receives an assigned flag. A variable’s final flags are written to the processed (QCed) netCDF file. They are designated 0, 1, 2, or 3, denoting “good,” “suspect,” “warning,” and “failure,” respectively. Only data with flags of 0 and 1 are shown as valid in the final processed netCDF files that are disseminated to general users and displayed on the NYSM public website; all data assigned flags of 2 or 3 are listed as missing values in the processed netCDF file. However, “raw” netCDF files that include all data regardless of their flags are also generated and archived. These raw observations are archived permanently and are never modified. The automated QA is run in real time, is rerun reviewing the most recent 30 min of data, and is run nightly reviewing the last several weeks of data to identify any longer-term irregularities. Automated QA can also be rerun manually over other time periods as needed. Automated QA also generates real-time and daily reports that list all data with nonzero flags. These reports are electronically mailed to relevant individuals for further review. The data QA manager uses the reports to identify and examine the flagged data.
The QA process: raw observations are reviewed using a variety of tests, flags are generated on the basis of the outcomes, the flags are reviewed, and then a final decision is made that determines whether the data are of high enough quality to release. A processed file is then released to the public. The color of the arrows represents the flow of observations, flags, and logs.
Citation: Journal of Atmospheric and Oceanic Technology 37, 10; 10.1175/JTECH-D-19-0220.1
To more easily investigate flagged data, a variety of data and products are displayed. A QA/QC monitoring web page generates plots of data flags (referred to as “status” plots), time series plots on both spatial and temporal domains for the data from the last two days, the first of the month to current, and the last month, as well as difference fields.
As an example, one of the automated QA routines utilizes a “like test.” To make the network more robust, redundant sensors are deployed; e.g., the NYSM has two wind sensors at 10 m AGL and three temperature sensors (two at 2 m and one at 9 m AGL; see Fig. 2). The like test compares wind speed and direction measured by the propeller and sonic anemometers at 10 m and compares temperatures from the aspirated platinum resistance temperature detectors (RTDs) at 2 and 9 m and the nonaspirated HMP155 at 2 m. An example of the wind speed differences between the propeller and sonic anemometers is shown in Fig. 6 for 15–16 April 2018 for the whole network. Each row represents one station, and stations are grouped by climate division. This grouping by geographic regions helps showcase the impact from environmental factors. This case study shows large wind speed differences (larger than the like-test threshold of 1 m s−1) at many sites, so it deserves a more detailed investigation made possible by the manual QA procedures (as will be described in section 5b). For this particular event, differences between the wind sensors (as shown in shades of blue in Fig. 6) highlight a freezing-rain event that slowed or stopped the propeller but did not impact the sonic anemometer.
The 5-min difference (m s−1; color filled) in wind speed between the wind propeller and sonic anemometer is shown at 126 stations (y axis) for 15–16 Apr 2018 (UTC hours; x axis); each row represents a different station, and stations are grouped by their climate division.
Citation: Journal of Atmospheric and Oceanic Technology 37, 10; 10.1175/JTECH-D-19-0220.1
b. Manual QA
Manual QA includes daily, monthly, and annual reviews and is primarily completed by the QA manager with help from Operations Center students. The daily routine allows data to be reviewed multiple ways with a goal of detecting and resolving problems in a timely fashion. The day starts with a morning system report prepared by the student interns in the operational center, who note any communication, power, or camera failures. The QA manager evaluates all flagged data shown in the daily QA report from the prior day and reviews the real-time QA report. Throughout the day, staff and students continue to review 2-day plots on the QA/QC monitoring web page and investigate any special issues that come up. A daily QA summary is captured on a dedicated QC “wiki” web page, and an evening system report is prepared at the end of the day.
If any issues are found that cannot be resolved remotely, a TT is issued through the instrument database website; each TT describes the problem and possible solutions along with an assigned priority number. The TT is electronically mailed to the field technician (and backup technician) responsible for the site and other relevant staff. Technicians prioritize TTs to resolve issues in an optimized manner. Each Wednesday the QA manager conducts weekly QA, approves completed TTs, and composes a prioritized list of outstanding TTs to help technicians plan their trip schedules for the following week.
The sensor repair process is illustrated in Fig. 7. The top panel shows comparisons from three temperature sensors on 18 February 2016 at the Otisville site. On 16 February, it was noticed visually that the temperature at 2 m measured by the RTD agreed well with the other two sensors, but it was a couple of degrees warmer than the other two when the sun was shining. The difference is smaller than the like-test threshold of 3°C set in the automated QA and therefore was not detected by the automated QA. After confirming that this was a repeated issue, a TT was issued and was sent to the technicians on 18 February, and the data were manually flagged with a flag value of “2” (“warning”). On 22 February, a technician visited the site and found that the bottom part of the radiation shield had fallen off (see the photographs in Fig. 7). As a result, the temperature sensor was directly exposed to the sun, artificially inflating the measured temperature. The technician replaced the radiation shield, and the sensor behaved normally thereafter. Without the manual QA, the issue may have gone unnoticed for a while, and the data would have been flagged for a longer period of time.
An example incident showing the sensor repair process. (top) Time series of 5-min temperatures from three temperature sensors (black, red, and green lines from 2-m aspirated RTD, 2-m HMP, and 9-m aspirated RTD, respectively) and solar radiation (orange line on left axis) on 18 Feb 2016 at the Otisville site. The inset photographs shows the problem—a defective radiation shield that fell off (see text for details).
Citation: Journal of Atmospheric and Oceanic Technology 37, 10; 10.1175/JTECH-D-19-0220.1
Another example that demonstrates the manual QA protocol is how wind monitor data are flagged during freezing-rain or wet-snow events. During such events the propeller significantly underestimates wind speed or completely stops working because of ice or wet snow on the propeller. As mentioned previously, the intercomparison between the propeller and sonic anemometers also provides a unique opportunity to identify freezing-rain events and to study their characteristics.
NYSM cameras have also proven very useful in assisting the data QA and QC process. As an example, Fig. 8 shows a time series of 5-min snow depth and daily accumulated precipitation from 1 to 15 February 2017 at the Redfield site. Snow-depth sensor data indicated a large snow accumulation from 90 cm at 0000 UTC to 150 cm at 2300 UTC 3 February 2017. The video from the onsite camera (https://youtu.be/p4wfjBDWiUs; Fig. 8) clearly show the heavy snowfall and quick accumulation, which reached to the top of the 1.8-m (6 ft) fence. However, the precipitation gauge failed to record any precipitation falling into the gauge. From this information, it was determined that the gauge data were erroneous and as a result were flagged, but we were puzzled by how the gauge could have malfunctioned, especially with the gauge heater turned on. When the NYSM technician visited the site on 5 February, he found that the top of the Pluvio gauge was completely buried in snow (Fig. 8). The gauge top orifice was heated, but the snow came down so heavily and quickly that the gauge was capped and buried in a cocoon of snow. Therefore, no more snow could be collected into the bucket, and so no additional precipitation was recorded. Looking further at the time series in Fig. 8, temperatures warmed above freezing for 7–8 February. Relatively warm temperatures combined with rain falling on the snowpack quickly reduced the snow depth. Additional days show minor snowfalls with slow settling and melting of the snowpack during subsequent days.
(top) Time series of 5-min snow depth (cm; black line) and daily accumulated precipitation (mm; blue line on right y-axis scale) from 1 to 15 Feb 2017 at the Redfield site. (bottom left) Picture taken by the station camera at 1650 EST 3 Feb 2017. (lower right) Picture taken by our technician on 5 Feb 2019. The video of this heavy lake-effect snow from the station camera can be seen online (https://youtu.be/p4wfjBDWiUs). Note that the precipitation gauge resets to zero at 0000 UTC each day.
Citation: Journal of Atmospheric and Oceanic Technology 37, 10; 10.1175/JTECH-D-19-0220.1
Monthly QA routines include examining monthly data plots, reviewing old, new, and recurring issues and flagging erroneous data undetected by daily QA. Additional statistical analyses are computed monthly, such as calculating the percentage of good data (with flag = 0) per sensor per site, along with the monthly mean, maximum, minimum, and standard deviation of each variable. A monthly data quality report summary is prepared, presented, and discussed at monthly QA meetings. At the monthly meetings, overall monthly network performance and data problems are discussed, possible solutions are brainstormed, and action items are created. Annual QA is conducted in the beginning of each year, and the data review is similar to what is done for the monthly QA. In addition, annual data are analyzed to document and study the changes and variabilities for each variable during the year.
c. Metadata
A major innovation of the NYSM is to make system metadata readily available and accessible to users. The NYSM used the recommendations set forth by Muller et al. (2013) as a guide for the metadata it should prioritize. As a result, extensive network, station, and sensor metadata are available on the NYSM website (e.g., Fig. 9). Network metadata include a general history, descriptions of each network (standard, profiler, flux, and snow), and network maps. Individual station metadata include latitude, longitude, county, general location, elevation, climate division, site type, soil type and composition, installation date, NWS weather forecast office (WFO) region, a list of nearby obstructions within 100 m, and a description of nearby topography and landscape. Each station was also evaluated for each variable against WMO guidelines for siting, and those classifications are also available. In addition, a set of station photographs is archived and updated every spring and autumn. Sensor information is also documented, including vendor, model, accuracy and reliability, measurement height, sampling rate, sensing range, and brief description of sensing limitations. A “readme” file is provided to users that describes the data file format, variables, and units. Additional metadata are kept internally for system operations, such as station host contact information and sensor histories.
An example of the metadata that are available online for each of the NYSM’s stations.
Citation: Journal of Atmospheric and Oceanic Technology 37, 10; 10.1175/JTECH-D-19-0220.1
6. Data dissemination
Data and products are disseminated through a variety of ways. Real-time data and products are displayed on the NYSM website (http://nysmesonet.org), with interesting cases and high-impact events highlighted on the mesonet Facebook and Twitter accounts. The NYSM also distributes a quarterly newsletter to site hosts and users to provide a more focused platform for communicating network updates and interesting news events.
Real-time data are disseminated directly to several partnering organizations, including the NWS, the New York State Division of Homeland Security and Emergency Services (DHSES), and the New York Independent System Operator (NYISO). The NWS has a real-time data feed available via NOAA Meteorological Assimilation Data Ingest System (MADIS), and the DHSES accesses NYSM products within its emergency operations center. NWS operational models assimilate NYSM standard data via MADIS. Local NWS WFOs utilize real-time data and photographs in numerous forecast discussions, alerts, and warnings. Archived data are available upon request; data requests are submitted through an online form (http://nysmesonet.org/data/requestdata). A data recovery fee is charged for most commercial and academic use. To date, over 1000 requests for data have been received for a wide variety of uses. Requests for archived data have come from the transportation, water management, public health, and utility sectors. Data requests have also come from the private commercial sector for agriculture, civil engineering, and forensic meteorology, and there have been a number of requests for private use. Data are used extensively by K–12 schools and colleges and universities for education and research.
7. Example applications
The primary purpose of the NYSM is to provide comprehensive situational awareness of the environment for emergency management (EM) operations and, given the initial feedback from the EM community, this has indeed occurred. Three examples follow that demonstrate both the needs and value of the statewide network.
On 8 August 2019, a strong cold front moved rapidly east across the state with moderate convection along the front, followed by strong gusty winds just behind the frontal boundary. In partnership with the NWS, the NYSM developed a guidance tool for displaying relevant data and trends most pertinent for monitoring convective weather and the prestorm environment (Fig. 10a). WSR-88D output is overlaid on objectively analyzed NYSM data. Wind gusts recorded within the previous 3 h that exceed a given threshold [e.g., 11 m s−1 (25 mi h−1)] are plotted. Site photos provide additional situational awareness (Fig. 10b). In this case, a wind gust of 24.4 m s−1 was recorded at the time of the photograph. A waterspout over Lake Ontario was seen by the site camera at Oswego later in the day.
(a) A statewide analysis of severe weather parameters and trends using NYSM data with WSR-88D imagery overlaid. (b) A camera photograph from the Scipio (SCIP) station looking north toward a significant storm cell as identified by radar.
Citation: Journal of Atmospheric and Oceanic Technology 37, 10; 10.1175/JTECH-D-19-0220.1
A second example event demonstrates the value of the NYSM for winter weather. A significant early March storm with relatively high precipitation was forecast for most of New York for 1–2 March 2018. However, most global and many mesoscale models suggested it would be mostly or all rain. In actuality, many locations received snow the entire time. Camera, snow-depth, and precipitation data from NYSM provided real-time situational awareness to NWS, transportation, and EM officials, alerting them early on that surface conditions were deteriorating more rapidly than expected. Snowfall accumulation maps of the event highlight the widespread significant snows received across a large portion of the state (e.g., Fig. 11a). Camera photographs provide verification of automated precipitation and snow-depth measurements (Fig. 11b). The NYSM surface air temperature data showed temperatures cooled to the freezing mark, which suggested a significant warm bias in model forecasts.
(a) Objectively analyzed snow totals from the 1–2 Mar 2018 storm as analyzed by the NYSM. (b) Camera photograph from the Cobleskill site, where the state’s highest snow total was confirmed by the camera image. Bands on the snow stick are marked every 6 in. (1 in. = 2.54 cm), and the photo shows a total exceeding the 24 in. mark.
Citation: Journal of Atmospheric and Oceanic Technology 37, 10; 10.1175/JTECH-D-19-0220.1
A third example shows the value of NYSM data for flash-flood warning. From 31 October to 1 November 2019, a strengthening low pressure system approached New York from the west. Southerly winds pulled warm, moist air north across the state ahead of an eastward-moving cold front. During the early evening hours of 31 October, two very intense rainbands moved across northern New York with significant training of cells over a narrow swath across the southern Adirondacks. However, as is common in complex terrain, radar-derived precipitation totals underestimated surface rainfall, in part because of radar blockage and overshooting of the radar beam as a result of the relatively long distance of NWS radars from the southern Adirondacks. The Multi-Radar Multi-Sensor (MRMS; Zhang et al. 2016) precipitation algorithm estimated maximum 24-h rainfall totals of just over 3 in. (76 mm); however, NYSM-observed rainfall totals well exceeded 5 in. (127 mm; Fig. 12a). The MRMS did not include NYSM data at that time. Fortunately, NYSM data are available to local NWS WFOs in real time, and the appropriate warnings were issued quickly, in part on the basis of the NYSM reports. An example timeline from a NYSM station located near Cold Brook shows two very intense rainfall bands (Fig. 12b). The first band was associated with a 5-min precipitation rate of 5.1 in. h−1 (130 mm h−1) at 1945 LT; this contributed to issuance of an NWS flash-flood warning within 30 min. A precipitation rate of 7.54 in. h−1 (192 mm h−1) was recorded for a 5-min period at the same site with a second band at 2200 LT. An NWS flash-flood emergency was issued about an hour later. Following the storm, the local NWS WFO in Albany stated, “The NY State Mesonet observations have been key to assessing near term flood potential and wind threats with the storm.”
NWS and NYSM precipitation measurements from 31 Oct through 1 Nov 2019: (a) Statewide 24-h precipitation map; the background colors are the MRMS estimates from the NWS, with NYSM observations plotted using the same scale and color coding. Black circles represent stations that recorded amounts greater than 3.5 in. (the maximum shown by the scale). (b) Accumulated precipitation and precipitation intensity recorded at Cold Brook (COLD) plotted as a function of time, with NWS advisories and warnings overlaid. This figure is provided through the courtesy of A. Lunavictoria.
Citation: Journal of Atmospheric and Oceanic Technology 37, 10; 10.1175/JTECH-D-19-0220.1
8. Summary and conclusions
The NYSM is a comprehensive, multipurpose, end-to-end system of data collection, transmission, processing, and dissemination. Real-time environmental information is collected at 126 standard stations across the state, with files and graphics generated every 5 min for immediate distribution and display.
Significant challenges were posed by New York’s geography, land cover, population density, and extreme weather. To address siting obstacles, the NYSM implemented the WMO standards and FEMA guidelines for siting; solicited input from the NWS, emergency management, and agricultural communities; and made extensive use of documentation and metadata for recording site properties. To ensure the continuance of system operation during high-impact weather, the NYSM adopted several innovations, including use of redundant communication networks, some duplicate measurements, and vigorous preventive maintenance. To date, these advances have been successful, with a network uptime of over 99.73% during 2019 and over one billion observations now collected and archived. To obtain accurate, representative hydrometeorological measurements, particularly during winter weather when data are often most critical to emergency management operations, several unique features were added to the NYSM, including weighing precipitation gauges with a double Alter shield, snow-depth sensors, soil moisture and temperature sensors, cameras, and snow sticks (within camera view). Redundant wind sensors also provide a reliable indication of ice buildup on surfaces.
A multilayered approach to sensor repairs, calibration rotations, and quality control has been implemented to reduce sensor down time and maintain high-quality data. Automated and manual quality control are designed to identify suspect issues quickly, and the staff of full-time field technicians restore failed sensors usually within 1–2 weeks. Extensive metadata are available online and within data files to clearly document network changes and outages.
With the NYSM being fully operational, several state and federal agencies now utilize NYSM data routinely for their operations and for research and product development (R&D). Building upon this foundation, the NYSM is now working to create an in-house calibration laboratory, develop an outdoor laboratory for sensor testing and R&D, and expand its online products. With its frequent, dense, homogeneous measurements collected across a uniquely diverse and complex region, the NYSM offers a long-term, reliable platform for use by the operational and research communities for a wide range of activities.
Acknowledgments
This research is made possible by the New York State (NYS) Mesonet. Original funding for the NYS Mesonet was provided by Federal Emergency Management Agency Grant FEMA-4085-DR-NY, with the continued support of the NYS Division of Homeland Security and Emergency Services; the state of New York; the Research Foundation for the State University of New York (SUNY); the University at Albany, SUNY; the Atmospheric Sciences Research Center (ASRC) at SUNY Albany; and the Department of Atmospheric and Environmental Sciences (DAES) at SUNY Albany. The National Center for Atmospheric Research is sponsored by the National Science Foundation.
REFERENCES
Brotzge, J., and Coauthors, 2016: The New York State Mesonet: Siting process and challenges. 18th Symp. on Meteorological Observation and Instrumentation, New Orleans, LA, Amer. Meteor. Soc., 2.3, https://ams.confex.com/ams/96Annual/webprogram/Paper283653.html.
FEMA, 2019: 2019 disasters. Department of Homeland Security, https://www.fema.gov/disasters.
Fiebrich, C., C. Morgan, A. McCombs, P. Hall Jr., and R. McPherson, 2010: Quality assurance procedures for mesoscale meteorological data. J. Atmos. Oceanic Technol., 27, 1565–1582, https://doi.org/10.1175/2010JTECHA1433.1.
Kimball, S. K., M. S. Mulekar, S. Cummings, and J. Stamates, 2010: The University of South Alabama Mesonet and coastal observing system: A technical and statistical overview. J. Atmos. Oceanic Technol., 27, 1417–1439, https://doi.org/10.1175/2010JTECHA1376.1.
Lazo, J., M. Lawson, P. Larsen, and D. Waldman, 2011: U. S. economic sensitivity to weather variability. Bull. Amer. Meteor. Soc., 92, 709–720, https://doi.org/10.1175/2011BAMS2928.1.
Mahmood, R., and S. A. Foster, 2008: Mesoscale weather and climate observations in Kentucky for societal benefit. Focus Geogr., 50, 32–36, https://doi.org/10.1111/j.1949-8535.2008.tb00210.x.
Mahmood, R., and Coauthors, 2017: Mesonets: Mesoscale weather and climate observations for the United States. Bull. Amer. Meteor. Soc., 98, 1349–1361, https://doi.org/10.1175/BAMS-D-15-00258.1.
Mahmood, R., M. Schargorodski, S. Foster, and A. Quilligan, 2019: A technical overview of the Kentucky Mesonet. J. Atmos. Oceanic Technol., 36, 1753–1771, https://doi.org/10.1175/JTECH-D-18-0198.1.
McPherson, R., and Coauthors, 2007: Statewide monitoring of the mesoscale environment: A technical update on the Oklahoma mesonet. J. Atmos. Oceanic Technol., 24, 301–321, https://doi.org/10.1175/JTECH1976.1.
Muller, C., L. Chapman, C. Grimmond, D. Young, and X.-M. Cai, 2013: Toward a standardized metadata protocol for urban meteorological networks. Bull. Amer. Meteor. Soc., 94, 1161–1185, https://doi.org/10.1175/BAMS-D-12-00096.1.
NOAA, 2019: Natural hazard statistics. accessed 24 May 2019, https://www.nws.noaa.gov/om/hazstats.shtml.
OTT, 2015: Operating instructions: OTT Pluvio2 precipitation gauge. OTT Hydromet Doc., 60 pp.
Rasmussen, R., and Coauthors, 2012: How well are we measuring snow: The NOAA/FAA/NCAR winter precipitation test bed. Bull. Amer. Meteor. Soc., 93, 811–829, https://doi.org/10.1175/BAMS-D-11-00052.1.
Schroeder, J. L., W. S. Burgett, K. B. Haynie, I. Sonmez, G. D. Skwira, A. L. Doggett, and J. W. Lipe, 2005: The West Texas Mesonet: A technical overview. J. Atmos. Oceanic Technol., 22, 211–222, https://doi.org/10.1175/JTECH-1690.1.
Shulski, M., S. Cooper, G. Roebke, and A. Dutcher, 2018: The Nebraska Mesonet: Technical overview of an automated state weather network. J. Atmos. Oceanic Technol., 35, 2189–2200, https://doi.org/10.1175/JTECH-D-17-0181.1.
WMO, 2012: Annex 1.B siting classification for surface observing stations on land. Guide to meteorological instruments and methods of observation. WMO Rep. 8, 30–42.
Zhang, J., and Coauthors, 2016: Multi-Radar Multi-Sensor (MRMS) quantitative precipitation estimation: Initial operating capabilities. Bull. Amer. Meteor. Soc., 97, 621–638, https://doi.org/10.1175/BAMS-D-14-00174.1.
