Evaluation and Intercomparison of Small Uncrewed Aircraft Systems Used for Atmospheric Research
1. Introduction
The lower atmosphere plays a significant role in the modulation of weather and climate. Not only is this the portion of the atmosphere that interacts most directly with Earth’s inhabitants, but it is also the portion that regulates energy transfer between land, ice, or ocean and the overlying air. Through these interactions, the atmospheric boundary layer is a key contributor to significant weather events, such as tropical cyclones (e.g., Ooyama 1969; Emanuel 1986; Rotunno et al. 2009), fog (e.g., Barker 1977; Fitzjarrald and Lala 1989; Koračin et al. 2005), and thunderstorms (e.g., Weaver 1979; Wilson and Megenhardt 1997; Trapp et al. 2007), as well as to climate-relevant processes such as drought (e.g., Charney 1975; Meng et al. 2013), sea ice melt (e.g., Maykut 1978; Kay and Gettelman 2009; Kay et al. 2011), and warming of land and ocean temperatures (e.g., Larson et al. 1999; Bony and Dufresne 2005).
Understanding the importance of this critical regime, the scientific community has worked for decades to improve the representation of the atmospheric boundary layer and surface–atmosphere exchange in numerical prediction tools across a variety of spatial scales (e.g., Martin et al. 2000; Pleim 2007; Brown et al. 2008). Such work has traditionally relied upon input from detailed observations collected using remote sensors, weather balloons, and research aircraft (e.g., Stull and Eloranta 1984; Cuxart et al. 2000; Lothon et al. 2014). Together, these sensor systems have provided statistics necessary for both evaluating the performance of currently used parameterizations (e.g., Holt and Raman 1988; Nolan et al. 2009; Hu et al. 2010), as well as for supporting the development of new parameterizations (e.g., Estournel and Guedalia 1987; Lock et al. 2000) with the goal of improving prediction of weather and climate.
Recent decades have seen significant progress in the advancement of sensing systems to support these efforts. One example has been the introduction and proliferation of small uncrewed aircraft systems (sUAS) into the atmospheric and Earth science communities. Initial work to leverage such systems was undertaken in the 1980s, primarily leveraging hobby-grade remote controlled aircraft (e.g., Tomlins 1983). Subsequent decades saw the advancement of sUAS technology, particularly for military applications, resulting in the introduction of more sophisticated and larger sUAS platforms to support studies of the atmosphere (e.g., Stephens et al. 2000; Holland et al. 2001; Intrieri et al. 2014). Those efforts provided ground-breaking perspectives on atmospheric and surface conditions without the need for large research aircraft and the associated infrastructure (e.g., airports, large instruments). These larger and complex sUAS are still regularly used for scientific research, including by the U.S. Department of Energy, National Oceanic and Atmospheric Administration, and the National Aeronautics and Space Administration, even though their cost and complexity generally makes it challenging for individual research teams to stand up operational capabilities using these systems.
Over the past decade or so, substantial advancement in consumer electronics have supported the miniaturization of numerous sensor components. At the same time, significant progress has been made in the hardening and development of autopilot systems for smaller remotely piloted aircraft, allowing for the development of both commercially available and custom-developed small (<55 lb; <25 kg) sUAS platforms. These factors have supported the development of a new class of research-grade sUAS, with operators who range from university teams to government laboratories, to branches of the U.S. military and to the public as part of citizen science efforts. This advancement has resulted in a sUAS revolution in atmospheric and Earth sciences, including frequent deployment of such systems for atmospheric boundary layer (ABL) research, and in the development of conference sessions and a separate international society focused on this specific topic [the International Society for Atmospheric Research using Remotely piloted Aircraft (ISARRA); de Boer et al. 2019].
Measurements provided by these systems are used to advance process-level understanding of physical (e.g., Mayer et al. 2012; Reuder and Jonassen 2012; de Boer et al. 2016; Al-Ghussain and Bailey 2022; Bailey et al. 2023) and chemical (e.g., Chen et al. 2018; Rüdiger et al. 2018; Brus et al. 2021) processes in the ABL and beyond. Such studies are being conducted across geographic regimes, including in the tropics (e.g., de Boer et al. 2022a), at high latitudes (e.g., de Boer et al. 2018; Kral et al. 2021; de Boer et al. 2022b), and in between (e.g., de Boer et al. 2020). They have allowed for in situ profiling of the lower atmosphere in areas with little infrastructure to support more traditional instrumentation and in weather conditions that have been deemed unsuitable or unsafe for larger research aircraft (e.g., Roadman et al. 2012; Xi et al. 2016; Cione et al. 2020). Recent work has also been undertaken to understand the potential impact that observations from these systems may have on numerical weather prediction should such observations be available in near–real time for assimilation into operational weather prediction systems (e.g., Jensen et al. 2021; Koch et al. 2018), with promising results.
Given the increased use of these systems and the amount of diversity in sensors and integration techniques, natural questions have arisen about the quality of the data collected by sUAS. In many cases, significant work has gone into ensuring that the measurements collected by these systems are equivalent to those collected using “standard” systems such as radiosondes and meteorological towers. For example, Greene et al. (2018) carefully evaluated the impacts of sensor placement when attempting to measure temperature from a rotary-winged system, finding that that sensor placement underneath a propeller on such a platform, at one-quarter the length of the propeller from the tip minimizes influences of turbulence and motor, compressional, and frictional heating while still maintaining adequate airflow. Additionally, a limited number of intercomparison efforts have been conducted to help shed light on the quality and intercomparability of sUAS observations for atmospheric research. One such study (Barbieri et al. 2019) leveraged a vehicle-mounted 15 m meteorological mast to intercompare observations from numerous different sUAS platforms. Measurements of temperature, humidity, winds, and pressure from many of the systems were found to compare well with those from the tower, though there were also notable differences resulting from sensor integration techniques (e.g., use of solar shielding and/or ventilation techniques) and the type of platform (fixed- versus rotary-wing) used.
In the current study, measurements from several sUAS platforms are compared to those from different observing facilities at the U.S. Department of Energy’s (DOE) Atmospheric Radiation Measurement (ARM) Southern Great Plains (SGP) facility in Oklahoma (United States). Over the course of two weeks, teams from five different institutions conducted flights for the sole purpose of system evaluation and intercomparison. This included two fixed-wing and three rotary-wing platforms, as will be described in the next section, and included three university-developed aircraft and two systems developed in the commercial sector. Between the various platforms, 153 flights were used for the current intercomparison for a combined total of 43.2 flight hours. These flights were primarily conducted at the DOE ARM SGP facility, though there were also several flights conducted at an Oklahoma Mesonet site in Marshall, Oklahoma. In section 3 of the manuscript, we provide detailed information on the intercomparison effort, including information on weather conditions, and the structure of the comparison between different platforms and sensors. Section 4 offers discussion on the outcomes and results of these comparisons, and discussion on the potential causes of noted discrepancies between sensors and systems. Finally, section 5 provides a summary and outlook on the implications of the results presented and recommendations on further efforts to advance sUAS-based atmospheric sensing.
2. Description of systems and sensors
Data collected in support of the current analysis were obtained using various sUAS and ground-based and in situ sensors deployed at the U.S. DOE ARM program SGP facility in Lamont, Oklahoma (36.6044°N, 97.4859°W, 314 m MSL). In this section, we provide descriptions of the sUAS and their instrumentation, as well as of the U.S. DOE ARM instrumentation that was used in development of comparison datasets.
a. Uncrewed aircraft systems
1) University of Colorado RAAVEN
The RAAVEN sUAS (Fig. 1) is a fixed-wing sUAS with a wingspan of 2.3 m and has been operated by the University of Colorado Boulder since 2019. The RAAVEN’s body is a modified version of a commercially available DRAK model from RiteWing RC. Modifications to the airframe include integration of a tail boom to enhance longitudinal stability and improve the platform’s performance. The RAAVEN is controlled through a PixHawk2 autopilot system and employs an 8S 21 000 mAh lithium ion (Li-Ion) battery pack, resulting in a maximum endurance of around 2.5 h. The aircraft can travel up to approximately 36 m s−1 through the air, though operations during the flights discussed in this manuscript were almost exclusively conducted in the 15–22 m s−1 range.
Sensing systems used in this intercomparison, including (clockwise from top left) the University of Colorado RAAVEN, the ARM SGP 60-m tower, the University of Nebraska M-600, the University of Oklahoma CopterSonde 3D, the University of Nebraska Meteodrone, and the Black Swift Technologies S0.
Citation: Journal of Atmospheric and Oceanic Technology 41, 2; 10.1175/JTECH-D-23-0067.1
For the current effort, the RAAVEN carried an instrument suite derived from the miniFlux payload, which was codeveloped by the National Oceanic and Atmospheric Administration (NOAA), the Cooperative Institute for Research in Environmental Sciences (CIRES) and IRISS at the University of Colorado [more details on the payload, postprocessing, wind estimation, and data acquisition are available in de Boer et al. (2022a) and Cleary et al. (2022)]. In this configuration, RAAVEN is set up to measure atmospheric and surface properties, evaluate thermodynamic state, kinematic state, and turbulent fluxes of heat and momentum. This includes a multihole pressure probe (MHP) from Black Swift Technologies, LLC (BST), a pair of RSS421 pressure, temperature, humidity (PTH) sensors from Vaisala, Inc., a custom finewire array, developed and manufactured at the University of Colorado Boulder, a pair of Melexis MLX90614 IR thermometers, and a VectorNav VN-300 inertial navigation system (INS) (see Fig. 1). This sensor suite is logged using a custom-designed FlexLogger datalogging system at rates between 1 and 250 Hz, with logging time stamps saved to provide very accurate aggregation of variables to common time stamps in postprocessing.
The Vaisala RSS421 sensors are identical to those used in the Vaisala RD41 dropsonde. For temperature measurements, this unit features a resolution of 0.01°C, repeatability of 0.1°C, and a response time of 0.5 s at 1000 hPa when moving at 6 m s−1. For relative humidity (RH), the RSS421 has a resolution of 0.1% RH and a repeatability of 2% RH, with a temperature-dependent response time of better than 0.3 s at 20°C (again, as measured within the RS41, with 6 m s−1 airflow at 1000 hPa). Finally, the pressure sensor has a resolution of 0.01 hPa and a repeatability of 0.4 hPa. Some additional details on these sensors can be found in de Boer et al. (2022a) or Cleary et al. (2022). For the flights conducted for the current analysis, a pair of these sensor modules were mounted to the top of the RAAVEN’s fuselage, between the nose and the tail of the aircraft on the port side, with the sensor mounting angles offset to reduce the possibility of coincident solar exposure to both sensors. Some additional flights were completed where the RSS421 sensors were mounted inside a tubular housing integrated into the nose of the aircraft. These flights were meant to evaluate a configuration that offers some mechanical protection of the sensors from precipitation and other particles.
Additional information on atmospheric thermodynamic state from RAAVEN is available from an E+E EE03 sensor that is integrated into the BST MHP and from a Sensiron SHT-85 sensor that is integrated in the finewire array. The EE03 has a manufacturer-stated temperature accuracy of 0.3°C (at 20°C), while the humidity sensor’s accuracy is stated to be 3% RH at 21°C. The SHT-85 has a temperature accuracy of 0.1°C (from 20° to 50°C) and repeatability of 0.08 C, while the humidity sensor has a stated accuracy of 1.5% RH and a repeatability of 0.15% RH. Both sensors have slower response times than the RSS421 sensor and are therefore only used in case of a complete failure of the RSS421.
In addition to the SHT-85 sensor, the University of Colorado–designed finewire array contains two 5-μm-diameter platinum wires that are extended into the free stream over a 2 mm length between supporting prongs. One wire is operated as a hotwire (100°C overheating) anemometer, while the other is operated as a coldwire (1°C overheating) thermometer. These wires have thermal time constants of 0.5 m s in a 15 m s−1 airflow regime and support a sampling frequency of up to 800 Hz. Changes in wire resistance due to velocity or temperature variability are electronically converted to amplified voltages that can be used to calculate air temperature and velocity through derivation of a polynomial fit to information from other onboard sensors.
In addition to information available from the EE03 PTH, the BST five-hole probe supports measurement of airspeed, angle of attack (α) and sideslip angle (β). These measurements support the calculation of three-component winds when combined with GPS-based ground velocities and aircraft attitude from the VectorNav VN-300 [see de Boer et al. (2022a) for details]. Under dynamic conditions, the VN-300 system has a stated yaw accuracy of 0.3°, 0.1° in pitch and roll, 2.5 m horizontal position accuracy, 2.5 m vertical position accuracy when integrating information from the barometric pressure sensor, and 0.05 m s−1 accuracy in inertial velocity. Input from the VN-300’s gyroscope, accelerometer, GNSS receiver, magnetometer and pressure sensor are filtered through an extended Kalman filter (EKF) to produce a navigation solution.
Finally, RAAVEN deploys a pair of Melexis MLX90614 IR thermometers (one upward looking and the other surface viewing). These sensors are factory calibrated to measure target brightness temperatures between −70° and 380°C. They have a high accuracy (0.5°C) and a measurement resolution of 0.02°C. For the current campaign, the RAAVEN carried the “I” version of this sensor, which has a 5° field of view. These sensors have a broad passband range of 5–14 μm, meaning that while it covers the infrared atmospheric window, it is also subject to radiation emitted by water vapor and other radiatively active gases, meaning that a significant depth of atmosphere between the aircraft and a given target (e.g., cloud, surface), atmospheric gases influence the temperature reading.
2) Black Swift Technologies S0
The S0 sUAS was designed by Black Swift Technologies for generation of rapid wind profiles, providing the capability to determine the thermodynamic properties of the lower 4.5 km of the atmosphere at a high resolution in under 20 min. It is derived from a tube-launched variant designed for use in hurricane observation after being deployed by the NOAA P3 and features the same SwiftCore autopilot system and tightly coupled multihole probe for three-dimensional wind measurements. The S0 has an anticipated dash speed of up to 44 m s−1 and cruises at 19 m s−1. Flight times vary due to atmospheric conditions, but the maximum endurance at sea level is 80 min. The vehicle is designed to be simple to transport and operate, and consists of a simple lightweight tube fuselage, removable wing set and three-dimensional printed tails. Contained within the aircraft is a panel providing USB access for high-rate onboard recorded data, a charge port that enables field charging without specialized equipment, and two LEDs to indicate charging status and operational status. This limits the other required deployment equipment to a tablet, small ground station, and a charging cable.
The sensing core of the S0 is composed of the Vaisala RSS421 mentioned in the RAAVEN description above, along with a five-hole probe pressure sensor. When combined with the onboard RTK GPS and inertial sensors, this provides wind measurements with the accuracy of 0.3 m s−1 for the horizontal components and 0.4 m s−1 in the vertical axis. These measurements are made at 100 Hz and are telemetered to the ground station for near-real-time observations. The raw sensors are also recorded at frequencies up to 1 kHz and are available through a USB interface for postprocessing, allowing for more accurate measurements to be obtained than those relayed during flight given the ability to better adjust for error sources such as clock drift and variable delay in the position solution.
The S0 is flown using the SwiftCore flight management system. This consists of the SwiftCore autopilot, SwiftStation, and SwiftTab user interface. This system was designed from the ground up to perform atmospheric sampling missions and features many unique capabilities not available in other platforms intended for typical commercial uses, such as photogrammetry. Of the unique features in the system, the advantage for campaigns such as the one described in this publication is the ability to generate patterns quickly and easily, including racetracks, barbells, and volumetric grids with a minimal set of parameters and an outline of the desired sampling area on a map. These patterns can be created and adjusted during flight, allowing significant flexibility in measurement campaigns.
3) University of Oklahoma CopterSonde
The CopterSonde 3D (hereafter CopterSonde) sUAS (Segales et al. 2020; Segales 2022), was designed and manufactured by the University of Oklahoma (OU) and maintained by the Cooperative Institute for High-Impact and Severe Weather Research and Operations (CIWRO). The data collected with the CopterSonde provide the same information as radiosondes but with higher spatiotemporal resolution besides having full control of the desired sampling location. This system has been proven to be a powerful reusable tool for collecting thermodynamic and kinematic profiles of the planetary boundary layer in a variety of environments from summer preconvective environments in the Southern Great Plains (Koch et al. 2018), high-altitude valleys (Pillar-Little et al. 2021; Lappin et al. 2022), winter stable boundary layers in the Arctic (Kral et al. 2021; Greene et al. 2022), and leading up to and during frozen precipitation events (Tripp et al. 2021).
The CopterSonde is a rotary-wing platform that is based on a modified version of the Lynxmotion HQuad500 (LHQ500) wide-X type quadcopter with fixed-pitch rotors. The platform is powered by a 4S 5870 mAh LiPo smart battery. Structural modifications were made to the LHQ500 airframe to fit additional electronics using custom 3D-printed parts, with the arrangement of components optimized for vertical profiling operations. The CopterSonde shell was designed to have less air resistance when the platform is facing into the wind, allowing the system to achieve high wind speed tolerance with reduced power consumption. The CopterSonde is controlled via a CubePilot autopilot system, running a modified ArduPilot code developed by OU-CIWRO. The CopterSonde has a maximum ascent speed of 12.2 m s−1, a maximum stable descent speed of 7 m s−1, and a maximum wind tolerance of 22 m s−1. Based on the energy budget analysis described in Segales (2022), the CopterSonde was operated with ascent speeds on the order of 1–4 m s−1 and descent speeds of 1–6 m s−1.
The CopterSonde is outfitted with a modular thermodynamic scoop consisting of a trapezoidal opening that leads into an L-shaped duct that has three iMet-XF bead thermistors and three IST HYT-271 capacitive humidity sensors in an “inverted V” configuration along the back of the duct. According to manufacturer specifications, the bead thermistors have a resolution of 0.01°C, accuracy of ±0.3°C, and a time response of less than 2 s while the humidity sensors have a 0.1% RH resolution, ±0.1% accuracy, and a time response of less than 5 s. These sensors are aspirated by a fan at the base of the duct, calibrated to draw air over the sensors at 12 m s−1. It was also programmed to switch on/off after takeoff/before landing to prevent the fan from blowing dust and debris from the ground onto the sensors. Additionally, there is a built-in MS5611 microelectromechanical system (MEMS) in the CubePilot system that provides pressure measurements and supports altitude control. It has a resolution of 0.012 hPa, an accuracy of ±1.5 hPa, and a time response of less than 8.22 m s. All atmospheric sensors on board the CopterSonde were programmed to sample and log data at 10 Hz tagged with GPS locations and time stamps. The scoop design together with the CopterSonde configuration have been demonstrated to be quite comparable to conventional radiosondes, with temperature and relative humidity measurements being accurate within ±0.1°C and ±2%, respectively (Bell et al. 2020). Further information regarding the development of sensor placement and calibration strategies for the thermodynamic sensors can be found in Greene et al. (2018, 2019) while the design of the scoop system is outlined in Segales et al. (2020).
The roll, pitch, and yaw of the quadcopter are estimated by the autopilot’s inertial measurement unit (IMU). These measurements are used to derive the horizontal wind vector based on the pitch angle and the projected area normal to the wind. A preliminary estimation of the wind vector is completed on board the aircraft in near–real time while using the wind vane algorithm. This custom algorithm, dubbed the wind vane flight mode (WVFM), directs the sUAS to turn into the wind while profiling, allowing the CopterSonde to maintain an efficient and constant aerodynamic configuration for wind speed estimation. The WVFM also ensures that the thermodynamic scoop is sampling air that is the most representative of the environment and not disturbed by the sUAS (Segales 2022). The final horizontal wind product was shown to be accurate to ±0.65 m s−1 and ±4° relative to Doppler wind lidars and radiosondes (Bell et al. 2020). The complete wind vector is derived during postprocessing. Readers are directed to Segales (2022) for more information regarding the wind vane algorithm, horizontal wind vector calculation, and calibration process.
4) University of Nebraska Meteodrone
The University of Nebraska–Lincoln (UNL) operated Meteodrone (Model SSE MM-641) is a small (1.1 kg takeoff weight, ∼40 cm across) hexacopter sUAS manufactured by Meteomatics. The maximum endurance is approximately 30 min and estimated maximum operating altitude is 1500 m. Profiles can be executed at ascent/descent rates up to 10 m s−1. The Meteodrone is capable of measuring temperature, relative humidity, wind velocity, and GPS position time stamped with GPS time. Temperature and dual relative humidity sensors are shielded from solar exposure by a housing with a vertically oriented air intake and horizontal exhaust. Sensor aspiration is driven by rotor wash across the exhaust. In its current configuration data are recorded only when executing vertical profiles. Attitude and position are updated at a data rate of 20 Hz while meteorological observations are updated at 10 Hz. Based on evaluations of Koch et al. (2018) and Leuenberger et al. (2020) temperature has an accuracy of ±0.1 K with a warm bias of 0.4 K and response time < 1 s, relative humidity has an accuracy of ±1.8% with a dry bias of 7% and response time < 4 s, and wind velocity has a positive wind bias of +2 m s−1 and direction bias of 7°. Additional tests, including the current study, are being undertaken to refine these values. Real-time data were available for situational awareness with full datasets stored on board and synced with the GCS postflight.
5) University of Nebraska M600P
A second sUAS used by University of Nebraska is a “DJI Matrice 600 Pro” (M600P) hexacopter manufactured by DJI. Flight endurance of the M600P is around 25 min with the sensor payload attached to it. It is capable of both vertical and horizontal profiling with speeds up to 3 and 10 m s−1, respectively. The maximum flight altitude of the M600P is 500 m, and profiling flights are conducted automatically by setting up a waypoint mission in the DJI Ground Station Pro app. These waypoints are programmed with appropriate speed, height, and hover times.
The M600P is equipped with a temperature–humidity sensor housing to aspirate system sensors, shield them from radiation, and source air from outside of the M600P’s turbulent rotor wake region (Islam et al. 2019). The housing is held by a support structure such that the inlet is pointing outside the sUAS, and the outlet sits above the propellers. The sensor housing exploits the negative pressure created by M600P’s propellers to draw high-speed air through the tubes and aspirate the sensors. An additional sensor housing configuration using a similar aspiration principle but with the inlet of the sensor housing pointing toward the center of the sUAS is also used for data collection. A third, much smaller sensor housing configuration, which does not require additional support structure was also used for data collection.
The temperature–humidity sensor on board the sUAS is an iMet-XQ2. The sensors provide data at 1 Hz, with a manufacturer-claimed accuracy for temperature and humidity of ±0.3°C and ±5% RH, respectively. The temperature sensor response time is 1 s with a 5 m s−1 second aspiration speed. The response time of the humidity sensors varies in response to temperature, with values of 0.6 s at 25°C, 5.2 s at 5°C, and 10.9 s at −10°C.
The M600P carries an onboard data acquisition (DAQ) computer. The DAQ uses the Robot Operating System (ROS) to interface with the sUAS flight controller and sensors over a serial connection. The DAQ allows different sources of data to be output at their own rate and logs time stamps for all the recorded data. These different time stamps can later be used to synchronize the data as necessary. The M600P’s ground station computer uses a wireless XBEE module to initiate and terminate data collection. An ethernet connection with a ground computer is used to transfer files for processing and archival.
b. DOE ARM SGP sensor systems
1) ARM tower
The U.S. DOE ARM program SGP site includes a 60 m triangular tower that is used to provide meteorological, radiometric, and other measurements (Cook 2016, see Fig. 1). The base of the tower is at 310 m MSL. The tower holds a variety of instruments, mounted primarily at 25 and 60 m AGL, on two separate elevators that are in place on the west and southeast sides of the tower. The sensors described here are limited to those that collected data used to evaluate sUAS sensor performance in the current study, and this description does not include all sensors installed on this tower.
Temperature and relative humidity are measured at both 25 and 60 m on both elevators (west and southeast sides) of the SGP 60-m tower. However, the aspirators used on the two sides are different: the west-side aspirators are made of metal and have a low flow rate of 1.8 L min−1, whereas the southeast-side aspirators are made of plastic and have a flow rate closer to 3 L min−1. The temperature sensors on the two sides are also different: the southeast side uses the internal platinum resistance temperature detector (PRTD) in the Vaisala HMP45D T/RH sensor, whereas the west side uses a PRTD (Minco Products Inc., Model S853PD60 × 72 100-ohm sensor) that is separate from the T/RH sensor. Both temperature sensors have an expected accuracy of 0.2°C. Both elevators leverage the Vaisala HMP45D for relative humidity measurements, with an anticipated accuracy of 2% between 0% and 90% RH and 3% between 90% and 100% RH. Vendor-recommended calibration procedures and calibration checks by the mentor are used to maintain the accuracy of the temperature and relative humidity sensors on the SGP CF tower. Details on the calibration procedures can be found in the Tower instrument handbook (Cook 2016).
In addition to these sensors for measuring thermodynamic state, the tower also supports an infrared thermometer (IRT) mounted at 25 m on the southeast side to observe surface temperature from this altitude. There are also carbon dioxide flux measurement systems (CO2FLX) sensors installed on the west side of the tower at both 25 and 60 m AGL. These systems include a sonic anemometer and open path gas analyzer, and provide measurements of vertical and horizontal winds, temperature, and CO2 and H2O concentrations in support of measuring turbulent fluxes of heat, moisture, and momentum at different altitudes. Because of the wind directions experienced during this campaign (N, S, W) and the availability of the sonic anemometer data from the west side of the tower, we used data from sensors mounted on the west side of the tower for comparison to sUAS data.
2) Radiosondes
In addition to the tower instrumentation, another ARM sensor system that is heavily leveraged in the current analysis is the radiosondes launched from the SGP facility. ARM currently uses the Vaisala RS-41 radiosonde, which measures pressure, temperature, relative humidity, wind direction and wind speed, along with position information. Measurement uncertainty associated with these sensors is 1.0 hPa, 0.3°C, and 4% RH for pressure, temperature, and relative humidity, respectively. Wind information is obtained through GPS wind finding. This technique is thought to be very precise, and the accuracy of the wind velocity is estimated to be 0.2 m s−1. Extra radiosondes were launched during the field campaign supporting this intercomparison. In addition to the standard 4-times-daily radiosondes launched at approximately 0530, 1130, 1730, and 2330 UTC (0030, 0630, 1230, and 1830 CDT) daily, the ARM program increased the launch frequency to also launch radiosondes at approximately 0930, 1045, 1345, 1530, and 1645 CDT daily. This provided opportunities to compare to radiosonde observations six times per day during daylight hours.
3. Overview of evaluation
Flight data used in the current intercomparison and evaluation effort were collected between 29 March and 8 April 2021. Weather conditions (Fig. 2) during this period were generally good, with very limited precipitation and good visibility. Given the mostly clear conditions, there was a significant diurnal cycle in near-surface air temperatures, with early morning temperatures around 5°–15°C, and afternoon high temperatures between 15° and 25°C. The only exception to this pattern was 7–8 April, when the passage of a cold front associated with a synoptic low pressure system reduced the high temperatures to around 10°C. Relative humidities experienced at the site ranged from values around 20% during the day to around 95% at night during the second week of the project. Winds also featured a diurnal pattern, with winds generally stronger during the day and slightly weaker during the night. One-minute average wind speeds were as high as 18 m s−1 on some days, and winds were generally higher (12–18 m s−1) and gustier during the first two days and second week of the campaign, with weaker winds (2–6 m s−1) during the two flight days spent at the Marshall Mesonet site. Winds were initially from the south, shifting to northerly during the first week, before shifting back to southerly for most of the second week. The weather system that passed through on 7–8 April resulted in a shift of the winds to west-northwesterly. Despite the moderately strong winds, weather conditions did not impact the flight operations significantly.
Time series of surface meteorological parameters from the ARM SGP site during the time period of the intercomparison flights, including (from top to bottom) air pressure, 2-m air temperature, 2-m relative humidity, 10-m wind speed, and 10-m wind direction.
Citation: Journal of Atmospheric and Oceanic Technology 41, 2; 10.1175/JTECH-D-23-0067.1
a. Comparison with ARM radiosondes
Evaluation of the different platforms was completed through a variety of different means. As discussed above, additional radiosondes were launched by the DOE ARM program during the intercomparison campaign. These launches facilitated a comparison between the sUAS and the widely used Vaisala RS-41 radiosonde package. To support this comparison, the sUAS platforms were programmed to conduct a vertical profile to the top of the airspace available for each platform based on the flight permissions obtained (120 m AGL for the S0, M600, and Meteodrone; 610 m AGL for the RAAVEN and CopterSonde). For the fixed-wing platforms, this profile was flown as an ascending and descending spiral flight with the fixed rate for each individual profile fixed. To investigate the impact of aircraft climb rate, this parameter was varied between one profile and the next, with a range covering 1–4 m s−1. For the rotary-wing platforms, the profile was conducted as a straight ascent–descent pair above a fixed location on the surface. For all platforms, the profile was initiated a few seconds after the release of the balloon to minimize the potential for conflict between the sUAS and balloon. To facilitate comparison between the sUAS platform and the radiosonde, data were gridded to a common altitude (above mean sea level) vector, with values for each sensor at a given altitude representing a height-binned mean over 10 m windows between 300 and 1100 m MSL (approximately −10 to 790 m AGL, depending on local topography).
The above-stated approach resulted in several profiles for comparison for each platform. While these profiles were conducted in close temporal succession, there are still spatial gradients to consider when using the radiosonde data for direct comparison. These spatial considerations result from both horizontal heterogeneity in the atmosphere, and the tendency of the radiosonde to drift horizontally during flight. The latter was particularly true given the significant wind speeds experienced during the campaign. This horizontal drift was a larger factor for the second week of operations with the RAAVEN and CopterSonde, given the higher altitude range covered by the permissions in place for these platforms. Because of the potential for spatial gradients, we look less at individual profiles, but rather focus on statistics across all the profiles conducted for any given aircraft. In total, the flights conducted resulted in 11, 15, 10, 17, and 14 radiosonde profiles for comparison against the RAAVEN, CopterSonde, S0, Meteodrone, and M600, respectively.
b. Comparison with ARM 60 m tower
In addition to radiosonde comparisons, statistical evaluation of the sUAS data was conducted through comparison with measurements from the SGP 60 m tower. For these evaluations, most of the aircraft were operated at the same height as the sensors on the tower for extended periods of time. For the fixed-wing aircraft, this entailed extended flight at 60 m above ground level, either in a loiter (circular) pattern, or in extended racetrack patterns. A variety of different types of patterns were executed to evaluate the impact of flight pattern on the quality of measurements from a given platform. This type of flight would be done for up to almost two hours at a time in the case of the RAAVEN, and around 40 min worth of time for the S0. For the copter platforms, the tower comparison was conducted by having the platform hover at the altitude of the tower instrumentation for an extended period. Typically, this would be for 15–20 min due to the shorter endurance of these platforms. One notable exception was that the Meteodrone’s autopilot system did not support single altitude hovering, meaning that tower comparisons with this platform were limited to repeated passes through the tower altitude.
These comparison flights provide rich data for statistical analysis on the performance of the sensors. As with the radiosonde comparisons, there are spatial differences to consider. However, it is assumed that over 20 min of flight time spatial heterogeneity on the scales being considered (approximately hundreds of meters) is negligible and that the statistics of the boundary layer structure will be similar at a given altitude. A second challenge that arises in this comparison is getting the altitude to exactly match that of the tower-mounted sensors. This is a multifaceted challenge, with part of the problem resulting from how the sUAS autopilot systems navigate and maintain altitude, and another part of the problem resulting from not being able to get a precise measurement of the altitude of the tower instrumentation. To derive the latter, we placed two different aircraft at the base of the tower to get a base altitude, and then added 60 m to this number to get an estimated sensor altitude relative to mean sea level. However, the aircraft typically navigate relative to the altitude obtained before launch, meaning that if the aircraft was flying from a different surface elevation that 60 m above that height would not align perfectly with the tower height. Therefore, we used mean sea level altitude in the execution of our flights. In addition to this, the fixed wing platforms tend to vary altitude in flight, sometimes by several meters if the atmosphere is turbulent. Therefore, some of the variability experienced by these aircraft is the result of altitude changes on the order of 5 m or so, and not a result of spatial variability in the atmosphere. Unfortunately, the ARM SGP tower does not report pressure for the sensor heights, making evaluation of altitude offsets more challenging.
The tower intercomparison flights support the evaluation of thermodynamic quantities, including temperature and moisture, as well as evaluation of wind variables, including vertical velocity. In addition, the combination of sensors on the tower allows us to evaluate the ability of the sUAS to provide information on derived properties such as turbulent fluxes of heat and momentum.
c. Platform intercomparison at Marshall site
The flight hours spent alongside the tower also allow opportunities for intercomparison between different platforms. Since the flights at SGP were split over a 2-week period, with the S0, M600, and Meteodrone flying in the first week, and the RAAVEN and CopterSonde flying in the second week, direct comparisons between all platforms were not possible alongside the tower. However, the four teams also conducted limited flight operations at the Marshall Mesonet site, allowing for direct intercomparison across all the different platforms. These latter flights included side-by-side loiter circles for the two fixed-wing aircraft, as well as close-proximity hovering for the different multirotors. They also included coordinated profiling of the lowest 400 m of the atmosphere. A map of flights is included in Fig. 3. The bottom panels of Fig. 3 illustrate the proximity of the different platforms during the Marshall flights. To our knowledge, these flights represent some of the densest sampling ever conducted by sUAS specifically equipped to observe the atmosphere, resulting in opportunities for direct intercomparison between the platforms. These flights are leveraged to evaluate similar platforms (e.g., S0 versus RAAVEN; CopterSonde versus Meteodrone).
(top left) A map illustrating the locations of the two flight locations relative to Oklahoma cities. Also included are (top right) maps illustrating the spatial extent of flight patterns conducted at the Marshall mesonet site, (middle left) the first week of flights at the SGP facility, and (middle right) the second week of flights at the SGP facility. (bottom) The altitudes covered by platforms at the SGP facility during the (bottom left) first week and (bottom right) second week. CU RAAVEN flights are shown in yellow, BST S0 flights in black, the OU CopterSonde flight location as a maroon dot, the UNL Meteodrone flight location as a red dot, and the UNL M600 flight location as a salmon dot.
Citation: Journal of Atmospheric and Oceanic Technology 41, 2; 10.1175/JTECH-D-23-0067.1
4. Results
a. Comparison with ARM radiosondes
We first compare observations from the sUAS platforms to those from radiosondes launched from the ARM SGP facility. Figure 4 provides an overview of these comparisons, in the form of scatterplots comparing sUAS-derived values to those from the radiosondes. A statistical summary of these comparisons, reporting the mean bias and standard deviation of the biases, is reported in Table 1. Temperature values reported by the different sUAS platforms reveal some differences from the radiosondes and from each other. The M600, RAAVEN, and CopterSonde all show relatively low mean biases (<0.5°C), given that the measurements from the sUAS and the radiosonde were not collocated, though the M600 demonstrated substantially greater variability, with a standard deviation that is more than double that of the other two platforms. The S0 and Meteodrone had significantly larger biases (both approaching 2°C) with the difference between the Meteodrone and radiosonde showing significantly larger variability than other platforms (standard deviation of 0.7°C). Looking at the relative humidity observations, measurements from all sUAS platforms are biased low, ranging from around −1.6% (RAAVEN) to −5.3% (S0). This dichotomy is a bit surprising, as both platforms leverage the same sensor (Vaisala RSS-421), with the sensor accounting for RH sensor temperature in its measurement, and for both platforms the sensor was reconditioned at the start of each flight day. The other platforms featured biases that were in between these extremes, and generally in the −3% to −4.5% range. As with the relative humidity values, all the pressure comparisons resulted in negative mean biases. These ranged between −0.70 hPa (CopterSonde) to −3.58 hPa (Meteodrone), with the RAAVEN and CopterSonde having substantially smaller mean biases than the other three platforms. It is possible that these differences are the result of meteorological conditions or range of altitudes sampled, given that the RAAVEN and CopterSonde both flew at the SGP facility during the second week of sampling and covered larger altitude ranges. Putting these different measured variables together allows for the calculation of water vapor mixing ratios for both the sUAS platforms and the radiosondes. Comparison between these calculated values shows a range of negative biases ranging from −0.03 g kg−1 (RAAVEN) to −0.61 g kg−1 (CopterSonde). Standard deviations of the differences between sUAS and radiosonde mixing ratios are all below 0.5 g kg−1.
Scatterplots showing comparison of sUAS observations with radiosondes, including (clockwise from top left) air temperature, air pressure, relative humidity, wind speed, wind direction, and water vapor mixing ratio.
Citation: Journal of Atmospheric and Oceanic Technology 41, 2; 10.1175/JTECH-D-23-0067.1
Mean and standard deviation of the bias values derived from comparison with radiosondes for each platform. The “total comps” row at the bottom is the number of comparison points (altitude bins) used in calculating these statistics.
Of the five platforms that were operated at the SGP facility, three provided estimated wind speed and direction values that could be compared with the radiosonde estimated quantities. It is important to note that the different types of platforms all calculate winds differently, yet the resulting values are all expected to be of good quality. It is also important to note that the radiosonde-based wind estimates may also be challenged in the lowest hundred meters or so as a result of the sensor string unspooling. Looking at the comparisons, the two fixed-wing platforms (RAAVEN and S0) and the CopterSonde had wind speed biases of comparable magnitude (0.58, 0.64, and −0.70 m s−1, respectively) with the Meteodrone having the smallest mean wind speed bias (0.07 m s−1). Having said that, the amount of variability in these differences is higher for the rotary-wing platforms (standard deviations of 2.14 and 2.12 m s−1 for the Meteodrone and CopterSonde, respectively) than for the fixed-wing platforms (1.7 and 0.73 m s−1 for the RAAVEN and S0, respectively). The fixed-wing platforms and the CopterSonde offered better performance in estimation of wind direction with mean biases of −3.76°, 3.17°, and 1.15° for the RAAVEN, S0, and CopterSonde, respectively. The Meteodrone had significantly larger wind direction biases (6.3° mean bias). The RAAVEN exhibited substantially less variability in estimating wind direction than the other three platforms, with a standard deviation of 6.45°, versus 11°, 13.5473°, and 14.75° for the S0, CopterSonde, and Meteodrone. Again, some of this could be the result of changing meteorological conditions and sampling altitudes covered during the sampling period, as the S0 and Meteodrone flew during the same week and only covered lower altitudes near the ground, where the radiosonde estimates may be less consistent due to unfurling of the string that attaches the balloon to the sensor package and near-surface turbulence. However, this does not explain the large standard deviation in CopterSonde wind direction measurements.
b. Comparison with ARM 60 m tower
1) Basic meteorological variables
Because of the significant amount of spatial variability in the atmospheric boundary layer, it is not possible to simply compare time series of data collected by sUAS and instruments situated on a stationary tower. Figure 5 illustrates such a comparison for the three wind components and the temperature deviation from a mean value over the recorded time period. While there is coherence in the larger signals measured by both platforms, the sUAS platform shows greater variability than the tower-mounted instrumentation. This is expected, as the sUAS is constantly resampling features by flying through them in orbital or racetrack patterns, while the tower measurements have each feature advect past the sensors once.
An example time series (∼1 h) of (top left) zonal and (top right) meridional wind speeds and (bottom left) vertical velocity and (bottom right) temperature anomaly, illustrating a comparison between the RAAVEN (gold) flight at 60 m AGL and tower observations (gray) from a sonic anemometer mounted at 60 m AGL on the ARM tower.
Citation: Journal of Atmospheric and Oceanic Technology 41, 2; 10.1175/JTECH-D-23-0067.1
To account for the differences in sampling, it is better to conduct a statistical comparison between the two datastreams, assessing similarity between different moments of the distribution of observations collected by each sensor system. Figure 6 illustrates such comparisons for the three wind components, atmospheric pressure, atmospheric temperature, and atmospheric potential temperature. These figures show the ratio of the standard deviation of the sUAS dataset to the tower standard deviation on the horizontal axis, where a value of one implies that the two datasets show the same standard deviation, and values exceeding one demonstrating greater standard deviation in the sUAS-measured distribution. The vertical axis shows the bias of the mean value measured by the sUAS relative to that measured at the tower, with a zero value indicating that the two datasets have the same mean value, and positive values indicating that the sUAS-derived mean is greater than the tower-derived value. To aid in the interpretation of the results, green and yellow boxes have been included in the figures. The horizontal component of the yellow boxes indicates a variance ratio between 0.5 and 1.5 for all variables, and the horizontal component of the green boxes indicates a variance ratio between 0.75 and 1.25 for all variables. The vertical component of the yellow boxes indicate that the mean values are within 1.5 m s−1 in the zonal and meridional winds, 0.5 m s−1 in the vertical winds, 2 K in temperature or potential temperature, and 2 hPa in pressure. The vertical component of the green boxes indicate that the mean values are within 0.75 m s−1 in the zonal and meridional winds, 0.25 m s−1 in the vertical winds, 1 K in temperature or potential temperature, and 1 hPa in pressure.
A statistical comparison between the sUAS observations and measurements collected by instrumentation on the ARM tower. The horizontal axis compares the standard deviation in the sUAS observation to the standard deviation observed by the tower, and the vertical axis shows the mean bias, with each point representing data from one flight leg at tower height. Included are comparisons of (clockwise from top left) zonal wind, meridional wind, pressure, potential temperature, temperature, and vertical velocity.
Citation: Journal of Atmospheric and Oceanic Technology 41, 2; 10.1175/JTECH-D-23-0067.1
Each point on the panels of Fig. 6 represents a comparison from a single flight during time periods where the aircraft was between 50 and 70 m AGL. Because the sUAS platforms move around vertically during flight, it is not possible to limit the altitudes to only a meter or two from 60 m. It is also challenging to have the mean aircraft altitude and tower altitude match up exactly, which can impact these comparisons. In general, the sUAS observations featured greater variance. This is partly to be expected because the sUAS are moving through the atmosphere while the tower observations are stationary, though in some cases that variance is clearly excessive (e.g., vertical velocity from S0), and for some platforms it is larger than others. Most of the data points for all variables fall in the yellow box, though there are some notable exceptions. Specifically, the pressure observations seem to have significantly more variability, which again is likely a result of the platforms moving around vertically during flight, while the tower is fixed in place. Additionally, there seem to be significant warm biases in the S0 measurements.
2) Turbulent fluxes of heat and momentum
For both sUAS we identified all flight legs flown within ±20 m vertically of the eddy covariance system (60 m AGL) on the SGP tower. The horizontal separation between sUAS flights and the SGP tower was an average of 600 ± 180 m for the RAAVEN and 450 ± 170 m for the S0. Because the flux measurements were not collocated, we did not expect the flux time series to match precisely. However, given the similar nature of the terrain across the flux footprints of both flux measurements, we did expect fluxes to be of a similar magnitude.
Figure 7 shows comparisons between τ measured from sUAS and from the SGP tower. The RAAVEN and SGP agree reasonably well, while the S0 versus SGP correspondence is less clear (Fig. 7c). Time series of the flux (Figs. 7a,b) showed that, while sUAS τ values broadly follow the diurnal trends in momentum flux displayed by the AmeriFlux 30-min average, any given interval may deviate by a substantial amount. However, the tower-measured τ with shorter flux averaging periods corresponding to the sUAS measurements (i.e., 10 and 15 min for S0 and RAAVEN, respectively) showed a similar amount of variability. This is illustrated by the scatterplot of sUAS versus tower τ (Fig. 7c), where bars represent the range of sUAS and tower-measured fluxes within the same flight leg. This highlights that longer flux averaging periods smooth out actual real variation occurring at smaller scale. This fact, in conjunction with the horizontal separation between the measurements, makes it challenging to use the flux comparisons alone to confirm the quality of the sUAS flux measurements.
(a) Time series of τ measured by S0 and SGP tower, with points representing equivalent 10-min flux intervals between both platforms and the line representing the AmeriFlux 30-min SGP tower fluxes, (b) as in (a), but for RAAVEN 15-min τ, (c) scatterplot of UAV vs tower τ (where bars represent the range of sUAS and tower-measured fluxes within the same flight leg and squares represent flight legs with only one flux interval), and (d) mean frequency-weighted uw cospectra for all platforms. Due to differences, separate mean SGP cospectra are shown for flux intervals corresponding to S0 and RAAVEN measurements.
Citation: Journal of Atmospheric and Oceanic Technology 41, 2; 10.1175/JTECH-D-23-0067.1
For a closer examination we calculated cospectra. Ensemble mean frequency-weighted uw cospectra (Fig. 7d) were presented as functions of the normalized frequency n = f z/U, where U is horizontal wind speed for the tower and the true airspeed for the sUAS. This enabled comparison between the stationary and moving platforms (Desjardins et al. 1989). The RAAVEN cospectra showed more noise than the SGP tower. Overall, however, the RAAVEN and SGP cospectra matched well, with both platforms showing the characteristic shape for turbulent momentum fluxes (e.g., Kaimal et al. 1972). The mean S0 cospectra did not show any consistent behavior in its cospectral shape. This may have been due to the weaker winds on the S0 measurement days. It is difficult to measure τ under such conditions (e.g., several of the periods failed the AmeriFlux quality control criterion). The unusual cospectral shape could also be the result of measurement error. The corrected wind speeds showed a residual correlation to orbital position, suggesting an incomplete motion correction of the 3D wind vector. This is expected to reduce the accuracy of the flux measurement.
Figure 8 shows comparisons between sUAS and SGP-measured HS. The agreement between sUAS and SGP is better for HS than τ, particularly for the RAAVEN (Fig. 8c). As was observed for τ, there are deviations from the 30-min fluxes by both sUAS and the SGP tower when shorter averaging intervals are used (Figs. 8a,b). However, within-flight-leg variation showed that such deviations straddle the one-to-one relationship (Fig. 8c). The S0 appeared to slightly underestimate HS on average, but with few data points it cannot be determined statistically.
(a) Time series of HS measured by S0 and SGP tower, with points representing equivalent 10-min flux intervals between both platforms and the line representing the AmeriFlux 30-min SGP tower fluxes, (b) as in (a), but for RAAVEN 15-min HS, (c) scatterplot of UAV vs tower HS (where bars represent the range of sUAS and tower-measured fluxes within the same flight leg and squares represent flight legs with only one flux interval), and (d) mean frequency-weighted wT cospectra for all platforms. The SGP curve represents the mean cospectrum for intervals corresponding with both S0 and RAAVEN flux measurements.
Citation: Journal of Atmospheric and Oceanic Technology 41, 2; 10.1175/JTECH-D-23-0067.1
An examination of the frequency-weighted wT cospectra revealed both sUAS to have narrow, but normally shaped cospectral curves, where contributions to the flux were more concentrated in the middle frequencies (Fig. 8d). This contrasted with the SGP tower, which had a broader curve, very closely followed the idealized wT cospectral curve of Kaimal et al. (1972). The RAAVEN showed a greater contribution than SGP at peak turbulent flux frequencies (i.e., n = 0.1), but lower contributions at higher frequencies (i.e., n = 1 and higher). The S0 showed a similar drop-off in contribution from higher frequencies, like the RAAVEN, though had lower overall magnitude f COwT. And examination of w and T power spectra revealed that T measurements had a noise floor beginning at roughly n = 1, which may explain the steeper drop-off of the sUAS cospectra at higher frequencies.
The low-frequency end of the cospectra also provided important information. Here the S0 curve failed to close to zero, indicating some degree of low-frequency contribution was being excluded in the S0’s measurements. This suggests that the 10-min averaging period of the S0 was too short. Unfortunately, most of the flight legs at 60 m AGL were less than 15 min, which meant that lengthening the averaging period was not possible. The 15-min averaging period of the RAAVEN appeared to be long enough, as indicated by its closure at lower frequencies. The S0 results are a reminder that future studies aimed at using sUAS to measure fluxes should carefully consider flight plans to ensure scientific objectives can be achieved.
Overall, the wT cospectra were noticeably less noisy than the uw cospectra (Fig. 7d). This was anticipated because the covariance for HS was derived from observations measured by different sensors. Therefore, noise in the two signals is expected to be uncorrelated, unlike for τ where noise and/or systematic error (e.g., residual motion effects) will be correlated and thereby be incorporated into the flux. Put simply—it is particularly challenging to make momentum flux measurements from a moving platform. However, here we observed agreement in τ magnitudes and uw cospectral shapes between the RAAVEN and SGP. The additional agreement of the HS measurements indicates the capability of sUAS for making flux measurements.
c. Platform intercomparison at Marshall site
In addition to flights conducted at the SGP facility, the teams took 2 days to conduct simultaneous flights in close proximity at a site in Marshall. Here, we use a roughly 2-h period from that intercomparison activity to demonstrate differences between platforms. Figure 9 shows the comparison from the Marshall flight. The top-left panel demonstrates the altitudes flown during that window by the different platforms. The distance between platforms was never more than a few hundred meters during this time (see top-right map in Fig. 2). Over this time period, differences between the different systems were shown to be largely consistent with those detected from their individual comparisons to the ARM instrumentation. For example, both the UNL Meteodrone and BST S0 were shown to measure lower pressures than the RAAVEN. Similarly, both of these platforms measured higher temperatures than the RAAVEN. Interestingly, the Meteodrone appears to have temperatures increasing over the sampling period, a feature not observed by other platforms. It is speculated that this results from motor heat contamination, as the flights were conducted in close succession which likely resulted in significant heating of the motors. However, this would need to be confirmed with additional testing. This increase in temperatures also results in an underobservation of relative humidity. Interestingly, the BST S0 measures both higher temperatures and higher RH values than the RAAVEN. This is inconsistent with the radiosonde comparison, which saw the S0 measuring similarly to the Meteodrone (temperatures too high, RH too low). It is not immediately clear what would cause such a discrepancy. Looking at the measured winds, it is clear that there is significant variability in the wind speed and direction on this day. The three platforms reporting winds (RAAVEN, S0, Meteodrone) appear to have approximately the same amount of variability, with the Meteodrone reporting slightly lower wind speeds. This figure also clearly demonstrates an advantage of using sUAS for profiling over radiosondes—the sUAS can provide important insight into the variability of the atmosphere at a given level, whereas the balloon only provides an instantaneous snapshot. Such information can be extremely useful when attempting to evaluate model performance or provide data for assimilation as it offers a more holistic view on the state of the atmosphere around a given time.
A direct comparison between the different platforms over a 2-h period of simultaneous flight activity at the Marshall site. (clockwise from the top left) This includes comparisons of altitude of the platforms as a function of time, air pressure, relative humidity, wind direction, wind speed, and temperature.
Citation: Journal of Atmospheric and Oceanic Technology 41, 2; 10.1175/JTECH-D-23-0067.1
5. Summary and outlook
This work describes the outcomes of a targeted effort to document the performance of five different sUAS platforms used broadly for atmospheric science experiments. Such efforts represent an important step toward widespread adaptation and use of sUAS by the atmospheric science field, including in field campaigns designed to advance process-level understanding and in coordinated efforts to support assimilation of sUAS data in numerical weather prediction. The platforms evaluated here include two fixed-wing systems and three rotary-wing systems, and include both custom systems developed in university settings and commercially available aircraft. The evaluations themselves include comparisons between sUAS observations and those from radiosondes launched at the same location as the sUAS operations and an instrumented 60 m tower on site, and intercomparison between different platforms.
These efforts reveal that generally all platforms provide reasonable measurements of the state of the atmosphere. A summary of the comparisons to radiosondes is included in Table 1. The absolute values of mean temperature biases over these flights spanned a range from 0.24 to 1.88 K with standard deviations between 0.17 and 0.7 K. Absolute values of air pressure mean biases ranged from −3.58 to −0.57 hPa, with standard deviations ranging from 0.23 to 1.33 hPa. Absolute values of mean mixing ratio biases ranged between 0.03 and 0.61 g kg−1 with standard deviations between 0.17 and 0.45 g kg−1. When combined with pressure and temperature measurements, this resulted in mean absolute value RH biases between 1.62% and 5.34% (standard deviations between 1.37% and 5.82%). For the four platforms reporting wind estimates, mean absolute value wind speed biases ranged between 0.07 and 0.7 m s−1, with standard deviations between 0.73 and 2.14 m s−1, and mean absolute value wind direction biases ranged from 1.15° to 6.3° (standard deviations from 6.45° to 14.75°). These values were derived over several hundred (360–1360) points of comparison per platform. Comparisons to the tall tower generally showed similar results (see Fig. 6), though there were some challenges related to ensuring that platforms were maintained at the tower altitude over extended time periods, and it is possible that some of the biases in the tower comparison were the result of small altitude offsets.
In addition to these evaluations, this paper provides some of the first detailed comparisons of turbulent fluxes of heat and momentum calculated through eddy-covariance techniques for platforms that provide high-resolution data of the vertical velocities (see Figs. 7 and 8). These evaluations revealed that at least one of the platforms (CU RAAVEN) provided reasonable estimates of momentum and heat fluxes and their variability over time, relative to the ARM tower. Comparisons of the cospectra derived from these flight legs generally show comparable spectral power to that derived from the sonic anemometer on the tower across different sampling frequencies, though there was a bit more noise in the sUAS spectra than the tower. The cause of this elevated noise is a continued subject of investigation.
It is important to remember that these flights covered a broad, but still limited set of conditions. In general, a wide range of wind speed and turbulence regimes were sampled, though there were no measurements obtained in a strongly stratified condition. Ideally, these activities would be repeated for a series of stable boundary layer days to understand the extent to which these results (particularly the turbulent flux results) translate to stratified cases. Additionally, there was no effort here to demonstrate that biases for a given platform cleanly translated across different versions of the same aircraft. In other words, the question of consistency between multiple comparable aircraft has not been evaluated and should be explored for each aircraft type, since often multiple aircraft are used in a campaign.
Ultimately, we believe that this activity was worthwhile and informative, and supported the notion that sUAS are a research-ready observational capability that should continue to be considered for future field campaigns targeting atmospheric phenomena. Additionally, we believe that such intercomparisons are valuable and should be supported, wherever possible, in connection with funded field projects to ensure that the sUAS to be operated are providing reasonable information. This also requires investment in infrastructure for such intercomparison. In this case, we leveraged sensors deployed as part of routine operations at the DOE ARM SGP facility, though it would be beneficial to have several such sites distributed across the country and world to foster additional verification efforts.
Acknowledgments.
This work was supported by the U.S. Department of Energy Atmospheric System Research (ASR) program (DE-SC0021381), the U.S. Air Force (FA8730-20-C-0021), and the NOAA Physical Sciences Laboratory. Additional support was provided by the U.S. National Science Foundation (IIS-1925052).
Data availability statement.
Data collected as a part of this effort are available for public download from the U.S. Department of Energy Atmospheric Radiation Measurement (ARM) program data portal. The sUAS data used for this intercomparison are archived on Zenodo (https://doi.org/10.5281/zenodo.7916656).
REFERENCES
Al-Ghussain, L., and S. C. C. Bailey, 2022: Uncrewed aircraft system measurements of atmospheric surface-layer structure during morning transition. Bound.-Layer Meteor., 185, 229–258, https://doi.org/10.1007/s10546-022-00729-2.
Bailey, S. C. C., S. W. Smith, M. P. Sama, L. Al-Ghussain, and G. de Boer, 2023: Shallow katabatic flow in a complex valley: An observational case study leveraging uncrewed aircraft systems. Bound.-Layer Meteor., 186, 399–422, https://doi.org/10.1007/s10546-022-00783-w.
Barbieri, L., and Coauthors, 2019: Intercomparison of small unmanned aircraft system (sUAS) measurements for atmospheric science during the LAPSE-RATE campaign. Sensors, 19, 2179, https://doi.org/10.3390/s19092179.
Barker, E. H., 1977: A maritime boundary-layer model for the prediction of fog. Bound.-Layer Meteor., 11, 267–294, https://doi.org/10.1007/BF02186082.
Bell, T. M., B. R. Greene, P. M. Klein, M. Carney, and P. B. Chilson, 2020: Confronting the boundary layer data gap: Evaluating new and existing methodologies of probing the lower atmosphere. Atmos. Meas. Tech., 13, 3855–3872, https://doi.org/10.5194/amt-13-3855-2020.
Bony, S., and J.-L. Dufresne, 2005: Marine boundary layer clouds at the heart of tropical cloud feedback uncertainties in climate models. Geophys. Res. Lett., 32, L20806, https://doi.org/10.1029/2005GL023851.
Brown, A. R., R. J. Beare, J. M. Edwards, A. P. Lock, S. J. Keogh, S. F. Milton, and D. N. Walters, 2008: Upgrades to the boundary-layer scheme in the Met Office numerical weather prediction model. Bound.-Layer Meteor., 128, 117–132, https://doi.org/10.1007/s10546-008-9275-0.
Brus, D., J. Gustafsson, V. Vakkari, O. Kemppinen, G. de Boer, and A. Hirsikko, 2021: Measurement report: Properties of aerosol and gases in the vertical profile during LAPSE-RATE campaign. Atmos. Chem. Phys., 21, 517–533, https://doi.org/10.5194/acp-21-517-2021.
Charney, J. G., 1975: Dynamics of deserts and drought in the Sahel. Quart. J. Roy. Meteor. Soc., 101, 193–202, https://doi.org/10.1002/qj.49710142802.
Chen, Y.-C., C.-C. Chang, W.-N. Chen, Y.-J. Tsai, and S.-Y. Chang, 2018: Determination of the vertical profile of aerosol chemical species in the microscale urban environment. Environ. Pollut., 243, 1360–1367, https://doi.org/10.1016/j.envpol.2018.09.081.
Cione, J. J., and Coauthors, 2020: Eye of the storm: Observing hurricanes with a small unmanned aircraft system. Bull. Amer. Meteor. Soc., 101, E186–E205, https://doi.org/10.1175/BAMS-D-19-0169.1.
Cleary, P. A., and Coauthors, 2022: Observations of the lower atmosphere from the 2021 WiscoDISCO campaign. Earth Syst. Sci. Data, 14, 2129–2145, https://doi.org/10.5194/essd-14-2129-2022.
Cook, D. R., 2016: Towers handbook. U.S. Dept. of Energy Tech. Rep. DOE/SC-ARM/TR-050, 30 pp., https://www.arm.gov/publications/tech_reports/handbooks/twr_handbook.pdf.
Cuxart, J., and Coauthors, 2000: Stable Atmospheric Boundary-Layer Experiment in Spain (SABLES 98): A report. Bound.-Layer Meteor., 96, 337–370, https://doi.org/10.1023/A:1002609509707.
de Boer, G., and Coauthors, 2016: The Pilatus unmanned aircraft system for lower atmospheric research. Atmos. Meas. Tech., 9, 1845–1857, https://doi.org/10.5194/amt-9-1845-2016.
de Boer, G., and Coauthors, 2018: A bird’s-eye view: Development of an operational ARM unmanned aerial systems capability for atmospheric research in Arctic Alaska. Bull. Amer. Meteor. Soc., 99, 1197–1212, https://doi.org/10.1175/BAMS-D-17-0156.1.
de Boer, G., B. Argrow, J. Cassano, J. Cione, E. Frew, D. Lawrence, G. Wick, and C. Wolff, 2019: Advancing unmanned aerial capabilities for atmospheric research. Bull. Amer. Meteor. Soc., 100, ES105–ES108, https://doi.org/10.1175/BAMS-D-18-0254.1.
de Boer, G., and Coauthors, 2020: Development of community, capabilities and understanding through unmanned aircraft-based atmospheric research: The LAPSE-RATE campaign. Bull. Amer. Meteor. Soc., 101, E684–E699, https://doi.org/10.1175/BAMS-D-19-0050.1.
de Boer, G., and Coauthors, 2022a: Measurements from the university of Colorado RAAVEN uncrewed aircraft system during ATOMIC. Earth Syst. Sci. Data, 14, 19–31, https://doi.org/10.5194/essd-14-19-2022.
de Boer, G., and Coauthors, 2022b: Observing the central Arctic atmosphere and surface with University of Colorado uncrewed aircraft systems. Sci. Data, 9, 439, https://doi.org/10.1038/s41597-022-01526-9.
Desjardins, R. L., J. I. MacPherson, P. H. Schuepp, and F. Karanja, 1989: An evaluation of aircraft flux measurements of CO2, water vapor and sensible heat. Bound.-Layer Meteor., 47, 55–69, https://doi.org/10.1007/BF00122322.
Emanuel, K. A., 1986: An air–sea interaction theory for tropical cyclones. Part I: Steady-state maintenance. J. Atmos. Sci., 43, 585–605, https://doi.org/10.1175/1520-0469(1986)043<0585:AASITF>2.0.CO;2.
Estournel, C., and D. Guedalia, 1987: A new parameterization of eddy diffusivities for nocturnal boundary-layer modeling. Bound.-Layer Meteor., 39, 191–203, https://doi.org/10.1007/BF00121874.
Fitzjarrald, D. R., and G. G. Lala, 1989: Hudson valley fog environments. J. Appl. Meteor., 28, 1303–1328, https://doi.org/10.1175/1520-0450(1989)028<1303:HVFE>2.0.CO;2.
Greene, B. R., A. R. Segales, S. Waugh, S. Duthoit, and P. B. Chilson, 2018: Considerations for temperature sensor placement on rotary-wing unmanned aircraft systems. Atmos. Meas. Tech., 11, 5519–5530, https://doi.org/10.5194/amt-11-5519-2018.
Greene, B. R., A. R. Segales, T. M. Bell, E. A. Pillar-Little, and P. B. Chilson, 2019: Environmental and sensor integration influences on temperature measurements by rotary-wing unmanned aircraft systems. Sensors, 19, 1470, https://doi.org/10.3390/s19061470.
Greene, B. R., S. T. Kral, P. B. Chilson, and J. Reuder, 2022: Gradient-based turbulence estimates from multicopter profiles in the Arctic stable boundary layer. Bound.-Layer Meteor., 183, 321–353, https://doi.org/10.1007/s10546-022-00693-x.
Holland, G. J., and Coauthors, 2001: The Aerosonde robotic aircraft: A new paradigm for environmental observations. Bull. Amer. Meteor. Soc., 82, 889–902, https://doi.org/10.1175/1520-0477(2001)082<0889:TARAAN>2.3.CO;2.
Holt, T., and S. Raman, 1988: A review and comparative evaluation of multilevel boundary layer parameterizations for first-order and turbulent kinetic energy closure schemes. Rev. Geophys., 26, 761–780, https://doi.org/10.1029/RG026i004p00761.
Hu, X.-M., J. W. Nielsen-Gammon, and F. Zhang, 2010: Evaluation of three planetary boundary layer schemes in the WRF Model. J. Appl. Meteor. Climatol., 49, 1831–1844, https://doi.org/10.1175/2010JAMC2432.1.
Intrieri, J. M., and Coauthors, 2014: Global hawk dropsonde observations of the Arctic atmosphere obtained during the Winter Storms and Pacific Atmospheric Rivers (WISPAR) field campaign. Atmos. Meas. Tech., 7, 3917–3926, https://doi.org/10.5194/amt-7-3917-2014.
Islam, A., A. L. Houston, A. Shankar, and C. Detweiler, 2019: Design and evaluation of sensor housing for boundary layer profiling using multirotors. Sensors, 19, 2481, https://doi.org/10.3390/s19112481.
Jensen, A. A., and Coauthors, 2021: Assimilation of a coordinated fleet of uncrewed aircraft system observations in complex terrain: EnKF system design and preliminary assessment. Mon. Wea. Rev., 149, 1459–1480, https://doi.org/10.1175/MWR-D-20-0359.1.
Kaimal, J. C., J. C. Wyngaard, Y. Izumi, and O. R. Coté, 1972: Spectral characteristics of surface-layer turbulence. Quart. J. Roy. Meteor. Soc., 98, 563–589, https://doi.org/10.1002/qj.49709841707.
Kay, J. E., and A. Gettelman, 2009: Cloud influence on and response to seasonal Arctic sea ice loss. J. Geophys. Res., 114, D18204, https://doi.org/10.1029/2009JD011773.
Kay, J. E., K. Raeder, A. Gettelman, and J. Anderson, 2011: The boundary layer response to recent Arctic sea ice loss and implications for high-latitude climate feedbacks. J. Climate, 24, 428–447, https://doi.org/10.1175/2010JCLI3651.1.
Koch, S. E., M. Fengler, P. B. Chilson, K. L. Elmore, B. Argrow, D. L. Andra Jr., and T. Lindley, 2018: On the use of unmanned aircraft for sampling mesoscale phenomena in the preconvective boundary layer. J. Atmos. Oceanic Technol., 35, 2265–2288, https://doi.org/10.1175/JTECH-D-18-0101.1.
Koračin, D., J. A. Businger, C. E. Dorman, and J. M. Lewis, 2005: Formation, evolution, and dissipation of coastal sea fog. Bound.-Layer Meteor., 117, 447–478, https://doi.org/10.1007/s10546-005-2772-5.
Kral, S. T., and Coauthors, 2021: The Innovative Strategies for Observations in the Arctic Atmospheric Boundary Layer Project (ISOBAR): Unique finescale observations under stable and very stable conditions. Bull. Amer. Meteor. Soc., 102, E218–E243, https://doi.org/10.1175/BAMS-D-19-0212.1.
Lappin, F. M., T. M. Bell, E. A. Pillar-Little, and P. B. Chilson, 2022: Low-level buoyancy as a tool to understand boundary layer transitions. Atmos. Meas. Tech., 15, 1185–1200, https://doi.org/10.5194/amt-15-1185-2022.
Larson, K., D. L. Hartmann, and S. A. Klein, 1999: The role of clouds, water vapor, circulation, and boundary layer structure in the sensitivity of the tropical climate. J. Climate, 12, 2359–2374, https://doi.org/10.1175/1520-0442(1999)012<2359:TROCWV>2.0.CO;2.
Leuenberger, D., A. Haefele, N. Omanovic, M. Fengler, G. Martucci, B. Calpini, O. Fuhrer, and A. Rossa, 2020: Improving high-impact numerical weather prediction with lidar and drone observations. Bull. Amer. Meteor. Soc., 101, E1036–E1051, https://doi.org/10.1175/BAMS-D-19-0119.1.
Lock, A. P., A. R. Brown, M. R. Bush, G. M. Martin, and R. N. B. Smith, 2000: A new boundary layer mixing scheme. Part I: Scheme description and single-column model tests. Mon. Wea. Rev., 128, 3187–3199, https://doi.org/10.1175/1520-0493(2000)128<3187:ANBLMS>2.0.CO;2.
Lothon, M., and Coauthors, 2014: The BLLAST field experiment: Boundary-layer late afternoon and sunset turbulence. Atmos. Chem. Phys., 14, 10 931–10 960, https://doi.org/10.5194/acp-14-10931-2014.
Martin, G. M., M. R. Bush, A. R. Brown, A. P. Lock, and R. N. B. Smith, 2000: A new boundary layer mixing scheme. Part II: Tests in climate and mesoscale models. Mon. Wea. Rev., 128, 3200–3217, https://doi.org/10.1175/1520-0493(2000)128<3200:ANBLMS>2.0.CO;2.
Mayer, S., A. Sandvik, M. O. Jonassen, and J. Reuder, 2012: Atmospheric profiling with the sUAS SUMO: A new perspective for the evaluation of fine-scale atmospheric models. Meteor. Atmos. Phys., 116, 15–26, https://doi.org/10.1007/s00703-010-0063-2.
Maykut, G. A., 1978: Energy exchange over young sea ice in the central Arctic. J. Geophys. Res., 83, 3646–3658, https://doi.org/10.1029/JC083iC07p03646.
Meng, X. H., J. P. Evans, and M. F. McCabe, 2013: The influence of inter-annually varying albedo on regional climate and drought. Climate Dyn., 42, 787–803, https://doi.org/10.1007/s00382-013-1790-0.
Nolan, D. S., J. A. Zhang, and D. P. Stern, 2009: Evaluation of planetary boundary layer parameterizations in tropical cyclones by comparison of in situ observations and high-resolution simulations of Hurricane Isabel (2003). Part I: Initialization, maximum winds, and the outer-core boundary layer. Mon. Wea. Rev., 137, 3651–3674, https://doi.org/10.1175/2009MWR2785.1.
Ooyama, K. V., 1969: Numerical simulation of the life cycle of tropical cyclones. J. Atmos. Sci., 26, 3–40, https://doi.org/10.1175/1520-0469(1969)026<0003:NSOTLC>2.0.CO;2.
Pillar-Little, E. A., and Coauthors, 2021: Observations of the thermodynamic and kinematic state of the atmospheric boundary layer over the San Luis Valley, CO, using the CopterSonde 2 remotely piloted aircraft system in support of the LAPSE-RATE field campaign. Earth Syst. Sci. Data, 13, 269–280, https://doi.org/10.5194/essd-13-269-2021.
Pleim, J. E., 2007: A combined local and nonlocal closure model for the atmospheric boundary layer. Part II: Application and evaluation in a mesoscale meteorological model. J. Appl. Meteor. Climatol., 46, 1396–1409, https://doi.org/10.1175/JAM2534.1.
Reuder, J., and M. O. Jonassen, 2012: First results of turbulence measurements in a wind park with the Small Unmanned Meteorological Observer SUMO. Energy Procedia, 24, 176–185, https://doi.org/10.1016/j.egypro.2012.06.099.
Roadman, J., J. Elston, B. Argrow, and E. Frew, 2012: Mission performance of the Tempest unmanned aircraft system in supercell storms. J. Aircr., 49, 1821–1830, https://doi.org/10.2514/1.C031655.
Rotunno, R., Y. Chen, W. Wang, C. Davis, J. Dudhia, and G. J. Holland, 2009: Large-eddy simulation of an idealized tropical cyclone. Bull. Amer. Meteor. Soc., 90, 1783–1788, https://doi.org/10.1175/2009BAMS2884.1.
Rüdiger, J., J.-L. Tirpitz, J. M. de Moor, N. Bobrowski, A. Gutmann, M. Liuzzo, M. Ibarra, and T. Hoffmann, 2018: Implementation of electrochemical, optical and denuder-based sensors and sampling techniques on UAV for volcanic gas measurements: Examples from Masaya, Turrialba and Stromboli volcanoes. Atmos. Meas. Tech., 11, 2441–2457, https://doi.org/10.5194/amt-11-2441-2018.
Segales, A. R., 2022: Design and implementation of a novel multicopter unmanned aircraft system for quantitative studies of the atmosphere. Ph.D. dissertation, University of Oklahoma, 242 pp., https://shareok.org/handle/11244/336925.
Segales, A. R., B. R. Greene, T. M. Bell, W. Doyle, J. J. Martin, E. A. Pillar-Little, and P. B. Chilson, 2020: The CopterSonde: An insight into the development of a smart unmanned aircraft system for atmospheric boundary layer research. Atmos. Meas. Tech., 13, 2833–2848, https://doi.org/10.5194/amt-13-2833-2020.
Stephens, G. L., and Coauthors, 2000: The department of energy’s Atmospheric Radiation Measurement (ARM) unmanned aerospace vehicle (UAV) program. Bull. Amer. Meteor. Soc., 81, 2915–2938, https://doi.org/10.1175/1520-0477(2000)081<2915:TDOESA>2.3.CO;2.
Stull, R. B., and E. W. Eloranta, 1984: Boundary Layer Experiment—1983. Bull. Amer. Meteor. Soc., 65, 450–456, https://doi.org/10.1175/1520-0477(1984)065<0450:BLE>2.0.CO;2.
Tomlins, G. F., 1983: Some considerations in the design of low cost remotely-piloted aircraft for civil remote sensing applications. Cancer Surv., 37, 157–167, https://doi.org/10.1139/tcs-1983-0020.
Trapp, R. J., N. S. Diffenbaugh, H. E. Brooks, M. E. Baldwin, E. D. Robinson, and J. S. Pal, 2007: Changes in severe thunderstorm environment frequency during the 21st century caused by anthropogenically enhanced global radiative forcing. Proc. Natl. Acad. Sci. USA, 104, 19 719–19 723, https://doi.org/10.1073/pnas.0705494104.
Tripp, D. D., E. R. Martin, and H. D. Reeves, 2021: Applications of uncrewed aerial vehicles (UAVs) in winter precipitation-type forecasts. J. Appl. Meteor. Climatol., 60, 361–375, https://doi.org/10.1175/JAMC-D-20-0047.1.
Weaver, J. F., 1979: Storm motion as related to boundary-layer convergence. Mon. Wea. Rev., 107, 612–619, https://doi.org/10.1175/1520-0493(1979)107<0612:SMARTB>2.0.CO;2.
Wilson, J. W., and D. L. Megenhardt, 1997: Thunderstorm initiation, organization, and lifetime associated with Florida boundary layer convergence lines. Mon. Wea. Rev., 125, 1507–1525, https://doi.org/10.1175/1520-0493(1997)125<1507:TIOALA>2.0.CO;2.
Xi, X., M. S. Johnson, S. Jeong, M. Fladeland, D. Pieri, J. Andres Diaz, and G. L. Bland, 2016: Constraining the sulfur dioxide degassing flux from Turrialba Volcano, Costa Rica using unmanned aerial system measurements. J. Volcanol. Geotherm. Res., 325, 110–118, https://doi.org/10.1016/j.jvolgeores.2016.06.023.
