The ongoing urbanization and climate change urges further understanding and monitoring of weather in cities. Two case studies during a 17-day period over the Amsterdam metropolitan area, the Netherlands, are used to illustrate the potential and limitations of hydrometeorological monitoring using nontraditional and opportunistic sensors. We employ three types of opportunistic sensing networks to monitor six important environmental variables: 1) air temperature estimates from smartphone batteries and personal weather stations, 2) rainfall from commercial microwave links and personal weather stations, 3) solar radiation from smartphones, 4) wind speed from personal weather stations, 5) air pressure from smartphones and personal weather stations, and 6) humidity from personal weather stations. These observations are compared to dedicated, traditional observations where possible, although such networks are typically sparse in urban areas. First, we show that the passage of a front can be successfully monitored using data from several types of nontraditional sensors in a complementary fashion. Also, we demonstrate the added value of opportunistic measurements in quantifying the urban heat island (UHI) effect during a hot episode. The UHI can be clearly determined from personal weather stations, though UHI values tend to be high compared to records from a traditional network. Overall, this study illustrates the enormous potential for hydrometeorological monitoring in urban areas using nontraditional and opportunistic sensing networks.
Traditionally, hydrologists and meteorologists, scientists and practitioners alike, have relied on dedicated measurement equipment in their research and operations. Such instruments are typically owned and operated by governmental agencies. Installed and maintained according to (inter)national standards, they offer accurate and reliable information about the state of environment we study, monitor and manage. Standard instruments are often based on novel measurement techniques that originate in the research community and have been tested extensively during dedicated field campaigns.
Unfortunately, the operational measurement networks available to the hydrometeorological community today often lack the required spatial and/or temporal density for high-resolution monitoring or forecasting of rapidly responding environmental systems. Apart from the high installation and maintenance costs of such dedicated networks, it can be challenging, if not impossible, to install meteorological monitoring instruments according to the official requirements in urban areas (Oke 2006).
Yet sensors are omnipresent in our environment nowadays, often related to the rapid development in wireless communication networks (e.g., McCabe et al. 2017; Balsamo et al. 2018; Tauro et al. 2018; Zheng et al. 2018). To make use of such opportunistic sensors could be greatly beneficial to (meteorological) science and environmental monitoring and management operations. Opportunistic sensors are devices that were not installed with the intention to generate large-scale observations, but can be used as such. They may not be as accurate or reliable as the dedicated equipment we are used to, let alone meet official international standards. However, they are typically available in large numbers and are often readily accessible online. Hence, combined with smart retrieval algorithms and statistical treatment, opportunistic sensors may provide a valuable complementary source of information regarding the state of our environment.
This article surveys recent opportunistic sensing techniques in meteorology, from 1) rainfall monitoring using commercial microwave links (CMLs) from cellular communication networks, via 2) crowdsourcing urban air temperature, pressure, and solar radiation using smartphones to 3) high-resolution urban monitoring of air temperature, pressure, humidity, wind speed, and rainfall using personal weather stations (PWSs). Other opportunistic sensing examples are using security cameras as rainfall indicators (Allamano et al. 2015), rainfall information from sensors in driving cars (Rabiei et al. 2013), deriving the urban heat island (UHI) from measurements of gradients of shallow groundwater (Buik et al. 2004), using fiber-optic cables (Bense et al. 2016), using airplanes to measure upper-air wind and temperature (de Haan 2011), using hot-air balloons to measure boundary layer winds (de Bruijn et al. 2016), smartphone anemometers (Hintz et al. 2017), or using networks of solar panels for radiation monitoring. Muller et al. (2015) and Zheng et al. (2018) provide excellent overviews of past and ongoing projects making use of opportunistic sensing techniques, and USAID (2013) showcases practical applications of crowdsourcing projects for agricultural purposes in Africa. We limit ourselves to the presented techniques since these are relatively established even in developing countries, discussed in detail in literature, and observe near Earth’s surface.
We present a 17-day analysis for the Amsterdam metropolitan area, the Netherlands, where these opportunistic sensors are employed in a complementary fashion, in particular to provide detailed monitoring (both time series and spatially) of the passage of a front, as well as to demonstrate the potential of opportunistic sensors to quantify the UHI effect. This study aims to showcase the availability of several opportunistic sensing techniques and their ability to capture meteorological events.
Traditional sensing methods.
We use three traditional data sources as reference for the opportunistic sensing observations: a gauge-adjusted radar product, the WMO station at Amsterdam airport, and the Amsterdam Atmospheric Monitoring Supersite (AAMS; Ronda et al. 2017) urban network. Details on instrumentation and processing of these datasets are provided in the appendix.
Opportunistic sensing methods.
Smartphones contain many sensors to support their functionality, including sensors for light levels to adjust screen brightness, pressure sensors to complement the GPS for an accurate (vertical) location estimation, and thermometers for the battery to avoid damage from overheating. Readings from such sensors can be used for opportunistic environmental sensing by collecting them through mobile applications (apps). These apps sample the sensor readings with a certain frequency, along with the last-stored GPS coordinates. Examples of apps that collect and store smartphone sensor readings include Pressurenet (www.cumulonimbus.ca/) (Mass and Madaus 2014; Madaus and Mass 2017), OpenSignal (https://opensignal.com/), and Atmos (Niforatos et al. 2014, 2017).
Mass and Madaus (2014), Madaus and Mass (2017), and McNicholas and Mass (2018) show that assimilating smartphone pressure data into NWP models improves representation of convective events. Likewise, Hintz et al. (2019) show for a case in Denmark that assimilating smartphone pressure observations decreased the surface pressure bias in an NWP model. Different quality-control methods were applied: for Madaus and Mass (2017) the raw smartphone pressure readings were filtered to only include one value per smartphone per assimilation time step, and were also corrected for the terrain elevation and checked for spatial and statistical consistency. Hintz et al. (2019) additionally use a consistency check with synoptic observations. In McNicholas and Mass (2018) a machine learning algorithm was used to remove outliers. Niforatos et al. (2017) compared smartphone light sensor readings with manually reported classifications of weather, which showed light readings to be indicative of present weather conditions.
Citywide air temperatures can be estimated from smartphone battery temperature readings, as has been shown for eight major cities (Overeem et al. 2013b), for the city of Birmingham (Muller et al. 2015) for daily temperatures, and for São Paulo for hourly and daily temperatures (Droste et al. 2017). Statistical training with independent temperature measurements was performed based on a steady-state heat transfer model: a smartphone is typically carried close to the user’s body. The thermal energy generated by the smartphone must be balanced by heat exchange to the body and the environment. The conductive heat flow between two adjacent systems is assumed to be proportional to their temperature difference, and depends on the thermal insulation between smartphone and environment, and between smartphone and body. This principle allows us to estimate hourly averaged air temperatures from hourly averaged battery temperatures (Overeem et al. 2013b):
where is the hourly mean urban air temperature, is the hourly averaged battery temperature (both in space A and time), T0 a constant equilibrium temperature, is a coefficient, εj,h is a random disturbance, and h denotes the hour.
In this study we build upon a large dataset of observations obtained from the Android application OpenSignal, which crowdsources data relevant to wireless connectivity along with the aforementioned sensor readings. Compared to the previously mentioned studies, readings were obtained at a far higher frequency, that is, 15-s intervals whenever the smartphone screen is active, not requiring the app to be opened by the user. A total of 3.14 million smartphone observations are available for the entire study period for the Amsterdam metropolitan region (larger domain in Fig. 1a).
The OpenSignal dataset includes self-reported accuracy scores (1, 2, or 3) of the light and pressure readings, as determined by the sensor management software in the smartphones (Android 2019). Only readings with the highest possible accuracy were included in our analysis. All smartphone pressure sensor readings below 950 hPa are excluded, based on the lowest recorded pressure in the Netherlands, 954.2 hPa (de Haij 2009), which results in a dataset of 2.06 million pressure readings. Light sensor readings above 0 lux (lumen m−2) are taken into account, leaving 2.32 million light readings in the whole study period. We only include battery temperature readings between 10° and 47°C when the smartphone is not charging: 0.4 million temperature readings within the city center. Hourly battery temperature readings are averaged spatially over the city center domain (Fig. 1b); light and pressure are averaged over the entire region for each hour (Fig. 1a).
Ambient air temperatures are estimated from battery temperature [Eq. (1)]; the value of equilibrium temperature T0 as optimized by Overeem et al. (2013b), 39°C, is used. Figure 1b shows the positions of the underlying battery temperature readings. Two different datasets are derived: one without and one with optimizing the coefficients of the heat transfer model for the available dataset. The first dataset uses a fixed value of for all hours, 2.4, as found for a summer period in London based on daily averages (Overeem et al. 2013b). These results, without further model calibration, are presented in Fig. 2b, which also shows the 25th and 75th percentiles. For the second dataset, records from 0000 UTC 1 June to 0000 UTC 15 June are employed to calibrate a value of for each clock hour (24 in total, ranging between 2.0 and 2.6). These optimized values, found using a least squares regression, are applied to the validation dataset from 0000 UTC 16 June to 0000 UTC 23 June.
A smartphone light sensor measures illumination in lux, that is, irradiance weighted for the visible part of the electromagnetic spectrum, so a measure for the perceived brightness for the human eye. To estimate the equivalent solar radiation, we use an empirical factor of 0.0079 lux (W m−2)−1, based on the spectral distribution of sunlight (Chua 2009). By applying this transformation, the readings are treated as if they were measurements of solar radiation. This is a fairly strong assumption, as we expect that most readings will not be made in a representative manner: with the smartphone perpendicular toward the sun and in direct sunlight. User behavior plays a large role (e.g., indoor vs outdoor measurements), so one may expect that most light readings will underestimate the solar radiation, resulting in a skewed distribution. A light sensor in a smartphone has a limited view angle (<180°) and has a relatively poor cosine response. Additionally, the sensor can oversaturate at high light intensities (the sensor limit is typically around 200 W m−2). Therefore, it is desirable to have many readings to increase the probability of observations taken in favorable conditions (unshaded and perpendicular to direct sunlight).
Because smartphone measurements are taken when the smartphone is used, most data are available for those times where people are active. Since hundreds of smartphone measurements are required to obtain a good signal of air temperature (Droste et al. 2017), spatial detail is limited to local climate zone (LCZ; Stewart and Oke 2012) scale at best, and temporal resolution to roughly hourly. The data at this availability are useful to get a broad overview of urban temperature, but not for (spatially) detailed studies.
Commercial microwave links.
Cell phone communication relies on a telecommunication link network that consists of transmitting and receiving antennas, typically several kilometers apart, between which radio signals propagate. Telecom operators commonly use signal frequencies that are sensitive to hydrometeors. This causes attenuation of the microwave link signals when liquid precipitation occurs between the antennas. Upton et al. (2005) first suggested to use signal attenuation in CML networks, which is typically monitored for quality-control purposes, to determine rainfall. Soon after, this was shown to be successful with actual CML data (Messer et al. 2006; Leijnse et al. 2007). This was promising as microwave link networks are widespread, also in areas of the world with limited to no traditional rainfall sensors. Subsequent research has focused on improving the techniques to obtain accurate rainfall estimates from these datasets, (e.g., Leijnse et al. 2008; Zinevich et al. 2010; Overeem et al. 2011; Chwala et al. 2012) and produce rainfall maps (Overeem et al. 2013a, 2016b) with real-time applicability (Chwala et al. 2016; Andersson et al. 2017; Chwala et al. 2018). Comprehensive overviews of literature on this technique were provided by Messer and Sendik (2015), Uijlenhoet et al. (2018), and Chwala and Kunstmann (2019). Several tools have been developed, documented and made (freely) available for users to construct rainfall observations with CML data: “Rcmlrain” (https://github.com/fenclmar/Rcmlrain), “RAINLINK” (https://github.com/overeem11/RAINLINK), “pycomlink” (https://github.com/pycomlink/pycomlink), or for real-time CML data acquisition: “pySNMPdaq” (https://github.com/cchwala/pySNMPdaq).
The relation between rainfall attenuation and rainfall intensity can be described with a power law between path-averaged specific signal attenuation k (dB km−1) and link path-averaged rainfall intensity R (mm h−1) (Atlas and Ulbrich 1977):
Coefficients a (mm h−1 dB−b kmb) and b (nondimensional) are dependent on signal frequency and polarization (Olsen et al. 1978; Jameson 1991). TSL and RSL are the transmitted and received signal level (dBm), respectively, Aa is the attenuation due to wet antennas (dB) assumed as a fixed value, Awet and Adry are the attenuation under wet and dry weather conditions, respectively (dB), and L is the length of the link path (km). The specific attenuation due to rainfall is what remains when the attenuation due to other causes (i.e., dry weather conditions and wet antennas) are subtracted.
The time series shown in Fig. 2d originate from the T-Mobile CML network visualized in Fig. 1a. Between 0000 UTC 6 June and 1400 UTC 10 June, 74 links were operational in the study area. Power levels were instantaneously sampled every 15 min. Due to data transfer issues, no power levels were available at the end of the study period. Rainfall time series for each link were constructed with the open source package RAINLINK (Overeem et al. 2016a), using the approach and optimized parameters from de Vos et al. (2019b). The wet antenna attenuation makes up a larger fraction of the total attenuation for short links, meaning that a small error in Aa, a constant, will result in a relatively large error in k for short links, and the effect on the estimated value of R would subsequently be larger than for long links given the same error in Aa.
Crowdsourced personal weather stations.
PWSs allow anyone to measure weather variables in their direct environment. Many automatic PWSs can upload their measurements directly to online platforms where they can be visualized and shared. Weather Underground (www.wunderground.com/wundermap), WOW-NL (https://wow.knmi.nl/) and the Netatmo Weathermap (https://weathermap.netatmo.com/) are examples of platforms where weather observations are visualized in real time. Ideally, weather variables can be crowdsourced from such platforms in far higher spatial and temporal resolution than from traditional sensor networks.
The devices are often low cost with a lower expected measurement accuracy than typical sensors from meteorological institutes. The PWSs are installed by citizens without expert knowledge on sensor placement requirements and/or lacking available measurement site without interference from surroundings. Hence, we expect that many of the PWSs generate compromised measurements. For tipping-bucket rain gauges, obstructions (e.g., insects, twigs) and the device not being completely level with the ground could hinder the tipping mechanism. A shielded location will also lead to underestimation of rainfall. Overestimation of rainfall can result from PWS owners cleaning or handling the device, resulting in tipping-bucket tips, creating measurements of artificial rain. PWS wind measurements are also largely affected by their position in relation to obstacles and the shielding effect of buildings. Furthermore, PWSs with a sonic anemometer are sensitive to rain blocking the path of the sound waves, so data quality might be compromised during rain events. Urban wind is highly variable in space, and is often measured as profile using, for example, lidar (Drew et al. 2013), so spatial averaging of PWS wind measurements is needed to obtain useful data. Temperature readings are highly affected by direct radiation: the lack of a proper radiation screen in most PWSs can result in overestimation of temperature by several degrees when positioned in direct sunlight (Bell et al. 2015; Chapman et al. 2017). Finally, the updates of measurements to the platform can be infrequent, and connectivity problems will result in large gaps in the time series.
Only a few studies compared PWSs with high-end sensors: temperature, relative humidity, radiation, pressure, rainfall, wind speed, and direction: Jenkins (2014) and Bell et al. (2015); temperature: Meier et al. (2015); rainfall: de Vos et al. (2017). Other studies have benefited from available PWS temperature records in cities. The UHI is then defined as the difference between PWS temperatures and a rural reference station (Meier et al. 2017; Chapman et al. 2017; Fenner et al. 2017; Golroudbary et al. 2018; Napoly et al. 2018). Preliminary work has been performed on crowdsourced wind (Droste et al. 2018) and rainfall measurements (de Vos et al. 2017; Golroudbary et al. 2018; Chen et al. 2018) and explored with simulated PWS rainfall measurements by de Vos et al. (2018) as well. In other studies code has been developed and made available to apply quality control (QC) on crowdsourced PWS data (the CrowdQC R-package for PWS temperature observations, https://depositonce.tu-berlin.de//handle/11303/7520.3; TITAN, https://github.com/metno/TITAN/; and code to filter crowdsourced rainfall observations PWSQC, https://github.com/LottedeVos/PWSQC).
Measurements from all personal weather stations from the brand Netatmo in the Amsterdam study area (Fig. 1a) are evaluated. All devices measure temperature, pressure and humidity. Additionally, rain and/or wind are measured in case those optional modules are installed for that PWS. To standardize the variable time intervals, all measurements are attributed to the time stamp of the 5-min interval in which it occurred. If multiple measurements occurred within the 5-min interval they are averaged (or accumulated in case of rainfall). The measurements over the study period are shown in Fig. 2 (Figs. 2a,c,f,h), where Fig. 2i indicates the dewpoint depression (DPD) as calculated from the temperature and humidity measurements from the PWS. No QC treatment is applied on the PWS data to showcase the raw potential. DPD is here preferred over dewpoint temperature itself to identify the frontal passage.
Case selection and study area
We selected Amsterdam (capital of the Netherlands) and its surroundings and the period between 0000 UTC 6 June and 0000 UTC 23 June 2017 as case study period (local time is UTC + 2 h). This period contains both sufficient data from opportunistic sensing techniques, and interesting meteorological events to illustrate the potential of the opportunistic sensing techniques. The selected region is bound by 52.24°–52.44°N, 4.67°–5.05°E (22 km × 26 km). To be able to distinguish between the inner city and suburbs, the study area was divided into two parts: that is, the urban center dataset covering 52.34°–52.38°N, 4.83°–4.95°E and the suburban dataset covering 52.24°–52.44°N, 4.67°–5.05°E, excluding the urban center area (Fig. 1a).
The Netherlands has a temperate maritime climate (Köppen Cfb). With a mean temperature of 18.0°C and 50.5 mm of rainfall, June 2017 was about 2.5°C warmer and 10.5 mm drier than the climatological mean (based on the past 30 years of observations at station WMO 06240 Amsterdam airport, henceforth referred to as “Amsterdam airport”). The month had eight summer days and two tropical days (maximum temperatures above 25° and 30°C, respectively).
On 6 June, a small low pressure system developed over the North Sea off the coast of the Netherlands and passed over the country, resulting in a substantial pressure drop to 992 hPa, an hourly maximum wind speed of 54 km h−1 (7 Beaufort; Bft) and 12 mm of rainfall measured at Amsterdam airport. On the morning of 9 June, an active cold front brought in relatively cold air, which resulted in 27 mm of rainfall. A clear-sky episode occurred 9–11 June, while another cold front passed in the early morning of 12 June (Fig. 2i). In the following period, no rainfall occurred, and temperatures were mild (daily maximum temperatures below 25°C), followed by a warm episode between 16 and 19 June. On 19 June the maximum air temperature reached 29.8°C at Amsterdam airport. This warm episode ended with the passage of a cold front and associated rainfall and thunderstorms on 22 June. For the remainder of the paper we will focus on two cases, that is, case A, describing the passing front and resulting rainfall at the start of the study period, and case B, containing the hot summer period, with a focus on UHI detection.
For this study, the UHI is defined as the instantaneous urban air temperature difference between the city and the countryside (Stewart 2011). The UHI develops as a result of the relatively low albedo of cities, high heat capacity of the urban fabric, thermal radiation trapping, and low surface evapotranspiration. The UHI is favored by weather conditions with high solar insolation (low cloud cover) and low wind speeds (Oke 1982; Theeuwes et al. 2017). Earlier crowdsourcing observations indicated that Dutch urban areas experiences a mean daily maximum UHI of 2.3°C and the 95th percentile amounts to 5.3°C (Steeneveld et al. 2011). Ronda et al. (2017) found a mean evening UHI of ∼1°C, and a maximum of 4.5°C in Amsterdam for the summer of 2015 as a whole.
Case A: Weather front.
First, we focus on the passage of a cold front over the study area on 9 June. At 0600 UTC the operational model analysis provided by KNMI locates the frontal zone to the west of Amsterdam (not shown), and by 1200 UTC the front has passed the city. Prior to the frontal passage itself, an upper-air disturbance passed over Amsterdam between 0300 and 0400 UTC, bringing strong convection and rainfall. Such frontal zones cause distinctive behavior in various meteorological variables, which we expect to be distinguishable in the crowdsourced data (Fig. 2).
The passage of the front is clearly visible in the observed DPD and the wind speed (Fig. 3a). The DPD steadily drops during the approach and passage of the cold front, reaching a minimum of 1.4°C at 0900 UTC. Between 1000 and 1100 UTC, when the front has passed, the DPD increases again up to 6.8°C, indicating the cold and dry air mass brought in by the cold front. Crowdsourced and reference wind speed steadily increase as the front passes (from 2 to over 4 km h−1), before reaching its maximum (5 km h−1) directly after the passage. The convection associated with the upper-air disturbance at around 0300–0400 UTC generates a strong peak in the wind speed. Despite the unknown measurement setup of the PWS anemometers, the average signal of all PWSs corresponds well to that of the quality-controlled reference AAMS network (mean bias of 0.4 km h−1), which shows the same behavior for the upper-air disturbance and the front passing. However, the AAMS signal indicates a delayed onset of the wind speed increase (at around 0900 UTC) and takes longer to reach a higher maximum wind speed.
The ambient air pressure (Fig. 3b), measured by PWSs and smartphones, starts increasing at the moment the front passes (0800 UTC). Typically, air pressure decreases before a cold front, rapidly increases during the passage, and increases at a slower rate afterward. The expected drop prior to the frontal passage is not very pronounced in the measurements: there is a slight decrease in pressure between 0000 and 0200 UTC (1.7 hPa decrease for PWS; 3.5 hPa for smartphone). The latter is more likely associated with the upper-air disturbance. After the frontal passage at 0800 UTC, the pressure rises, from 1,006 to 1,008 hPa (PWS to smartphone) up to a maximum of 1,013–1,016 hPa at midnight. The pressure tendency remains roughly 1 hPa h−1 after the front has passed.
The light intensity as measured by smartphones shows a distinct diurnal pattern in Fig. 2e, following the course of solar radiation. The measured data are strongly skewed, so the median light intensity values are low (Fig. 2e). Figure 4 shows the 99th percentile of light readings to capture the readings made in the most favorable light conditions (see “Smartphone data” section). The sky on 9 June is overcast (8 oktas) until 1100 UTC, at which time the front has passed over Amsterdam and the sky clears up to scattered cloudiness (Fig. 4). The light intensity is also very low until 1000 UTC, even though this is well within daylight hours. Compared to 18 June (a clear day) the light intensity is roughly halved, and the shape of the line is not as symmetrical (as we would expect from the diurnal cycle of global radiation). The green lines in Fig. 4 indicate the other days over the study period, showing the strong variability in the daily course of light intensity. The light intensity measured by smartphones not only depends on incoming radiation, but also strongly on user behavior (indoors vs outdoors, the angle of the phone) and the type of light sensor in the smartphone, which can differ between brands. The light sensor may also be oversaturated during high light intensities, resulting in flattened peak values.
The light intensity peak at 0400 UTC coincides with the upper-air disturbance seen in Fig. 3, but is actually an artifact of the low number of observations. The number of available observations is higher during the day than during night and early morning, since it is related to user activity whether the smartphone logs an observation (as detailed in the “Smartphone data” section). At 0400 UTC there are only 502 smartphone observations, compared to 10,373 at 1500 UTC (1700 local time, the typical end of the working day), so the data are more sensitive to outliers.
The upper-air disturbance, and subsequent frontal passage, of 9 June results in 27 mm of rainfall as measured by the gauge-adjusted radar reference. Figures 2c and 2d show that the peak of rainfall occurs after sunrise, coinciding with the timing of the frontal passage. Figures 5a and 5b depict the cumulative rain over 9 June, measured by CML and PWS, against the reference. Total amounts differ between the two methods, but both show the same time response. The relatively short links (<2 km) overestimate rainfall, with the majority reporting >30 mm of rainfall (relative bias is 87%). The longer links (≥2 km) also tend to overestimate, but much less extreme (relative bias is 12%). Although the expected uncertainty in rainfall estimates is higher for short links, the larger systematic bias (54% relative bias, or 0.13-mm absolute bias, for all links) indicates that the methods to derive rainfall (RAINLINK) were not ideal for this rainfall event, especially for short links.
PWS measurements tend to underestimate the rain as measured by the reference, with some occurrences of large reported rainfall values that are not otherwise captured (Fig. 5b). Nevertheless, the majority of PWSs seem to agree overall with the reference (Figs. 5b,d). The spatial distribution of rainfall (Fig. 6) measured by PWS and CML corresponds to that of the gauge-adjusted radar reference. We find that areas with high rainfall in the reference also yield high accumulations in the CML and PWS data in these areas. The overestimation by short links up to 8 mm is visible to the northwest of the band with high rainfall. The rainfall observations by PWSs correspond well to the spatial pattern of rain, although a number measure little (<1 mm) rain during the hour represented in Fig. 6. These stations are mainly clustered in the city center. The large amount of obstructions inside the city center could reduce the rainfall received by the stations, which may partly explain the underestimation tendency already seen in Figs. 5b and 5d.
Case B: Urban heat island.
The last days of the study period are characterized by high temperatures and generally clear, sunny weather, leading to higher urban temperatures (PWS median air temperature up to 30°C on 19 June; Fig. 7). Air temperature is measured by PWS, and derived through the smartphone battery temperature using the second, calibrated dataset (“Smartphone data” section). The AAMS network serves as urban reference, and the Amsterdam airport measurements are used as rural reference for the UHI (Fig. 7b). The smartphone-derived air temperature differs clearly from the PWS and AAMS measurements, with more erratic behavior and strong minimum values at night and early morning (as low as 7°C when the AAMS values are above 16°C). Figure 7c showcases this larger spread, also indicated by the large standard deviation (2.82°C compared to 0.66° and 1.08°C for the PWS). During daytime the smartphone-derived temperatures correspond better with the PWS and AAMS measurements than at night. The diurnal cycle is clearly visible: the low values at night are most likely due to a low number of measurements available, increasing the sensitivity to outliers. Despite these occasional large deviations, the bias amounts to −0.6°C compared to AAMS (Fig. 7c), which is relatively small. A large positive bias (2.0°C) is found when a fixed literature value for is used, for the time series shown in Fig. 2b, whereas the other model statistics are mostly uninfluenced by optimizing .
The diurnal pattern of air temperature between center and suburban PWSs is similar, although the center stations tend to be warmer at night, and colder during the day (Fig. 7a) The suburban stations contain a higher spread and bias than the center stations, though both show good agreement to the reference (Fig. 7c). The AAMS air temperature is typically about 2°–3°C lower during the day: this could partially be caused by the unknown setup of the Netatmo station, which is likely be exposed to direct sunlight or close to walls, making it sensitive to radiation errors. Figure 7b depicts the UHI estimated by subtracting the center and suburban PWS (red dashed line). This particular PWS–UHI shows spatial variability within the PWS data, which is most pronounced during daytime, where the difference can be up to −1.5°C (i.e., the center is 1.5°C colder than the suburban area). Higher urban shading in the morning, and the faster heating rate of the relatively thin rural boundary layer compared to the deeper urban boundary layer cause this urban cool island in the morning (Theeuwes et al. 2015).
The other two UHI estimates are constructed using Amsterdam airport as rural background, showing that the city center is indeed much warmer at night than the rural surroundings. Urban cool islands typically form in the morning, persisting for several hours before the city heats up more. A remarkable 6°C UHI peak is visible on 22 June, in the afternoon (1300–1400 UTC). This seems to be mainly caused by the Amsterdam airport temperature, since the PWS–UHI (which has no true rural reference) shows a value close to 0°C at that time. This is visible in Fig. 7a, where temperatures rapidly decrease in the course of a few hours on the afternoon of 22 June. Thunderstorms were reported on this day, and several mm of rain were measured at Amsterdam airport (according to radar) between 1400 and 1500 UTC. The UHI in this case is likely caused by the sudden cooling of the rural reference, rather than strong urban heating.
Figure 8 presents the spatial variability in the AAMS and PWS temperature recordings between 0200 and 0300 UTC 18 June, when the UHI is typically largest. The cluster of stations in the center yields higher values than the suburban stations, although in both areas many stations deviate from this trend. The center PWS report an average UHI of 4.0°C, the AAMS UHI is 3.6°C, whereas the suburban areas have an average UHI of 2.7°C. Variability between measurement sites is high: some stations report a temperature difference of up to 12.4°C, and even a few with negative UHI (up to −0.6°C).
Discussion and conclusions
We have shown that even though each technique has considerable limitations regarding accuracy, the data from opportunistic sources can be used to monitor meteorological phenomena. The potential of these techniques lies in the high spatial density of such observations, especially in urban areas.
We explicitly consider observations that can be obtained near directly from the opportunistic sensors, without applying many correction schemes, to illustrate their inherent potential: validation using the available quality assurance schemes was not the aim of this research. We use temperature from smartphone batteries and personal weather stations (PWSs), rainfall from commercial microwave links (CMLs) and PWS, solar radiation from smartphones, wind speed from PWS, air pressure from smartphones and PWS, and humidity from PWS. Two case studies in a 17-day period over the city of Amsterdam, the Netherlands, are explored. In the first case study we show how the passage of a front is apparent from many of the data sources. The second case study shows that these measurements can be valuable in monitoring the urban heat island (UHI) effect, especially given the fact that WMO stations in urban areas are very rare.
The passage of a cold front is visible in all of the studied opportunistic sensing data sources. The dynamics of the temperature (especially from PWS, less so from smartphones), rainfall, solar radiation, wind speed, air pressure, and humidity all show the passage of the front. However, not every aspect of the weather events is sufficiently captured by the data: techniques using smartphone observations can only estimate a variable as a spatial average over the city and cannot be used to describe detailed spatial variability. Also, the PWS wind observations were too noisy to describe spatial patterns in the city with confidence.
A rural reference station is needed to quantify the UHI, for which we use the WMO station Amsterdam Airport. Even the PWS locations outside the city center of Amsterdam (suburban, see Figs. 1 and 8) are mostly in built-up areas, and are hence expected to experience the UHI, although less severely. This is supported by Fig. 7b, where the temperature difference between the city center and the suburban areas is much less pronounced than when the WMO station is used as rural reference. Using a single rural reference to quantify the UHI, instead of multiple background stations, is a good practice when the main interest is in the intra-urban variability of temperature, as in this work and, for example, Fenner et al. (2017). However, a limitation of this practice is evident from the artificially high UHI we see in Fig. 7 on 22 June, which was caused by local cooling at the rural site. Finally, we note that the distinction between center and suburban in this study was made rather crudely. In future UHI studies we recommend a more sophisticated partitioning of the stations into different classes (such as local climate zones; Stewart and Oke 2012).
The PWSs are suitable for monitoring the UHI. When compared to the AAMS urban reference network, these PWS show an UHI on the same order of magnitude (2°–4°C), especially during the night. We observe urban cool islands during the period between sunrise and local noon. Air temperatures derived from smartphone battery temperatures exhibit much more noise than PWS temperatures, which limits their use for UHI measurement. We note that the PWS thermometers are not shielded from solar radiation or ventilated, whereas the AAMS are. This is clearly visible in Fig. 7, where the PWS temperature (and derived UHI) increases much more quickly than the AAMS temperature. This corresponds to the findings of Bell et al. (2015).
Both smartphones and PWSs provide good estimations of pressure. Pressure fields are relatively constant in time and space, and both opportunistic sensing techniques show ability to describe them.
Light estimations derived from smartphones are highly variable in time. The indirect nature of the measurement and the typical suboptimal conditions during sampling result in merely an indicative observation of light. Such measurements should only be considered in the absence of dedicated sensor observations and considered with caution.
Average wind speed from the PWSs are very low compared to what would be expected during the passage of a front. This may partly be due to how the PWS anemometers are installed. However, the carefully installed AAMS stations show average wind velocities on the same order of magnitude (around 5 km h−1), indicating that the placement of the anemometers does not play a large role here. The reasons for these wind speeds to be lower than expected lies in the fact that the urban wind measurements are made at a lower level, and the urban fabric greatly reduces wind speeds at street level (Macdonald 2000). This also means that wind speeds are expected to be highly variable across the city, which is clear from Fig. 2f. Spatial averaging over the city is therefore needed in order to see clear signals in the wind speed. Spatial averages of wind speed show the same behavior between PWS and AAMS, indicating their use to measure the urban wind as a whole. Note that the wind sensor on the Netatmo PWS is a sonic anemometer, which is negatively affected by precipitation, hence wind observations during rainfall can be less reliable. This illustrates the need of a quality-control procedure that could improve overall data quality by filtering out precipitation events (Droste et al. 2018).
Data from PWS and CML are shown to provide useful information on both rainfall amount and space–time variation. Their ability to show detailed variations in space and time makes them useful for qualitative use in rainfall monitoring. The CML network overestimated rainfall in case A (Fig. 5a), although the relative bias of long links (≥2 km) was 71% smaller than that of short links (<2 km). This is likely related to the larger error contribution wet antennas have for shorter links, and that the correction was calibrated on a different dataset, possibly with more long links (de Vos et al. 2019b).
The PWS also show a good agreement with the reference, although most stations underestimate rainfall (Fig. 5b). This may be due to the higher wind speed above the urban fabric, which could cause buildings to act as a shield for the PWS rain gauges. It is also apparent from Fig. 5d that some PWSs report either zero rainfall when there is clearly rain or large amounts of rainfall where there was none. Such, and other errors could be corrected by using automated filters (de Vos et al. 2019a). The difference in accumulations between the city-averaged CML and PWS rainfall data (see Fig. 2) is partly caused by the overestimation by CML. However, differences may also be due to the spatial variation of rain and the respective locations of the PWS and CML. Figure 6 shows that for the examined hour the CML are more abundant in high-rainfall areas, whereas the PWS are more clustered in the city center, where less rainfall was observed.
The method used to derive rainfall estimates from CML data (RAINLINK) is one of many possible methods (see “Commercial microwave links” section). Our dataset consists of instantaneously sampled CML data, which is more prone to errors than CML data obtained with a min/max sampling strategy (de Vos et al. 2019b) and likely also than strategies more frequently than every 15 min or those providing mean values.
Our study shows that the research opportunistic sensing techniques all yield meaningful results. However, without quality-control procedures, PWS data perform better than smartphone or CML measurements. The PWS sensors are designed to measure hydrometeorological variables and are less reliant on quality control than the indirect CML or smartphone observations. A thorough procedure that removes error sources will therefore be most effective for the CML (in addition to the quality-control steps in the RAINLINK tool) and smartphone data, which can strongly improve with regards to the unfiltered signal. This may change in the future when the expected measurement density of smartphones increases and their hardware (sensor capability) improves. The observations contain large errors, as found by the larger spread in the data than would be explained by spatial or temporal variability. However, the opportunistic sensors provide information in time scales and areas that cannot be achieved with traditional sensing techniques.
Many PWSs are found in densely populated areas, where also many smartphones are operational. This is mainly true for urban areas in parts of the world where people have funds to invest in these devices (although smartphones are considered so important that they are essentially ubiquitous, independent of living standards). CML networks differ as well, in sampling strategy and frequency (which affects the accuracy of rainfall estimates) and in network density (depending on replacement by fiber optic technology). The availability of opportunistic sensing observations should be explored in order to judge their usefulness, especially as their accuracy heavily relies on the quantity of observations. Because traditional meteorological measurements are generally absent in urban areas, these new data provide a welcome addition. This is particularly important for monitoring the UHI, and wind and rainfall at street level. We therefore urge the scientific community to keep investigating new sources of data, and to study the uncertainties therein. In combination with reference networks of meteorological measurements or stand-alone, these new sources will provide much needed hydrometeorological information for citizens and scientists, in any part of the world.
Gert-Jan Steeneveld and Arjan Droste acknowledge funding from the Netherlands Organisation for Scientific Research (NWO) Project 864.14.007. Lotte de Vos is financially supported by KNMI MSO Project 2013.09. The reference stations of the Amsterdam Atmospheric Monitoring Supersite have been financially supported by the Amsterdam Institute for Advanced Metropolitan Solutions (AMS), which has also provided cofunding (Project VIR17006). We thank James Robinson from OpenSignal (London, United Kingdom) for providing the smartphone dataset. We gratefully acknowledge Ronald Kloeg and Ralph Koppelaar from T-Mobile NL for providing the cellular telecommunication link data. All PWS owners who have contributed to the Netatmo dataset are greatly appreciated, as well as the Netatmo company for making the measurements available to us (see https://dev.netatmo.com/apidocumentation/weather for information on how to obtain Netatmo data). Datasets retrieved from third parties (CML and OpenSignal) are available upon request. The gauge-adjusted radar rainfall dataset is freely available in netCDF4 format, “Radar precipitation climatology” (via http://climate4impact.eu) or in HDF5 format at the KNMI Data Centre (https://data.knmi.nl/datasets/rad_nl25_rac_mfbs_em_5min/2.0?q=radar). The AAMS data are available upon request (contact: firstname.lastname@example.org).
Appendix: Traditional sensing methods
Gauge-adjusted radar dataset.
The Royal Netherlands Meteorological Institute (KNMI) operates two C-band Doppler weather radars. The 5-min reflectivity data from these radars are combined into one composite using a weighing factor as a function of distance from the radar. Beekhuis and Mathijssen (2018) provide detailed characteristics on the radars and the processing of their data. Reflectivity factors Z (mm6 m−3) are converted to rainfall intensities R (mm h−1) with a fixed Z–R relationship (Marshall et al. 1955), Z = 200R1.6, and, subsequently, accumulated to rainfall depths for different durations. The two KNMI rain gauge networks are employed to adjust the radar-based accumulated rainfall depths: an automatic network with 1-h rainfall depths for each hour (∼1 station per 1,000 km2) and a manual network with 24-h, 0800–0800 UTC rainfall depths (∼1 station per 100 km2). A daily spatial adjustment utilizing the manual gauge data is combined with an hourly mean field bias adjustment employing the automatic gauge data. The resulting gauge-adjusted radar rainfall dataset has a spatial resolution of 0.9 km2, with no missing data for the study period. Overeem et al. (2011) provide a more detailed description of this radar dataset, which largely uses the methodology developed by Overeem et al. (2009a,b). Finally, 15-min path-averaged rainfall intensities are derived from the radar pixels covering each link path of the CML dataset (described in “Commercial microwave links” section). The gauge-adjusted radar rainfall dataset is used as a reference to validate rainfall estimates from CMLs and PWSs.
WMO station Amsterdam airport.
The WMO station Amsterdam airport, WMO 06240 (52.32°N, 4.78°E; Fig. 1a) provides hourly air temperature and cloud cover observations. This surface synoptic station is operated by KNMI, situated in a polder (4.18 m below MSL) and surrounded by meadows, arable land, and buildings as well as infrastructure from Amsterdam airport. Air temperature is observed at 1.5-m height above short mowed grass. The sensor is covered by a radiation screen and well ventilated. Cloud cover aloft is obtained from a LD40 ceilometer, which uses lidar to detect the height and concentration of particles, such as cloud droplets. KNMI (2000) provides more information on the temperature observation.
Amsterdam Atmospheric Monitoring Supersite.
As an urban reference network, we utilize the observations from the Amsterdam Atmospheric Monitoring Supersite (AAMS; Ronda et al. 2017), which consists of 30 weather stations across the city. The network consists of temperature and humidity sensors (Decagon VP-3, United States) mounted inside a 184-mm aspirated radiation shield (Davis, United States). The ventilation fan is powered by two small solar panels mounted on top of the shield. The fans work at global radiation levels >100 W m−2. The radiation screens are mounted onto lantern posts using a boom to mount the center of the radiation screen 0.46 m away from the edge of the lantern post at a height of 4.0 m above ground level. The sonic anemometer (Decagon DS-2, United States) has an accuracy of 0.30 m s−1) or 3% (whichever is larger). The anemometers were mounted above the radiation screens 0.50 m away from the lantern post edges and at heights of 4.30 m (from ground level to center of the anemometer).
Contributed equally to this work.