Atmospheric stability plays an essential role in the evolution of weather events. While the upper troposphere is sampled by satellite sensors, and in situ sensors measure the atmospheric state close to the surface, only sporadic information from radiosondes or aircraft observations is available in the planetary boundary layer. Ground-based remote sensing offers the possibility to continuously and automatically monitor the atmospheric state in the boundary layer. Microwave radiometers (MWR) provide temporally resolved temperature and humidity profiles in the boundary layer and accurate values of integrated water vapor and liquid water path, and the differential absorption lidar (DIAL) measures humidity profiles with high vertical and temporal resolution up to 3000-m height. Both instruments have the potential to complement satellite observations by additional information from the lowest atmospheric layers, particularly under cloudy conditions. This study presents a neural network retrieval for stability indices, integrated water vapor, and liquid water path from simulated satellite- and ground-based measurements based on the COSMO regional reanalysis (COSMO-REA2). Focusing on the temporal resolution, the satellite-based instruments considered in the study are the currently operational Spinning Enhanced Visible and Infrared Imager (SEVIRI) and the future Infrared Sounder (IRS), both in geostationary orbit. Relative to the retrieval based on satellite observations, the additional ground-based MWR/DIAL measurements provide valuable improvements not only in the presence of clouds, which represent a limiting factor for infrared SEVIRI/IRS, but also under clear-sky conditions. The root-mean-square error for convective available potential energy, for instance, is reduced by 24% if IRS observations are complemented by ground-based MWR measurements.
Short-term forecasts of current high-resolution numerical weather prediction (NWP) models still have large deficits in forecasting the exact temporal and spatial location of severe, locally influenced weather such as summertime convective storms, cool season lifted stratus, or ground fog. Often the thermodynamic instability and the availability of moisture, especially in the boundary layer, play an essential role in the evolution of these weather events.
Thermodynamic instability is determined by vertical distribution of temperature and humidity. One way to assess the atmospheric instability is through forecast or stability indices (Peppler 1988). The atmospheric stability indices (STI) are derived from vertical profiles and usually combine humidity and temperature at different pressure levels. Each index has an empirically determined threshold. In case the index exceeds or falls below this threshold, the potential for convection, thunderstorm, or fog, depending on the index, is given. Maps of stability indices provided by NWP models help the operational forecasters identify regions with unstable conditions favorable to the development of deep convection and severe weather. Note that there is no “global” index that can serve as the most reliable predictor of convection for different regions. Most of the STI were developed for specific atmospheric phenomena or for certain geographic regions. Several studies have focused on the evaluation of STI and associated thresholds for specific geographic regions (Kunz 2007; Haklander and Van Delden 2003; Manzato 2003, 2012). The purpose of such studies is to find a STI or a set of STI, along with their thresholds, appropriate for predicting particular weather event in the region of interest.
Traditionally, STI are calculated from radiosonde profiles of temperature and humidity. However, radiosondes are typically only available one–two times per day, which is not frequent enough to capture the temporal and spatial variability of the thermodynamic state of the atmosphere.
Currently the atmospheric stability indices are routinely calculated from geostationary satellite observations (e.g., Koenig and de Coning 2009). The Meteosat Second Generation global instability index product (MSG-GII) is based on SEVIRI measurements and covers Africa and Europe. It includes three instability indices—the lifted index (LI), the Konvektionsindex (KO), and the K index (KI)—as well as the total precipitable water (EUMETSAT 2013). The GII product is produced at the horizontal resolution of approximately 9 km at the subsatellite point. Although the horizontal resolution of the GII products decreases with increasing latitude, the full disk coverage and repeat cycle of 15 min still represent a significant improvement to the sparsely located radiosonde sounding sites.
However, the main limitation of the MSG-GII product is that it relies on infrared observations and is restricted to cloud-free and thus preconvective areas. Moreover, currently operational geostationary instruments have low spectral resolution and therefore provide less information on the vertical structure of the atmosphere, especially of the lowest layers (Schmit et al. 2008). Particularly, the clear-sky radiances of geostationary SEVIRI on board MSG are mainly sensitive to the water vapor in the mid- and upper troposphere (Stengel et al. 2009).
The Infrared Sounder (IRS) on board the future geostationary Meteosat Third Generation (MTG; https://www.eumetsat.int) is expected to provide a more detailed picture of four-dimensional water vapor and temperature structures by means of highly spectrally resolved observations (Wang et al. 2007). The first satellite of the MTG series carrying IRS is scheduled to launch in 2023 and will perform a full disk scan from the 0° nominal longitude.
Accurate retrievals of temperature and humidity profiles from hyperspectral infrared radiance require accurate information about the surface emissivity. Land surface emissivity is highly inhomogeneous in space and time and shows different angular dependency for different land surface types (Li et al. 2013). Although progress has been made in the simultaneous retrieval of surface temperature, surface emissivity, and atmospheric profiles from hyperspectral polar orbiting and from geostationary infrared observations (Yao et al. 2011; Masiello et al. 2018), the surface emissivity still remains a high uncertainty for retrieved atmospheric profiles. Furthermore, since the IRS will measure in the longwave and midwave infrared, optically thick clouds will represent a limiting factor for thermodynamic profiling of the lower troposphere (Zhou et al. 2005).
Several studies focused on the evaluation of STI calculated from hyperspectral infrared and combined infrared and microwave observations from polar-orbiting platforms (Iturbide-Sanchez et al. 2018). Gartzke et al. (2017) showed that there is a poor correlation between the surface-based convective available potential energy (SBCAPE) derived from Atmospheric Infrared Sounder (AIRS) and from radiosonde profiles. It could be shown that the differences in SBCAPE from satellite and radiosonde profiles are primarily explained by the error in the surface-parcel temperature and humidity in the satellite soundings. Atmospheric profiles obtained from infrared and also from combined infrared and microwave satellite observations often show dry and cold bias in the lowest levels, especially over land and under warm, moist conditions (Tobin et al. 2006; Bloch et al. 2019). The replacement of the surface-parcel properties in the satellite soundings with the surface or the radiosonde observations leads to significant improvements in accuracy of SBCAPE (Gartzke et al. 2017; Bloch et al. 2019).
Networks of ground-based instruments that are operated on the basis of 24 h per day/7 days per week have the potential to provide information on thermodynamic conditions below and above clouds as well as close to the surface and thus to complement satellite observations. Particularly, ground-based microwave radiometers (MWR), low-cost and network-suitable instruments, are well established for observing the atmospheric temperature and humidity at high temporal resolution during all weather conditions except during precipitation when water on the radome disturbs the measurement (Rose et al. 2005; Crewell and Löhnert 2007). Most common MWRs measure brightness temperature at selected channels in the 20–60-GHz frequency range, where the atmospheric radiation is less affected by clouds than in infrared. Thus the retrieval of thermodynamic profiles below, within, and above clouds is possible, albeit with lower vertical resolution when compared with hyperspectral infrared observations (Löhnert et al. 2009).
The STI calculated from temperature and humidity profiles measured by MWR were shown to agree well with those computed from radiosonde soundings, with correlation coefficients (CORR) above 0.8 (Cimini et al. 2015). The accuracy of the obtained STI depends on the approach used for retrieval of temperature and humidity profiles from passive microwave observations. A comprehensive overview and evaluation of different retrieval techniques developed in the past can be found in Cimini et al. (2006). The optimal estimation theory allows for the assessment of information content of observations in terms of degree of freedom for signal (DOF): the number of independent pieces of information about atmospheric state variables (e.g., temperature profile) that can be extracted for a given set of measurements (Rodgers 2000). It was shown that 90% of temperature information provided by ground-based MWR originate from the lowest 500 hPa, with the maximum information in the lowest 200 hPa. The maximum of humidity information in terms of DOF comes from the layer between 500 and 800 hPa, while 80% of humidity information is from heights below 500 hPa (Ebell et al. 2013; Löhnert and Maier 2012). This explains the fact that the resolution and accuracy of thermodynamic profiles obtained from ground-based MWR observations degrades with increasing height. Thus the MWR are mainly suited for continuous observations in the boundary layer.
The latest advances in calibration techniques (Küchler et al. 2016) make MWR suitable for long-term, unattended operation within a ground-based network. The potential benefit of assimilating the temperature and humidity profiles retrieved from single and network MWR observations in operational convective-scale NWP models in both clear-sky and cloudy conditions was demonstrated by several studies (Caumont et al. 2016; Martinet et al. 2017; Otkin et al. 2011; Hartung et al. 2011; Cimini et al. 2012). The development of the ground-based version of the fast radiative transfer model RTTOV [De Angelis et al. 2016; RRTOV indicates Radiative Transfer for the Television and Infrared Observation Satellite (TIROS) Operational Vertical Sounder (TOVS)] offers the possibility of direct assimilation of brightness temperatures instead of retrievals, which should lead to more positive impact of MWR observations within assimilation.
The low resolution of MWR humidity profiles can be compensated by collocated observations of ground-based water vapor differential absorption lidar (DIAL) (Spuler et al. 2015). Recently developed water vapor DIALs perform humidity profiling up to 3 km or to cloud base with 10% uncertainty. The capability of these compact, network-suitable instruments for unattended, continuous water vapor profiling was demonstrated during several measurement campaigns (Weckwerth et al. 2016; Roininen 2016).
Considering the advantages and disadvantages of geostationary satellite (e.g., high temporal and horizontal resolution, but low resolution and accuracy in the boundary layer and dependency on surface emissivity and cloud cover) and ground-based remote sensing observations (high resolution in the boundary layer, but sparse spatial coverage) it is expected that combination of both will improve the retrieval of atmospheric state.
In this paper we analyze the potential of geostationary satellite sensors (SEVIRI and IRS), ground-based instruments (MWR and water vapor DIAL) and their synergy for the assessment of atmospheric stability in both clear-sky and cloudy conditions. Clear-sky and cloudy cases were treated separately to demonstrate the strength and weaknesses of each instrument. The theoretical study is performed for a typical midlatitude station and is based on the high-resolution Consortium for Small-Scale Modeling regional reanalysis (COSMO-REA2; Bollmeyer 2015). Reanalysis profiles were used for simulation of ground-based and satellite observations and for calculation of a set of parameters including seven STI, integrated water vapor (IWV), and liquid water path (LWP). Simulated observations along with STI were used to train and test the neural networks for single instruments and instrument combinations. Because of the assumptions made for simulation of satellite observations, the results of this study are only valid for the specified midlatitude site and for atmospheric conditions that are represented by selected reanalysis profiles.
The geostationary MSG and MTG observations over Europe are and will be performed with the basic repeat cycle of 15 and 30 min, respectively, and thus with the currently highest operational resolution (regarding temperature and humidity profiles). Therefore, the SEVIRI and IRS retrievals are considered as the baseline and are compared to the retrievals from ground-based and combined ground- and satellite-based observations. The synergy benefit is defined here as increase in correlation and reduction of retrieval uncertainty achieved through additional ground-based observations compared to the retrieval from satellite observations only.
The set of STI considered in this work includes KI, KO, total totals (TT), LI, Showalter index (SI), most unstable convective available potential energy (CAPE), and fog threat (FT). In addition, IWV and LWP are retrieved as further useful airmass parameters.
This paper is organized as follows. The reanalysis and dataset used in this study are presented in section 2. The satellite- and ground-based sensors along with measurement geometry and channel selection are described in the section 3. Section 4 presents the forward model simulations and the neural network retrieval. Section 5 discusses the results for single instruments and synergy of instruments. Conclusions and perspectives are given in section 6.
2. Reanalysis data
The study was performed using the COSMO-REA2 reanalysis data. The regional reanalysis COSMO-REA2 was developed within the Hans-Ertel Centre for Weather Research (HErZ) and is currently available for seven years from 2007 to 2013 (Bollmeyer et al. 2015). COSMO-REA2 covers nine European countries (Austria, Belgium, Denmark, Germany, Liechtenstein, Luxemburg, the Netherlands, Slovenia, and Switzerland) and parts of the Czech Republic, France, Italy, Poland, and the United Kingdom, with horizontal grid spacing of 2 km and temporal resolution of 1 h. The system is based on the nonhydrostatic limited-area COSMO model of the German Weather Service [COSMOS-DE; see online at http://www.cosmo-model.org/ or Doms and Baldauf (2015) and Doms et al. (2011)]. The setup of COSMO-REA2 follows the operational COSMO-DE version that runs without parameterization of deep moist convection and uses 50 vertical levels with the lowest and highest levels at 10 m and 22 km, respectively (Baldauf et al. 2011). COSMO-REA2 is nested into the COSMO-REA6 reanalysis that covers the European domain with a horizontal grid spacing of 6 km. The evaluation of both reanalyses focusing on precipitation, global horizontal irradiance, temperature, and wind speed showed the superior performance of both reanalyses when compared with the European Centre for Medium-Range Weather Forecasts (ECMWF) interim reanalysis dataset (ERA-Interim) (Wahl et al. 2017; Frank et al. 2018; Bollmeyer 2015; Kaiser-Weiss et al. 2015).
Conventional observations are assimilated using a nudging approach and comprise surface-level observations obtained from synoptic (SYNOP) stations, ships, and drifting buoys, as well as observations from radiosondes, aircraft, and wind profilers. In addition to conventional observations, the radar-derived rain rates are assimilated using latent heat nudging. Furthermore, the snow and sea surface temperature analysis are applied every 6 and 24 h, respectively (Bollmeyer 2015).
This study is based on the COSMO-REA2 output at the Jülich Observatory for Cloud Evolution (JOYCE) in Germany, which is located at 50°30″N, 6°48″E at 111 m MSL (Löhnert et al. 2015). The atmospheric profiles for the time period from May to September from 2007–13 were selected. According to the total amount of ice and water in each profile the whole profile set was divided into clear-sky (9430 profiles) and cloudy (14 294 profiles) datasets, in which the cloudy dataset comprises ice, liquid, and mixed-phase clouds. Both datasets were used for the simulation of ground-based and satellite remote sensing observations and for the calculation of stability indices, IWV, and LWP. Stability indices and corresponding thresholds included in this study are summarized in Table 1. The detailed descriptions of the indices and their thresholds are given in Cimini et al. (2015) and Haklander and Van Delden (2003).
3. Instruments and channel selection
SEVIRI is the main instrument on board four MSG satellites. The last satellite of the MSG series, Meteosat-11, has been operational since 2018 at a position of 0° longitude providing a view on Europe and Africa. SEVIRI observes the full disc of Earth with a repeat cycle of 15 min in 12 spectral channels. The horizontal resolution of the eight thermal IR, two visible, and one near-infrared channels is 3 km × 3 km at nadir enlarging to 4 km × 6 km at the midlatitudes. The broadband high-resolution visible channel covers one-half of the full disk with a 1-km spatial sampling distance at nadir (Schmetz et al. 2002).
The high temporal resolution of SEVIRI observations allows, among other things, the continuous monitoring of weather patterns and rapidly changing phenomena such as deep convection, fog occurrence, and fog dissipation. However, the main operational limitation in terms of vertical atmospheric profiling is the broad band, low spectral resolution [as compared with the instruments on board polar-orbiting satellites; e.g., Infrared Atmospheric Sounding Interferometer (IASI)], and the strong sensitivity to clouds. Thus, no information on stability is available for cloudy pixels (Koenig and de Coning 2009).
In this study, seven IR channels were used in the statistical retrievals: two water vapor channels (WV6.2 and WV7.3) sensitive to the water vapor distribution in the middle and upper troposphere, three window channels (IR8.7, IR10.8, IR12.0) providing information on surface and cloud-top temperature, and IR9.7 and IR13.4 channels, which are located at the O3 and CO2 absorption bands, respectively, and provide information on atmospheric air mass and temperature. The SEVIRI noise is simulated by normal distribution with a standard deviation (STD) equal to the noise equivalent differential temperatures between 0.1 and 0.37 K for the considered channels (Schmetz et al. 2002).
The next generation of Meteosat satellites, MTG, will comprise four imaging (MTG-I) and two sounding (MTG-S) satellites. The latter will replace Meteosat-11 at 0° longitude and bring an operational hyperspectral instrument into geostationary orbit. The IRS is a sounding Fourier transform spectrometer that will perform highly spectrally resolved measurements of Earth-emitted radiation in 1738 channels. According to the MTG mission requirement document (EUMETSAT 2018), IRS will perform observations in two bands, in the longwave infrared (LWIR: 700–1210 cm−1) and in the midinfrared (MWIR: 1600–2175 cm−1) band with a spectral resolution of 0.625 cm−1 and a spatial sampling distance of 4 km at nadir. The basic repeat cycle of IRS will take 60 min with increased frequency of 30 min over Europe.
The channels in the LWIR band are mostly sensitive to surface and cloud properties, atmospheric temperature, and ozone, whereas MWIR channels provide information on humidity and temperature. The main objective of the IRS mission is the monitoring of the evolution of vertically resolved water vapor, temperature, and wind structures. Thus the IRS data will be particularly important for nowcasting and short-term forecast of advection and convergence of low-level moisture, which is often accompanied by severe storm development.
Since a detailed channel selection is beyond the scope of this study, a subset of IRS channels that give information on atmospheric temperature and humidity was selected. The subset of 1113 total channels consists of the following: 130 channels along the longwave CO2 absorption band between 700 and 780 cm−1, every second channel between 780 and 1210 cm−1 (344 in total), and 639 channels in the water vapor absorption band between 1600 and 2000 cm−1. The simulated spectra were perturbed with normally distributed noise, which varies between 0.2 and 0.9 K (EUMETSAT 2018). Figure 1 shows a typical IRS brightness temperature spectrum for a midlatitude standard atmosphere with corresponding radiometric noise in terms of noise equivalent differential temperature at a scene temperature of 280 K. Channels used in this study are shown in red.
Most retrieval methods including neural networks are often unable to deal with the high number of observations provided by hyperspectral instruments (Aires et al. 2002). Different methods such as Jacobian-based channel selection (Rabier et al. 2002; Collard 2007) or compression using principal component analysis (PCA) were developed to deal with this problem (Aires et al. 2016). In addition to data compression, the PCA is often used to improve the signal-to-noise ratio and to suppress the random uncorrelated noise in the reconstructed radiances (Antonelli et al. 2004; Huang and Antonelli 2001). In this study, we apply PCA to reduce the dimensionality of dataset and to optimally extract atmospheric profile information from simulated observations. PCA makes use of redundant information in hyperspectral observations and transforms highly correlated observations to an uncorrelated set of principal components. The first principal components represent the most dominant atmospheric signal contained in the original spectrum, whereas the last principal components consist mostly of random instrument noise and can be discarded. Different ways to determine the optimal number of PCs are discussed by Turner et al. (2006). We calculated the factor indicator function (IND) and the percent cumulative variance and found the first 15 principal components to explain more than 99% of the variance in the dataset and to lead to the minimum of the IND function.
The humidity and temperature profiler (HATPRO) is a passive multifrequency MWR measuring downwelling radiation emitted by the atmospheric components, mainly oxygen, water vapor, and cloud liquid, in two bands (Rose et al. 2005). HATPRO offers a high-speed simultaneous detection at 14 channels by utilizing two receivers. The seven K-band channels (22.24, 23.04, 23.84, 25.44, 26.24, 27.84, and 31.40 GHz) are located at the right slope of the pressure-broadened water vapor absorption line and are used to derive low-resolution humidity profiles and very accurate values of integrated water vapor and liquid water path (Löhnert and Crewell 2003). The seven channels of the V band (51.26, 52.28, 53.86, 54.94, 56.66, 57.30, and 58 GHz) are located within the oxygen absorption band and contain information about the vertical profile of temperature in the lower and middle troposphere. Assuming a horizontally homogeneous atmosphere, elevation scanning enhances the temperature profiling accuracy, especially in the case of a low-level temperature inversion, but does not improve humidity profiling. The accuracy of the retrieved temperature profiles is between 0.5 and 2 K close to the surface and in the lower troposphere, respectively, whereas humidity profile accuracies are in the range of 0.8 g m−3 for the midlatitudes (Crewell and Löhnert 2007) and increase up to 1.6 g m−3 in more humid environments (Löhnert et al. 2009; Zhang et al. 2018).
The HATPRO measurement vector in this work consists of 30 brightness temperatures: zenith observations at 14 frequencies and additional nonzenith measurements at the 4 most opaque frequencies in the V band. To take into account the typical radiometric noise and calibration uncertainties of real HATPRO measurements, the normally distributed random errors in the range of 0.2–0.5 K for zenith and of 0.2 K for scanning observations were added to simulated brightness temperatures (Löhnert et al. 2009).
DIAL is an active remote sensing technique providing number density profiles of trace gases, and in our case, water vapor. A typical DIAL system alternately emits two laser pulses and receives the attenuated backscattered signal. The wavelength of the first pulse, the so-called online wavelength, is centered on a water vapor absorption line. The second, the offline wavelength, is positioned close to the first one but outside the influence of absorption line so that the difference between returned signals originates only from the absorption of molecule of interest. The wavelength commonly used are between 700 and 950 nm. The principal disadvantage of this technique is that DIAL systems place high demands on the properties of laser transmitter, detector system, and data-acquisition system, making DIAL instruments large and expensive devices to develop and operate (Wulfmeyer and Walther 2001). On the other hand, the DIAL transmitter uses a ratio of the returned signals. Thus, the demanding calibration, which is needed for most other humidity sensors (e.g., Raman lidar), can be avoided, and the direct retrieval of water vapor density is possible.
In recent years, progress has been made in developing relatively small, low-cost, and network-suitable diode-laser-based DIAL systems (Spuler et al. 2015; Roininen 2016). These systems are capable of continuous, automated water vapor profiling during day and night, as well as in cloudy conditions.
Depending on the measurement setup and the telescope design of different DIAL systems, observations in the range of 0–3000 m (Roininen et al. 2017) or from 75 to 300 m up to 6000 m above ground level (Weckwerth et al. 2016) are possible. As for the infrared satellite sensors, the DIAL observations are hindered by optically thick liquid clouds, and derived profiles extend up to the cloud base. The accuracy and the maximum range of achieved water vapor profiles are different for day and nighttime and clear and cloudy conditions. The mean errors of less than 10% relative to radiosondes can be achieved for the lowest 2–3 km (Weckwerth et al. 2016). The vertical and temporal resolution as well as the lowest range are adjustable to specific user needs.
To create the hypothetical DIAL measurement vector, the COSMO-REA2 mixing ratio profiles up to the height of 2000 m under clear sky, or up to the cloud base under cloudy conditions, were perturbed with 10% mean error. The vertical resolution of assumed profiles varies from 90 to 230 m. Higher vertical resolution of 75–100 m is possible for DIAL profiles but is not available from reanalysis.
4. Forward simulation of microwave ground-based and infrared satellite observations
For the simulation of ground-based and satellite observations, radiation transfer models were applied. The simulations of satellite observations were performed with the widely used fast radiative transfer model RTTOV, version 12 (Saunders et al. 2018). Continuously developed and extensively validated since the 1990s, the RTTOV model now allows simulations of observations for around 90 sensors, including retired and future instruments, measuring in the IR, MW, and visible parts of the spectrum.
For the simulation of clear-sky infrared observations, atmospheric profiles of temperature, humidity, and trace gases (CO2, O3, N2O, CH4, SO2, CO) along with surface properties are required. In this study, the profiles of trace gases are set to the RTTOV reference profiles and assumed to be constant. The surface emissivity values for satellite instruments were taken from the RTTOV University of Wisconsin Global Infrared Land Surface Emissivity (UWIREMIS) atlas, which provides monthly climatological mean emissivity values (Borbas and Ruston 2010).
For the simulation of cloudy observations, additional profiles of cloud liquid/ice water and cloud fraction are required. For water clouds, five different cloud types are considered. For ice clouds, we used the Baran ice scheme (Vidot et al. 2015), which allows a direct parameterization of ice optical properties from the ambient temperature and the ice water content.
The input profiles are interpolated within the RTTOV simulations from the 71 original levels to the 54 and 101 RTTOV levels for SEVIRI and IRS, respectively, spanning from 0.005 hPa to the surface. Further, RTTOV applies regression coefficients derived from the line-by-line model simulations to a set of profile parameters and calculates level-to-space transmittances from which the brightness temperatures are estimated. The parameterization of transmittances makes the RTTOV model more computationally efficient and does not increase the errors that are introduced by the line-by-line model simulations (Matricardi et al. 2004). The regression coefficients utilized by RTTOV, version 12, has been derived from calculations with Line-by-Line Radiative Transfer Model (LBLRTM, version 12.2; Clough et al. 2005), which uses the Mlawer–Tobin–Clough–Kneizys–Davies (MT-CKD, version 2.5.2; Mlawer et al. 2012) continuum spectra for IR optical depth calculations.
The satellite zenith angle is assumed to be the same for both instruments on board geostationary satellites (MTG-IRS and MSG-SEVIRI). Further, the atmosphere is assumed to be horizontally homogeneous and aerosol-free, and wavelength dependence of diffraction is ignored so that both instruments sample the same volume of air at all channels.
Note that the simulated spectra may deviate from the real spectra because of the forward model error that results from representation of the atmospheric column state by only 54 and 101 layers, respectively, and due to the assumption of constant trace gas profiles.
The ground-based MWR observations were simulated using RTTOV-gb (De Angelis et al. 2016). RTTOV-gb is based on the original RTTOV (version 11), which was adapted to handle ground-based MWR observations. The regression coefficients of RTTOV-gb have been calculated with the gas absorption model described by Rosenkranz (1998) and the cloud liquid water model described in Liebe et al. (1993). RTTOV-gb uses 101 pressure levels from 0.005 to 1050 hPa, which are selected for its ground-based perspective and are denser near the ground than those used by RTTOV itself. The validation of the RTTOV-gb model showed that the root-mean-square errors of RTTOV-gb brightness temperatures compared to the temperatures simulated with the reference line-by-line model are below the typical uncertainty of ground-based MWRs (~0.5 K; Rose et al. 2005; De Angelis et al. 2016). This is valid for all 14 frequencies and elevation angles from 10° to 90°.
To produce a realistic observation dataset, radiometric noise was added to the simulated brightness temperatures according to the instrument specifications. The simulated errors are assumed to be normally distributed with the appropriate standard deviation and not correlated between channels. Furthermore, PCA was applied to the obtained IRS spectra in order to reduce the dimensionality of IRS dataset as described in section 3b.
5. Statistical retrieval using neural networks
A neural network (NN) approach has been widely used for retrieval of atmospheric profiles and cloud properties from satellite (Aires et al. 2002) and ground-based microwave observations (Marke et al. 2016; Cadeddu et al. 2009; Jacob et al. 2019). The task of the neural network is to find a relationship between a set of input (simulated observations) and output (calculated STI) vector pairs. It was demonstrated that a neural network with only one hidden layer of sufficient number of nodes and a nonlinear activation function is able to reproduce any nonlinear statistical relationship (Hornik et al. 1989). In this study, for each instrument and combination of instruments one two-layer feed-forward backpropagation network per STI was trained, validated and applied to the independent set of input parameters (test set). The size of the input layer of the networks is determined by the number of channels of the instrument and consists of 30 nodes for MWR, 15 nodes for IRS, 7 nodes for SEVIRI, and 10 additional nodes for DIAL. The output consists of only one neuron (one STI). Each node of the hidden layer is connected to the nodes of the input through a sigmoid activation function of the form
where a1 is the output of the hidden layer, p is the input vector, 1 is the matrix of input weights, and b1 is called bias vector. The connection between the hidden layer and output layer is linear. The optimal values of weights and biases were achieved by minimizing a mean-square error in the network outputs. The backpropagation of the errors was performed according to Levenberg–Marquard algorithm in the MATLAB software neural network toolbox (Hagan and Menhaj 1994).
The whole dataset, consisting of brightness temperatures for satellite sensors and MWR or of humidity profiles for DIAL, was sorted in descending order by the value of corresponding STI and divided into a training (50% of cases), a validation (20% of cases), and a test set (30% of cases). The division was performed using interleaved indices to guarantee that all sets have similar statistical properties and contain adequate representation of rare events (high or low STI values).
A good generalization of the networks is achieved by the early stopping technique. During the iterative training process, the network is applied to the validation set and the error on the validation set is monitored. When the validation error increases for several iteration steps, the training is stopped and the network properties at the minimum of the validation error are returned, stored, and applied to the test dataset. The resulting network offers a trade-off between learning (i.g. small error on the training dataset) and generalization (i.e., smallest possible error on the validation set). The evaluation of the retrieval performance was carried out using only the independent test dataset.
The number of neurons in the hidden layer is different for each instrument. To find the optimal size of the hidden layer, each network was trained repeatedly with number of neurons varying between 5 and 25 with a step of 5 neurons. For too few neurons, the complexity of network was not sufficient to represent the training dataset, resulting in the large training error. On the other hand, with increasing size of the hidden layer the generalization error and the time needed for training increases. The criterion for the optimal network configuration were the small final error on the training set and the small generalization error. At the end of the calculations, networks with 15–25 neurons in the hidden layer were found to be a good compromise.
This section presents results of a comparison between STI calculated from reanalysis profiles (“truth”) and STI as retrieved from simulated measurements. First, statistical scores, such as the bias, STD, and root-mean-square error (RMSE) of the difference between the truth and retrieved STI values were calculated. Then, taking into account the threshold values of STI, the performance of the retrieval can be verified by considering the contingency table (Table 2). In the case of event/nonevent forecasts, the four entries of the contingency table are the number of correct event forecasts (hits h), correct nonevent forecasts (zeros z), false alarms f, and not-predicted events (misses m). A perfect forecast system would produce only hits and zeros, with no misses or false alarms. Observed values correspond to STI calculated from reanalysis profiles; predicted values correspond to retrieved STIs. On the basis of these four outcomes different verification parameters can be derived. Here we show three of them, the probability of detection (POD), the false alarm ratio (FAR), and the Heidke skill score (HSS), as defined in Table 2.
The POD gives the percentage of all events that could be forecast. This parameter is sensitive to hits (correct predicted events) and misses (not-predicted events) but ignores false alarms. So, in the case in which the event is forecast too often (that means no misses but a great number of false alarms), the POD will be 100%. The FAR reveals the false predicted events among all predictions.
Both scores are highly dependent on the ratio of events and nonevents. Therefore, we use the HSS as a further verification parameter. The HSS includes all elements of the contingency table and is considered to be an appropriate score in the case of forecasting of rare events, when correct forecasts of nonevents dominate the contingency table. Moreover, statistical skill scores like HSS make it possible to compare results on the basis of different datasets such as clear-sky and cloudy cases in this study (Doswell et al. 1990). HSS measures the relative forecasting skill, giving the accuracy of the forecast relative to that of random chance. The range of HSS is between −1 and 1, with negative values indicating that a forecast is worse than a randomly generated forecast. A value of 0 means no forecast skill, and a perfect forecast results in an HSS of 1.
a. Single-instrument performance under clear-sky and cloudy conditions
First, we investigate the ability of every single instrument (SEVIRI, MWR, and IRS) and of a combination of ground-based instruments (MWR + DIAL) to provide STI under clear-sky and cloudy conditions. Table 3 shows the statistics of the difference between STI calculated from reanalyses and NN retrievals for clear-sky conditions. Since the bias is relatively small for all indices and instruments, the root-mean-square error values are close to that of standard deviation and are not shown. In addition to the STI discussed above both tables show the statistics for IWV and LWP.
Among all instruments, the SEVIRI provides lowest CORR and highest STD values for all indices and both clear-sky and cloudy conditions (Tables 3 and 4). Under clear-sky conditions, the future IRS outperforms the SEVIRI instrument in terms of CORR with an improvement varying between 8% for FT and 54% for CAPE. As both instruments were assumed to have the same observation geometry, the improvements can be clearly attributed to the increased information content of highly spectrally resolved IRS measurements.
For both geostationary instruments, the lowest correlation values under clear-sky conditions were achieved for CAPE, which is dependent on the entire temperature profile and highly sensitive to temperature and humidity below the lifted condensation level, and for FT index, which is strongly dependent on the humidity gradient and on the near-surface temperature and dewpoint. This indicates the insufficient capability of infrared satellite instruments to sample the lowest atmospheric layers. However, in the case of the IRS instrument, improvements could be made by a more precise selection of channels sensitive to surface temperature and humidity. Moreover, better results for IRS can be expected for lower latitudes due to smaller zenith angle.
Relative to IRS, the ground-based MWR provides slightly lower CORR values (and higher STD values, respectively) for four of the STI (KI, TT, LI, and SI) and higher CORR for the remaining three STI (KO, CAPE, and FT) and IWV. Note that ground-based MWR observations are most beneficial for CAPE, achieving 13% higher CORR values than IRS and pointing out better performance of MWR in the boundary layer.
Additional humidity information from DIAL leads to even higher CORR values, making the combination of both ground-based instruments comparable with IRS for four of the STI (KI, TT, LI, and SI) and IWV and considerably better for three STI (KO, CAPE, and FT).
The categorical parameters POD, FAR, and HSS calculated for seven STI and clear-sky conditions are shown in Fig. 2. As with the CORR, the hyperspectral observations of IRS lead to significant improvements in the statistics compared to SEVIRI. The measurement skill of IRS in terms of HSS ranges between 0.62 and 0.73 for the first five STI. For CAPE and FT, the measurement skill achieved by IRS is only 0.50 and 0.32, respectively. The MWR provides the values of POD, FAR, and HSS comparable to those achieved by IRS for three of the STI: KI, KO, and LI. For TT and SI, the lower values of POD and higher number of false alarms retrieved by MWR result in lower HSS relative to IRS retrieval. The higher HSS values achieved by MWR for FT and CAPE in comparison with IRS (0.56 and 0.51, respectively) are notable.
If the DIAL humidity profile is added to the MWR observations, the statistics are further improved for all indices with the exception of TT. The FT index in particular benefits from additional humidity information in the lowest atmospheric layers. Altogether, in terms of POD and HSS, the combination of both ground-based instruments outperforms the IRS for five STI (KI, KO, LI, CAPE, and FT).
Table 4 and Fig. 3 show the statistics for single instruments and the combination of ground-based instruments for cloudy conditions. Again, there are apparent differences in CORR between SEVIRI and IRS, as well as the decrease of CORR of up to 30% for both satellite instruments relative to results for clear sky. The reason for this decrease lies in the saturation of infrared channels in the presence of optically thick clouds.
In contrast, the CORR values achieved by ground-based MWR and MWR + DIAL remain almost the same, with changes within 6% relative to the clear-sky retrieval, revealing an advantage of ground-based microwave over satellite infrared observations under cloudy conditions. The main advantage of the geostationary satellite observations is still the spatial coverage.
Under cloudy conditions the ground-based MWR outperforms the IRS for all STI, IWV, and LWP, and shows significantly higher CORR, with the improvement between 10% for KI, 30% for CAPE, and 90% for LWP. However, despite an almost unchanged CORR, the bias and STD for CAPE calculated from MWR observations increase for the cloudy retrieval.
The ground-based MWR provides the best results under all sky conditions for the integrated atmospheric parameters IWV and LWP. The high CORR values for the combination MWR + DIAL result mostly from information contained in the microwave observations. Neural network retrieval applied to single DIAL observations leads to CORR values of 0.37 for LWP and of 0.91 and 0.88 for IWV under clear-sky and cloudy conditions, respectively (not shown). Despite the worsening due to clouds, the IRS achieves high CORR value of 0.9 for IWV. On the other hand, a CORR of only 0.52 is achieved by IRS for LWP. It is important to note that in the case of SEVIRI, only observations at infrared channels were used for LWP retrieval in this study. The visible (0.6 μm) and near-infrared (1.6 μm) SEVIRI channels offer the possibility to retrieve cloud particle effective radius and cloud optical thickness, which can be used for LWP calculation (Stengel et al. 2014).
The POD, FAR, and HSS values calculated for cloudy cases clearly illustrate the advantage of ground-based microwave and combined MWR + DIAL observations in the presence of clouds (Fig. 3). Similar to CORR, the POD, FAR, and HSS values for ground-based instruments remain almost the same as for clear-sky conditions, whereas the POD and HSS for satellite sensors decrease significantly for six STI. Therefore, the MWR-only retrieval and the MWR + DIAL retrieval outperform that of satellite instruments for all STI and integrated values IWV and LWP.
b. Benefit from synergy of satellite- and ground-based sensors
The potential of ground-based instruments to complement highly temporally resolved geostationary observations was investigated. Tables 5 and 6 show the statistics between STI calculated from reanalysis and STI retrieved from synergistic observations for clear-sky and cloudy conditions, respectively.
First, we compare the statistics for single SEVIRI instrument and combinations SEV + MWR and SEV + MWR + DIAL. For both clear-sky and cloudy conditions, additional microwave ground-based observations improve the statistics significantly. However, taking into account high CORR values of single MWR retrieval, we conclude that in case of SEV + MWR and SEV + MWR + DIAL retrievals, the improvements are mostly due to information contained in MWR and MWR + DIAL observations, respectively.
From a ground-based point of view, additional SEVIRI observations only slightly improve MWR-only and MWR + DIAL retrievals for all STI, except for FT under both clear-sky and cloudy conditions. The categorical parameters POD, FAR, and HSS show also very slight improvements for SEV + MWR and SEV + MWR + DIAL combinations relative to MWR-only and MWR + DIAL retrievals (cf. Figs. 2 and 4 and Figs. 3 and 5).
A closer look at the CORR values achieved by MWR-only, MWR + DIAL, and SEV + MWR combinations shows that the contribution of SEVIRI in the synergistic retrieval is smaller than (KO, CAPE, and FT) or comparable to (KI, TT, SI, and LI) that of DIAL under both clear-sky and cloudy conditions.
Further, the comparison of results from IRS-only and MWR-only retrieval with those from combined IRS + MWR observations shows that under clear-sky conditions both instruments complement each other, making the combination of IRS + MWR more efficient than each sensor alone. The CORR for all STI and IWV increases by values between 2% and 23% relative to the IRS-only retrieval. At the same time, the STD decreases by 6%–37% for STI and by 88% (from 1.25 to 0.14 kg m−2) for IWV.
In the presence of clouds, ground-based MWR added to IRS observations significantly improve the statistics relative to IRS-only retrieval with increments in CORR between 10% and 40% for STI and IWV and about 90% for LWP. The CORR values for all indices except for CAPE and FT exceed 0.9. The STD values decrease by 20%–50% for STI and by 90% for integrated values IWV and LWP.
From the ground-based point of view, the additional information from the higher atmospheric levels contained in the IRS observations also leads to a slight increase in CORR values relative to the MWR-only retrieval. This is valid for both clear-sky and cloudy conditions and all indices except for FT and integrated values IWV and LWP.
The combination of IRS + MWR shows slightly better results than MWR + DIAL for all STI except for CAPE under cloudy conditions and FT under clear-sky conditions and in the presence of clouds. For these two indices, the DIAL is able to provide more valuable information on humidity close to the ground, which is needed for calculation. Thus adding the DIAL humidity profiles to IRS + MWR leads to higher CORR values for all STI, with maximum improvements for CAPE and FT under cloudy conditions.
For all indices and all skies, the synergy of hyperspectral IRS with one (MWR) or two (MWR + DIAL) ground-based instruments leads to higher POD and HSS and lower FAR values than corresponding combinations with SEVIRI (Figs. 4 and 5). The best results for the first five indices are provided by the combination IRS + MWR + DIAL, with the HSS lying around 0.8 and corresponding POD in the range between 0.8 and 0.87 under clear-sky conditions. The false alarms appear in 10%–15% of predicted cases. In the presence of clouds, the HSS achieved by this instrument combination varies between 0.68 and 0.81 with corresponding POD between 0.78 and 0.9 and false alarms in 9%–20% of cases.
However, in the case of the FT index, the combinations IRS + MWR and SEV + MWR achieve CORR, POD, FAR, and HSS values equal to that of the MWR-only retrieval. The same is valid for combinations IRS + MWR + DIAL and SEV + MWR + DIAL compared to the MWR + DIAL retrieval. These results show that for the assessment of potential for radiation fog, the broadband and hyperspectral infrared satellite observations cannot add complementary information in a synergistic retrieval and thus can neither replace nor complement the ground-based observation, even under clear-sky conditions.
c. Time series of STI
The developed NN retrieval was applied to a time series of brightness temperatures simulated for single instruments and synergy of instruments. Figure 6 shows the time series of KO for August 2012. The KO describes the potential instability between lower and higher atmospheric levels by comparing the equivalent potential temperatures at low (1000 and 850 hPa) and mid- (700 and 500 hPa) levels. Designed for estimating thunderstorm potential in Europe (Andersson et al. 1989), KO is smallest if cold, dry air lies above warm and humid air. Values below 1.9 K indicate strong thunderstorm potential. The KO values computed from reanalysis show alternating stable and unstable periods, with very unstable conditions in the time from 18 to 22 August. The unstable periods are mostly accompanied by clouds. Thus the observations under cloudy conditions are crucial for assessment of atmospheric stability.
In general, the KO values retrieved from observations of MWR, IRS, and combinations IRS + DIAL and MWR + DIAL follow the trend given by reanalysis. However, the lowest KO values during unstable periods on 2, 5, 9, 14, and 21 August could not be captured well by all instruments. The IRS-only and MWR-only retrievals provide CORR values of 0.76 and 0.87, respectively, for the whole period, whereas both combinations IRS + MWR and MWR + DIAL achieve CORR values of 0.93.
The time series of CAPE for the unstable period from 18 to 20 August and corresponding satellite images are shown in Figs. 7 and 8. The period starts with stable, cloud-free conditions where the trend is best captured by the IRS + MWR combination. The instability increases starting from 1100 UTC 19 August and leads to cloud formation. In the presence of clouds, the CAPE values retrieved from IRS observations differ significantly from the reanalysis. The ground-based MWR as well as the MWR + DIAL combination underestimates the highest CAPE values in the night of 19–20 August in cloudy as well as in clear-sky cases. The CAPE calculated from synergistic IRS + MWR observations follows the trend well and captures the maximum CAPE values in the night of 19–20 August and in the morning of 20 August.
7. Summary and conclusions
Stability indices are a useful tool for assessment of local instability of the atmosphere. The temporal and spatial resolution of radiosonde soundings, which are traditionally used for calculation of STI, is not sufficient to capture the initiation and the development of convection. Ground-based remote sensing instruments and instruments on board geostationary satellites provide temporally resolved information on vertical structure of the atmosphere and can be used for monitoring of atmospheric stability.
On the basis of regional reanalysis and using the neural network approach, this study demonstrates the potential of ground-based microwave radiometer and water vapor differential absorption lidar and geostationary satellites, as well as of their synergy, to provide STI, integrated water vapor, and liquid water path.
In agreement with expectations, our results show that the hyperspectral geostationary IRS observations contain significantly more information on vertical humidity and temperature than broadband SEVIRI measurements and can improve the monitoring of atmospheric stability especially in the clear-sky, preconvective environment. The correlation between retrieved STI and so-called truth increased by up to 54% for all STI, IWV, and LWP. However, lack of information from the boundary layer in the IRS observations results in the lower accuracy of retrieved CAPE and FT indices, which are highly dependent on the near-surface temperature and humidity. Consistent with previous studies (Gartzke et al. 2017; Bloch et al. 2019), additional ground-based observations lead to significant improvements in the retrieval of these two indices. Moreover, since the MWR and DIAL observations provide information from the entire boundary layer, the indices that are not directly dependent on surface properties (KI, KO, TT, and SI) also benefit from additional ground-based observations. Thus, under clear-sky conditions, satellite and ground-based observations complement each other in an optimal way and provide more accurate retrievals of STI than each individual sensor.
In the presence of clouds, the IR channels of both satellite sensors IRS and SEVIRI become saturated and provide only limited profile information. In contrast, the impact of clouds on the results from ground-based instruments is negligible. Thus, under cloudy conditions, the synergy benefit is dominated by MWR and DIAL, highlighting the potential and necessity of ground-based observations.
In summary, our results supplement the outcomes of the previous study by Ebell et al. (2013), who demonstrated the synergy benefit in terms of degree of freedom within a physical retrieval. Moreover, it can be expected that constraining the observations with a priori information within a physical retrieval would lead to more accurate results (Cimini et al. 2015). However, the neural network approach is reliable and in some cases is preferable to computationally demanding physical retrievals. The learning algorithm (the most expensive computational part) is performed offline only once. Further application of neural network provides real-time retrievals: no information on initial conditions, no forward model simulations, and no Jacobian computations are required. If calculated with statistical methods from satellite observations and hence independent of NWP models, the stability indices and integrated water vapor provide forecasters with additional information on location of unstable air masses and convection potential, which may differ from forecasts.
It is important to note that, as for each statistical retrieval, the results are entirely dependent on the complexity and the range of dataset used for the training. Since the satellite measurements are highly dependent on the viewing angle, the results are only valid for a specified station (JOYCE, or Jülich). To make the retrieval applicable at other geographical positions, a correction of all simulated radiances to one particular viewing angle and also consideration of different altitudes would be necessary. In addition, it is important to ensure that the dataset used for neural network training covers all kinds of possible atmospheric situations, including rare events.
Moreover, our results are sensitive to assumptions made concerning measurement errors. In the case of MWR, the error added to simulated brightness temperatures includes measurement and calibration so that only assumption of horizontal homogeneity is not fulfilled. On the other hand, in the case of satellite instruments, only instrument error (or theoretical error requirements for IRS) was added to the simulated measurements. In this theoretical study we also did not account for diffraction issues and horizontal inhomogeneity of the atmosphere. Further, surface emissivity and vertical distribution of trace gases used for simulation of satellite measurements are more variable in reality and would introduce more uncertainty into the retrieval. For satellite instruments, our study shows, rather, the best possible results, which would most likely not be achieved by real observations.
In the future, we plan to extend the algorithm for retrieval of temperature and humidity profiles and compare the results for MWR with the physical retrieval. The application to real observations of MWR-HATPRO collecting measurements at the JOYCE station is possible.
Although the ground-based observations and their synergy with future satellite observations were shown to be very beneficial for assessment of instability, IWV, and LWP, the horizontal resolution of currently available instrument networks is not sufficient and much lower than that of geostationary instruments. Further analysis of spatial representativeness of single ground-based observation is essential and will be performed to determine the optimal density of future network of ground-based instruments.
This study was carried out within the ARON-project (A Virtual Remote Sensing Observation Network for continuous, near-real-time monitoring of atmospheric instability) and funded by the Extramurale Research Programme of the German Weather Service under Grant 2015EMF-09. Responsibility for the content of the publication lies with the authors. The authors gratefully thank Domenico Cimini and Francesco De Angelis for providing the radiative transfer model RTTOV-gb and support in its application. Christopher W. Frank is thanked for his help in preparing the reanalysis data. We are also grateful to Susanne Crewell, Volker Lehmann, Christine Knist, and Annika Schomburg for their support and helpful discussions.
Data availability statement: The whole COSMO-REA2 dataset is archived in the ECMWF file storage system (ECFS). Data for selected sites are available at the Institute for Geophysics and Meteorology of the University of Cologne.
Denotes content that is immediately available upon publication as open access.