Two comparative studies have been performed to evaluate the accuracy of Chinese Aircraft Meteorological Data Relay (AMDAR) weather reports. The comparison between AMDAR reports and radiosonde observations shows that the root-mean-square differences (RMSDs) in temperature, wind speed, and wind direction are 1.06°C, 1.95 m s−1, and 22°, respectively, within a spatial range of ≤20 km and a temporal window of ≤15 min. The comparison between AMDAR reports collected by different aircraft reveals that observation uncertainties in temperature, wind speed, and wind direction are 0.59°C, 0.90 m s−1, and 12°, respectively. The spatial and temporal representativeness as well as the environmental factors that may affect the evaluation results are also discussed in detail in the two comparative studies. The results of the present study provide valuable information on and high confidence in the application of Chinese AMDAR in numerical weather prediction models.
The purpose of Aircraft Meteorological Data Relay (AMDAR) is to use automatic meteorological instruments installed on modern aircraft and associated sophisticated data acquisition and processing systems to acquire real-time upper-air meteorological data. Through the Aircraft Communications, Addressing, and Reporting System (ACARS) and other communication systems, these meteorological observations are sent to the ground (Moninger et al. 2003). The World Meteorological Organization (WMO) AMDAR Panel established a cooperation program for sharing AMDAR data since 1998 (WMO 2003, 2017). In the past few years, AMDAR temperature and wind reports have already been widely used in numerical weather predictions (NWP) and have been proved to have significant positive effects. Their contributions to short-range regional NWP are especially significant in areas with dense AMDAR report coverage (Cardinali et al. 2003; Petersen 2016). James and Benjamin (2017) showed that aircraft observations have the strongest impact on short-range forecasts, and AMDAR has become one of the most important components of the North American observing system.
Following the rapid expansion of AMDAR observations, the quality of these data is worth more attention and further analysis. Schwartz and Benjamin (1995) collected AMDAR reports and radiosonde observations around Denver airport, and compared temperature and wind observations between these two datasets. They found that the standard deviation was 0.59 K for temperature and 2.84 m s−1 for wind speed within a spatial range of ≤25 km and a temporal window of ≤15 min. Ballish and Kumar (2008) found an overall positive bias in aircraft temperature observations when compared to radiosonde observations. Meanwhile, systematic differences in temperature observations exist between different aircraft types (Drüe et al. 2008). Zhu et al. (2015) has indicated that the aircraft ascent/descent rate can be used to correct biases in temperature observations. Benjamin et al. (1999) performed a study of AMDAR observations by collocating the reports from different aircraft with a small separation in time (≤10 min) and space (≤10 km). They found an observation uncertainty of 1.1 m s−1 for a single horizontal component of wind and 0.5 K for temperature above the boundary layer.
The Chinese AMDAR program started in 2003 and developed rapidly following the rapid growth of the Chinese aviation industry. In a previous study (Ding et al. 2015), the Chinese AMDAR reports were compared with radiosonde observations collected at 10 sounding stations. The results confirmed the reliability of the Chinese AMDAR reports. However, despite the encouraging results, there are still some shortcomings that need to be improved. First, the estimated root-mean-square differences (RMSDs) contain both AMDAR observation uncertainty and radiosonde observation uncertainty, which makes it hard to distinguish the uncertainties in AMDAR and radiosonde observations. Second, the results may not be representative enough, since the samples used for comparison are unevenly distributed over space (only observations collected at 10 radiosonde stations were used). Third, the spatial and temporal windows used are relatively large, and thus the representativeness uncertainty cannot be ignored.
Two methods are used in the present study to evaluate the accuracy of Chinese AMDAR. One method is to make comparisons between AMDAR reports and radiosonde observations collected at all the sounding stations in China and to objectively evaluate the quantity of Chinese AMDAR. The other method is adopted from Benjamin et al. (1999). AMDAR reports collected by different aircraft are compared to quantitatively reveal the observation uncertainty of Chinese AMDAR. In addition, spatial and temporal representativeness as well as some other environmental factors that may affect the results are also considered in the present study.
2. Data description and experimental design
a. Chinese AMDAR reports
The original AMDAR reports in China are collected by various airline companies, and then transmitted to National Meteorological Information Center (NMIC) of China Meteorological Administration (CMA). The proportion of the reports shared on the WMO Global Telecommunications System for each airline company in China is different (Table 1). Since the observation instruments and calibration methods vary with the aircraft type and airline companies, an error analysis and bias correction should be done (Liao and Xiong 2010). Currently, the quality control scheme employed by NMIC of CMA for Chinese AMDAR reports (Tao et al. 2009) is derived from the Meteorological Assimilation Data Ingest System (MADIS; Miller et al. 2005) automated aircraft reports quality control scheme. It includes gross validity checks, temporal consistency checks, internal (physical) consistency checks, and spatial checks. Through quality control, the error rate has been found to be 0.3% for temperature and 1.3% for wind; these erroneous reports have been rejected (Liao and Xiong 2010). Up to now, however, bias correction has not been applied to the data.
The reports used in the present study cover the whole year of 2015 over Mainland China (15°–55°N, 70°–140°E). The variables contained in the reports are time, latitude, longitude, pressure altitude, flight level, temperature, wind speed, wind direction, and flight phase (i.e., ascent, descent, cruise, and turbulence). Figures 1 and 2 show the spatial and temporal distributions of the AMDAR reports, respectively. Compared to the AMDAR data of the years 2004–10 (Ding et al. 2015), the reports of recent years have increased significantly in both the number and the spatial coverage.
b. Chinese radiosonde data
Starting in 2002, CMA phased in the use of the GTS1 radiosondes (mainly manufactured by Nanjing Daqiao Machine company) described in Nash et al. (2011). Chinese radiosondes are launched from 119 stations (denoted by black circles in Fig. 1) twice a day at 2315 and 1115 UTC. The observations are recorded during the ascent of the balloons with a temporal resolution of one second, and transmitted in real time to NMIC/CMA. All these observations during the year of 2015 are used in this study.
c. Two methods of creating comparison data pairs
In this study, two methods of comparison are used to estimate the accuracy of Chinese AMDAR reports.
1) AMDAR reports versus radiosonde observations
In the first comparison, the radiosonde observations are treated as reference values. Since the vertical resolutions of the radiosonde and AMDAR observations are different, the radiosonde observations are interpolated to the pressure altitude of the AMDAR report by a log-linear interpolation method. After that, the horizontal and temporal distances between radiosonde and AMDAR observations are analyzed. An AMDAR–radiosonde data pair is identified when one AMDAR report is located in the spatial and temporal window of a radiosonde observation. By comparing the AMDAR reports to radiosonde observations in a small spatial and temporal window, quantitative differences between these two data (such as RMSDs and biases) can be obtained. The RMSD is written as
where is the RMSD between these two data, which can be calculated immediately by comparison; and are the observation uncertainties of AMDAR and radiosonde, respectively; and is the uncertainty introduced into the data by other factors, such as the spatial and temporal representativeness or environmental influences.
2) AMDAR reports versus AMDAR reports
In the second comparison, the AMDAR reports collected by different aircraft are treated as independent observations. If one AMDAR report falls into the spatial and temporal window of another AMDAR report that is recorded by a different aircraft, then these two reports are set as an AMDAR–AMDAR data pair. The RMSD can be estimated by
Assuming in a very small spatial and temporal window, the observation uncertainty in AMDAR can be given by
3. Comparison results
a. AMDAR–radiosonde comparison results
1) Overall results
The RMSDs between AMDAR reports and radiosonde observations mainly arise from observation uncertainties in AMDAR and radiosonde datasets, spatial and temporal representativeness of the observations, environmental factors, and so on. Several earlier studies have confirmed that the stability of the aircraft platform influences observation quality. Therefore, the reports recorded under turbulent conditions were not used in the present study. To reduce the impact of representativeness during comparison, a spatial and temporal window (20 km, 0 m, and 15 min for horizontal, vertical, and time separation, respectively) has been specified. Within this window, 8971 pairs of AMDAR–radiosonde data are created. The biases and RMSDs in temperature, zonal wind (u), meridional wind (υ), wind direction, and vector are shown in Table 2. The dependence of biases and RMSDs on the flight phase is also given in Table 2. The results indicate that AMDAR temperatures are 0.32°C colder than radiosonde observations. However, the earlier studies (Zhu et al. 2015; Ballish and Kumar 2008; Benjamin and Moninger 2016) indicated that aircraft temperature observations are on average warmer than radiosonde observations. This difference will be discussed below. During this set of comparisons, the best match is found in the cruise phase. The comparison between ascent and descent data shows that the sample size of ascent data is much larger than that of descent data, but the cruise sample is quite small. For temperature, RMSD is better for ascent data. However, descent data are slightly better than ascent data for wind observations, and this is more marked for wind speed (including a bias) than for vector wind. As we know that aircraft do more turns on descent (flying around holding patterns), this probably makes the observations worse on both temperature and wind observations. The reason why the temperature statistic is better and the wind statistic is worse for ascent needs to be investigated in the future.
These RMSDs show some significant improvements compared to the results from the earlier study of Ding et al. (2015), which used the same spatial and temporal window but yielded RMSD values of 1.23°C, 2.77 m s−1, and 27°, for temperature, wind speed, and wind direction, respectively. A higher vertical resolution of the radiosonde observations reduces the interpolation error during the comparison [one part of the radiosonde data used in the earlier study of Ding et al. (2015) has the resolution coarser than 1 s], while larger comparison samples also reduce the RMSD uncertainty. The abovementioned two factors make the most important contribution to the improvement shown in the present study. Despite all this, the temperature observation quality is still worse than the result given by Schwartz and Benjamin (1995).
2) RMSD dependence on separation
In Ding et al. (2015), the RMSDs associated with the spatial separation have already been discussed. The RMSDs associated with temporal separation are also considered in the present study with a strict limit to reduce mutual interference between time and space. Variations of RMSDs with spatial separation are shown in Table 3 with a temporal window of 5 min. For temperature and wind (including speed, u, υ, and vector), RMSDs all increase rapidly with spatial separation. For direction, the RMSD seems less sensitive to spatial separation. A possible explanation is that the Chinese radiosonde uses wind-finding radar, which is susceptible to small errors in orientation that can affect wind direction but has little impact on temperature and wind speed observations. Variations of RMSDs with temporal separation are shown in Table 4 with the spatial window specified to be 10 km. It is clear to see that RMSDs of all these variables increase with temporal separation. The abovementioned results indicate that the representative differences are dependent on the spatial and temporal windows.
3) RMSD dependences on height (above sea level)
Schwartz and Benjamin (1995) have shown that the RMSDs between AMDAR and radiosonde observations display larger differences at lower altitudes presumably as a result of boundary layer turbulence. In the present study, the dependence of RMSDs on height is calculated within a window of (20 km, 0 m, and 15 min), and data pairs recorded below the height of 1 km are divided into five 200-m-interval subgroups according to their heights. The results are shown in Table 5. In the lower atmosphere, all these variables have very large RMSDs and the RMSDs all decrease sharply with increasing height. This result demonstrates the impact of atmospheric variability, including low-level inversions that enhance the vertical gradient, underlying the surface factors that introduce more local changes, and boundary layer turbulence and other factors. Note that the sharp decrease in RMSD of wind direction is also related to the wind speed, which is due to the high correlation in average wind speed and height (Schwartz and Benjamin 1995; Gao et al. 2012; Ding et al. 2015). Above 1000 m, the RMSDs in wind speed and direction still show a decrease tendency with height, but the RMSDs in temperature start to increase again with height. Meanwhile, we find that the temperature biases become more negative as height increases; this trend is opposite of the results given by Benjamin and Moninger (2016), who show that the biases of U.S. aircraft from rawinsonde are negative near the surface and then become positive in the mid- to upper troposphere (600–150 hPa). Benjamin and Moninger (2016) have summarized other research results (Zhu et al. 2015; Ballish and Kumar 2008) that indicated that aircraft temperature observations are on average warmer than radiosonde observations in the upper troposphere.
b. AMDAR–AMDAR comparison results
1) Overall results
In this comparison 3.61 million data pairs are created within the window (20 km, 50 m, and 15 min). The RMSDs for temperature, wind speed, and wind direction are 0.92°C, 1.74 m s−1, and 20°, respectively. All the biases are almost equal to 0—as expected. Compared to the RMSDs calculated based on AMDAR–radiosonde data pairs (i.e., 1.06°C, 1.95 m s−1, and 22°), the AMDAR–AMDAR data pairs have lower RMSDs. Furthermore, when a very small window (1 km, 50 m, and 1 min) is used and the mesoscale variability is assumed to be nearly zero at this time, the RMSDs calculated from 16 107 data pairs reduce to 0.83°C, 1.27 m s−1, and 17°, respectively. Based on Eq. (3), the observation uncertainties in AMDAR are 0.59°C, 0.90 m s−1, and 12°, respectively.
2) RMSD dependence on separation
The RMSD dependence on separation is calculated here using the AMDAR–AMDAR data pairs. The sample size of the AMDAR–AMDAR data pairs is large; therefore, an even more stringent limit is used. First, the temporal window is set to 5 min and the variations of RMSDs with spatial separation in the horizontal direction (≤10 km) are shown in Table 6. For all these variables, even in this tiny window, all the RMSDs increase with spatial separation, especially wind direction, which was not sensitive to spatial separation as shown in Table 3. Second, the spatial window is set to 1 km and the variations of RMSDs with temporal separation are shown in Table 7. It is obvious that all these RMSDs increase with temporal separation, and the results are similar to the comparison made by the AMDAR–radiosonde data pairs. These results indicate that the spatial and temporal representativeness is a key factor even in small scale and that it should be fully taken into account when the AMDAR reports are used in data assimilation or other applications.
3) RMSD dependence on wind speed
In the earlier studies (Schwartz and Benjamin 1995; Gao et al. 2012; Ding et al. 2015), the RMSDs between AMDAR and radiosonde data were demonstrated to be dependent on wind speed. Their results show that the RMSD of wind speed increases with the magnitude of wind speed, while the RMSD of wind direction decreases sharply with the magnitude of wind speed. This issue is discussed in the present study within the window (1 km, 50 m, and 1 min), and the results are shown in Table 8. The increase in vector wind RMSD with speed is probably due to both representativeness and measurement uncertainty. Meanwhile, the observation uncertainty in the wind direction is almost negligible when the wind speed is greater than 8 m s−1.
Chinese AMDAR has advanced rapidly in recent years, with both the data volume and quality rising steadily. On average there are more than 40 000 observations per day for the year 2015. Such large amounts of observations provide a favorable condition for the evaluation of data quality. Two comparisons have been conducted in this study.
The differences between AMDAR and radiosonde data show good quality in Chinese AMDAR reports. The RMSDs in temperature, speed, and direction are 1.06°C, 1.95 m s−1, and 22°, respectively, within the window of (20 km, 0 m, 15 min), and the biases are −0.32°C, −0.31 m s−1, and 2°, respectively (AMDAR minus radiosonde). The comparison between AMDAR–AMDAR data pairs shows better quality in AMDAR reports than ever before. The RMSDs are 0.92°C, 1.74 m s−1, and 20°, respectively, within the window of (20 km, 50 m, and 15 min), and 0.83°C, 1.27 m s−1, and 17°, respectively, within the window of (1 km, 50 m, 1 min) at the same time; the observation uncertainties of temperature, speed, and direction in AMDAR are 0.59°C, 0.90 m s−1, and 12°, respectively. The temperature uncertainty is large compared to the values of 0.3°–0.4°C given by WMO (2017).
The two comparisons illustrate the uncertainty resulting from spatial and temporal representativeness. In particular, the spatial representativeness has significant impacts even in a very small window. Meanwhile, the RMSDs in the lower levels (a few hundred meters above the ground) are much larger, which reflects the impacts of atmospheric variability, including low-level inversions, underlying surface conditions, boundary turbulence, etc. In addition, the comparison results demonstrate that the RMSDs in wind speed (wind direction) increase (decrease) with wind speed. This study found that Chinese AMDAR reports had lower average temperatures than collocated radiosondes (except below 200 m). Benjamin and Moninger (2016) and others have found AMDAR temperatures to be higher than radiosonde temperatures in the middle and upper troposphere. Further work is needed to understand this and the relatively large uncertainties for temperature. A possible approach is to monitor the differences between observation and forecast in an NWP system, and this should also allow timely feedback to data producers of problems with individual aircraft.
This evaluation will attract more attention in and awareness of Chinese AMDAR. The quantitative uncertainty estimation of AMDAR observations can provide direct guidance for applying these data to numerical data assimilation and NWP. Besides, this study proves that some improvement in the quality control system in China is needed.
The authors thank the two reviewers for their comments and suggestions on the manuscript. This research work is supported by the Jiangsu Provincial Natural Science Foundation of China (BK20170751), the Nanjing Joint Center of Atmospheric Research (NJCAR) Program (NJCAR2016MS01), and the National Fundamental Research 973 Programs of China (2015CB452800).