1. Introduction
Observations of wind and temperature from commercial aircraft have increased significantly in the last few years over the United States and, to a lesser extent, worldwide. This dataset has been critical in allowing the development and operation of high-frequency data assimilation systems covering the United States. These reports are transmitted throughout the United States over a communication system used by many air carriers, primarily for nonmeteorological information. This system is called ACARS (Aircraft Communications, Addressing, and Reporting System), which is operated by Aeronautical Radio, Inc.
Aircraft observations relayed through ACARS are used to improve the initialization of operational numerical prediction models and thereby improve the accuracy of the numerical forecasts themselves (Benjamin et al. 1991, 1997, 1998, 1999a). This improvement benefits all users of numerical products, including the aviation industry. Operational data assimilation systems running at the National Centers for Environmental Prediction (NCEP) that use ACARS observations include the Global Data Assimilation System (Kanamitsu et al. 1991), the Eta Data Assimilation System (Rogers et al. 1996), and the Rapid Update Cycle (RUC; Benjamin et al. 1994, 1998). Of these systems, the RUC is run at the highest frequency, every 1 h in a new version (RUC-2) run at NCEP starting in April 1998, replacing the previous version, RUC-1 (which ran every 3 h). In this paper, we provide a comprehensive summary of comparisons between ACARS wind observations and RUC-1 and RUC-2 wind forecasts.
The purposes of this study were to establish 1) conditions under which wind forecast errors are likely to be large, 2) the degree of improvement from RUC-1 to RUC-2 wind forecasts, and 3) the magnitude of ascent, cruise, and descent timing errors (trajectory predictions) for RUC-2 forecasts. Although this study includes details pertinent to air traffic management users, its results should be of interest to all operational users of RUC forecast winds.
This study was carried out under the auspices of the National Aeronautics and Space Administration’s (NASA’s) Terminal Airspace Productivity (TAP) program with the specific intent of examining wind forecast accuracy in the context of automated air traffic management (ATM) decision support tools. In this automated environment (Green and Vivona 1996; Williams and Green 1998), forecast winds from the Rapid Update Cycle are used to predict aircraft trajectories in departure, cruise, and arrival phases of flight. The primary applications of these ATM tools include the metering of arrivals into high-density airspace and the prediction/resolution of conflicts (i.e., the loss of minimum desired separation between crossing flights). For these ATM tool applications, typically involving time horizons of 20–40 min, trajectory prediction errors in excess of 20–30 s may be disruptive and decrease the efficiency of ATM service (Green and Vivona 1996; Paielli and Erzberger 1996). The NASA TAP study also includes the examination of improvements in en route wind forecasts through a nowcast (frequency of 5–30 min) update of RUC forecasts using the latest ACARS observations available since the last RUC analysis. This aspect of the study, conducted by the Massachusetts Institute of Technology’s (MIT) Lincoln Laboratory, was based on an experimental system that applied algorithms from their Integrated Terminal Weather System (ITWS) to en route airspace. Cole et al. (1998, 1999, manuscript submitted to Wea. Forecasting, hereafter CGJ) provide a detailed, separate study of ITWS and RUC-1 forecasts for the same time period and region discussed here.
In this study, we collected all ACARS reports and RUC-1 and RUC-2 wind forecasts for a 13-month period (August 1996–August 1997) for a rectangular region overlaying the Denver Air Route Traffic Control Center area (Fig. 1). This is a fairly sizable region (over 1 800 000 km2, about 1300 km on each side) covering much of the central and western United States, with many en route ACARS observations available. It also includes Denver, where high-frequency ascent/descent ACARS observations were available.
This study included not just an overall assessment of RUC-1 and RUC-2 wind forecast accuracy, with respect to ACARS observations, but also a number of stratifications, including those by flight level, forecast duration, time of day, local weather conditions, and observed wind speed. Also, an assessment of flight time accuracy from RUC-2 forecasts was performed for the comparison period. This assessment, which is more directly applicable to ATM, is equivalent to a horizontal averaging of forecast errors over short time periods corresponding to mesoalpha to mesobeta spatial scales. These spatial scales correspond to the 20-min timescale that is critical for air traffic management in the terminal airspace.
Descriptions of the RUC-1 and RUC-2 are presented in section 2, followed by a description of the datasets used in section 3. A summary of the overall RUC-1 versus RUC-2 accuracy is described in section 4, with more specific stratifications under varied conditions in section 5. The accuracy of aircraft trajectory timing estimates is discussed in section 6. Section 7 describes conclusions of this study.
2. The Rapid Update Cycle
The Rapid Update Cycle is a regional numerical weather prediction and data assimilation system that runs at the National Centers for Environmental Prediction to provide high-frequency, three-dimensional analyses and short-range (out to 12 h) forecasts in support of aviation and other mesoscale forecast users. The RUC differs from other forecast models runs at NCEP in that it runs at a higher frequency, taking advantage of recent observations to produce frequently updated estimates of atmospheric conditions. The high-frequency atmospheric observations that allow this rapid updating include those from commercial aircraft, wind profiles from various kinds of vertically pointing radars, surface observations, and estimates of moisture and winds from satellites. The RUC horizontal domain covers the 48 lower United States and adjacent parts of Canada, Mexico, and oceanic areas. The RUC is also unique in that it uses an isentropic-hybrid vertical coordinate that gives additional resolution in frontal regions and in the vicinity of jet streams.
The initial operational version of the RUC was implemented at NCEP in September 1994 with a 60-km horizontal resolution. A major upgrade was implemented in April 1998 as the 40-km RUC-2. One purpose of this study was to determine improvements in wind forecast accuracy between RUC-1 (Benjamin et al. 1994) and RUC-2 (Benjamin et al. 1998). Forecast skill for the operational RUC-2 is slightly better than shown here, due to improvements added during and after this 13-month test and before April 1998. Basic characteristics of these two versions of the RUC are shown below.
a. The 60-km RUC-1
The RUC-1 used a 81 × 62 horizontal grid with 60-km resolution on a polar stereographic projection, 25 vertical levels, and a 3-h assimilation cycle, using rawinsonde, profiler, aircraft, and surface data. It used relatively simple physics, including supersaturation removal for stable precipitation.
b. The 40-km RUC-2
The RUC-2 uses a 151 × 113 horizontal grid with 40-km resolution on a Lambert conformal projection, 40 vertical levels, and a 1-h assimilation cycle. The RUC-2 uses the same data as in RUC-1 plus velocity azimuth display (VAD) wind profiles [calculated from radial winds at different elevation angles from National Weather Service Weather Surveillance Radar-1988 Doppler (WSR-88D) radars], boundary layer wind profilers, satellite cloud-drift winds, and precipitable water data. Of these, only VAD winds were of significance in the area of data collection. Hourly VAD wind profiles are available at over 100 sites over the United States. More than 1000 cloud-drift wind observations are assimilated into the RUC-2 every 3 h, but only over water regions (as of September 1998).
The use of ascent/descent aircraft data was improved in the RUC-2. The analysis algorithm was modified so that a higher percentage of the ascent/descent data are used. The RUC-2 has more sophisticated physics than RUC-1, including a mixed-phase bulk cloud microphysics scheme with five types of hydrometeors (Brown et al. 1998; Reisner et al. 1998), a multilevel soil–vegetation model (Smirnova et al. 1997, 1999), an improved turbulence parameterization, and a full atmospheric radiation package with sensitivity to cloud hydrometeor mixing ratios.
3. Data collection
a. TAP area definition
A verification region for this study (henceforth referred to as the TAP box and displayed in Fig. 1) was subdivided into nine approximately equal sectors. The area, roughly the size of the Denver center airspace, was chosen to be sufficiently large to sample many aircraft flying into, out of, and over the Denver International Airport (DIA) terminal area. The following chart shows specific information as it relates to both the RUC-1 and RUC-2 RUC grids. As shown in Table 1, the TAP box is not exactly the same size in both the RUC-1 and RUC-2 grids since the two model grids use different map projections.
b. ACARS
The description and processing of ACARS observations is described in an associated article by Benjamin et al. (1999b, referred to hereafter as BSC99). ACARS reports within the TAP box were saved for a 13-month period from August 1996 through August 1997. Four different carriers provided the data used in this study: United Airlines, United Parcel Service (UPS), Delta Air Lines, and Northwest Airlines. United Airlines enhanced the number of ascent/descent profiles in and out of Denver specifically for this study after April 1997. The results presented in this paper for absolute RUC performance may therefore be better than those that would occur for other locations with less ACARS data. However, since the additional reports were available to both the RUC-1 and RUC-2, the improvements in RUC-2 over RUC-1 described later are generally independent of this factor.
Most aircraft relayed a report at 5-min intervals at cruising altitudes. However, United Airlines and UPS aircraft also reported more frequent observations on ascent and descent for part of the observational period. The United data were reported at 2000-ft (609.6 m) intervals from 6000 ft (1828.8 m) to 30 000 ft (914.1 m). The levels correspond to pressure altitudes above mean sea level in the U.S. Standard Atmosphere, 1976 (see Saucier 1955, appendix, Table H). The UPS ascent and descent data were reported at higher frequency, approximately every 100 ft (30 m) at low altitudes.
The following variables from the ACARS reports were used in this study: latitude–longitude [tenths of a minute (∼150 m); actual reporting precision for some of the aircraft used in this study was about 4 min lat/long (∼6 km)], time (nearest minute), flight level [nearest hundred feet (∼30 m), corresponding to pressure altitude], wind direction (to the nearest degree), and wind speed (to the nearest knot).
A limited quality control procedure was performed on the raw incoming data at the Forecast Systems Laboratory (FSL). This procedure is discussed in Moninger and Miller (1994). Significant errors in ACARS meteorological observations are almost always due to report formatting problems in some aircraft with certain digital flight data acquisition units rather than actual problems with the data sensors. These errors include inaccurate position, wind, or temperature. The Moninger–Miller checks include temporal consistency and reasonable range checks. Observations found to be in error from these checks were not used in this study.
A total of 1 757 066 ACARS observations were collected in the TAP box from 1 August 1996 through 31 August 1997. Table 2 shows the number of ACARS reports broken down by nine area sectors comprising the TAP box and by pressure altitude. (The sectors correspond to those displayed in Fig. 1.) The distribution of reports is highly skewed toward the sectors closest to DIA with the most reports received (37% of total) in the sector containing DIA (sector 22). In addition, most of the observations were received from aircraft at cruising altitudes. Most of the relatively small number of remaining reports from aircraft on ascent or descent are contained in the DIA sector. As a result, stratification of statistics by airspace is problematic. In particular, results for noncruising altitudes were limited to the airspace immediately surrounding the DIA terminal.
On average, approximately 4650 ACARS reports per day in the TAP box were received. Prior to 1 May 1997, the average number of reports was about 3000 per day and increased to approximately 7000 per day on 1 May when United Airlines increased the frequency of their data reporting to assist the NASA TAP study.
c. METAR data
Routinely available surface weather observations (meteorological aviation reports—METARs) were also archived for the DIA terminal site. These observations were scanned for the occurrence of the following weather phenomena: precipitation at or near the station, thunder and/or towering cumulus (TCU), and altocumulus standing lenticular cloud observations (possible indicator of mountain-wave activity). The use of the scanned METAR data is discussed in section 5g.
d. Model forecast data
Model forecast data from both the RUC-1 and RUC-2 were routinely archived for the area covering the TAP box. The data for the RUC-2 were available from an experimental version of the model run locally at the Forecast Systems Laboratory. For both models, we archived the highest-frequency data available at the time of the data collection period. Forecasts were produced and archived from runs initialized every 3 h (0000, 0300, . . . , 2100 UTC). For the RUC-1 runs, forecasts were available for the 0-, 1-, 2-, 3-, 4-, 5-, and 6-h forecast projections; for the RUC-2, forecasts were available only for the 0-, 3-, and 6-h forecast projections. In the operational RUC-2, forecast output is available at 1-, 2-, and 3-h projections from each hourly run. In this study, we performed a simple linear interpolation in time to produce forecast values for the 1-, 2-, 4-, and 5-h forecast projections for the RUC-2 model runs.
Gridded data were vertically interpolated from the RUC–MAPS native hybrid-b vertical coordinate to evenly spaced 25-hPa isobaric surfaces from 1000 to 100 hPa. The vertical interpolation was performed in order to more closely match the vertical coordinate of the wind observations contained in the ACARS data [pressure altitude (hPa)]. Also, Federal Aviation Administration (FAA) and ITWS users of RUC data receive the data on isobaric levels, so the treatment here of the gridded data matches how they are available to those users. Approximately 90% of all possible model runs during the data collection period were successfully archived and used to produce matched forecast–observation data matrices.
Using the ACARS and model-forecast data saved over the TAP box, we produced daily matrices of matched RUC-1 and RUC-2 forecasts with ACARS observations. All possible forecasts (out to the 6-h forecast projection) that were available were matched with ACARS observations in time and space. The forecasts were matched in space to the latitude and longitude of the ACARS observation using bilinear interpolation. The vertical interpolation of the forecasts to the altitude of the ACARS observation was a two-step process. First, we converted the reported altitude to the pressure altitude in the U.S. Standard Atmosphere, 1976 that it actually represents. We then did a simple linear interpolation in log p using the model data on the 25-hPa pressure surfaces. Observations were matched to the forecast verifying within a maximum of 30 min of the observation. In other words, we did not interpolate between two different forecasts to match with the observation time, except, as previously described, to do a time interpolation to derive forecasts for the 1-, 2-, 4-, and 5-h projections for the RUC-2.
Assuming all data were available, either two or three different forecast cycles contained a forecast that could be matched to each observation. Three forecasts were matched with observations that occurred within 30 min of RUC/MAPS run initialization times, that is, 0000, 0300, 0600, . . . , 2100 UTC. For example, if an observation time was 0323 UTC, forecast data from the following cycles/forecast projections that verify at 0300 UTC were available for comparison: 0300 UTC/0 h, 0000 UTC/3 h, 2100 UTC/6 h (previous day). On the other hand, if the observation time was 0345 UTC, there were only two matching forecasts that verify at 0400 UTC: 0300 UTC/1 h and 0000 UTC/4 h.
A data matrix containing all ACARS observations along with their corresponding forecasts, interpolated in space, was produced for each day. All data used to produce the matrix were also saved. Using the data matrices, we produced various statistics regarding the differences between the observations and forecasts. A discussion of the various statistical measures follows.
4. RUC-1 versus RUC-2 wind forecast accuracy with matched forecast projections
For each day, an overall root-mean-square vector difference (rmsvd) statistic between observed and forecast wind was computed from the data matrices described above. The rmsvd includes errors from both forecasts and from the verifying observations themselves. As shown in BSC99, the rms errors from the verifying ACARS observations are estimated at 1.8 m s−1 overall at the smallest time–space collocation window, up to 0.7 m s−1 higher below 800 hPa (perhaps due in part to more frequent aircraft maneuvers), and up to 0.2 lower at 300–400 hPa.
These daily statistics combine forecast–observation pairs from all forecast projections, altitudes, aircraft, and over the entire TAP airspace. Table 3 shows some simple distributions of the rmsvd for all possible forecast–observation pairs from the dataset and indicates the variability of individual days over the 13-month period. The data were stratified for each model separately and for a matched sample (days when both RUC-1 and RUC-2 forecasts were available). As shown in Table 3, the 40-km RUC-2 wind forecasts were superior to the 60-km forecasts by a substantial margin, about 0.6 m s−1 for the mean daily rmsvd values. To test for the effect of the temporal interpolation performed for some RUC-2 forecasts (section 3d), a second comparison was made eliminating any interpolated values and using only 0-, 3-, and 6-h forecasts (Table 3b). The improvement of RUC-2 over RUC-1 forecasts was unchanged in this comparison. Thus, for other comparisons in this paper, the time-interpolated RUC-2 values were used. The effect of matched versus unmatched samples was also examined and found to be negligible (Table 3). A histogram showing the number of days with different daily rmsvd values for the 40-km RUC-2 is shown in Fig. 2. The predominant daily value is in the 4.5–5.5 m s−1 range, accounting for about 164 of the 395 days.
Although the difference in rmsvd between RUC-1 and RUC-2 forecasts is less than the estimated ACARS error, it was found to be statistically significant because of the very large number of events considered and because the ACARS observation errors are unbiased with respect to forecast errors from either model. We performed two tests for significance. The first was the parameteric t-test, which assumed that the individual daily means for both RUC-1 and RUC-2 were normally distributed. The t value of 7.29 indicates significance well in excess of the 95% confidence level (equivalent to a t value of 2.576). However, since the 353 daily means were not normally distributed (not shown), a nonparametric test (Kruskal–Wallis, or chi-square approximation) was necessary to confirm that the RUC-2 group of daily means was significant different from the RUC-1 group of means. This test gave a chi-square value of 57.146 (significant to 99.99% confidence level). This statistic clearly indicates that the distributions from RUC-1 and RUC-2 are independent and are not the same.
5. Statistical analysis of possible factors for wind forecast accuracy
In this section, we present different measures of the differences between ACARS observations and RUC wind forecasts. To better understand the factors that affect forecast errors, a number of different stratifications are provided, including month of year, wind speed, forecast projection, altitude, time of day, horizontal location, weather events, and forecast availability. The choice of stratifications was made by hypothesizing what influences might be most significant on wind forecast errors. These breakdowns were calculated to provide guidance for RUC wind forecast users (such as in air traffic management) regarding conditions in which wind forecast errors might be expected to be higher or lower than average.
a. Month of year
Rms differences by month for both the RUC-1 and RUC-2 runs are shown in Fig. 3. There was a clear seasonal variation of forecast errors in general, with lower rmsvd values in summer and higher values in winter. This seasonal variation of forecast errors is commonly observed for all forecast models in the extratropics as wind speeds change with the seasonal movement of the mean jet stream position. In this particular period, December and April were particularly active weather periods, with relative maxima indicated in the rmsvd values. Again, the superiority of the 40-km runs is evident in this monthly breakdown, with the largest improvement from the 40-km RUC-2 in November and December. As is shown in section 6a, this improvement was more accentuated when examining a smaller subset of high error cases.
b. Wind speed
One should view the above monthly results and others that will be shown with the knowledge that the rms differences are dependent upon the observed wind speeds. Table 4 shows the distribution of mean rms differences with observed ACARS wind speed categories. (The “oper” columns are discussed in section 5h.) As wind speeds increased, so did rms vector differences. Again, this result is consistent with statistics for all operational forecast models (e.g., DiMego 1988). One reason for this behavior is that at large wind speeds, small direction differences can easily result in very large vector differences. Again, the improvement of the RUC-2 over the RUC-1 was most pronounced in more difficult forecast situations. For observed winds of over 50 m s−1 with the matched forecast sample, this improvement in rmsvd was over 1.3 m s−1.
c. Forecast projection
It is not surprising that forecast errors should grow with the length of the forecast projection, but the ACARS dataset provides a unique opportunity to examine this error growth on an hour-by-hour basis. Figure 4 shows the rms vector difference by forecast projection for the entire sample. Rmsvd values increase by about 1.5 m s−1 from 1 to 6 h for both the RUC-1 and RUC-2 models. (In the sample for this study, 3- and 6-h forecast–observation pairs had slightly higher mean altitudes than those at other forecast projections, resulting in the slight peaks of the differences at these times in Fig. 4.)
d. Altitude
The effect of altitude on rms vector difference between observations and forecasts is depicted in Fig. 5. The average wind speed for each altitude category is also shown, again illustrating the dependence of rms vector differences on wind speed. However, note that maximum differences also occur near the ground as well as at upper levels. Because of the ACARS collocation study described in BSC99, we attribute the maximum differences near the ground primarily to larger ACARS observational errors from aircraft maneuvers and secondarily to larger forecast errors in the boundary layer where eddies cause large local variations. Erroneous ACARS observations on ascent and descent are discussed in Shwartz and Benjamin (1995) and BSC99.
e. Time of day
Since the proportion of reports aloft versus reports near the surface is highly correlated with the time of day, we examined rmsvd’s by time of day (Fig. 6) using only observations between 210 and 220 hPa. (Flight level 370 is at 216 hPa.) The average wind speed from this sample showed a minimum for the 1200–1500 UTC window, possibly related to the location of flights within the TAP box at that time. However, the range of average wind speed was reduced from using observations at all levels, as hoped for, and the variation for 3-h windows other than 1200–1500 UTC is only 1.0 m s−1. The results in Fig. 6 show reduced forecast error during the daytime, when more ACARS observations are available.
It is likely that RUC/MAPS forecast accuracy during the day (1200–2100 UTC) is slightly higher because of the larger ACARS data volume assimilated into RUC/MAPS at these times. This result is also evident in improved verification (using rawinsonde data) of 1- to 3-h RUC forecasts valid at 0000 UTC compared to that for forecasts valid at 1200 UTC (Benjamin et al. 1999a).
f. Horizontal location and data density
A difficult goal of this study was to examine forecast errors for dependence on data density at analysis time. In general, the forecast accuracy was expected to increase with the density of observations available for assimilation into the analysis. The ACARS data density varies with horizontal location and time of day depending on each participating airline’s schedule and route structure.
An effort was made to determine if wind forecast errors were dependent on horizontal location. This was done by stratifying rms vector differences between ACARS observations and RUC-2 forecasts by the nine sectors comprising the TAP box (Fig. 1). The ACARS reports are not evenly distributed among these sectors. As shown in Table 2, the distribution of ACARS observations was highly skewed toward the center sector containing Denver (sector 22). Moreover, the Denver sector had a much higher proportion of ascent/descent data, whereas the other sectors had predominantly en route reports. Therefore, a meaningful result could not be obtained about the effect of horizontal location by simply determining the rms vector difference for all reports in each sector.
Although the dataset did not support a study of horizontal variation of mid- and lower-tropospheric wind forecasts, it did allow a relatively fair evaluation of horizontal location (and data density) effects on en route wind errors. This was done by determining rms vector difference values for reports at pressure less than 250 hPa in each sector (Fig. 7). There was some evidence of slightly larger errors in the northwest and north-central sectors, perhaps due to climatologically stronger winds farther to the north. This also may be due to a lower volume of ACARS reports in the northern sectors, which would lead to less certainty in the forecasts due to less accurately defined initial conditions. Overall, these results do not contradict the hypothesis that higher ACARS volumes lead to some improvements in RUC forecasts a few hours later, but do not unambiguously support it either.
Another approach to examine sensitivity of short-range forecasts to ACARS data density would be to run full parallel RUC cycles with and without extra ACARS data. Although such an experiment was beyond the scope of this study, CGJ show improved ITWS wind nowcasts with higher density of ACARS observations.
g. Differences associated with various types of weather events
Wind forecast errors from the RUC-2 near Denver were stratified by various surface conditions using the Denver (DEN) surface observation dataset described in section 3c. We computed rms vector differences for ACARS reports within 1.0° lat–long of DEN and calculated results for the presence or absence of precipitation, thunder, and wave clouds. Results were computed for “high” (pressure < 250 hPa) and “low” (pressure > 250 hPa) altitudes.
As can be seen from Figs. 8–10, it appears that the differences between forecasts and observations were greatest during convective weather events. This correlation between convective weather and larger wind errors was probably even larger than indicated here since the data collection area (2° or 150–220 km on each side) was somewhat larger than the radius for TCU or thunder reports at a given surface station. It is interesting that the differences were actually smaller for hours with precipitation during the cool season, perhaps because the jet position was usually south of DEN in such situations.
h. RUC-2 improvements in forecast availability
Previous figures show that there was in an improvement in wind forecast accuracy from RUC-1 to RUC-2. There was, additionally, an improvement in forecast availability from RUC-2 that is particularly important from an air traffic perspective. The RUC-2 model runs more frequently (every 1-h as opposed to every 3-h for the RUC-1) and is available sooner than the old RUC-1. With the RUC-1, the data cut-off time was not until 80 min after the initial time, and new cycles were run only every 3 h. Thus, air traffic control tools were forced to use 3–5 h RUC-1 forecasts as estimates of winds for 1 h ahead of real time (Williams and Green 1997; Jardin and Green 1998; Jardin and Erzberger 1996).
With the RUC-2, the data cutoff time is about 20 min after the initial time and new runs with at least a 3-h forecast are run hourly. As a result, a 1–2-h forecast from RUC-2 can be used to support air traffic management, compared to the 3–5-h forecast previously available in the same time period from RUC-1. Figure 11 shows the monthly rms vector differences for 3–5-h 60-km RUC-1 forecasts against the 1–2-h forecasts from RUC-2. With the improvement in forecast availability added to the improvement in accuracy, the rms vector difference improved by about 1.2 m s−1 overall from RUC-1 to RUC-2, a very substantial improvement. Assuming that the perfect forecast score would be close to the 1.8 m s−1 estimate of ACARS vector wind error (BSC99) and using Eq. (1) to calculate true forecast error, the RUC-1 to RUC-2 improvement of 1.2 m s−1 (accuracy plus timeliness improvements) constitutes about a 22% reduction in wind forecast error at short timescales. In the peak error month, December 1996, the improvement was greater than 1.7 m s−1. The impact of more frequent availability is also accentuated in peak wind speed situations (see “oper” columns in Table 4). In comparing the RUC-1 (3–5-h forecasts) and RUC-2 (1–2-h forecasts), the improvement of the latter was greater than 2.2 m s−1 for situations with observed wind speed greater than 50 m s−1. Accounting for these operational timing considerations, the improvement of RUC-2 over RUC-1 for short-range users such as air traffic is magnified.
6. RUC wind errors from an air traffic management perspective
In this section, we discuss wind forecast errors from two perspectives important to air traffic management: the frequency of particularly large errors compared to individual observations (peak error events), and errors in calculating exact arrival times for an aircraft position about 15 min in the future.
a. Peak error events
For air traffic automation, the peak error periods for wind forecasts are critical. In defining a metric for peak errors, it is useful to consider that the FAA standard for en route radar separation is 5 n mil (under instrument flight rules). A 15-kt (7.5 m s−1) mean error in alongtrack wind component over a 20-min trajectory prediction for an aircraft will result in a 5–n mi position error. In this section, we consider the number of individual reports where observation-minus-forecast vector differences (not necessarily along the flight path) are at least 10 m s−1, which we defined as peak error events. In reality, these peak errors are likely to be significant to ATM applications only if they persist along a predicted trajectory of 20-min duration or greater (approximately 200–300 km or more). In section 6b, we examine such mean errors that persist over actual trajectories evaluated in the TAP study dataset.
Figure 12 shows the percentage of total forecast–observation differences greater than 10 m s−1 for both versions of the RUC system as a function of month. The forecast projections used here are the operational ones as in Fig. 11 (1–2 h for RUC-2; 3–5 h for RUC-1). Overall, the percentage of such peak errors decreased from 8% to 3% using the 1–2-h forecasts from RUC-2. In December 1996, peak errors decreased from 17% to 7%.
b. Effect of wind forecasts on aircraft arrival timing accuracy
As seen in the previous section, large differences can occur between forecast wind and observed wind velocities at any given position. However, for air traffic management applications, it is important to consider wind prediction errors that persist along the predicted aircraft trajectory (i.e., component errors along the trajectory that do not average out). In this study, the actual flight paths from strings of ACARS observations were evaluated to determine the impact of the mean alongtrack errors on arrival time (i.e., the estimated elapsed time for a given flight segment). Results are presented in terms of time, as opposed to distance, to facilitate comparisons to actual flight tests referred to later in this section.
Arrival time errors were computed for 15-min trajectory segments for all flights participating in our data collection. Although a 20-min time horizon is desirable for ATM predictions, the time horizon was limited to 15 min to correspond to the typical duration of ascent/descent segments; segments were calculated from strings of reports. A segment was started at takeoff or when the aircraft entered the TAP study airspace or after the last 15-min segment.
The time errors for each subsegment may be positive or negative in calculating the cumulative time error, ΔTerr. For these computations, we used the 1–2-h forecast projections for the RUC-2 runs only, since 1–2-h forecasts are now used for this application from the hourly RUC-2. We note that by using a string of reports from the same aircraft in this comparison, the overall error may be inflated since some aircraft may have systematic errors. However, by using this string from the same aircraft, we gained meaningful statistics in being able to follow actual flights with fairly high vertical resolution.
Figure 13 shows histograms of the number of occurrences of different arrival time errors for over 17 000 ascent/descent segments (Fig. 13a) and over 150 000 en route segments (Fig. 13b). The average time error was 10.2 s (σ = 8.9 s) for en route segments, and smaller errors, mean of 9.3 s (σ = 10.2 s), for ascent/descent segments. Even with the simple air speed approximation used in this analysis, these results are generally consistent with results from earlier field tests of ATM tool trajectory prediction accuracy reported by Williams and Green (1997) and Green and Vivona (1996). Those studies evaluated en route and descent trajectory prediction accuracy under the influence of operational error sources including not only wind predictions, but also radar tracking, aircraft performance modeling, and pilot navigation. Overall arrival time accuracy for descents of approximately 15-min duration in the earlier studies was typically within 20 s (mean plus standard deviation).
7. Conclusions
First, the study showed that rms vector differences between observations and forecasts from either RUC-1 or RUC-2 increased as wind speed increased, and correspondingly, as altitude increased and toward winter months. There was a slight increase of en route errors in regions with limited ACARS data, but this may have been related to climatologically stronger wind in these areas. Wind errors increased when thunderstorms were nearby and were smaller in wintertime precipitation situations.
Second, this study revealed that considerable progress has been made in the accuracy of wind forecasts used for air traffic management by the introduction of the RUC-2 system, replacing the previous RUC-1 system. This new version was implemented operationally at NCEP in early April 1998. Improvement was made both in the intrinsic accuracy (10% reduction of forecast error) as well as in the time availability, both contributing to the overall improvement (22% reduction of wind forecast error) in the actual wind forecast available for air traffic management purposes. These RUC-2 accuracy improvements were determined from a comparison made between RUC forecasts and ACARS observations in the central and western United States over a 13-month period from August 1996 to August 1997.
It was also found that the degree of improvement from the RUC-2 increased substantially for periods with larger than normal wind errors. For example, increased improvement was shown for higher wind speeds and the percentage of individual vector errors greater than 10 m s−1 was reduced by RUC-2 from 8% (RUC-1) to 3% overall and from 17% to 7% during the worst month.
Third, it was found that estimated trajectory projection errors from the RUC-2 using 1–2-h forecasts averaged only 9 s for ascent/descent flight segments of approximately 15 min, and about 10 s for en route segments of the same duration. These trajectory prediction errors were smaller than combined errors from wind errors and other sources reported in previous air traffic management experiments. It should be noted that this accuracy was achieved with an enhanced volume of ACARS data for the last part of this experiment, so the trajectory prediction errors may be larger near other hubs without as much ACARS data.
These improvements in RUC-2 are significant in a direct sense for the many programs in the FAA that use RUC data, but they also are important in providing improved background fields for the Integrated Terminal Weather System planned to run locally near large hubs across the United States. Cole et al. (1998) show a considerable accuracy increment of ITWS nowcasts over its background fields from RUC-1 forecasts, so the accuracy of ITWS using RUC-2 backgrounds should be significantly enhanced over the RUC-1 version.
Considerable effort was made in this study to develop an ACARS observational database with most of the erroneous data excluded. There was still some ambiguity in some cases as to whether large observation-minus-forecast differences result from erroneous observations or forecasts. Here, additional quality control procedures based on knowledge of maneuver-induced ACARS wind errors (Jardin and Green 1998) may be useful. In addition, we advocate that some information about aircraft maneuvers such as a flag for roll angle exceeding 5° be added to the ACARS meteorological report format for all reporting aircraft (UPS is already providing such information).
Acknowledgments
We appreciate the careful reviews of this paper given by Rodney Cole (MIT Lincoln Laboratory), Michael Kraus (NOAA/FSL), William Moninger (NOAA/FSL), and anonymous reviewers. We are grateful to Carl Knable of United Airlines for his cooperation in this study. We also thank Jamie Riggs of FSL for her help with significance testing. This work was supported by the NASA Terminal Airspace Productivity project. The development of the RUC-2 system has been supported by the FAA Aviation Weather Research Program and by NOAA.
REFERENCES
Benjamin, S. G., K. A. Brewster, R. L. Brummer, B. F. Jewett, T. W. Schlatter, T. L. Smith, and P. A. Stamus, 1991: An isentropic three-hourly data assimilation system using ACARS aircraft observations. Mon. Wea. Rev.,119, 888–906.
——, K. J. Brundage, P. A. Miller, T. L. Smith, G. A. Grell, D. Kim, J. M Brown, and T. W. Schlatter, 1994: The Rapid Update Cycle at NMC. Preprints, 10th Conf. on Numerical Weather Prediction, Portland, OR, Amer. Meteor. Soc., 566–568.
——, and Coauthors, 1997: Improvements in aviation forecasts from the 40-km RUC. Preprints, Seventh Conf. on Aviation, Range, and Aerospace Meteorology, Long Beach, CA, Amer. Meteor. Soc., 411–416.
——, J. M. Brown, K. J. Brundage, B. E. Schwartz, T. G. Smirnova, and T. L. Smith, 1998: The operational RUC-2. Preprints, 16th Conf. on Weather Analysis and Forecasting, Phoenix, AZ, Amer. Meteor. Soc., 249–252.
——, ——, K. J. Brundage, D. Kim, B. E. Schwartz, T. G. Smirnova, and T. L. Smith, 1999a: Aviation forecasts from the RUC2. Preprints, Eighth Conf. on Aviation, Range, and Aerospace Meteorology, Dallas, TX, Amer. Meteor. Soc., 486–490.
——, B. E. Schwartz, and R. E. Cole, 1999b: Accuracy of ACARS wind and temperature observations determined by collocation. Wea. Forecasting,14, 1032–1038.
Brown, J. M., T. G. Smirnova, and S. G. Benjamin, 1998: Introduction of MM5 level 4 microphysics into the RUC-2. Preprints, 12th Conf. on Numerical. Weather Prediction, Phoenix, AZ, Amer. Meteor. Soc., 113–115.
Cole, R. E., C. Richard, S. Kim, and D. Bailey, 1998: An assessment of wind field accuracy for ATM DST using the 60-km RUC augmented with real-time aircraft reports via the ITWS terminal wind algorithm. Project Rep. NASA/A-1, MIT Lincoln Laboratory, 63 pp. [Available from National Technical Information Service, 5285 Port Royal Rd., Springfield, VA 22161.].
——, S. Green, and M. Jardin, 2000: Improving RUC-1 wind estimates by incorporating near-real-time aircraft reports. Wea. Forecasting, in press.
DiMego, G. J., 1988: The National Meteorological Center Regional Analysis System. Mon. Wea. Rev.,116, 977–1000.
Green, S., and R. Vivona, 1996: Field evaluation of Descent Advisor trajectory prediction accuracy. AIAA Guidance Navigation and Control Conf., AIAA Publ. 96-3764, 13 pp. [Available from American Institute of Aeronautics and Astronautics, 1801 Alexander Bell Dr., Suite 500, Reston, VA 22091.].
Jardin, M. R., and H. Erzberger, 1996: Atmospheric data acquisition and interpolation for enhanced trajectory-prediction accuracy in the Center-TRACON Automation System. AIAA Aerospace Sciences Conf., AIAA Publ. 96-0271, 7 pp. [Available from American Institute of Aeronautics and Astronautics, 1801 Alexander Bell Dr., Suite 500, Reston, VA 22091.].
——, and S. M. Green, 1998: Atmospheric data error analysis for the 1994 CTAS Descent Advisor preliminary test. NASA Tech. Memo. TM-1998-112228, 24 pp. [Available from NASA Center for Aerospace Information, 7121 Standard Drive, Hanover, MD 21076.].
Kanamitsu, M., and Coauthors, 1991: Recent changes implemented into the global forecast system at NMC. Wea. Forecasting,6, 425–435.
Moninger, W. R., and P. A. Miller, 1994: ACARS quality control, monitoring, and correction. Preprints, 10th Conf. on Numerical Weather Prediction, Portland, OR, Amer. Meteor. Soc., 4–6.
Paielli, R. A., and H. Erzberger, 1996: Conflict probability estimation for free flight. NASA Tech. Memo. TM-110411, 15 pp. [Available from NASA Center for Aerospace Information, 7121 Standard Drive, Hanover, MD 21076.].
Reisner, J., R. M. Rasmussen, and R. T. Bruintjes, 1998: Explicit forecasting of supercooled liquid water in winter storms using the MM5 mesoscale model. Quart. J. Roy. Meteor. Soc.,124, 1071–1107.
Rogers, E., T. M. Black, D. G. Deaven, G. J. DiMego, Q. Zhao, M. Baldwin, N. W. Junker, and Y. Lin, 1996: Changes to the operational “early” eta analysis/forecast system at the National Centers for Environmental Prediction. Wea. Forecasting,11, 391–413.
Saucier, W. J., 1955: Principles of Meteorological Analysis. Dover Publications, 438 pp.
Schwartz, B. E., and S. G. Benjamin, 1995: A comparison of temperature and wind measurements from ACARS-equipped aircraft and rawinsondes. Wea. Forecasting,10, 528–544.
Smirnova, T. G., J. M. Brown, and S. G. Benjamin, 1997: Performance of different soil model configurations in simulating ground surface temperature and surface fluxes. Mon. Wea. Rev.,125, 1870–1884.
——, ——, and ——, 1999: Parameterization of frozen soil physics in MAPS. J. Geophys. Res., in press.
Williams, D. H., and S. M. Green, 1998: Flight evaluation of the center/TRACON automation system trajectory prediction process. NASA Tech. Paper TP-1998-208439, 88 pp. [Available from NASA Center for Aerospace Information, 7121 Standard Drive, Hanover, MD 21076.].
The area (TAP box) in the central United States used in the TAP study. Sector numbers correspond to those used in Table 1.
Citation: Weather and Forecasting 15, 3; 10.1175/1520-0434(2000)015<0313:AORARW>2.0.CO;2
Histogram showing the distribution of the daily rms wind vector differences between RUC-2 forecasts and ACARS observations over the 13-month period.
Citation: Weather and Forecasting 15, 3; 10.1175/1520-0434(2000)015<0313:AORARW>2.0.CO;2
Monthly rms vector differences between ACARS wind observations and forecasts for RUC-1 and RUC-2 forecasts.
Citation: Weather and Forecasting 15, 3; 10.1175/1520-0434(2000)015<0313:AORARW>2.0.CO;2
Rms vector differences by forecast projection (0 h analysis to 6 h) for RUC-1 and RUC-2 forecasts.
Citation: Weather and Forecasting 15, 3; 10.1175/1520-0434(2000)015<0313:AORARW>2.0.CO;2
Rms vector differences by altitude. The dark dashed line indicates the average observed wind speed.
Citation: Weather and Forecasting 15, 3; 10.1175/1520-0434(2000)015<0313:AORARW>2.0.CO;2
Rms vector differences by time of day only for observations between 210 and 220 hPa (approximately flight level 370)
Citation: Weather and Forecasting 15, 3; 10.1175/1520-0434(2000)015<0313:AORARW>2.0.CO;2
Rms vector differences by TAP airspace sector for reports with pressure <250 hPa.
Citation: Weather and Forecasting 15, 3; 10.1175/1520-0434(2000)015<0313:AORARW>2.0.CO;2
Rms vector differences for RUC-2 forecasts with and without thunder reports at Denver. Results are broken down into high altitude (p < 250 hPa) and low altitude (p < 250 hPa).
Citation: Weather and Forecasting 15, 3; 10.1175/1520-0434(2000)015<0313:AORARW>2.0.CO;2
Same as Fig. 10 but for reports of TCU at Denver.
Citation: Weather and Forecasting 15, 3; 10.1175/1520-0434(2000)015<0313:AORARW>2.0.CO;2
Same as for Fig. 10 but for reports of precipitation at Denver. This figure is further stratified for cool season and warm season events.
Citation: Weather and Forecasting 15, 3; 10.1175/1520-0434(2000)015<0313:AORARW>2.0.CO;2
Rms vector differences for the operational comparison of RUC-1 (3–5-h forecasts) and RUC-2 (1–2-h forecasts)
Citation: Weather and Forecasting 15, 3; 10.1175/1520-0434(2000)015<0313:AORARW>2.0.CO;2
Percentage of occurrence for individual ACARS reports with vector differences from RUC-1 (3–5 h) and RUC-2 (1–2 h) forecasts of >10 m s−1.
Citation: Weather and Forecasting 15, 3; 10.1175/1520-0434(2000)015<0313:AORARW>2.0.CO;2
Histogram displaying the distribution of estimated arrival time errors from RUC-2 1–2-h forecasts for segments of at least 15 min in duration. (a) Ascent/descent segments, (b) en route segments.
Citation: Weather and Forecasting 15, 3; 10.1175/1520-0434(2000)015<0313:AORARW>2.0.CO;2
Description of the TAP area within the RUC-1 and RUC-2 domains.
Number of ACARS reports by sector and pressure altitude (hPa). (Sector numbers correspond to those used in Fig. 1.)
Table 3a. Daily rms vector difference between ACARS observations and RUC–MAPS forecasts for all forecast projections (all values in m s−1).
Table 3b. Same as Table 3a except for the 0-, 3-, and 6-h projections.
Comparison of Rmsvd by speed category (all values m s−1) [operational (oper) comparison: RUC-1 (3–5-h forecasts) RUC-2 (1–2-h forecasts)].
Estimate of the actual rms vector error in RUC forecasts (all values in m s−1).
