• Baker, N. L., and R. Daley, 2000: Observation and background adjoint sensitivity in the adaptive observation-targeting problem. Quart. J. Roy. Meteor. Soc., 126 , 14311454.

    • Search Google Scholar
    • Export Citation
  • Benjamin, S. G., K. A. Brewster, R. L. Brummer, B. F. Jewett, T. W. Schlatter, T. L. Smith, and P. A. Stamus, 1991: An isentropic 3-hourly data assimilation system using ACARS aircraft observations. Mon. Wea. Rev., 119 , 888906.

    • Search Google Scholar
    • Export Citation
  • Benjamin, S. G., and Coauthors, 2004a: An hourly assimilation/forecast cycle: The RUC. Mon. Wea. Rev., 132 , 495518.

  • Benjamin, S. G., G. A. Grell, J. M. Brown, T. G. Smirnova, and R. Bleck, 2004b: Mesoscale weather prediction with the RUC hybrid isentropic/terrain-following coordinate model. Mon. Wea. Rev., 132 , 473494.

    • Search Google Scholar
    • Export Citation
  • Benjamin, S. G., B. E. Schwartz, E. J. Szoke, and S. E. Koch, 2004c: The value of wind profiler data in U.S. weather forecasting. Bull. Amer. Meteor. Soc., 85 , 18711886.

    • Search Google Scholar
    • Export Citation
  • Benjamin, S. G., and Coauthors, 2004d: A 13-km RUC and beyond: Recent developments and future plans. Preprints, 11th Conf. on Aviation, Range, and Aerospace, Hyannis, MA, Amer. Meteor. Soc., J1.6.

    • Search Google Scholar
    • Export Citation
  • Benjamin, S. G., J. M. Brown, S. S. Weygandt, T. L. Smith, B. Schwartz, and W. R. Moninger, 2004e: Assimilation of surface cloud, visibility, and current weather observations in the RUC. Preprints, 16th Conf. on Numerical Weather Prediction, Seattle, WA, Amer. Meteor. Soc., P1.47.

    • Search Google Scholar
    • Export Citation
  • Benjamin, S. G., S. Weygandt, D. Devenyi, J. M. Brown, G. Manikin, T. L. Smith, and T. Smirnova, 2004f: Improved moisture and PBL initialization in the RUC using METAR data. Preprints, 11th Conf. on Aviation, Range, and Aerospace, Hyannis, MA, Amer. Meteor. Soc., 17.3.

    • Search Google Scholar
    • Export Citation
  • Benjamin, S. G., W. R. Moninger, S. R. Sahm, and T. L. Smith, 2007: Mesonet wind quality monitoring allowing assimilation in the RUC and other NCEP models. Preprints, 22nd Conf. on Weather Analysis and Forecasting/18th Conf. on Numerical Weather Prediction, Park City, UT, Amer. Meteor. Soc., P1.33.

    • Search Google Scholar
    • Export Citation
  • Bouttier, F., and G. Kelly, 2001: Observing-system experiments in the ECMWF 4D-Var data assimilation system. Quart. J. Roy. Meteor. Soc., 127 , 14691488.

    • Search Google Scholar
    • Export Citation
  • Cardinali, C., 2009: Monitoring the observation impact on the short-range forecast. Quart. J. Roy. Meteor. Soc., 135 , 239258.

  • Dee, D. P., and A. M. da Silva, 2003: The choice of variable for atmospheric moisture analysis. Mon. Wea. Rev., 131 , 155171.

  • Devenyi, D., and S. G. Benjamin, 2003: A variational assimilation technique in a hybrid isentropic-sigma coordinate. Meteor. Atmos. Phys., 82 , 245257.

    • Search Google Scholar
    • Export Citation
  • Graham, R. J., S. R. Anderson, and M. J. Bader, 2000: The relative utility of current observations systems to global-scale NWP forecasts. Quart. J. Roy. Meteor. Soc., 126 , 24352460.

    • Search Google Scholar
    • Export Citation
  • Lord, S. J., T. Zapotocny, and J. Jung, 2004: Observing system experiments with NCEP’s Global Forecast System. Proc. Third WMO Workshop on Impact of Various Observing Systems on Numerical Weather Prediction, WMO/TD 1228, Alpbach, Austria, WMO, 56–62.

    • Search Google Scholar
    • Export Citation
  • Moninger, W. R., R. D. Mamrosh, and P. M. Pauley, 2003: Automated meteorological reports from commercial aircraft. Bull. Amer. Meteor. Soc., 84 , 203216.

    • Search Google Scholar
    • Export Citation
  • Moninger, W. R., S. G. Benjamin, B. D. Jamison, T. W. Schlatter, T. L. Smith, and E. J. Szoke, 2010: Evaluation of regional aircraft observations using TAMDAR. Wea. Forecasting, in press.

    • Search Google Scholar
    • Export Citation
  • Nutter, P., D. Stensrud, and M. Xue, 2004: Effects of coarsely resolved and temporally interpolated lateral boundary conditions on the dispersion of limited-area ensemble forecasts. Mon. Wea. Rev., 132 , 23582377.

    • Search Google Scholar
    • Export Citation
  • Schwartz, B., and S. Benjamin, 2004: Observation sensitivity experiments using the Rapid Update Cycle. Preprints, Eighth Symp. on Integrated Observing and Assimilation Systems for Atmosphere, Oceans, and Land Surface, Seattle, WA, Amer. Meteor. Soc., 6.3.

    • Search Google Scholar
    • Export Citation
  • Smith, T. L., S. G. Benjamin, S. I. Gutman, and S. Sahm, 2007: Short-range forecast impact from assimilation of GPS-IPW observations into the Rapid Update Cycle. Mon. Wea. Rev., 135 , 29142930.

    • Search Google Scholar
    • Export Citation
  • Weatherhead, E. C., and Coauthors, 1998: Factors affecting the detection of trends: Statistical considerations and applications to environmental data. J. Geophys. Res., 103 , 1714917161.

    • Search Google Scholar
    • Export Citation
  • Weygandt, S. S., and Coauthors, 2004: Potential forecast impacts from space-based lidar winds: Results from a regional observing system simulation experiments. Preprints, Eighth Symp. on Integrated Observing Systems, Seattle, WA, Amer. Meteor. Soc., 6.4.

    • Search Google Scholar
    • Export Citation
  • Weygandt, S. S., S. G. Benjamin, T. G. Smirnova, and J. M. Brown, 2008: Assimilation of radar reflectivity data using a diabatic digital filter within the Rapid Update Cycle. Preprints, 12th Conf. on IOAS-AOLS, New Orleans, LA, Amer. Meteor. Soc., 8.4.

    • Search Google Scholar
    • Export Citation
  • White, A. B., F. M. Ralph, J. R. Jordan, C. W. King, D. J. Gottas, P. J. Neiman, L. Bianco, and D. E. White, 2007: Expanding the NOAA Profiler Network: Technology evaluation and new applications for the coastal environment. Preprints, Seventh Conf. on Coastal Atmospheric and Oceanic Prediction and Processes, San Diego, CA, Amer. Meteor. Soc., 8.6.

    • Search Google Scholar
    • Export Citation
  • Zapotocny, T. H., W. P. Menzel, J. P. Nelson, and J. A. Jung, 2002: An impact study of five remotely sensed and five in situ data types in the Eta Data Assimilation System. Wea. Forecasting, 17 , 263285.

    • Search Google Scholar
    • Export Citation
  • Zapotocny, T. H., J. Jung, J. Le Marshall, and R. Treadon, 2007: A two-season impact study of satellite and in situ data in the NCEP Global Data Assimilation System. Wea. Forecasting, 22 , 887909.

    • Search Google Scholar
    • Export Citation
  • Zhu, Y., and R. Gelaro, 2008: Observation sensitivity calculations using the adjoint of the Gridpoint Statistical Interpolation (GSI) analysis system. Mon. Wea. Rev., 136 , 335351.

    • Search Google Scholar
    • Export Citation
  • View in gallery

    Surface analyses for (a) 1200 UTC 1 Dec 2006, in middle of winter experiment period, and (b) 1200 UTC 20 Aug 2007, in the middle of the summer experiment period (courtesy of NOAA/NCEP/HPC).

  • View in gallery

    Distribution of data sources. Sample data from 12 Nov 2009 valid within 1 h of 0000 UTC (data available in retrospective periods is similar, but that TAMDAR data are not shown here). Color coded as in the histogram plots below: (a) AMDAR; (b) profilers (national network plus cooperating agency profilers); (c) VAD, from NEXRAD radars; (d) raobs; (e) GPS; (f) AMV; (g) all surface (METARs plus mesonet); and (h) METAR, color coded by altitude.

  • View in gallery

    Midwest–Great Lakes (red rectangle) and national (gray area) verification regions. Also shown is the location of the NOAA profilers (blue rectangles) and verifying raobs (brown triangles).

  • View in gallery

    Differences in rms error (vs radiosonde) between observation denial experiments listed in Table 2 and control run for 1000–400 hPa for relative humidity (% RH) for national domain. Results for each of the eight observational denial experiments are coded with a different color (aircraft: red, profiler: blue, VAD: pink, raob: tan, surface: light blue, GPS-PW: green, mesonet: black, satellite AMV winds: brown). Graphs at top are for winter results; those at bottom are for summer. (left) The three adjacent columns for each OSE are for 3-, 6-, and 12-h forecasts. (right) The same results are organized by observation type for each forecast projection (3, 6, and 12 h). The impact of AMV data was not tested during the summer period. Statistical uncertainties are indicated for each observation denial experiment by narrow black lines showing ±1 standard error from the mean impact.

  • View in gallery

    As in Fig. 4, but for temperature error (K) for 1000–100-hPa layer over national domain.

  • View in gallery

    As in Fig. 4, but for wind vector difference (m s−1), still for 1000–100-hPa layer over national domain.

  • View in gallery

    Vector wind error (vs radiosondes) for RUC 9- (black), 3- (blue), and 1-h (red) forecasts averaged over 400–200-hPa layer and for a 30-day running mean for January 2007–April 2009. Also shown are the forecast error increments (between 0 and 1) from assimilation of recent observations for 9- to 1-h (red, near lower axis) and 3- to 1-h (tan) pairs.

  • View in gallery

    As in Fig. 6, but for 1000–800-hPa only (still wind, national domain).

  • View in gallery

    As in Fig. 8, but for 800–400-hPa layer (still wind, national domain).

  • View in gallery

    As in Fig. 8, but for 400–100-hPa layer (still winds for national domain).

  • View in gallery

    As in Fig. 4, but for the Midwest (or Great Lakes) regional verification domain.

  • View in gallery

    As in Fig. 5, but for the Midwest verification region.

  • View in gallery

    As in Fig. 12, but for the 1000–800-hPa layer only.

  • View in gallery

    As in Fig. 12, (temperature, Midwest domain), but for the 800–400-hPa layer.

  • View in gallery

    As in Fig. 6, but for the Midwest domain.

  • View in gallery

    As in Fig. 15, but for the 1000–800-hPa layer.

  • View in gallery

    As in Fig. 15, but for the 800–400-hPa layer.

  • View in gallery

    As in Fig. 15, but for the 400–100-hPa layer.

  • View in gallery

    Differences in 3-h vector wind errors between No-profile minus Control (N − C, blue line) and 8-km (quarter-scale) minus Control (8 − C, red line) over the Midwest region.

  • View in gallery

    As in Fig. 19, but for 6-h forecasts instead of 3-h forecasts.

  • View in gallery

    As in Fig. 20, but for 3-h wind forecast impact in the Midwest domain, but with results added for 12-km profilers. Green line (12 − C) is for difference between experiments with 12-km profilers (12) vs Control (C).

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 236 217 28
PDF Downloads 194 185 22

Relative Short-Range Forecast Impact from Aircraft, Profiler, Radiosonde, VAD, GPS-PW, METAR, and Mesonet Observations via the RUC Hourly Assimilation Cycle

View More View Less
  • 1 NOAA/Earth System Research Laboratory, Boulder, Colorado
© Get Permissions
Full access

Abstract

An assessment is presented on the relative forecast impact on the performance of a numerical weather prediction model from eight different observation data types: aircraft, profiler, radiosonde, velocity azimuth display (VAD), GPS-derived precipitable water, aviation routine weather report (METAR; surface), surface mesonet, and satellite-based atmospheric motion vectors. A series of observation sensitivity experiments was conducted using the Rapid Update Cycle (RUC) model/assimilation system in which various data sources were denied to assess the relative importance of the different data types for short-range (3–12 h) wind, temperature, and relative humidity forecasts at different vertical levels and near the surface. These experiments were conducted for two 10-day periods, one in November–December 2006 and one in August 2007. These experiments show positive short-range forecast impacts from most of the contributors to the heterogeneous observing system over the RUC domain. In particular, aircraft observations had the largest overall impact for forecasts initialized 3–6 h before 0000 or 1200 UTC, considered over the full depth (1000–100 hPa), followed by radiosonde observations, even though the latter are available only every 12 h. Profiler data (including at a hypothetical 8-km depth), GPS-precipitable water estimates, and surface observations also led to significant improvements in short-range forecast skill.

* Additional affiliation: Cooperative Institute for Research in the Atmosphere, Fort Collins, Colorado

+ Additional affiliation: Systems Research Group, Colorado Springs, Colorado

# Retired

Corresponding author address: Stanley G. Benjamin, NOAA/ESRL/GSD, 325 Broadway, R/GSD1, Boulder, CO 80304. Email: stan.benjamin@noaa.gov

Abstract

An assessment is presented on the relative forecast impact on the performance of a numerical weather prediction model from eight different observation data types: aircraft, profiler, radiosonde, velocity azimuth display (VAD), GPS-derived precipitable water, aviation routine weather report (METAR; surface), surface mesonet, and satellite-based atmospheric motion vectors. A series of observation sensitivity experiments was conducted using the Rapid Update Cycle (RUC) model/assimilation system in which various data sources were denied to assess the relative importance of the different data types for short-range (3–12 h) wind, temperature, and relative humidity forecasts at different vertical levels and near the surface. These experiments were conducted for two 10-day periods, one in November–December 2006 and one in August 2007. These experiments show positive short-range forecast impacts from most of the contributors to the heterogeneous observing system over the RUC domain. In particular, aircraft observations had the largest overall impact for forecasts initialized 3–6 h before 0000 or 1200 UTC, considered over the full depth (1000–100 hPa), followed by radiosonde observations, even though the latter are available only every 12 h. Profiler data (including at a hypothetical 8-km depth), GPS-precipitable water estimates, and surface observations also led to significant improvements in short-range forecast skill.

* Additional affiliation: Cooperative Institute for Research in the Atmosphere, Fort Collins, Colorado

+ Additional affiliation: Systems Research Group, Colorado Springs, Colorado

# Retired

Corresponding author address: Stanley G. Benjamin, NOAA/ESRL/GSD, 325 Broadway, R/GSD1, Boulder, CO 80304. Email: stan.benjamin@noaa.gov

1. Introduction

An increasing number of atmospheric observation systems are used to initialize operational numerical weather prediction (NWP) models. Observation system experiments (OSEs) have been found very useful in determining the impact of particular observation types on operational NWP systems (e.g., Graham et al. 2000; Bouttier and Kelly 2001; Zapotocny et al. 2002, 2007; Lord et al. 2004; Cardinali 2009). OSEs can provide a basis for decisions regarding the design and implementation of current and future observing systems.

Such studies have provided valuable guidance on relative expenditures for different observational systems where expansions of current limited-areal deployments for certain observing systems [e.g., the National Oceanic and Atmospheric Administration (NOAA) Profiler Network (NPN)] might be most helpful toward improved NWP guidance. As heterogeneity of the overall composite observing system increases and as data assimilation and modeling techniques are improved, new OSEs will be needed to evaluate these new configurations.

This study uses a commonly used OSE design, with different observation types being excluded from the data assimilation system for separate experiments to measure effects on subsequent NWP forecasts, with the control experiment using all available observation types. This study differs from adjoint-based observation sensitivity experiments (e.g., Cardinali 2009; Zhu and Gelaro 2008; Baker and Daley 2000, among others). As described by Cardinali (2009), the adjoint-based sensitivity method tests the impact of all observations from a given time (or a short time over which a tangent linear model is run). An OSE by comparison shows impact over a longer period and also requires a much larger number of experiments (a separate experiment for each denial of a given observation type or subset, as done here). In contrast to those experiments mentioned above, this OSE study is performed using a regional model/assimilation system, whereas those previous listed (except for Zapotocny et al. 2002) were performed using global systems. Finally, the OSE data denial approach used in this study also differs from a data addition approach using a baseline control with, for instance, radiosondes only and adding other observation types to this control, one at a time.

This new OSE study is also unique in that it considers the very short-range forecast (3–12 h) effects from most of the currently assimilated high-frequency observing systems in a 1-h assimilation cycle, the Rapid Update Cycle (RUC; Benjamin et al. 2004a), which runs at the highest assimilation frequency of operational NWP models operated at the NOAA/National Centers for Environmental Prediction (NCEP). RUC short-range forecasts are heavily used as guidance for aviation, severe weather, energy, and other applications, some applying automated decision support algorithms suitable for hourly-updated NWP systems. Therefore, consideration of observation impact on very short-range (1–12 h) forecasts is important in considering investment in these observation systems, both from regional and global perspectives.

The new study is similar to the previous observation impact experiments also using the RUC reported in Benjamin et al. (2004c), which consider only wind forecast impact from wind profilers over a 13-day winter period. This new impact study is much broader than the previous study; it is now for a greater number of observation types over both summer and winter experiment periods, and for three fields—wind, temperature, and moisture. Other previous work on effects of high-frequency (hourly) observations on short-range forecasts include those reported by Smith et al. (2007) for GPS precipitable water observations and Weygandt et al. (2004) for simulated lidar wind observations [a regional observing system simulation experiment (OSSE)].

The observation sensitivity experiments reported here were carried out using a 2007 version of the RUC, including both assimilation system and forecast model components. The observing systems considered in this study include seven primary wind/temperature observation types over the United States: radiosonde observations (raobs), aircraft (Moninger et al. 2003), aviation routine weather report (METAR; surface), mesonet (automated surface observations from non-METAR networks), wind profilers (Benjamin et al. 2004c), velocity azimuth display (VAD) vertical wind profiles from NOAA Weather Surveillance Radar-1988 Doppler (WSR-88D) radar radial winds, and satellite atmospheric motion vectors (AMVs, sometimes less precisely called cloud-drift winds). All these observing systems except radiosondes provide hourly data. This study also includes the primary tropospheric moisture observation types [radiosondes, GPS ground-based precipitable water (PW; Smith et al. 2007)]. Relative effects of METAR and mesonet surface observations are also considered. We do not consider effects of satellite-measured radiances or retrieved soundings from satellite radiances in this study (neither are assimilated in the RUC; Benjamin et al. 2004b, section 2). Finally, this study also includes the relative impact of actual 16-km (full depth) NOAA profilers versus hypothetical profilers with only an 8-km vertical range.

This paper accompanies a report by Moninger et al. (2010) that focuses on a multiyear data impact study specifically for the (Tropospheric Aircraft Meteorological Data and Recording) TAMDAR-based observations from regional commercial aircraft. The companion paper includes results from a TAMDAR-denial experiment to measure TAMDAR impact during the same test periods used in this paper.

2. RUC version used for OSEs

The version of the RUC used in these experiments employs the same code as the 13-km version run operationally at NCEP as of March 2007, including 50 hybrid isentropic-sigma vertical levels and model physical parameterizations as described by Benjamin et al. (2004b), including five-species mixed-phase bulk cloud microphysics, Grell–Devenyi convective parameterization, and RUC–Smirnova land surface model. For computational efficiency, these experiments were run at 20-km resolution with no other modifications except for this resolution modification via a single parameter. The hourly intermittent assimilation cycle in the RUC (Benjamin et al. 2004a) allows full use of hourly observational datasets. The analysis method is the three-dimensional variational (3DVAR) technique implemented in the operational RUC in May 2003 (Devenyi and Benjamin 2003; Benjamin et al. 2004a, section 4), but with subsequent improvements listed below.

The key RUC modifications used in these OSE experiments made since the version of the RUC described by Benjamin et al. (2004a,b) include the following:

  • Modification of moisture analysis variable from lnq (natural logarithm of water vapor mixing ratio) to pseudo relative humidity (pseudo-RH), defined as q/q-saturation-background (Dee and da Silva 2003). Assimilation of all integrated precipitable water observations [GPS-PW and the Geostationary Operational Environmental Satellite (GOES)] was applied to the RUC 3DVAR using a forward model for vertically integrated pseudo-RH with respect to precipitable water (Benjamin et al. 2004d). A small modification in moisture background error specification was made between the winter and summer seasons that did not appear to modify observation impact results (Moninger et al. 2010).

  • Assimilation of GPS precipitable water data added in 2005 (Smith et al. 2007).

  • Fractional application of lowest temperature analysis increment to top two levels in soil–vegetation–snow model used in RUC.

  • Assimilation of METAR ceiling and visibility observations modifying the 3D RUC 3D hydrometeor (five species) and 3D water vapor mixing ratio fields (Benjamin et al. 2004e).

  • Assimilation of pseudoresiduals for surface observations distributed within the planetary boundary layer (PBL) using the background (RUC 1-h forecast) PBL depth, using certain constraints (Benjamin et al. 2004f).

  • Extension of digital filter initialization (DFI) used in RUC model to a two-pass diabatic DFI.

Changes were also made in RUC model physics using the Thompson mixed-phase cloud microphysics and Grell–Devenyi convective parameterization as described by Benjamin et al. (2004f).

Observational data assimilated in the version of the RUC used in this OSE study are listed in Table 1. GOES-based cloud-top temperature/pressure retrievals, AMVs, and precipitable water are also assimilated in the RUC 1-h cycle.

3. Experiment design for observation impact experiments

A series of experiments was conducted using the RUC model/assimilation system in which various data sources were denied to assess relative importance of the different data types for short-range (3–12-h duration) wind, temperature, and relative humidity forecasts at different vertical levels. This assessment was carried out for 10-day periods in cold season (November–December 2006) and warm season (August 2007).

The same boundary conditions were used in all experiments, damping the signal in differences between experiments, more than might be expected in similar OSEs performed with global assimilation and models. The damping effect by lateral boundary conditions becomes stronger as the model/assimilation domain is reduced, and therefore is larger in this study using the RUC domain than that for the (North American Mesoscale) NAM-based (larger regional domain) described by Zapotocny et al. (2002). Nutter et al. (2004) also show a similar effect from lateral boundary conditions limiting the spread of regional ensemble forecasts; the same effect occurs in the OSEs described here limiting variation between experiments more than in global OSEs. In a regional OSSE study for simulated lidar wind observations, Weygandt et al. (2004) found the observational impact from simulated lidar winds interior to the RUC regional domain about equal to that from variations in lateral boundary conditions from associated global OSSE experiments with and without lidar. In this study, the observations considered are generally denser over the United States than over oceans and other land areas, but the actual impact is underestimated in this study because of the common lateral boundary conditions prescribed in these experiments.

a. Experiments performed

A control experiment was performed for each of two seasonal 10-day test periods in which all available observations were used, similar to the operational RUC. In subsequent experiments, different observation types were withheld, as shown in Table 2. Most of these observation types were available over the full RUC horizontal domain covering the lower 48 U.S. states and adjacent Canada and Mexico (approximately that shown in Fig. 1). As shown in Fig. 2, some of the observation types (profiler, TAMDAR aircraft) were available only in the midwestern United States, motivating us to also employ a verification subregion in that area, as discussed in the next section. In the RUC, GOES AMVs are assimilated only over oceanic areas, since aircraft data (generally of higher quality) are predominant over land area in the RUC domain. Impact experiments for AMVs, 8-km profilers, and 12-km profilers were performed only for the winter period (Table 2).

Lateral boundary conditions were specified from the NCEP NAM, initialized every 6 h and available with 3-h output frequency. NAM boundary conditions were specified in the same delayed manner as with the operational RUC: RUC model runs at 0000, 0600, 1200, and 1800 UTC use NAM boundary conditions from the previous NAM cycle (1800, 0000, 0600, and 1200 UTC, respectively).

The experiments for the winter and summer 10-day data assimilation periods are shown in Table 3. The November–December 2006 winter period was synoptically active in the northern United States, especially in the upper Midwest and Great Lakes area. An example of surface conditions during this period (1200 UTC 1 December 2006) is shown in Fig. 1, with a strong winter storm centered over Indiana. The 10-day summer experiment period spanned 15–25 August 2007, and was chosen because it included considerable intense weather in the Great Lakes region. The period started with a warm front producing heavy precipitation in that region; later, flooding occurred in Minnesota and Wisconsin. Severe storms continued to appear, and generally move toward the east, throughout the period.

b. Verification

We verified model forecasts against conventional, twice-daily radiosonde data over the two domains depicted in Fig. 3. The first domain contains all the radiosonde sites located within the RUC domain; the second (the red rectangle) is a limited area over the data-rich Midwest.

Verification results for the national region reflect the impact of observations over the full RUC domain, covering the lower 48 contiguous U.S. region and significant proportions of Canada and Mexico. The Midwest verification region shown in Fig. 3 has special interest because of the NOAA profiler network (marked in green) and TAMDAR aircraft coverage at that time (see Moninger et al. 2010 for TAMDAR coverage). With the United States considering expenditures for wider deployment of profilers and regional aircraft observations, the Midwest verification domain corresponds to the density that might be expected nationally over the next few to several years.

For each RUC experiment, residuals [forecast minus observed ( fo) differences] for temperature (T), relative humidity (RH), and wind (V) were computed at all radiosonde locations located within each verification domain. These fo residuals were calculated for 3-, 6-, 9-, and 12-h forecasts. The rms (root-mean-square) difference between forecasts and observations was computed for each 12-h radiosonde verification time (0000 and 1200 UTC). This difference is sometimes referred to below as the “forecast error,” or “RMSE,” but in fact also contains a contribution from the observation error (including a representativeness “error” from the inability of a grid to resolve subgrid variations sometimes evident in observations).

In the following results, increase in forecast error from denying a given observation type can be considered equivalent to the added forecast skill when that observation type is added to other existing observations. Benjamin et al. (2004c) explain this verification procedure.

Verification in this paper uses 10-hPa vertical resolution, including significant-level radiosonde observations and native model levels interpolated to that resolution, for calculating fo differences using a verification capability explained in Moninger et al. (2010). This high vertical resolution of forecast errors allows clearer attribution of differences to physical mechanisms than verifying against radiosonde data only at mandatory levels (e.g., 850, 700, 500, 400, 300, 250, 200, and 150 hPa). For example, higher vertical resolution in verification revealed a peak near 900 hPa in temperature forecast error and aircraft impact at that level, subsequently related to boundary layer depth as described by Moninger et al. (2010). The 10-hPa verification also increases the number of fo data points over what would have been available with mandatory level raob data only, numbering about 5200 for a 200-hPa layer in the Midwest domain (10 days × 2 times/day × 13 raob sites × 20 vertical points) and about 32 000 fo points for the national domain (80 raobs), increasing significance of results shown later.

For quality control of radiosonde data used in verification, fo values from the control experiment were subjectively screened for egregiously large values and removed when found. While some erroneous values may have escaped detection, they were used uniformly in verifying all experiments and therefore do not contribute to the relative impact results shown below.

We looked for impact on precipitation forecasts in control versus denial experiments for the two observation types most likely to show them, GPS-PW, and TAMDAR aircraft observations, and found only negligible effect. Assimilation of radar reflectivity data, by contrast, has shown a strong effect on RUC precipitation forecasts (Weygandt et al. 2008).

c. Statistical significance of results

Results that follow present differences in rms forecast error (RMSE) for model runs with and without specific observation types. Each estimate has an associated uncertainty due, in part, to the small number of days we examined. We present overall RMSE differences for each period (winter and summer), but we can estimate the uncertainty in these RMSE differences by considering the variations in RMSE over each of the 20 raob times in each seasonal period. The uncertainty on the mean (“standard error”) is estimated as
i1520-0493-138-4-1319-eq1
where σ is the standard deviation, n is the number of RMSE forecast values, x is the set of RMSE forecast differences, and φ is the lag 1 autocorrelation derived from the time series x. This is empirically derived from the RMSE values with the following approximation:
i1520-0493-138-4-1319-eq2
The estimate of the standard error on the mean is distinct and separate from the standard deviation from the sample. The standard error is an estimate of how well we understand the underlying, fundamental differences in RMSE between using the additional data and ignoring the additional data. The standard deviation is an estimate of how far off the mean value any one RMSE forecast may be (Weatherhead et al. 1998). Thus, were our experiment to be repeated in a similar season and for a similar duration, we can say that the mean RMSE difference has a 67% likelihood of being within 1 standard error of our results, and a 95% likelihood of being within 2 standard errors of our results.

This approach at least partially accommodates the fact that the pairwise differences in RMSE are autocorrelated. Physically, this implies that in some situations, the added data have more influence than others. Those situations can last for more than one day, thus the sequential forecast RMSE differences are not independent estimates of the effect of the added data, but represent an oversampling of the system. The standard error equation above accounts for these. It should be noted that the lag 1 autoregressive assumption in this case refers to a 12-h lag as the most significant approximation to the autocorrelation. It should also be noted that aggregation of 0000 and 1200 UTC results allows for a larger sample size, but may result in combining different physical causes of differences as well as different statistical properties of the time series. These more finescale effects are beyond the scope of the research presented in this paper.

In the figures that follow, standard errors are indicated where relevant. Differences of 1 standard error are significant at the 67% confidence level; differences of 2 standard errors are significant at the 95% confidence level.

d. Procedure for 8-km (quarter scale) versus full-scale profiler experiments

For these experiments, we extracted 8-km (quarter scale) and 12-km profiler data from actual 16-km full-scale profiler data by removing data for all vertical gates higher than 8 km (or 12 km) above station elevation. The hypothetical 8-km profilers provide half the vertical coverage of wind observations compared with the full-scale 404-MHz profilers and approximate data from proposed 8-km 449-MHz profilers. With the 16-km full-scale profilers, winds are available from 36 high-mode gates and 36 low-mode gates, with a slight overlap near 8-km elevation above ground level (AGL). We extracted quarter-scale (8 km) and 12-km profiler data at the 30 profiler sites shown in Fig. 3. In removing winds above 8 km AGL, data were left for 33 low-mode gates, and 5 high-mode gates. We assumed that the observation error in these hypothetical future profilers would be the same as that used for the existing 16-km profilers.

The profiler stations assimilated by the RUC in these experiments included about 21 Cooperative Agency Profiler (CAP) sites operating at 915 MHz with a vertical range of about 4 km, and these data were not truncated in 8- and 12-km profiler experiments. These CAP profilers include 12 in California, 4 in Texas, and 1 each in New Mexico, Arizona, Minnesota, New Jersey, and Nova Scotia), as shown by White et al. (2007, their Fig. 3).

4. Results for impact from existing observations

a. Stratification

To summarize the complexity of the OSE results from this study, we considered the five verification stratifications: experiment, regions, layers, seasons, and forecast duration, as shown in Table 4. Rather than examine detailed vertical profiles of forecast errors (e.g., Benjamin et al. 2004c; Moninger et al. 2010), we found it effective to break down the full 1000–100-hPa vertical domain into three layers: 1000–800 hPa (dominated by boundary layer and surface effects), 800–400 hPa (middle troposphere), and 400–100 hPa (upper troposphere to lower stratosphere including tropopause and upper-level jet maxima).

We developed a composite graphical format used throughout the rest of this section that is introduced in Fig. 4 to summarize results for all OSE experiments for a given domain (national or Midwest) and vertical layer (1000–100, 1000–800, 800–400, or 400–100 hPa). Because of known raob moisture sensor limitations above approximately 400 hPa, where temperatures are commonly below −30°C, we used 1000–400 hPa for the “full troposphere” results for RH.

b. National, full troposphere

We begin with the broadest view by examining results on the national domain for vertically integrated layers: 1000–100 hPa for temperature and wind, and 1000–400 hPa for RH. In section 4c, we shall show stratifications over different vertical layers, and in section 4d, results specifically from the Midwest region where observations are dense.

In the first graphical composite (Fig. 4), we consider impact results for RH for 1000–400 hPa. Results are for differences between experiments in which various observation types were withheld, as well as the control experiment in which all observations were assimilated (similar to the operational RUC). We use different colors to depict results for each of the eight observation denial experiments. Results in the top graphs are for winter; those in the bottom two boxes are for summer. For graphs at left, the three adjacent columns for each OSE are for 3-, 6-, and 12-h forecasts, respectively. The graphs on the right show the same information as those on the left, but organized by forecast projection to allow easier interpretation from that perspective. Again, as stated in the last section, increase in forecast error from denying a given observation type can be considered equivalent to the added forecast skill (“forecast impact”) when that observation type is added to other existing observations.

The black bars indicate ± one standard error (section 3c) from the forecast impact of each observation type. Differences of 1 standard error are significant at the 67% confidence level; differences of 2 standard errors are significant at the 95% confidence level.

For RH over the 1000–400-hPa layer for 3–12-h RUC forecasts over the full domain (Fig. 4), the observation type with the largest impact is clearly raobs, for which the impact is 1%–2% RH for all forecast durations (3, 6, and 12 h) in both summer and winter. GPS-PW (Smith et al. 2007) had the second largest impact especially in winter (0.6%–0.9% for 3- and 6-h forecasts). The third most important observation source is aircraft in winter [<0.5%, presumably primarily from TAMDAR reports (including moisture) in the Midwest] and surface observations in summer. In the summer period, each of the five observation types that provide moisture observations are shown to have varying degrees of at least small positive impacts on the short-range RUC RH forecasts over the full national domain.

The impact of raobs at 12 h on RH forecasts is large, sometimes even larger than that at 3 and 6 h. This is so because 12-h forecasts valid at 0000 and 1200 UTC have the direct benefit of raob data in the initial conditions whereas 3-h and 6-h forecasts do not. Even so, the impact at 3 and 6 h is substantial. We attribute this to the “memory” in the assimilation system of raobs incorporated several cycles before the start of these forecasts.

For temperature forecasts over the full (1000–100 hPa) atmospheric depth (Fig. 5), in winter, on average, raobs and aircraft observations had about equal effect (0.05–0.15 K) on average over 3–6-h impact, more from aircraft at 3 h, equal at 6 h, and much more impact from radiosondes at 12 h. In summer, surface observations have nearly equal impact as both aircraft and raobs over the full 1000–100 hPa. The explanation is that a deeper mixed layer in summer extends the potential vertical influence of surface observations. This PBL effect is accounted for in the RUC 3DVAR design, as discussed in section 2 and in Benjamin et al. (2004f).

For the vector wind difference (Fig. 6), aircraft observations have the strongest overall impact for 3- and 6-h forecast projections for both for the summer (0.3–0.6 m s−1) and winter periods (0.15–0.20 m s−1) and for 12-h forecasts in the summer season. Radiosondes have the greatest impact for winds at 12 h in winter only. Satellite AMVs provide a small positive impact (<0.05 m s−1) at 12 h in winter, in third place after radiosondes and aircraft. All observation types tested showed at least a small positive impact except for VAD winds in summer at 12-h duration (perhaps due to bird migration problems not detected by the RUC bird detection algorithm; Benjamin et al. 2004a, their section 4e) and mesonet observations, which frequently have unrepresentative siting for wind measurement (Benjamin et al. 2007).

Our results indicate that aircraft observation impact (Fig. 6) was stronger in summer (0.3–0.6 m s−1 over full layer) than winter (0.15–0.20 m s−1), which was surprising to us since upper-level wind forecast errors are usually larger in winter than in summer over the United States. To examine this behavior a bit further, we first looked at seasonal variations of upper-level wind (400–200 hPa) forecast error for the RUC at 9-, 3-, and 1-h forecast duration (Fig. 7, 30-day running mean) over a period from January 2007 to May 2009. For RUC 9-h wind forecasts for the 400–200-hPa layer, error (versus raobs) was about 5.8–6.0 m s−1 in winter for 2007–09 and lower, about 5.2–5.3 m s−1 on average, in summer for 2007–08. However, the short-range increment in forecast skill (e.g., 9- to 1-h forecast skill difference, bottom in Fig. 7), largely from assimilation of recent observations (Benjamin et al. 2004a), does not vary drastically over season, although the 30-day running mean show some apparent shorter-period regime-dependent variations. Therefore, we consider the larger aircraft impact for wind forecasts in the summer August 2007 period than in winter to be slightly unusual but plausible, consistent with the particular synoptic-scale regimes of those separate 10-day periods.

c. National, but stratified into three layers (for wind only)

Next, we stratify the OSE results within three layers as described in section 4a, 1000–800 (near surface), 800–400 (midtroposphere), and 400–100 hPa. We start with the lower tropospheric (1000–800 hPa) layer for wind forecasts (Fig. 8).

For the lower-tropospheric 1000–800-hPa layer (Fig. 8), aircraft, VAD, and surface observations have about equal impact for 3-h wind forecasts in winter, when the PBL is typically shallow and inversions are common. In summer, surface observations have the most impact. We attribute this to deeper PBL mixing and the addition of PBL-depth pseudoresiduals for surface observations in the RUC 3D assimilation (discussed in section 2 and Benjamin et al. 2004f). Mesonet observations were found to add little or no impact to 3- or 6-h lower-tropospheric wind forecasts to other surface observations (primarily METARs) even in summer when stronger effects are shown from surface observations, but have a very small positive effect at 12 h in summer and winter. The increasing impact of aircraft observations with forecast projection in summer may result from better midtropospheric winds (next section) that are mixed down over time in the typically deeper PBL.

For midtroposphere winds over the national verification domain (Fig. 9), aircraft observations had the strongest impact overall, especially in summer (0.25–0.40 m s−1), followed by raobs. Raobs had the strongest impact for 12-h forecasts in winter for midtropospheric winds. Profilers, VAD winds, and AMVs all have a small positive effect for midtropospheric winds. The slight positive impact from GOES AMVs shows a slight increase with forecast projection as its offshore effect (assimilated only over water) propagates inland.

Figure 10 indicates that aircraft have a pronounced impact on upper-level wind forecast accuracy in the RUC domain for all forecast projections and both seasons, consistently larger than that for any other observation type, ∼0.3 m s−1 in this particular winter period and 0.8–0.9 m s−1 in the summer period for 3- and 6-h forecasts. Hourly automated aircraft reports over the United States were the original primary justification for the development and implementation of a rapidly updated data assimilation cycle to improve short-range upper-level wind forecasts (Benjamin et al. 1991). The results depicted in Fig. 10 are still consistent with that justification, despite the addition of many other observation types since 1991. Raob data had the second largest impact on upper-level wind forecasts over the national verification domain, with the profiler also making a very small positive impact over this larger domain. As with the 800–400-hPa layer, AMVs (“cloud drift” winds) had a small but positive effect on upper-level winds, larger at 12 h than at 3 h, again a consequence of their assimilation in RUC only over oceanic regions.

d. Midwest (very data rich area)

The Midwest region has exceptional upper-air observational coverage, denser than any other area in the United States because of the region’s proximity to the NOAA profiler network, and the initial deployment of TAMDAR sensors on regional aircraft in this area (see Moninger et al. 2010). Therefore, we considered it useful to examine the relative impact of different observation types for short-range RUC forecasts specifically in this region.

1) Relative humidity impact in Midwest region

In the Midwest verification region, we start again with the overall observational impact on relative humidity forecasts starting with the 1000–400-hPa layer (Fig. 11). Here, radiosondes still show the largest impact in winter (1%–2% RH), but with nearly equal impact from aircraft observations in summer (all forecast projections) and in winter 3-h forecasts. The availability of aircraft-based moisture observations from TAMDAR clearly contributed strongly in this region, comparing Fig. 11 with corresponding RH impact for the national domain (Fig. 4) showing much less aircraft impact. OSE results (control–noTAMDAR) in Moninger et al. (2010) confirm its very large impact, averaging about 2% for the 1000–400-hPa layer in both the November–December 2006 winter period and from fall 2008 onward. The RH impact from GPS-PW observations followed closely that from aircraft data in both summer and winter test periods. Even profilers made a positive contribution to RH forecasts (0.2%–0.5%) although they do not measure moisture, presumably because of improved vertical motion and horizontal transport fields.

2) Temperature forecast impact over the Midwest domain

For temperature impact over the full depth (1000–100 hPa) in the Midwest region (Fig. 12), results were similar to those shown in Fig. 5 for the national domain, with aircraft showing the greatest impact for 3- and 6-h temperature forecasts in both winter and summer. However, the average impact from assimilation of aircraft observations in winter in 3-h temperature forecasts was significantly stronger in the Midwest (0.25 K) than over the full national domain (0.15 K), likely due to the higher density of aircraft data in this region.

Regarding temperature forecasts in the lower troposphere (1000–800 hPa, Fig. 13), aircraft reports have the strongest impact (0.3–0.56 K) by far in winter for 3–6-h forecasts. In summer, the aircraft data have a slightly but significantly larger impact (0.12–0.20 K) than surface observations. The extra spatial density provided by TAMDAR aircraft observations, which include data from frequent ascents and descents into regional airports (Moninger et al. 2010), contributes to the forecast impact for lower-tropospheric temperatures, and especially during periods of wintertime lower-tropospheric temperature inversions.

Aircraft observations also exhibit the largest impact in the midtroposphere for both winter and summer seasons (Fig. 14), although the impact for all observation types is quite low in summer (<0.1 K for all types, all forecast projections), presumably due to a relatively low thermal stability and a general absence of midlevel frontal zones.

3) Wind forecast impact over the Midwest domain

The observation impact results for winds integrated over the full 1000–100-hPa layer within the data-rich Midwest domain is shown in Fig. 15, indicating a nearly equal impact for aircraft and profiler observations in winter at 3 and 6 h (0.13–0.20 m s−1). In summer, aircraft observations had the greatest overall impact at 3 h (0.35 m s−1) and 6 h (0.25 m s−1), followed by profiler, surface, and raobs, all with about the same effect. Note that surface observations have such a large effect on the fully integrated 1000–100-hPa layer, again indicating their representativeness in the deeper summertime boundary layer and the effectiveness of the RUC PBL-based pseudoresidual assimilation technique. The addition of mesonet observations, by contrast, had a slight negative effect on 1000–100-hPa 3-h wind forecasts in both winter and summer, again presumably due to widespread siting issues.

For lower-tropospheric (1000–800 hPa) wind forecasts in the Midwest domain (Fig. 16), it is not surprising that surface observations had the largest positive effect on both 3- and 6-h forecasts in both winter and summer periods. After surface observations, the largest effects in lower-tropospheric 3-h wind forecasts were from aircraft in winter, and from profiler in summer. As with the national domain (Fig. 9), aircraft data have very little effect in summer near-surface (1000–800 hPa) winds for 3- and 6-h duration, but have the largest effect by 12 h. VAD wind observations had the third-largest impact at 3 h in summer and for 3–6 h in winter.

The lowest gate in NOAA profilers is 500 m AGL, and sites within the Midwest verification domain are at 170–300-m elevation, limiting the profiler impact below 800 hPa, and perhaps contributing to negligible impact in winter in this layer. Over a different regional domain centered directly on the NOAA profiler network but using a similar RUC observation impact strategy for a profiler-only OSE with a 14-day test period in February 2001, Benjamin et al. (2004c) showed a larger 0.3 m s−1 impact from assimilation of profiler winds for 3-h wind forecasts at 850 hPa and a 0.1 m s−1 impact over a larger eastern U.S. verification domain. The smaller impact in this study is attributed to a shift in the Midwest domain not centered on the NPN (Fig. 3) and averaging over the 1000–800-hPa layer, essentially down to the surface.

For midtropospheric (800–400 hPa; Fig. 17) wind forecasts in the data-rich Midwest verification domain, aircraft, followed by profiler observations had the greatest impact. Clearly, these two observation types in the Midwest are not redundant, but together produce a larger reduction in forecast error. For upper-level winds (400–100 hPa; Fig. 18), profiler observations had the largest positive impact (reduction in forecast error) in winter at 3 and 6 h, followed by aircraft observations in winter. In the summer period, the opposite was true, with aircraft showing the greatest effect (e.g., >0.5 m s−1 at 3 h), followed by radiosondes and profilers at 3 h. At 6 h, aircraft had the most impact, with profiler and raobs second. Profiler impact for the NPN-centered verification domain shown by Benjamin et al. (2004c) from the February 2001 period was approximately 0.5 m s−1 for the 800–400-hPa layer for 3-h forecasts and about 0.1 m s−1 for 12-h forecasts. We are unable to explain the small negative impact at 12 h from aircraft in the Midwest layer for the 400–100-hPa layer evident in both winter and summer periods, except that it may be a sampling anomaly. The dropoff of profiler impact with projection time is attributed to propagation of that impact quickly outside of the limited extent of the NOAA profiler network (Fig. 3). A similar dropoff with time was shown in Benjamin et al. (2004c) for profiler impact.

5. Results from profiler height experiments—Impact from vertically truncated profiler heights

We added two additional experiments (Table 2), involving hypothetical 8- and 12-km profilers, referring to the vertical reach of the profiler antenna. The additional experiments give us a quantitative measure of the impacts that potential reduction in the vertical reach of existing NOAA network profilers will have on forecast accuracy. The operating frequency of network profilers must soon be changed from 404 to 449 MHz. Larger (and more expensive) antennas are required to reach 16 km than to reach lower altitudes. The experiment also relates to the cost of a possible expansion of the current network from the mid–United States to the entire lower 48 states.

The 8-km profilers have half the vertical reach of the full-scale (16 km) network profilers. The former are often called “quarter-scale profilers” because their antennas occupy only one quarter of the area. To manufacture data from quarter-scale wind profilers, we merely extracted actual data (surface to 8 km AGL) from the full-scale profilers.

The results for these experiments are depicted in Figs. 19 –21 for both 3- and 6-h forecast projections in the Midwest region for the 10-day winter period (Table 2). The vector wind error difference, No-Profiler minus Control (N-C, blue line in figures), shows the impact of profiler data themselves (equivalent to degradation if profiler data are missing). When profiler data are denied for the 10-day test period, 3-h forecasts of winds aloft (Fig. 19) from 600 to 300 hPa are worse by ∼0.4 m s−1 in the Midwest region and by 0.2–0.3 m s−1 for the same layer for 6-h forecasts (Fig. 20). The greatest improvement from the inclusion of profiler winds for 3-h wind forecasts was ∼0.55 m s−1 at 350 hPa (Fig. 19). These results are similar to those shown by Benjamin et al. (2004c) in a previous profiler impact study for the profiler (Midwest) domain and downscale domains, respectively.

The vector wind error difference, Quarter-scale minus Control (Q − C, red line in Figs. 19 and 20), shows the value of using 16-km full-scale profilers versus 8-km quarter-scale profilers. The Q − C difference is near zero at most altitudes, indicating that most of the value added to tropospheric wind forecasts from full-scale profilers is also added by quarter-scale profilers up to the jet levels where aircraft data are plentiful. Only at 200 hPa and above do full-scale profilers show value added (more accurate 3-h forecasts) that are not available with 8-km profilers. Quarter-scale profilers actually delivered somewhat better 3-h wind forecasts in the 900–750-hPa layer than full-scale profilers, possibly resulting from less lower-tropospheric geostrophic wind adjustment without the stratospheric (200 hPa and above) wind observations available only with 16-km profilers.

The final experiment was performed to simulate the inclusion of 12-km profilers, extracted from the actual 16-km profiler data. The results for 12-km profilers in Fig. 21 are very similar to those for the 8-km (quarter scale) profilers up to the 250-hPa level. However, as might be expected, the 12-km profilers do add forecast skill improvement over 8-km profilers for 3-h wind forecasts for the 150–200-hPa layer. The 12-km profilers do not capture the extra improvement in the 50–100-hPa layer available from the full 16-km profilers.

6. Conclusions

We performed extensive observation system experiments (OSEs) involving data denial for two 10-day periods, one in winter and one in summer, using the hourly-updating Rapid Update Cycle model/assimilation system. We examined forecast impact for relative humidity, temperature, and wind at 3, 6, 9, and 12 h.

We conclude from these experiments that the heterogeneous atmospheric observing system in the United States is effective for short-range (3–12 h) 1000–100-hPa forecasts for all three variables studied: relative humidity, temperature, and wind.

Overall, aircraft data were found to have the most impact on reducing error in short-range forecasts over the United States from the lower stratosphere down to the surface, but they are strongly and necessarily augmented by other observing systems. As shown by Moninger et al. (2010) in a companion article, TAMDAR aircraft observations (also including moisture) clearly improved forecast accuracy in the Midwest and eastern U.S. area when added to all other observations in a complementary experiment to those shown in this paper.

Radiosonde observations were second in importance overall, within the parameters defining this OSE for 3–6 h, and arguably most important for 12-h forecast impact on the national scale followed closely by aircraft. GPS-PW, surface, profiler, and VAD all provided value added to forecast accuracy, in roughly that order. GPS-PW was similar to raob contributions for short-range RH forecasts. Given that surface observations showed a significant additional value to lower-tropospheric forecasts, especially for the 1000–800-hPa layer and in summertime, we conclude that the RUC assimilation and use of PBL depth for pseudoresiduals is effective for 3D assimilation of these surface variables. The impact of profiler wind data was notably higher in the Midwest domain, where the NOAA network is located, than in the national domain, where their effect is heavily diluted. The relatively small impact from AMVs (used only over ocean areas) is attributable to the relatively small extent of the RUC domain over oceanic areas, limiting the possible AMV-related effect. Generally, the relative impact for profiler, aircraft, and raobs in this experiment was similar to that shown by Schwartz and Benjamin (2004) for an OSE using an earlier version of the RUC for a February 2001 test period. Midtroposphere wind forecast impact from profiler data in that earlier study was larger with a verification domain centered directly over the NOAA profiler network than in this study for the Midwest verification domain, shifted from the NPN area.

Experiments using hypothetical vertically truncated profiler data (with a vertical reach of 8- and 12-km AGL) were performed for the winter (November–December 2006) period. These experiments showed that 8-km (quarter scale) profilers provide 3- and 6-h wind forecast improvement about equal to that from full-scale (16 km) profilers from the surface up to 250 hPa, suggesting that 8-km profilers would complement aircraft data for short-range tropospheric forecasts.

We note once again that the magnitude of forecast impacts from different observation denial experiments is damped by the same lateral boundary conditions used in all experiments for the regional RUC domain. The 10-day periods used in this study for winter and summer seasons are barely long enough for robust results, but were limited by the logistics for the 1-h update cycle environment (unique to this study) and available computing and storage resources. These limitations were partially mitigated by performing verification every 10-hPa using full significant level radiosonde data, adding considerable data points. Standard error calculations for each experiment indicate that, even for our relatively short 10-day summer and winter periods, results are statistically robust, with many forecast impacts being significant at more than the 95% confidence level.

The observation impact results in this study often showed a decrease with forecast projection (see the example in Fig. 15). This was evident, in general, for aircraft and profiler data, for which this effect was enhanced by regional concentrations of those observations (aircraft enhanced by TAMDAR in Midwest area, profiler primarily in the NOAA profiler network). The impact of raob data was a prominent exception, showing an apparent increase with time, a statistical quirk from our verification only at 0000 and 1200 UTC for forecasts valid at those times, initialized at 0900 and 2100 for 3-h forecasts, for instance. Of course, in general, raobs are available only every 12 h, so their impact on the analyses that create the less-than-12-h forecasts valid at 0000 and 1200 UTC is indirect (only through the hourly-cycled background field) and degrades as the analysis time moves away from 0000 and 1200 UTC. Also, the overall impact of high-frequency observations is somewhat larger at analysis times when not competing with raobs.

Conducting OSEs can sometimes reveal flaws in the assimilation system from forward models or observation-error specification. In this study, relatively consistent positive (sometimes very small) or near-zero impacts were shown for nearly all observation types, presumably indicating no major flaws in treatment in the RUC for any observation types. But in initial experiments performed for this OSE, some counterintuitive results arose, leading to detection of assimilation design flaws for aircraft moisture observation error, moisture assimilation design, and too-small observation errors specified for radiosonde RH and wind observations. The results shown in this paper are dependent, for better or for worse, on the design of the RUC 3D variational analysis and modeling system as described in section 2, and we cannot rule out remaining design flaws or outright errors.

This OSE study included vertically stratified results for the data-rich Midwest verification domain. Even here, nearly all observation types contributed positive impact, with clear, positive, and complementary effects from profiler and aircraft data, indicating that this region is not oversampled by observations. A strong positive effect from surface observations over surprisingly deep layers was shown, especially but not solely in summer, for temperature, wind, and RH, but very little positive impact was shown when mesonet observations were added to METAR observations.

We intend to add new observation impact experiments using high-frequency assimilation of radar reflectivity (Weygandt et al. 2008), added to the operational RUC at NCEP in November 2008, and using hydrometeor assimilation from GOES and METAR cloud/visibility data (Benjamin et al. 2004e). We also intend to identify diurnal variations in observation impact (1200 versus 0000 UTC) and repeat similar OSEs with the upcoming Rapid Refresh soon replacing the RUC hourly assimilation/model cycle at NCEP.

Acknowledgments

We thank Betsy Weatherhead of NOAA/ESRL/GSD for her thoughtful and effective analysis of the statistical significance of our results. We also thank Annie Reiser and Dezso Devenyi of NOAA/ESRL/GSD for their excellent reviews of this manuscript.

REFERENCES

  • Baker, N. L., and R. Daley, 2000: Observation and background adjoint sensitivity in the adaptive observation-targeting problem. Quart. J. Roy. Meteor. Soc., 126 , 14311454.

    • Search Google Scholar
    • Export Citation
  • Benjamin, S. G., K. A. Brewster, R. L. Brummer, B. F. Jewett, T. W. Schlatter, T. L. Smith, and P. A. Stamus, 1991: An isentropic 3-hourly data assimilation system using ACARS aircraft observations. Mon. Wea. Rev., 119 , 888906.

    • Search Google Scholar
    • Export Citation
  • Benjamin, S. G., and Coauthors, 2004a: An hourly assimilation/forecast cycle: The RUC. Mon. Wea. Rev., 132 , 495518.

  • Benjamin, S. G., G. A. Grell, J. M. Brown, T. G. Smirnova, and R. Bleck, 2004b: Mesoscale weather prediction with the RUC hybrid isentropic/terrain-following coordinate model. Mon. Wea. Rev., 132 , 473494.

    • Search Google Scholar
    • Export Citation
  • Benjamin, S. G., B. E. Schwartz, E. J. Szoke, and S. E. Koch, 2004c: The value of wind profiler data in U.S. weather forecasting. Bull. Amer. Meteor. Soc., 85 , 18711886.

    • Search Google Scholar
    • Export Citation
  • Benjamin, S. G., and Coauthors, 2004d: A 13-km RUC and beyond: Recent developments and future plans. Preprints, 11th Conf. on Aviation, Range, and Aerospace, Hyannis, MA, Amer. Meteor. Soc., J1.6.

    • Search Google Scholar
    • Export Citation
  • Benjamin, S. G., J. M. Brown, S. S. Weygandt, T. L. Smith, B. Schwartz, and W. R. Moninger, 2004e: Assimilation of surface cloud, visibility, and current weather observations in the RUC. Preprints, 16th Conf. on Numerical Weather Prediction, Seattle, WA, Amer. Meteor. Soc., P1.47.

    • Search Google Scholar
    • Export Citation
  • Benjamin, S. G., S. Weygandt, D. Devenyi, J. M. Brown, G. Manikin, T. L. Smith, and T. Smirnova, 2004f: Improved moisture and PBL initialization in the RUC using METAR data. Preprints, 11th Conf. on Aviation, Range, and Aerospace, Hyannis, MA, Amer. Meteor. Soc., 17.3.

    • Search Google Scholar
    • Export Citation
  • Benjamin, S. G., W. R. Moninger, S. R. Sahm, and T. L. Smith, 2007: Mesonet wind quality monitoring allowing assimilation in the RUC and other NCEP models. Preprints, 22nd Conf. on Weather Analysis and Forecasting/18th Conf. on Numerical Weather Prediction, Park City, UT, Amer. Meteor. Soc., P1.33.

    • Search Google Scholar
    • Export Citation
  • Bouttier, F., and G. Kelly, 2001: Observing-system experiments in the ECMWF 4D-Var data assimilation system. Quart. J. Roy. Meteor. Soc., 127 , 14691488.

    • Search Google Scholar
    • Export Citation
  • Cardinali, C., 2009: Monitoring the observation impact on the short-range forecast. Quart. J. Roy. Meteor. Soc., 135 , 239258.

  • Dee, D. P., and A. M. da Silva, 2003: The choice of variable for atmospheric moisture analysis. Mon. Wea. Rev., 131 , 155171.

  • Devenyi, D., and S. G. Benjamin, 2003: A variational assimilation technique in a hybrid isentropic-sigma coordinate. Meteor. Atmos. Phys., 82 , 245257.

    • Search Google Scholar
    • Export Citation
  • Graham, R. J., S. R. Anderson, and M. J. Bader, 2000: The relative utility of current observations systems to global-scale NWP forecasts. Quart. J. Roy. Meteor. Soc., 126 , 24352460.

    • Search Google Scholar
    • Export Citation
  • Lord, S. J., T. Zapotocny, and J. Jung, 2004: Observing system experiments with NCEP’s Global Forecast System. Proc. Third WMO Workshop on Impact of Various Observing Systems on Numerical Weather Prediction, WMO/TD 1228, Alpbach, Austria, WMO, 56–62.

    • Search Google Scholar
    • Export Citation
  • Moninger, W. R., R. D. Mamrosh, and P. M. Pauley, 2003: Automated meteorological reports from commercial aircraft. Bull. Amer. Meteor. Soc., 84 , 203216.

    • Search Google Scholar
    • Export Citation
  • Moninger, W. R., S. G. Benjamin, B. D. Jamison, T. W. Schlatter, T. L. Smith, and E. J. Szoke, 2010: Evaluation of regional aircraft observations using TAMDAR. Wea. Forecasting, in press.

    • Search Google Scholar
    • Export Citation
  • Nutter, P., D. Stensrud, and M. Xue, 2004: Effects of coarsely resolved and temporally interpolated lateral boundary conditions on the dispersion of limited-area ensemble forecasts. Mon. Wea. Rev., 132 , 23582377.

    • Search Google Scholar
    • Export Citation
  • Schwartz, B., and S. Benjamin, 2004: Observation sensitivity experiments using the Rapid Update Cycle. Preprints, Eighth Symp. on Integrated Observing and Assimilation Systems for Atmosphere, Oceans, and Land Surface, Seattle, WA, Amer. Meteor. Soc., 6.3.

    • Search Google Scholar
    • Export Citation
  • Smith, T. L., S. G. Benjamin, S. I. Gutman, and S. Sahm, 2007: Short-range forecast impact from assimilation of GPS-IPW observations into the Rapid Update Cycle. Mon. Wea. Rev., 135 , 29142930.

    • Search Google Scholar
    • Export Citation
  • Weatherhead, E. C., and Coauthors, 1998: Factors affecting the detection of trends: Statistical considerations and applications to environmental data. J. Geophys. Res., 103 , 1714917161.

    • Search Google Scholar
    • Export Citation
  • Weygandt, S. S., and Coauthors, 2004: Potential forecast impacts from space-based lidar winds: Results from a regional observing system simulation experiments. Preprints, Eighth Symp. on Integrated Observing Systems, Seattle, WA, Amer. Meteor. Soc., 6.4.

    • Search Google Scholar
    • Export Citation
  • Weygandt, S. S., S. G. Benjamin, T. G. Smirnova, and J. M. Brown, 2008: Assimilation of radar reflectivity data using a diabatic digital filter within the Rapid Update Cycle. Preprints, 12th Conf. on IOAS-AOLS, New Orleans, LA, Amer. Meteor. Soc., 8.4.

    • Search Google Scholar
    • Export Citation
  • White, A. B., F. M. Ralph, J. R. Jordan, C. W. King, D. J. Gottas, P. J. Neiman, L. Bianco, and D. E. White, 2007: Expanding the NOAA Profiler Network: Technology evaluation and new applications for the coastal environment. Preprints, Seventh Conf. on Coastal Atmospheric and Oceanic Prediction and Processes, San Diego, CA, Amer. Meteor. Soc., 8.6.

    • Search Google Scholar
    • Export Citation
  • Zapotocny, T. H., W. P. Menzel, J. P. Nelson, and J. A. Jung, 2002: An impact study of five remotely sensed and five in situ data types in the Eta Data Assimilation System. Wea. Forecasting, 17 , 263285.

    • Search Google Scholar
    • Export Citation
  • Zapotocny, T. H., J. Jung, J. Le Marshall, and R. Treadon, 2007: A two-season impact study of satellite and in situ data in the NCEP Global Data Assimilation System. Wea. Forecasting, 22 , 887909.

    • Search Google Scholar
    • Export Citation
  • Zhu, Y., and R. Gelaro, 2008: Observation sensitivity calculations using the adjoint of the Gridpoint Statistical Interpolation (GSI) analysis system. Mon. Wea. Rev., 136 , 335351.

    • Search Google Scholar
    • Export Citation

Fig. 1.
Fig. 1.

Surface analyses for (a) 1200 UTC 1 Dec 2006, in middle of winter experiment period, and (b) 1200 UTC 20 Aug 2007, in the middle of the summer experiment period (courtesy of NOAA/NCEP/HPC).

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 2.
Fig. 2.

Distribution of data sources. Sample data from 12 Nov 2009 valid within 1 h of 0000 UTC (data available in retrospective periods is similar, but that TAMDAR data are not shown here). Color coded as in the histogram plots below: (a) AMDAR; (b) profilers (national network plus cooperating agency profilers); (c) VAD, from NEXRAD radars; (d) raobs; (e) GPS; (f) AMV; (g) all surface (METARs plus mesonet); and (h) METAR, color coded by altitude.

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 3.
Fig. 3.

Midwest–Great Lakes (red rectangle) and national (gray area) verification regions. Also shown is the location of the NOAA profilers (blue rectangles) and verifying raobs (brown triangles).

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 4.
Fig. 4.

Differences in rms error (vs radiosonde) between observation denial experiments listed in Table 2 and control run for 1000–400 hPa for relative humidity (% RH) for national domain. Results for each of the eight observational denial experiments are coded with a different color (aircraft: red, profiler: blue, VAD: pink, raob: tan, surface: light blue, GPS-PW: green, mesonet: black, satellite AMV winds: brown). Graphs at top are for winter results; those at bottom are for summer. (left) The three adjacent columns for each OSE are for 3-, 6-, and 12-h forecasts. (right) The same results are organized by observation type for each forecast projection (3, 6, and 12 h). The impact of AMV data was not tested during the summer period. Statistical uncertainties are indicated for each observation denial experiment by narrow black lines showing ±1 standard error from the mean impact.

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 5.
Fig. 5.

As in Fig. 4, but for temperature error (K) for 1000–100-hPa layer over national domain.

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 6.
Fig. 6.

As in Fig. 4, but for wind vector difference (m s−1), still for 1000–100-hPa layer over national domain.

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 7.
Fig. 7.

Vector wind error (vs radiosondes) for RUC 9- (black), 3- (blue), and 1-h (red) forecasts averaged over 400–200-hPa layer and for a 30-day running mean for January 2007–April 2009. Also shown are the forecast error increments (between 0 and 1) from assimilation of recent observations for 9- to 1-h (red, near lower axis) and 3- to 1-h (tan) pairs.

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 8.
Fig. 8.

As in Fig. 6, but for 1000–800-hPa only (still wind, national domain).

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 9.
Fig. 9.

As in Fig. 8, but for 800–400-hPa layer (still wind, national domain).

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 10.
Fig. 10.

As in Fig. 8, but for 400–100-hPa layer (still winds for national domain).

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 11.
Fig. 11.

As in Fig. 4, but for the Midwest (or Great Lakes) regional verification domain.

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 12.
Fig. 12.

As in Fig. 5, but for the Midwest verification region.

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 13.
Fig. 13.

As in Fig. 12, but for the 1000–800-hPa layer only.

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 14.
Fig. 14.

As in Fig. 12, (temperature, Midwest domain), but for the 800–400-hPa layer.

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 15.
Fig. 15.

As in Fig. 6, but for the Midwest domain.

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 16.
Fig. 16.

As in Fig. 15, but for the 1000–800-hPa layer.

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 17.
Fig. 17.

As in Fig. 15, but for the 800–400-hPa layer.

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 18.
Fig. 18.

As in Fig. 15, but for the 400–100-hPa layer.

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 19.
Fig. 19.

Differences in 3-h vector wind errors between No-profile minus Control (N − C, blue line) and 8-km (quarter-scale) minus Control (8 − C, red line) over the Midwest region.

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 20.
Fig. 20.

As in Fig. 19, but for 6-h forecasts instead of 3-h forecasts.

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Fig. 21.
Fig. 21.

As in Fig. 20, but for 3-h wind forecast impact in the Midwest domain, but with results added for 12-km profilers. Green line (12 − C) is for difference between experiments with 12-km profilers (12) vs Control (C).

Citation: Monthly Weather Review 138, 4; 10.1175/2009MWR3097.1

Table 1.

Observation types assimilated in the RUC for observation system experiments used in this study: pressure (P), height (Z), temperature (T), horizontal wind (V), relative humidity (RH), precipitable water (PW), virtual temperature (Tv), and dewpoint temperature (Td).

Table 1.
Table 2.

Observation impact experiments in this study. Those observational variables denied to the RUC are shown for each experiment: pressure (P), height (Z), temperature (T), horizontal wind (V), relative humidity (RH), precipitable water (PW), and dewpoint temperature (Td).

Table 2.
Table 3.

Experiment periods.

Table 3.
Table 4.

Verification stratifications for OSEs.

Table 4.
Save