1. Introduction
Carefully designed observing system simulation experiments (OSSEs) can provide a realistic simulated framework for the theoretical evaluation of data assimilation system (DAS) properties and the potential impact of a particular observing system, current or future, on the overall DAS performance. Development of an OSSE relies on the availability of a high-resolution free running model integration called the “nature run” providing a baseline (truth) with respect to which errors are defined. To emulate the real observing system, simulated observations are sampled from the nature run with realistic simulated errors added to them with the goal to create a framework that is able to mirror the overall behavior of the real DAS that uses real observations. This requires significant validation efforts to match the characteristics of the simulated observing system network, its temporal and spatial distribution, and its error statistics to that of real observations.
The premise of any OSSE is that it can be relied upon to be a reasonable representation of the real system it is trying to emulate. Besides the input observations, consistency in the data assimilation algorithm and other DAS-related configurations is also critical. The GMAO OSSE was originally designed and calibrated based on the 3D-Var data assimilation framework that was used in the GMAO’s quasi-operational NWP system at the time of its implementation (Errico et al. 2013, 2017). The latter has, however, long been upgraded to use a hybrid four-dimensional ensemble–variational (4D-EnVar) scheme (Todling and El Akkraoui 2018). The upgrade introduced a change in the representation of the prescribed background error covariance matrix used in the assimilation cycle which resulted in significant improvements both in the analysis quality and forecast skill. Moreover, the assimilation of observations in hourly bins in the hybrid 4D-EnVar using flow-dependent background error covariances and producing four-dimensional hourly analysis increments is in stark contrast to the 3D-Var treatment of the same observations, producing a single analysis increment in the 6-hourly assimilation window and using a climatological estimate of background error statistics. These differences have motivated the effort to tackle the same upgrade in the GMAO OSSE to keep it relevant for future observation impact studies.
This paper describes the extension of the GMAO OSSE framework to use a hybrid 4D-EnVar scheme in the assimilation process and provides a short intercomparison of the 3D/4D OSSEs to highlight aspects where this change matters to the OSSE community. In Kleist and Ide (2015a,b), some properties of the hybrid ensemble–variational approach were examined in an OSSE framework, which helped direct the development of the hybrid 3D-Var and hybrid 4D-EnVar systems at NCEP (Kleist 2012) and subsequently at GMAO (Todling and El Akkraoui 2018). While we expect corroboration of such results, the purpose of the present work is not to demonstrate the benefits of the hybrid 4D-EnVar methodology in data assimilation, but rather to set the grounds for interpreting OSSE results when the hybrid 4D-EnVar is employed instead of a 3D-Var. Validation of the 4D OSSE in regard to its effectiveness in mimicking the real DA system behavior builds on the extensive validation of GMAO’s 3D-Var OSSE in Errico et al. (2013) and Privé et al. (2013), and will focus here on aspects of data assimilation that are unique to the four dimensional case.
This paper is organized as follows. The framework of the original 3D-Var OSSE is described in section 2 along with details of the implementation of the new hybrid 4D-EnVar OSSE. In section 3, the experiment setting is presented followed by a discussion on OSSE validation and comparison with the real system. Intercomparison of system error estimates in the hybrid 4D-EnVar and 3D-Var OSSEs is presented in section 4, and finally, a short discussion on future work exploring the four dimension aspect of error estimates and observation thinning is presented.
2. A hybrid 4D-EnVar OSSE framework
a. GMAO nature run, and original 3D-Var OSSE
The GMAO OSSE framework consists of a nature run (NR) that is considered the “truth” of the atmospheric state, simulated observations derived from the NR with suitable simulated observation errors, and a numerical weather prediction (NWP) model with associated data assimilation system that ingests the simulated observations and produces forecasts. As every aspect of NWP is simulated in an OSSE framework, it is vital that the behavior of the OSSE is validated against the real world to ensure that the OSSE is a robust tool that can be used to infer characteristics of the real global observing network and operational NWP performance.
The nature run employed by the GMAO OSSE is a 2-yr free forecast using a nonhydrostatic global mesoscale simulation of the Goddard Earth Observing System Model, version 5 (GEOS-5) with approximately 7-km horizontal resolution (C1440), 72 vertical levels, and 30-min saved output states (Putman et al. 2014; Rienecker et al. 2008). The NR uses boundary conditions of sea surface temperature, sea ice, daily volcanic and biomass burning as well as high-resolution inventory of anthropogenic sources, all taken from the period of May 2005–07 but is otherwise not influenced by the real world, so that the climatology and synoptic states of the NR do not correspond to any particular real world dates. An extensive validation of the behavior of the NR is described in Gelaro et al. (2014); while there are some aspects of the NR that were shown to be unrealistic, the NR was found to be sufficiently realistic for use in NWP studies.
The NR is considered to be the true atmosphere both for verification and for generation of the simulated or “synthetic” observations. Most data types that are operationally ingested into the DAS are represented in the GMAO OSSE, including the Atmospheric Infrared Sounder (AIRS), the Advanced Microwave sounder Unit-A (AMSU-A), the High Resolution Infrared Radiation Sounder (HIRS) series, the Special Sensor Microwave Imager/Sounder (SSMI/S), the Global Precipitation Measurement (GPM) Microwave Imager (GMI), the Advanced Technology Microwave Sounder (ATMS), the Infrared Atmospheric Sounding Interferometer (IASI), the Cross-Track Infrared Sounder (CrIS), the Microwave Humidity Sounder (MHS), the Global Navigation Satellite System Radio Occultation (GNSS-RO), and conventional types such as surface observations, rawinsondes, aircraft measurements, scatterometers, and atmospheric motion vectors (AMVs). With the exception of AMVs and rawinsondes, the synthetic observations are based on the locations and times of real observations available during the basis period of June–August 2015, but with values determined by the NR fields at a corresponding time.
Simulated AMVs are based on the locations of clouds or water vapor features in the NR rather than the locations of real observations in 2015. An updated AMV simulation algorithm is used where observation times are produced in such a way that their temporal distributions closely resembled those of each real corresponding instrument (Errico et al. 2020). Simulated Rawinsondes are launched at times and locations from the 2015 base period but are advected by the NR wind fields. Radiance observations are generated using the Community Radiance Transfer Model (Han et al. 2006) using vertical column data from the NR and with cloud contamination determined by the locations and density of clouds in the nature run. GNSS-RO bending angles are calculated using the observation operator developed and described by Culverwell et al. (2015). All simulated observations are generated to match the counts and spatial distribution of real observations that are ingested in the real world system. Both correlated and uncorrelated simulated observation errors are added to the synthetic observations in such a way as to match the statistics of observation increments of real-world data. Vertically correlated errors are added to sounding observations such as rawinsondes, AMVs and GNSS-RO, horizontally correlated errors are added to microwave radiances and AMVs, and channel correlated errors are added to hyperspectral infrared radiances. The simulated errors were adjusted to match the standard deviation of the observation innovations of real data. Full details of the generation of synthetic observations and their errors are discussed in Errico et al. (2017).
The NWP model selected for generating forecasts is the Goddard Earth Observing System Model (GEOS) version 5.17 run at 25-km horizontal resolution. Although the GEOS model is used for both the NR and the forecasts, there are numerous differences between the two model versions, including the use of a two-moment microphysics scheme (Barahona et al. 2014) in the forecast model version, and some different choices of boundary layer and low level cloud parameterizations. This OSSE is best considered as a “fraternal twin” setup, with model error that is nonzero but substantially less than real world model error.
b. Extension of the OSSE framework to hybrid 4D-EnVar
Assimilation in the original OSSE is performed using a three-dimensional variational version of the Gridpoint Statistical Interpolation (GSI) data assimilation system (Kleist et al. 2009) with a first guess at appropriate time approach (3D-FGAT) whereby observation residuals are computed using 3-hourly backgrounds, valid at the beginning, center, and end of the 6-hourly assimilation window (Massart et al. 2010). The configuration of the hybrid 4D-EnVar OSSE used in this paper is consistent with version 5.17 of the real system and described hereafter.
In the real operational system, the ensemble component consists of 32 members cycled concurrently with the deterministic (central) variational DAS, and recentered around the central analysis after each analysis cycle. An ensemble square root Kalman filter (EnSRF) is used to update the ensemble members before recentering is applied and the same forecast model is used for the time integration of the member forecasts (Whitaker et al. 2008). An ensemble localization procedure with vertically varying and resolution-dependent scales is applied to filter out unrealistic correlations due to sampling size. The ensemble contributes to the estimation of the hybrid background error covariance in (2) with assigned vertically varying weights of βe = 0.5 and βc = 0.5 throughout the tropospheric and lower stratospheric model levels and transitioning to full static covariances (βe = 0, βc = 1) above 5 hPa as described in Fig. 25 of Todling and El Akkraoui (2018).
Minimization of the hybrid 4D-EnVar cost function (1) is performed using the biconjugate gradient algorithm (El Akkraoui et al. 2013) and yields hourly analysis increments used in the initialization of the forecast model through the incremental analysis update (IAU) procedure (Bloom et al. 1996; Lorenc et al. 2015; Takacs et al. 2018). The IAU was designed to filter out some high-frequency oscillations by introducing the increments gradually as weighted model forcing (tendency) at each model time step for the first 6 h of the model integration (corrector phase), and the model thereafter continues freely (analysis tendencies set to zero) during the subsequent 6 h (predictor phase) to produce the background fields for the next assimilation cycle. In 3D-Var, the process implies using uniform weights following Bloom et al. (1996), with a digital filter modulation described in Takacs et al. (2018). The hybrid 4D-EnVar implements a 4DIAU as described in Takacs et al. (2018). The EnSRF is implemented in its 3D configuration, consequently, the ensemble assimilation employs the digital filter modulation of Takacs et al. (2018) to the Bloom et al. (1996) IAU procedure.
The extension of the OSSE framework to hybrid 4D-EnVar assimilation follows the same approach as described above with a 32-member ensemble used to provide flow dependent estimates of the background error covariance matrix. The EnSRF procedure used in the OSSE assimilates the same synthetic observations as in the deterministic OSSE. Effectively, this means that the capability of the GMAO OSSE system has been extended for use in either of these configurations: 3D-Var, hybrid 3D-EnVar, hybrid 4D-EnVar, or simply a pure EnSRF-based ensemble data assimilation.
Aside from the use of real versus synthetic observations, the ensemble was set up to use the same configuration in the real and OSSE. That is, the same localization scales are imposed, the same inflation parameters are applied after the EnSRF analysis is performed, and the same weights are assigned for the ensemble-based covariances in the hybrid variational cost function. Arguably, most of these ensemble aspects are tunable quantities that should be adjusted to match the appropriate metrics in each application. For instance, the use of the same level of nonadaptive additive inflation may in reality result in inadequate ensemble spread in the OSSE, thus leading to inconsistent representation of the prescribed background error variances. Ideally, having identical real and OSSE configurations is most desirable because then we can keep a simple basis for direct comparison with the real system. On the other hand, if the performance statistics that validate the OSSE do not match the corresponding real ones well enough, then interpretation of the OSSE results become less clear.
3. Validation
a. Setup and experiments
Two OSSE experiments were carried out, one using the 3D-Var (denoted hereafter “3d_osse”) and another using the hybrid 4D-EnVar (“4d_osse”) algorithms. For the purpose of validation, two other equivalent experiments (“3d_real” and “4d_real”) were set up to use the real observations instead of the simulated ones (see Table 1). All four experiments are performed using a forecast model at 25-km horizontal resolution and a GSI analysis at 50-km horizontal resolution. The two hybrid experiments use ensembles at 100-km horizontal resolution. All experiments ran for two months after a 3-week spinup period. Results presented here cover the periods of July–August 2006 for the OSSE cases and July–August 2015 for the real cases. Both cases use the global observing network from July to August 2015, while the nature run period of 2006 indicates that this is the second year of the nature run period and otherwise is not representative of the actual year 2006.
Experiment configurations.
One goal of this validation study was to determine if the synthetic observations required different calibration of simulated errors for the 3D-Var versus hybrid 4D-EnVar setups. Because the variance of the observation innovation was used to calibrate the simulated observation errors, recalibration might be required if there is a substantial change to the background state depending on the assimilation methodology.
The same observation dataset was used for both the 3D-Var and hybrid 4D-EnVar OSSEs. The synthetic observations use prescribed errors that are tuned so that observation-related statistics such as data counts and departures (observation minus background; omf) in the OSSE match those of the real system. The simulated observations and their errors described in section 2 were calibrated using the 3D-Var setup and used without recalibration in the hybrid 4D-EnVar setup for the 4d_osse. Figure 1 shows a comparison of the standard deviations of the zonal wind departures from rawinsondes (Fig. 1a) and brightness temperature from active channels of AMSU-A on board MetOp-A (Fig. 1b) for all four cases. Statistics in the OSSE cases are in agreement with those in the real ones. This was verified for other observation types (not shown), and it was found that overall the OSSE framework had sufficiently realistic behavior with both variations of the DAS, and so no additional tuning of the observation error statistics was deemed necessary.
Standard deviations of observation innovations. (a) Zonal winds from rawinsonde (m s−1) with vertical pressure levels and (b) brightness temperature from AMSU-A channels on MetOp-A (K) for active channels. Statistics from the 3d_real and 3d_osse cases are shown in black circles and dots, respectively, and statistics from the 4d_real and 4d_osse cases are shown in red triangles and filled triangles, respectively.
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Standard deviations of observation innovations. (a) Zonal winds from rawinsonde (m s−1) with vertical pressure levels and (b) brightness temperature from AMSU-A channels on MetOp-A (K) for active channels. Statistics from the 3d_real and 3d_osse cases are shown in black circles and dots, respectively, and statistics from the 4d_real and 4d_osse cases are shown in red triangles and filled triangles, respectively.
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Standard deviations of observation innovations. (a) Zonal winds from rawinsonde (m s−1) with vertical pressure levels and (b) brightness temperature from AMSU-A channels on MetOp-A (K) for active channels. Statistics from the 3d_real and 3d_osse cases are shown in black circles and dots, respectively, and statistics from the 4d_real and 4d_osse cases are shown in red triangles and filled triangles, respectively.
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Figure 2 shows a time series of ensemble spread in the 4d_real (black circles) and 4d_osse (black dashed line) experiments for the July–August period. The global average of zonal wind spread at 250 hPa, and temperature and specific humidity spread at 850 hPa are shown in Figs. 2a, 2b, and 2c, respectively. The evolution of the ensemble during the two-months period is consistent between the real and OSSE cases.
Time series of the global mean ensemble spread for (a) the 250-hPa zonal wind (m s−1), (b) the 850-hPa temperature (K), and (c) specific humidity (g kg−1) for the July–August period. The 4d_real and 4d_osse cases are shown with black circles and dashed lines, respectively.
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Time series of the global mean ensemble spread for (a) the 250-hPa zonal wind (m s−1), (b) the 850-hPa temperature (K), and (c) specific humidity (g kg−1) for the July–August period. The 4d_real and 4d_osse cases are shown with black circles and dashed lines, respectively.
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Time series of the global mean ensemble spread for (a) the 250-hPa zonal wind (m s−1), (b) the 850-hPa temperature (K), and (c) specific humidity (g kg−1) for the July–August period. The 4d_real and 4d_osse cases are shown with black circles and dashed lines, respectively.
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Because of the flow-dependent property of the ensemble evolution, an important aspect to note is that while the time series in Fig. 2 show consistency between the level of spread in the real and OSSE ensembles, the OSSE spread at any given synoptic time is representative of the uncertainty around the atmospheric flow in the nature run, not the real world. This is illustrated in Fig. 3 showing the zonal mean of the ensemble spread for the 4d_real and the 4d_osse (left and right panels, respectively) for a chosen synoptic time 0000 UTC 1 August 2015 in the real case, and the bogus year 2006 in the OSSE case. The regional discrepancies in the amplitude of the spread can be explained, at least in part, by the fact that the two cases represent two different “realities of the day.” The OSSE is designed to mimic the real system in terms of statistics and properties, not in terms of day-to-day true flow. Here, the structure and amplitude of the spread are realistic and consistent between the real and OSSE for the variables shown (zonal wind, temperature, and specific humidity).
Example of the zonal mean of ensemble spread (left) at a chosen synoptic time 0000 UTC 1 Aug 2015 in the 4d_real case and (right) for the bogus year 2006 in the OSSE case. Rows show the tropospheric fields of (a),(b) zonal wind (m s−1); (c),(d) temperature (K); and (e),(f) specific humidity (g kg−1).
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Example of the zonal mean of ensemble spread (left) at a chosen synoptic time 0000 UTC 1 Aug 2015 in the 4d_real case and (right) for the bogus year 2006 in the OSSE case. Rows show the tropospheric fields of (a),(b) zonal wind (m s−1); (c),(d) temperature (K); and (e),(f) specific humidity (g kg−1).
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Example of the zonal mean of ensemble spread (left) at a chosen synoptic time 0000 UTC 1 Aug 2015 in the 4d_real case and (right) for the bogus year 2006 in the OSSE case. Rows show the tropospheric fields of (a),(b) zonal wind (m s−1); (c),(d) temperature (K); and (e),(f) specific humidity (g kg−1).
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
b. Analysis increment
One of the known suboptimal aspects of the original 3D-Var OSSE is the underrepresentation of model error owing to the use of similar but not identical versions of GEOS for the GMAO nature run and the experiment model (considered a fraternal twin OSSE). This results in smaller amplitudes of the analysis increments when compared with the real case (Privé et al. 2021). This unrealistic behavior is also observed in the case of the hybrid 4DenVar OSSE. A comparison of the zonal mean of standard deviations of the 3d_real and 3d_osse analysis increments is shown in Fig. 4 for the zonal wind, temperature and specific humidity fields (top, middle, and bottom panels, respectively), and a similar comparison but for the 4d_real and 4d_osse analysis increments is shown in Fig. 5. In both cases, the amplitudes are smaller in areas where model errors would normally contribute to the growth of system errors during the time integration of the model to produce the background states.
Zonal mean of the temporal standard deviations of analysis increment over the two-month period of July–August. (left) 3d_real; (right) 3d_osse. (a),(b) Zonal wind (m s−1); (c),(d) temperature (K); and (e),(f) specific humidity (g kg−1).
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Zonal mean of the temporal standard deviations of analysis increment over the two-month period of July–August. (left) 3d_real; (right) 3d_osse. (a),(b) Zonal wind (m s−1); (c),(d) temperature (K); and (e),(f) specific humidity (g kg−1).
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Zonal mean of the temporal standard deviations of analysis increment over the two-month period of July–August. (left) 3d_real; (right) 3d_osse. (a),(b) Zonal wind (m s−1); (c),(d) temperature (K); and (e),(f) specific humidity (g kg−1).
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
As in Fig. 4, but for the analysis increments in the 4d_real and 4d_osse cases.
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
As in Fig. 4, but for the analysis increments in the 4d_real and 4d_osse cases.
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
As in Fig. 4, but for the analysis increments in the 4d_real and 4d_osse cases.
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Note that while the ensemble spread is considered a proxy for the 6-h forecast error which encompasses both initial and model error growth, the lack of adequate representation of model error in the OSSE framework has no bearing on the ensemble itself since the spread is computed as a standard deviation from the ensemble mean rather than from the nature run. The ensemble could well be under- or overestimating the true error statistics but that would be unrelated to the OSSE configuration. As seen in Fig. 2, the OSSE ensemble spread is consistent with the real case, owing to the use of the same randomly selected perturbations as additive inflation in both the real and OSSE cases, the same prescribed observation error covariance matrix and consistent real and simulated observations in the EnSRF.
c. Observation impact
The GMAO OSSE framework includes a forecast sensitivity to observation impact (FSOI) tool for estimating observation impacts (Privé and Errico 2019). The GMAO OSSE framework includes a forecast sensitivity to observation impact (FSOI) tool for estimating the impact of ingested observations on forecast skill, based on a selected error norm as described in Privé and Errico (2019). The procedure follows the Trémolet (2008) extension of the Langland and Baker (2004) approach and uses pairs of forecasts initialized from analysis and background fields, respectively. Differences in forecast skills are then attributed to the impact of ingested observations. The calculation of these impacts employs the GEOS-5 adjoint model with a moist physics component that accounts for convective processes (Holdaway et al. 2014), and a total wet energy norm following Ehrendorfer and Errico (1995) and Holdaway et al. (2014). Once-daily 30- and 24-h forecast pairs from the 1800 UTC and the following 0000 UTC cycle are used to calculate FSOI over the two-month period of the experiment.
A comparison of the impact of observations as measured by FSOI is shown in Fig. 6 with the net daily observation impact with respect to various observing systems (Fig. 6a) for the real cases (red and orange for 3d_real and 4d_real, respectively) and the OSSE cases (cyan and blue for 3d_osse and 4d_osse, respectively). Negative (positive) values represent beneficial (detrimental) impact. Consistency between 3D-Var and hybrid 4D-EnVar can be seen for the observations having the larger impacts, AMSU-A, AMV, and raob winds. The impact values are smaller in the OSSE compared to the real cases. This can be explained by the underrepresentation of model error in the OSSE, which results in smaller error growth in the forecast used to compute the impact (Privé and Errico 2019).
Impact of observations as measured by the forecast sensitivity to observation impact. (a) Net daily observation impact with respect to various observing systems for the real cases (red and orange for 3d_real and 4d_real, respectively) and the OSSE cases (cyan and blue for 3d_osse and 4d_osse, respectively). (b) Fractional observation impact defined as the net impact for each observing type normalized by the sum of the impacts of all data types.
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Impact of observations as measured by the forecast sensitivity to observation impact. (a) Net daily observation impact with respect to various observing systems for the real cases (red and orange for 3d_real and 4d_real, respectively) and the OSSE cases (cyan and blue for 3d_osse and 4d_osse, respectively). (b) Fractional observation impact defined as the net impact for each observing type normalized by the sum of the impacts of all data types.
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Impact of observations as measured by the forecast sensitivity to observation impact. (a) Net daily observation impact with respect to various observing systems for the real cases (red and orange for 3d_real and 4d_real, respectively) and the OSSE cases (cyan and blue for 3d_osse and 4d_osse, respectively). (b) Fractional observation impact defined as the net impact for each observing type normalized by the sum of the impacts of all data types.
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
The fractional observation impact is shown in Fig. 6b. The net impact for each observing type is normalized by the sum of the impacts of all data types. This metric allows comparison of the relative ranking of different observation impacts for the real and OSSE cases. It is seen that there is relatively little difference in the fractional observation impact between any of the four experiment cases. Some exceptions are overestimation of AMV impacts in both OSSE cases as compared to real case, and larger AMSU-A impacts for 3D-Var compared to hybrid 4D-EnVar.
4. Intercomparison of the 3D-Var and hybrid 4D-EnVar OSSEs
Because of the Incremental Analysis Update procedure, we distinguish between “analysis” and “assimilation” states at any given time within the assimilation window. While the first is simply the corrected background with an analysis increment valid at the same time, the latter is the atmospheric state produced by a model integration that gradually ingests the analysis increment at any time during the IAU corrector interval. To some extent, the IAU here is analogous to a final outer loop whereby the nonlinear model is integrated from the beginning of the window with increment tendency adjustments applied at each model time step. In an IAU-based DAS, errors at the analysis time can be computed either with respect to the assimilation state or the analysis state; similarly forecast verification can be done with respect to the assimilation or the analysis state. For consistency with the GMAO’s real DAS conventions, we use the assimilation state in all cases to define errors at the analysis time, hereafter referred to as “assimilation error.”
a. Background and assimilation error
Four dimensional data assimilation allows for the analysis updates to be computed more frequently than in 3DVar where only one analysis increment is produced and is valid at the center of the assimilation window. For a direct comparison between the two OSSEs, we focus first on error estimates at the center of the window.
The vertical profiles of assimilation and background error standard deviations are shown in Fig. 7 for the 3d_osse and 4d_osse. Statistics are computed for the July–August period on a 6-hourly synoptic time interval, and results are shown for bands of latitudes representing the Northern Hemisphere extratropics [NHEX; (30°–90°N)], Southern Hemisphere extratropics [SHEX; (90°–30°S)], and the tropics [tropics; (30°S, 30°N)]. The assimilation error standard deviations in the 4D case (solid orange lines) are significantly smaller than in the 3D case (solid blue lines) for all three fields of zonal wind, temperature, and specific humidity and at all levels. This is consistent with the improvements seen in NWP metrics after the GMAO DAS was upgraded to use hybrid ensemble–variational data assimilation as documented in Todling and El Akkraoui (2018). Similar to the assimilation error, the background error standard deviations are also significantly reduced in the 4D (dashed orange lines) compared to the 3D case (dashed blue lines), owing to the systematic improvement in the assimilation states. Note though that because of the known underestimation of model error in the OSSE, the estimated background error standard deviations may be lower than they would be in a real system in both the 3D and 4D cases.
Regionally averaged assimilation (solid) and background (dashed) error standard deviation for the 3d_osse (blue) and the 4d_osse (orange) calculated for July–August. (left) NHEX (30°–90°N); (center) SHEX (90°–30°S); (right) tropics (30°S–30°N). (a)–(c) Zonal wind (m s−1); (d)–(f) temperature (K); and (g)–(i) specific humidity (g kg−1).
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Regionally averaged assimilation (solid) and background (dashed) error standard deviation for the 3d_osse (blue) and the 4d_osse (orange) calculated for July–August. (left) NHEX (30°–90°N); (center) SHEX (90°–30°S); (right) tropics (30°S–30°N). (a)–(c) Zonal wind (m s−1); (d)–(f) temperature (K); and (g)–(i) specific humidity (g kg−1).
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Regionally averaged assimilation (solid) and background (dashed) error standard deviation for the 3d_osse (blue) and the 4d_osse (orange) calculated for July–August. (left) NHEX (30°–90°N); (center) SHEX (90°–30°S); (right) tropics (30°S–30°N). (a)–(c) Zonal wind (m s−1); (d)–(f) temperature (K); and (g)–(i) specific humidity (g kg−1).
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
The clear separation of errors between the two data assimilation methods (blue versus orange lines) is also contrasted with the relatively small difference between the respective background and assimilation error standard deviation change (dashed versus solid lines). The latter difference reflects the contribution of the analysis increment corrections, halfway through the corrector-phase of the IAU, to the reduction of errors.
It is worth noting that in real applications, statistics of the analysis increment are sometimes used as a metric to measure the quality of the data assimilation system. Figure 7 illustrates why these statistics cannot encompass the full impact of DA changes, since they do not account for the feedback on the background state being also improved.
To further quantify the improvement in the assimilation (background) error with the hybrid 4D-EnVar when compared to 3D-Var, the relative change in error standard deviations, computed as
As in Fig. 7, but for the relative change in error standard deviations, computed as
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
As in Fig. 7, but for the relative change in error standard deviations, computed as
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
As in Fig. 7, but for the relative change in error standard deviations, computed as
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Unlike most other fields, areas, and levels, specific humidity errors show a degradation for the layers above 200 hPa. It is not clear whether this is due to a deficiency in the 4D scheme or the result of a known disagreement between the single-moment microphysics convection scheme used in the nature run and the two-moment scheme used in the experiment model which mainly affects the upper tropical troposphere. For this reason and as is the case in all previous GMAO OSSE published validation studies, performance assessment in this OSSE is limited to tropospheric variables.
Overall, even though the hybrid 4D-EnVar acts to produce significantly different initial conditions, this only results in a reduction in the amplitude of assimilation and background error standard deviations, but not in any noticeable change in their structure. The areas where the errors are larger remain the same. It is worth noting that a retuning of the ensemble can potentially lead to a change in the structure of the error statistics. However, as mentioned previously, the ensemble configuration was deliberately chosen to match the real system configuration for consistency’s sake. A possible separate study would aim to evaluate the impact of a tuning update on the error statistics in line with the work of Kleist and Ide (2015a,b) in order to inform on possible configuration changes to be implemented in the real system.
b. Forecast error
As for the assimilation and background errors, the forecast error is determined using the difference between a forecast and the nature run state valid at the same time. The relative change in the RMS forecast error between the 4d_osse and 3d_osse, normalized with the RMS forecast error of the 3d_osse is shown in solid colors in Fig. 9. Stippling indicates significance of the paired differences at the 90th percentile level. The forecast error is significantly smaller in the 4d_osse case (blue dotted areas), with improvements of the hybrid 4D-EnVar larger in the first hours of the forecast and persisting to 10 days in the Southern Hemisphere for all variables and to a good extent in the tropics. The specific humidity field, however, shows degradation above 200 hPa, consistent with a similar degradation at the analysis time shown in Fig. 8. The humidities are not assimilated above the tropopause, so these degradations may be related to noise or improper relationships between temperature and humidity increments in the stratosphere. In both the NR and the experiment forecast model, the stratospheric water vapor fields are damped to the same monthly averaged, zonally averaged climatologies. Prior OSSE work has also noted that stratospheric humidities were degraded during the assimilation process (Privé et al. 2022).
Relative change in the RMS forecast error between the 4d_osse and 3d_osse, normalized with the RMS forecast error of the 3d_osse in solid colors. Stippling indicates significant differences of the paired differences at the 90th percentile level. (left) NHEX (30°–90°N); (center) SHEX (90°–30°S); and (right) tropics (30°S–30°N). (a)–(c) Zonal wind (m s−1); (d)–(f) temperature (K); and (g)–(i) specific humidity (g kg−1).
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Relative change in the RMS forecast error between the 4d_osse and 3d_osse, normalized with the RMS forecast error of the 3d_osse in solid colors. Stippling indicates significant differences of the paired differences at the 90th percentile level. (left) NHEX (30°–90°N); (center) SHEX (90°–30°S); and (right) tropics (30°S–30°N). (a)–(c) Zonal wind (m s−1); (d)–(f) temperature (K); and (g)–(i) specific humidity (g kg−1).
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Relative change in the RMS forecast error between the 4d_osse and 3d_osse, normalized with the RMS forecast error of the 3d_osse in solid colors. Stippling indicates significant differences of the paired differences at the 90th percentile level. (left) NHEX (30°–90°N); (center) SHEX (90°–30°S); and (right) tropics (30°S–30°N). (a)–(c) Zonal wind (m s−1); (d)–(f) temperature (K); and (g)–(i) specific humidity (g kg−1).
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
While significant, the improvements in the Northern Hemisphere extratropics temperature only cover the first 48 h. In prior OSSE studies of new observation impacts, it has been noted that the forecast impacts are largest at the initial time and often become nonsignificant after 24–48 h, particularly in the tropics and Northern Hemisphere extratropics during summer (McCarty et al. 2021; McGrath-Spangler et al. 2022; Privé et al. 2022). This indicates that improvements to initial condition errors are most impactful at the initial forecast time, while model and chaotic errors begin to dominate after the initial forecast period. This is also consistent with Fig. 8, middle row panels, where the reduction in assimilation error standard deviations with the hybrid 4DEnVar is smaller in the NHEX compared to the tropics and the SHEX. This behavior can be explained by the difference in the driving processes in these regions; the summer tropics and NHEX are dominated by short time scale convective processes, while the winter SHEX has longer time scale baroclinic processes that contribute to the retention of improvement into midrange forecast.
The error growth shows some noticeable peculiarities in the fields. The zonal wind forecast error grows largely around the jet levels driven by the model dynamics. The 200-hPa temperature error in the extratropics is consistent with the slow growing phase error where synoptic features in the OSSE become out of sync with the nature run around the tropopause layer.
c. Forecast error nature run and self-verification
In the previous subsection, validation of the forecast error in the OSSE experiments is done with respect to the “truth” as represented by the nature run states. Another way to validate forecast skills is the self-verification method commonly used in real operational settings where the truth is unknown. The verification would then be done with respect to the corresponding assimilation state for each forecast, valid at the forecast time.
Since verifying assimilation states are available for both the OSSE and real cases, a direct comparison is possible here. Figure 10 shows the 10-day time series of the RMS of forecast error for the 3d_real and 4d_real with self-verification (denoted ASM; black lines) and for the 3d_osse and 4d_osse with both self (ASM; solid red and solid blue, respectively) and NR verification (dashed red and dashed blue, respectively), for select fields: 250-hPa zonal wind (Fig. 10a), 500-hPa temperature (Fig. 10b), and 850-hPa specific humidity (Fig. 10c). Consistent with the results shown before in Fig. 9, the self-verified curves show smaller RMSE in the 4d_osse and 4d_real when compared to the 3d_osse and 3d_real cases.
The 10-day time series of the RMS of forecast error verified against nature run (NR) and self-verified (ASM). The self-verified 3d_real and 4d_real are shown in solid black thick and thin lines, respectively. The 3d_osse and 4d_osse cases are shown in red and blue lines, respectively, with self-verification in solid lines and NR verification in dashed lines. (a) 250-hPa zonal wind (m s−1); (b) 500-hPa temperature (K); and (c) 850-hPa specific humidity (kg kg−1).
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
The 10-day time series of the RMS of forecast error verified against nature run (NR) and self-verified (ASM). The self-verified 3d_real and 4d_real are shown in solid black thick and thin lines, respectively. The 3d_osse and 4d_osse cases are shown in red and blue lines, respectively, with self-verification in solid lines and NR verification in dashed lines. (a) 250-hPa zonal wind (m s−1); (b) 500-hPa temperature (K); and (c) 850-hPa specific humidity (kg kg−1).
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
The 10-day time series of the RMS of forecast error verified against nature run (NR) and self-verified (ASM). The self-verified 3d_real and 4d_real are shown in solid black thick and thin lines, respectively. The 3d_osse and 4d_osse cases are shown in red and blue lines, respectively, with self-verification in solid lines and NR verification in dashed lines. (a) 250-hPa zonal wind (m s−1); (b) 500-hPa temperature (K); and (c) 850-hPa specific humidity (kg kg−1).
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
By construction, all self-verified curves (solid lines) start from the zero-line, then the RMS errors grow at a different rate for the first two days due to the interplay between growing and decaying modes of the system error—a combination of initial time and model errors—after the initialization step. The curve’s slope becomes similar later on in the forecast integration as the growth of model errors becomes dominant over the growth of initial condition error. This means that the growth then is a characteristic of the model used (the same) rather than the initialization state. The 3d_real and 4d_real (solid black and thin black lines, respectively) both converge to the same level for all three variables after about 8 days. Conversely, there is still a separation between the self-verified 3d_osse and 4d_osse curves (solid red and solid blue, respectively) after 10 days. Differences between the climatologies of the real and OSSE systems and underestimation of model error in the OSSE framework reflect in smaller estimated RMS errors in the OSSE compared to the real cases.
The NR-verified RMS forecast errors are also shown in Fig. 10 (dashed curves). After about 3–4 days, the NR 3d_osse and 4d_osse curves converge to the self-verified 3d_osse and 4d_osse for the zonal wind and temperature fields, and longer for specific humidity. It is important to note, however, that regardless of the data assimilation method used to produce the initialization state, all NR-verification curves (dashed) should in theory saturate to the “true” forecast error after a long enough model integration. How fast that convergence happens would depend on the variable, model level and the region. Except for the specific humidity field, both the temperature and zonal wind forecast errors still show a separation between the two assimilation methods after 10 days. It is not clear how long of an integration is required for the curves to fully converge. Privé and Errico (2013) found similar results, though at 5 days only, when comparing forecast error growths in cases with different initial conditions and/or different model error representations with the OSSE framework.
Overall, consistency between the 3d-Var and hybrid 4D-EnVar OSSE experiments holds in terms of both NR- and self-verified forecast error statistics.
5. Discussion on future work
The hybrid 4D-EnVar data assimilation approach offers a unique framework to examine the inner workings of the DAS in ways that are not available with 3D-Var. Exploring the time dimension within the assimilation window opens up new opportunities for the use of OSSEs. For instance, the adequacy of the treatment of observations with high-temporal frequency can be directly assessed and strategies of four-dimensional thinning of radiance data can also be explored to determine when such data have the most positive impact. Hereafter, we present two examples of potential DAS evaluations that are only possible in a hybrid 4D-EnVar OSSE.
a. Errors within the assimilation window
For the purpose of our intercomparison with the original 3D-Var OSSE, results shown previously were limited to the center of the assimilation window. In the hybrid 4D-EnVar OSSE, the availability of hourly incremental updates allows for a more frequent estimation of the errors so that DAS properties can be examined throughout the 6-h window. An example using results from the 4d_osse experiment is shown in Fig. 11 representing the hourly estimates of error standard deviations for the zonal wind at select pressure levels 250, 500, and 850 hPa (Figs. 11a, 11b, and 11c, respectively). The statistics are computed for the two-week period of 1–15 August, using all 15 daily analyses at their respective synoptic hour.
Hourly estimates of background (dashed), assimilation (dotted), and analysis (open circles) error standard deviations for zonal wind at select pressure levels of (a) 250, (b) 500, and (c) 850 hPa. The four daily assimilation windows are centered around synoptic times 0000, 0600, 1200, and 1800 UTC. Statistics computed using all 15 daily analyses at their respective synoptic hour for the two-week period of 1 Aug–15 Aug.
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Hourly estimates of background (dashed), assimilation (dotted), and analysis (open circles) error standard deviations for zonal wind at select pressure levels of (a) 250, (b) 500, and (c) 850 hPa. The four daily assimilation windows are centered around synoptic times 0000, 0600, 1200, and 1800 UTC. Statistics computed using all 15 daily analyses at their respective synoptic hour for the two-week period of 1 Aug–15 Aug.
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Hourly estimates of background (dashed), assimilation (dotted), and analysis (open circles) error standard deviations for zonal wind at select pressure levels of (a) 250, (b) 500, and (c) 850 hPa. The four daily assimilation windows are centered around synoptic times 0000, 0600, 1200, and 1800 UTC. Statistics computed using all 15 daily analyses at their respective synoptic hour for the two-week period of 1 Aug–15 Aug.
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
In each cycle, the assimilation error (dotted line) generally decreases in the first 3 h, is smallest around the center of the window and grows slowly toward the end of the assimilation window but remain considerably smaller than the errors in the background (dashed curve). This is a direct consequence of IAU: the assimilation errors arise within the corrector step of IAU when the model is forced with analysis tendencies for 6 h and the influence of observation is felt by the state; the background errors arise during the predictor step of IAU, when the model runs free from the influence of observations and errors grow monotonically in the next 6 h.
The figure also shows how the analysis error (open circles) behaves, in contrast to the assimilation error. We distinguish between analysis and assimilation states in an IAU-based data assimilation system. The former arises directly from the minimization of the cost function (1), independently of IAU; it is simply the direct correction of the individual hourly backgrounds by the corresponding hourly increments. Ideally, one would expect the analysis error to be smaller than the background error throughout the analysis window. However, the increments in the first hour of each window result in a consistent degradation that may reflect inadequate thinning and weighting of the observations or a misrepresentation of the background error covariances at the beginning of the window. Nevertheless, we see that the IAU process produces assimilated states that have systematically smaller errors than the backgrounds.
Using the hybrid 4D-EnVar OSSE, future work will study the four dimensional thinning of observations and the sensitivity of analysis error to the placement of observations within the assimilation window. Similar studies using 4D-Var systems for instance have shown that observations placed near the end of the window are more influential than those used in the first half of the window (Laroche and Gauthier 1998; McNally 2019).
b. Ensembles
Errico et al. (2015) used the original 3DVar OSSE to examine the properties of background error covariances estimated from the OSSE versus the prescribed climatological background error covariance matrix derived with the NMC method (Parrish and Derber 1992). The introduction of ensembles in the hybrid 4D-EnVar OSSE allows for the examination of additional aspects of representation of these covariances in the DAS. As explained in the introduction, the ensemble-based flow-dependent estimates are used in combination with the climatological NMC-based estimates in the hybrid 4D-EnVar formulation.
Figure 12 shows instantaneous vertical profiles of the regionally averaged zonal mean ensemble spread at 0000 UTC for each day during the July–August period (dashed blue curves) and the OSSE estimates of background error standard deviation (dashed black curves) for the zonal wind (Figs. 12a–c), temperature (Figs. 12d–f), and specific humidity (Figs. 12g–i).
Vertical profiles of the zonal mean of the ensemble spreads at 0000 UTC for each day during the July–August period (solid blue curves) and the OSSE estimates of background error standard deviation (dashed black curves). (a)–(c) Zonal wind (m s−1); (d)–(f) temperature (K); and (g)–(i) specific humidity (g kg−1). (left) NHEX; (center) SHEX; and (right) tropics.
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Vertical profiles of the zonal mean of the ensemble spreads at 0000 UTC for each day during the July–August period (solid blue curves) and the OSSE estimates of background error standard deviation (dashed black curves). (a)–(c) Zonal wind (m s−1); (d)–(f) temperature (K); and (g)–(i) specific humidity (g kg−1). (left) NHEX; (center) SHEX; and (right) tropics.
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Vertical profiles of the zonal mean of the ensemble spreads at 0000 UTC for each day during the July–August period (solid blue curves) and the OSSE estimates of background error standard deviation (dashed black curves). (a)–(c) Zonal wind (m s−1); (d)–(f) temperature (K); and (g)–(i) specific humidity (g kg−1). (left) NHEX; (center) SHEX; and (right) tropics.
Citation: Monthly Weather Review 151, 7; 10.1175/MWR-D-22-0254.1
Since the ensemble spread is a proxy for the 6-h forecast error standard deviation, a desirable result would be to find consistency between the magnitude and shape of ensemble spread and OSSE estimates of background error standard deviations. The zonal wind ensemble spread compares well with the corresponding OSSE errors, especially in the tropics, but overestimates the errors in the Southern Hemisphere extratropics and particularly around the 300-hPa layer. In the case of temperature, although there is agreement in the mid- and upper-tropospheric tropical region, significant inconsistencies are apparent around 200 hPa in the extratropics, and everywhere in the lower troposphere. The underestimation of the prescribed background error standard deviation in these areas contributes to the lack of improvement in the hybrid 4D-EnVar OSSE seen previously in Fig. 8. The specific humidity ensemble spread is consistent with the OSSE errors in the Northern Hemisphere extratropics but with a slight underestimation of the tropical errors.
Overall, the behavior illustrated in Fig. 12 is unlikely to be unique to the OSSE ensemble since the spread in the 4d_real experiment is about the same (Fig. 3). On the other hand, while the prescribed background error standard deviations are similar in the OSSE versus the real system, the effective error standard deviations are likely different (due to the underestimation of model error in the OSSE for instance), which may affect the consistency between the spread and error illustrated in Fig. 12. Regardless of these identified inconsistencies, it is important to remember that a good OSSE should be able to mimic the real system in its overall behavior (qualities and deficiencies alike). This is the main reason we chose not to retune the ensemble for this OSSE upgrade.
6. Conclusions
The GMAO OSSE framework has been updated to use a hybrid 4D-EnVar data assimilation technique instead of a 3D-Var. Implementation of this update follows the same methodology used in the real system and described in detail in Todling and El Akkraoui (2018). Since both the 3D-Var and hybrid 4D-EnVar OSSEs are using the same version of the DAS and the forward model, a direct comparison was possible, both in terms of the validation with respect to their corresponding real cases and also in terms of estimates of assimilation, background and forecast error statistics.
A short validation of the hybrid 4D-EnVar OSSE found that the previous assessments of the 3D-Var OSSE in its ability to mimic the behavior of the real system still hold with the same caveats. For instance, the underrepresentation of model error that is inherent to the choice of the Nature Run model is also impacting the 4D OSSE analysis increments in a similar way. Furthermore, the reliance on the same NMC-like perturbations in the ensemble additive inflation procedure as in the real system results in a consistent ensemble spread between the OSSE and real case. It was also shown that the use of the hybrid 4D-EnVar leads to a consistent reduction in the standard deviations of assimilation, background and forecast errors. However, even though this particular configuration of the hybrid 4D-EnVar acts to produce significantly different initial conditions, this only results in a reduction in the amplitude of the error standard deviations, but not in any noticeable change in their structure. This highlights the importance of other DAS components that can still benefit from further improvements such as data coverage, quality control and thinning, and other assimilation algorithms and configurations, among others. In the present paper, no attempt was made to evaluate metrics related to the ensemble used in this OSSE beyond the necessary sanity check and validation with the real case. Keeping the consistency between the GMAO tuning/setting of the ensemble was important. This OSSE can later be used as a framework to study the ensemble separately and examine the adequacy of the ensemble tuning used in this version of the real system. Future work can also evaluate the consistency of the hourly growth of ensemble spreads within the assimilation window as it pertains to the forecast error growth. Furthermore, a reexamination of the uniform variable scaling of the NMC-like perturbations used in the additive inflation would expound for the differences in error growth rates between model variables.
While we presented an intercomparison between the 3d and 4d cases, the purpose here is not to qualify and quantify the differences between the two DA methodologies, but rather to highlight aspects of relevance to the OSSE community and identify particular features of DA that can only be explored in a four dimensional OSSE framework. All things being equal, the importance of a hybrid 4D-EnVar OSSE lays rather when the temporal density of the observing system is considered. The change of the observation binning from 3-hourly bins to 1-hourly bins is the main difference in the treatment of observation in 3D-Var and hybrid 4D-EnVar. The OSSE community can explore various procedures for four dimensional thinning for instance to allow for better use of the observations when thinned differently at the beginning and the end of the assimilation window. Similarly, anticipated future instruments with a higher temporal resolution will benefit from an assessment of their impact in a 4D OSSE much more than 3D.
Acknowledgments.
Support for this project was provided by NASA/Global Modeling and Assimilation Office core funding. Resources supporting this work were provided by the NASA High-End Computing (HEC) Program through the NASA Center for Climate Simulation (NCCS) at Goddard Space Flight Center. The software for simulating GPSRO observations was provided by the Radio Occultation Processing Package (ROPP) of the Radio Occultation Meteorology (ROM) Satellite Applications Facility (SAF) of EUMETSAT, with the assistance of Sean Healy at ECMWF.
Data availability statement.
The dataset on which this paper is based is too large to be retained or publicly archived with available resources. Documentation and methods used to support this study are available from Amal El Akkraoui at NASA/GMAO.
REFERENCES
Barahona, D., A. Molod, J. Bacmeister, A. Nenes, A. Gettelman, H. Morrison, V. Phillips, and A. Eichmann, 2014: Development of two-moment cloud microphysics for liquid and ice within the NASA Goddard Earth Observing System model (GEOS-5). Geosci. Model Dev., 7, 1733–1766, https://doi.org/10.5194/gmd-7-1733-2014.
Bloom, S. C., L. L. Takacs, A. M. da Silva, and D. Ledvina, 1996: Data assimilation using incremental analysis updates. Mon. Wea. Rev., 124, 1256–1271, https://doi.org/10.1175/1520-0493(1996)124<1256:DAUIAU>2.0.CO;2.
Culverwell, I. D., H. W. Lewis, D. Offiler, C. Marquardt, and C. P. Burrows, 2015: The radio occultation processing package, ROPP. Atmos. Meas. Tech., 8, 1887–1899, https://doi.org/10.5194/amt-8-1887-2015.
Ehrendorfer, M., and R. M. Errico, 1995: Mesoscale predictability and the spectrum of optimal perturbations. J. Atmos. Sci., 52, 3475–3500, https://doi.org/10.1175/1520-0469(1995)052<3475:MPATSO>2.0.CO;2.
El Akkraoui, A., Y. Trémolet, and R. Todling, 2013: Preconditioning of variational data assimilation and the use of a bi-conjugate gradient method. Quart. J. Roy. Meteor. Soc., 139, 731–741, https://doi.org/10.1002/qj.1997.
Errico, R. M., R. Yang, N. C. Privé, K.-S. Tai, R. Todling, M. E. Sienkiewicz, and J. Guo, 2013: Development and validation of observing-system simulation experiments at NASA’s global modeling and assimilation office. Quart. J. Roy. Meteor. Soc., 139, 1162–1178, https://doi.org/10.1002/qj.2027.
Errico, R. M., N. C. Privé, and W. Gu, 2015: Use of an OSSE to evaluate background-error covariances estimated by the NMC method. Quart. J. Roy. Meteor. Soc., 141, 611–618, https://doi.org/10.1002/qj.2384.
Errico, R. M., and Coauthors, 2017: Description of the GMAO OSSE for weather analysis software package: Version 3. NASA Tech. Rep. NASA/TM-2017-104606, 156 pp., https://archive.org/details/NASA_NTRS_Archive_20170011277.
Errico, R. M., D. Carvalho, N. C. Privé, and M. Sienkiewicz, 2020: Simulation of atmospheric motion vectors for an observing system simulation experiment. J. Atmos. Oceanic Technol., 37, 489–505, https://doi.org/10.1175/JTECH-D-19-0079.1.
Gelaro, R., and Coauthors, 2014: Evaluation of the 7-km GEOS-5 nature run. NASA Tech. Rep. NASA/TM2014-104606, 305 pp., https://ntrs.nasa.gov/citations/20150011486.
Han, Y., P. van Delst, Q. Liu, F. Weng, B. Yan, R. Treadon, and J. Derber, 2006: JCSDA Community Radiative Transfer Model (CRTM)—Version 1. NOAA Tech. Rep. NESDIS 122, 33 pp., https://repository.library.noaa.gov/view/noaa/1157.
Holdaway, D., R. Errico, R. Gelaro, and J. G. Kim, 2014: Inclusion of linearized moist physics in NASA’s Goddard Earth Observing System data assimilation tools. Mon. Wea. Rev., 142, 414–433, https://doi.org/10.1175/MWR-D-13-00193.1.
Ide, K., P. Courtier, M. Ghil, and A. C. Lorenc, 1997: Unified notation for data assimilation: Operational, sequential and variational. J. Meteor. Soc. Japan, 75, 181–189, https://doi.org/10.2151/jmsj1965.75.1B_181.
Kleist, D. T., 2012: An evaluation of hybrid variational-ensemble data assimilation for the NCEP GFS. Ph.D. thesis, University of Maryland, College Park, 149 pp., http://drum.lib.umd.edu/handle/1903/13135.
Kleist, D. T., and K. Ide, 2015a: An OSSE-based evaluation of hybrid variational ensemble data assimilation for the NCEP GFS. Part I: System description and 3D-hybrid results. Mon. Wea. Rev., 143, 433–451, https://doi.org/10.1175/MWR-D-13-00351.1.
Kleist, D. T., and K. Ide, 2015b: An OSSE-based evaluation of hybrid variational ensemble data assimilation for the NCEP GFS. Part II: 4DEnVar and hybrid variants. Mon. Wea. Rev., 143, 452–470, https://doi.org/10.1175/MWR-D-13-00350.1.
Kleist, D. T., D. F. Parrish, J. C. Derber, R. Treadon, W.-S. Wu, and S. Lord, 2009: Introduction of the GSI into the NCEP Global Data Assimilation System. Wea. Forecasting, 24, 1691–1705, https://doi.org/10.1175/2009WAF2222201.1.
Langland, R. H., and N. L. Baker, 2004: Estimation of observation impact using the NRL atmospheric variational data assimilation adjoint system. Tellus, 56A, 189–201, https://doi.org/10.3402/tellusa.v56i3.14413.
Laroche, S., and P. Gauthier, 1998: A validation of the incremental formulation of 4D variational data assimilation in a nonlinear barotropic flow. Tellus, 50A, 557–572, https://doi.org/10.3402/tellusa.v50i5.14558.
Lorenc, A. C., and M. Jardak, 2018: A comparison of hybrid variational data assimilation methods for global NWP. Quart. J. Roy. Meteor. Soc., 144, 2748–2760, https://doi.org/10.1002/qj.3401.
Lorenc, A. C., N. E. Bowler, A. M. Clayton, S. R. Pring, and D. Fairbairn, 2015: Comparison of hybrid-4DEnVar and hybrid-4DVar data assimilation methods for global NWP. Mon. Wea. Rev., 143, 212–229, https://doi.org/10.1175/MWR-D-14-00195.1.
Massart, S., B. Pajot, A. Piacentini, and O. Pannekoucke, 2010: On the merits of using a 3D-FGAT assimilation scheme with an outer loop for atmospheric situations governed by transport. Mon. Wea. Rev., 138, 4509–4522, https://doi.org/10.1175/2010MWR3237.1.
McCarty, W., D. Carvalho, I. Moradi, and N. C. Privé, 2021: Observing system simulation experiments investigating atmospheric motion vectors and radiances from a constellation of 4–5-μm infrared sounders. J. Atmos. Oceanic Technol., 38, 331–347, https://doi.org/10.1175/JTECH-D-20-0109.1.
McGrath-Spangler, E. L., W. McCarty, N. C. Privé, I. Moradi, B. M. Karpowicz, and J. McCorkel, 2022: Using OSSEs to evaluate the impacts of geostationary infrared sounders. J. Atmos. Oceanic Technol., 39, 1903–1918, https://doi.org/10.1175/JTECH-D-22-0033.1.
McNally, A. P., 2019: On the sensitivity of a 4D-Var analysis system to satellite observations located at different times within the assimilation window. Quart. J. Roy. Meteor. Soc., 145, 2806–2816, https://doi.org/10.1002/qj.3596.
Parrish, D. F., and J. C. Derber, 1992: The National Meteorological Center’s spectral statistical-interpolation analysis system. Mon. Wea. Rev., 120, 1747–1763, https://doi.org/10.1175/1520-0493(1992)120%3c1747:TNMCSS%3e2.0.CO;2.
Privé, N. C., and R. M. Errico, 2013: The role of model and initial condition error in numerical weather forecasting investigated with an observing system simulation experiment. Tellus, 65A, 21740, https://doi.org/10.3402/tellusa.v65i0.21740.
Privé, N. C., and R. M. Errico, 2019: Uncertainty of observation impact estimation in an adjoint model investigated with an observing system simulation experiment. Mon. Wea. Rev., 147, 3191–3204, https://doi.org/10.1175/MWR-D-19-0097.1.
Privé, N. C., R. M. Errico, and K.-S. Tai, 2013: Validation of the forecast skill of the global modeling and assimilation office observing system simulation experiment. Quart. J. Roy. Meteor. Soc., 139, 1354–1363, https://doi.org/10.1002/qj.2029.
Privé, N. C., R. M. Errico, and W. McCarty, 2021: The importance of simulated errors in observing system simulation experiments. Tellus, 73A, 1886795, https://doi.org/10.1080/16000870.2021.1886795.
Privé, N. C., R. M. Errico, and A. EI Akkraoui, 2022: Investigation of the potential saturation of information from Global Navigation Satellite System radio occultation observations with an observing system simulation experiment. Mon. Wea. Rev., 150, 1293–1316, https://doi.org/10.1175/MWR-D-21-0230.1.
Putman, W. M., A. M. da Silva, L. E. Ott, and A. S. Darmenov, 2014: Model configuration for the 7-km GEOS-5 nature run, Ganymed release (non-hydrostatic 7 km global mesoscale simulation). GMAO Office Note 5 (version 1.0), 18 pp., https://gmao.gsfc.nasa.gov/pubs/docs/Putman727.pdf.
Rienecker, M., and Coauthors, 2008: The GEOS-5 data assimilation system—Documentation of versions 5.0.1, 5.1.0, and 5.2.0. NASA Tech. Rep. NASA/TM-2008-104606, Vol. 27, 101 pp., https://ntrs.nasa.gov/api/citations/20120011955/downloads/20120011955.pdf.
Takacs, L. L., M. J. Suárez, and R. Todling, 2016: Maintaining atmospheric mass and water balance in reanalyses. Quart. J. Roy. Meteor. Soc., 142, 1565–1573, https://doi.org/10.1002/qj.2763.
Takacs, L. L., M. J. Suárez, and R. Todling, 2018: The stability of incremental analysis update. Mon. Wea. Rev., 146, 3259–3275, https://doi.org/10.1175/MWR-D-18-0117.1.
Todling, R., and A. El Akkraoui, 2018: The GMAO hybrid ensemble-variational atmospheric data assimilation system: Version 2.0. NASA Tech. Rep. 50, NASA/TM-2018-104606-50, 184 pp., https://ntrs.nasa.gov/citations/20180002172.
Trémolet, Y., 2008: Computation of observation sensitivity and observation impact in incremental variational data assimilation. Tellus, 60A, 964–978, https://doi.org/10.1111/j.1600-0870.2008.00349.x.
Whitaker, J. S., T. M. Hamill, X. Wei, Y. Song, and Z. Toth, 2008: Ensemble data assimilation with the NCEP global forecast system. Mon. Wea. Rev., 136, 463–482, https://doi.org/10.1175/2007MWR2018.1.
