a. CWBGFS data assimilation system

All the experiments are conducted with the operational global NWP system at CWB, called CWB Global Forecast System (CWBGFS; Liou et al. 1997; Su et al. 2019). The spectral model utilizes a dynamical core developed at CWB, and is configured with a triangular truncation of T511 (equivalent to about 25-km horizontal resolution) and 60 vertical hybrid sigma-pressure levels topped at 0.1 hPa. The radiation scheme is the Rapid Radiative Transfer Model for GCM (RRTMG; Iacono et al. 2008). The convective parameterization and vertical diffusion are based on the simplified Arakawa–Schubert convection scheme (Pan and Wu 1995) and Han and Pan (2011). The gridscale precipitation is parameterized based on Zhao and Carr (1997). Other physical parameterizations include the Noah land surface model (Ek et al. 2003), Palmer et al. (1986) orographic gravity wave drag scheme, and Scinocca (2003) convective gravity wave drag scheme.

The data assimilation is performed using the NCEP Gridpoint Statistical Interpolation (GSI; Wu et al. 2002; Kleist et al. 2009) hybrid ensemble-variational (EnVar) system, which has been modified to interface with CWBGFS. At the time of the study, a GSI version in 2015 is used for operation. A typical 6-h forecast-analysis cycle is conducted, with a major (for forecast products)/post (for analysis cycles) workflow design very similar to NCEP’s GFS/GDAS system. The main (deterministic) analysis is conducted using a hybrid 3DEnVar scheme (Wang et al. 2013; Kleist and Ide 2015). For the ensemble component, 36-member ensemble forecasts are run with T319 resolution models. The function of time-lagged ensemble (Lorenc 2017) has been developed and enabled at CWB to double the ensemble size used in the hybrid 3DEnVar to 72, by combining the 6- and 12-h ensemble forecasts. The static background error covariance is estimated using the NMC method (Parrish and Derber 1992) with CWBGFS forecast samples. The weighting coefficients of the static and ensemble background error covariances are 0.25 and 0.75, respectively.

The conventional observations assimilated in the “major” (early) analysis are obtained from the GTS and processed (i.e., QC) by CWB, while most of the satellite radiance data are collected from the BUFR format files available on the NOAA Operational Model Archive and Distribution System (NOMADS). These assimilated radiance data include AMSU-A, ATMS, CrIS, MHS, AIRS, IASI, and SEVIRI. The GNSS RO data from satellites other than FS7/C2 (currently FS3/C1, MetOp-A/B, TerraSAR-X, and TanDEM-X data are assimilated in operation) are also directly obtained from the NOMADS. In addition to the conventional and satellite data, two types of bogus data are assimilated in the CWBGFS system: 1) the 2.5°-mesh wind data at 850-, 700-, 500-, and 200-hPa levels and 850-hPa temperature data taken from the ECMWF operational analysis (assimilated in the “post” analysis) and 24-h forecast (assimilated in the “major” analysis) products as bogus observations, available 12-hourly; 2) the tropical cyclone bogus data constructed assuming a Rankine vortex (in the same way described in Hsiao et al. 2010) for only the typhoons in the western North Pacific region. A vortex relocation scheme (Liu et al. 2000) is also implemented for the western North Pacific typhoons.

b. GNSS RO data assimilation

The 300 vertical levels in one RO bending angle profile in the BUFR format data are denser than the vertical resolution of the model (i.e., 60 levels), which may overemphasize the information from observations. Therefore, the given observation errors (described in section 3c) of the bending angle data within the same model layer are increased proportionally to the square root of the number of observations (in one profile) within the layer, to reduce their effective weights in the cost function and thus achieve a similar effect of “superobing” (Cucurull et al. 2007).

c. Observation errors of GNSS RO data

The GSI has its own “built-in” observation errors for GNSS RO bending angle. In this 2015 version, the observation errors are represented by a series of quadratic functions, piecewise for different height zones, in the form:

${\sigma }_{\text{BA}}^o={10}^{-3}\times e^{-\gamma },\gamma =c_0+c_1\left(a-r_c\right)+c_2{\left(a-r_c\right)}^2,$(2)

where ${\sigma }_{\text{BA}}^o$ is the observation errors of bending angles in radians, a is the impact parameter, and r_c is Earth’s local radius of curvature so that a − r_c is roughly the “elevation” of the tangent point of the ray path assuming no refraction, which is called “impact height.” It is noted that the impact height is generally higher than the “occultation height” or the perigee of the curved ray path, and this difference is especially large for low-level data: for the GNSS ray just passing the sea levels, the impact height is commonly approximately 2 km. Coefficients c₀, c₁, and c₂ are determined empirically for different latitudinal zones, impact heights, and data sources. The RO data are categorized by CDAAC and UKMET types because of their different height-level structures in the BUFR files. The CDAAC-type includes FS3/C1 and FS7/C2, whereas the UKMET-type includes MetOp-A/B, TerraSAR-X, and TanDEM-X. The coefficients defaulted in GSI are listed in Table 1, and the error curves are visualized with normalized values (in percentage; normalized by the one-month average bending angle observations for each height interval) in light red and blue lines in Fig. 2a. According to Cucurull (2010) and Cucurull et al. (2013), these observation error curves were determined based on statistics following the Desroziers et al. (2005) method for a sample collected with NCEP’s global NWP system.

Table 1.

Coefficients [in Eq. (2)] for definition of bending angle observation errors defaulted in GSI (separate for CDAAC- and UKMET-types) and estimated at CWB.

View Table

View Full Size

The given and estimated relative observation errors (%) of the RO bending angle data with respect to impact height (in the study conducted prior to the launch of FS7/C2). (a) Shown in lighter colors are the GSI-default error profiles for CDAAC-type data (dashed–dotted lines; FS3/C1) and UKMET-type data (short-dashed lines; MetOp-A/B, TerraSAR-X, and TanDEM-X) within 40°N/S latitudes (light red lines) and beyond 40°N/S latitudes (light blue lines). Shown in darker colors are the estimated observation errors by the Desroziers et al. (2005) method using all types of the RO data in the operational CWBGFS in a one-month period from 0000 UTC 9 Mar to 1800 UTC 8 Apr 2019, within 40°N/S latitudes (solid red line) and beyond 40°N/S latitudes (solid blue line), and the fitted quadratic curves by Eq. (2) for these estimated values (long-dashed red and blue lines). All the values are normalized by the average observation values in this month for each height interval. (b) As in (a), but for an assimilation experiment rerun using the estimated (fitted) RO observation errors for a month of August 2018. The error profiles used in the experiment are shown in short-dashed lines [same as the fitted curves in (a)], and the re-estimated RO observation errors from this experiment are shown in solid lines.

Citation: Monthly Weather Review 149, 7; 10.1175/MWR-D-20-0267.1

• Download Figure
• Download figure as PowerPoint slide

CWB had directly used the GSI’s default bending angle observation errors described above for the CWBGFS operation prior to the use of the FS7/C2 data. However, it is known that the optimal observation errors are usually dependent on the assimilation systems and associated numerical models because the observation errors in data assimilation contain not only the instrument errors but also the representativeness errors among other errors (e.g., Janjić et al. 2018). There have been various studies on the estimation of the GNSS RO observation errors using different approaches and in different systems (e.g., Kuo et al. 2004; Chen et al. 2011; Anthes and Rieckh 2018; Bowler 2020). Therefore, for a potentially better use of the GNSS RO data, prior to the launch of FS7/C2, we conducted a preparatory study to estimate the bending angle observation errors using one-month (9 March–8 April 2019) innovation statistics in the operational CWBGFS system based on the Desroziers et al. (2005) method. The FS7/C2 certainly did not exist in this period, but this was a reasonable way trying to improve the assimilation of general GNSS RO data in the preparatory period. The results are shown in Fig. 2a (solid red and blue lines). The raw estimations were then fitted using the same quadratic Eq. (2) and therefore implemented in the GSI (long-dashed red and blue lines in Fig. 2a and the lower part in Table 1). Since the Desroziers et al. (2005) method was an iterative method, to confirm if these estimated observation errors were close to convergence, we used them to conduct another one-month assimilation experiment for August 2018 (without FS7/C2 data), and estimated the observation errors again for this second experiment (Fig. 2b). The results supported that the first estimation was already good (i.e., the estimations from the first and second iterations are similar). Hence, we decided to retain the first estimation in Fig. 2a and Table 1 as the CWB-estimated observation errors. At the same time, the forecast verification of the second experiment also supported that the estimated observation errors can lead to a slightly better assimilation performance (figure not shown).

It is somewhat surprising that the estimated observation error profiles are about twice larger than the GSI-default ones (Fig. 2a), although their relative trends in different heights and latitudes are similar. Several questions are then raised: 1) Are the RO observation errors defaulted in the GSI, which were tuned for the NCEP’s operational forecast system almost a decade ago, are still optimal for the NCEP’s system at present? 2) Is the possible system dependence of the observation errors so significant such that, even if the observation errors remain optimal for the NCEP’s system, the same observation errors are suboptimal for the CWBGFS system? 3) Since these estimation and assimilation experiments were not conducted with the FS7/C2 RO data but other RO data available in a period prior to FS7/C2’s launch, how would it be applicable to the FS7/C2 assimilation? Obviously, not until the actual FS7/C2 RO data arrived for a certain period for a preliminary assimilation study to be conducted can we ascertain the suitability of using the newly estimated observation errors for the FS7/C2 assimilation. Therefore, we did not immediately use the estimated error profiles for the CWBGFS operation, nor for the parallel FS7/C2 assimilation experiment, but held them until we had assimilated FS7/C2 data for a sufficiently long period. Following the strategy, we first investigated the FS7/C2 assimilation using the original GSI-default observation errors. During this first period, necessary information about the characteristics of the FS7/C2 data was collected. Afterward, the CWB-estimated RO observation errors can be tested in the assimilation. The detailed descriptions of the FS7/C2 parallel experiment are given in section 3e. A sensitivity experiment designed to more cleanly study the impact of the RO observation error tuning is presented in section 6b with discussion on the optimal observation errors for the FS7/C2 RO data.

d. Quality control

With GSI’s default implementation, there are four major categories of QC checks to reject “bad” (i.e., not desirable for assimilation) GNSS RO bending angle observations, performed in the following order:

1. Out-of-model-boundary check (hereafter BD): It rejects observations below the model surface or above the model top.

2. Superrefraction check (hereafter SR): It rejects observations if either their observed profiles or the corresponding model background profiles are suspicious to be affected by superrefraction. This check only applies to observations at low levels (≤6-km impact height). Discussion about the superrefraction detection criteria can be found in Cucurull (2015).

3. Gross error check (hereafter GC): It rejects observations if their differences to the model background values (O–B) are unreasonably large. This check includes two parts: The first part is the traditional “n-time observation error” gross error check, using the ratio of O–B to the given observation errors to set the rejection threshold. Here, n is 10. The second part is specially designed for the RO data. It rejects observations if the O–B normalized by the observed values exceeds some thresholds that were predetermined from some statistics in a past period [a similar idea described in Cucurull (2010)]. In general, the criterion in the second part is stronger than that in the first part, so the rejection here is mainly detected by the second (statistical) part of the gross error check.

4. Height threshold (hereafter HT): It rejects observations brutally if the impact height of the observation is higher or lower than some fixed thresholds. In the current GSI setting, the upper threshold is set to 50 km (Cucurull et al. 2013) because data above this level are too close to the model top. On the other hand, the low-level threshold has been implemented mainly because of data quality concern. By default, a low-level threshold of 8-km impact height (i.e., rejecting all data below) is effective but only for the MetOp-A/B data because their profiles showed biases when the algorithm was developed. For other RO data such as FS3/C1 and FS7/C2, the strategy is to try to use all low-level data down to the surface as long as the previous three steps of the QC are passed. However, a low-level threshold of 4-km impact height is artificially added for the FS7/C2 data in a part of our experiment period, which will be described in section 3e.

e. Parallel semioperational experiment

The operational CWBGFS run, named OP, was unaffected by the FS7/C2 data in this study period. A parallel semioperational experiment, named FS7, was started as soon as we received the preliminary FS7/C2 data from TACC. It ran with all the assimilation settings same as OP, except for additional assimilation of the FS7/C2 data. To save the resource usage, the model forecast length for each cycle was limited to 7 days in this FS7 experiment, in contrast to 16 days in OP. Figure 3 shows the schematic of the experimental periods, and the associated experimental configurations are also summarized in the top half (OP and FS7 experiments) of Table 2. CWB received the first FS7/C2 data on 16 July 2019. Then 10 days later, the FS7 experiment was initialized at 0000 UTC 26 July by copying the first guess from OP and starting assimilating the FS7/C2 data. Considering that the FS7/C2 data were still not very stable at the beginning of the experiment, the first few day results until 0000 UTC 4 August were not included in any verification. Therefore, the first verification period, Period I, started from 0000 UTC 4 August to 1200 UTC 6 October 2019. During this period, the assimilation settings of the FS7/C2 data, including the observation errors and QC, were exactly the same as the GSI-default settings for the FS3/C1 data.

View Full Size

Schematic of the timeline of the operational (OP) and FS7/C2 parallel semioperational (FS7) experiments. Verification Periods I and II are also shown.

Citation: Monthly Weather Review 149, 7; 10.1175/MWR-D-20-0267.1

• Download Figure
• Download figure as PowerPoint slide

Table 2.

List of experimental configurations. For the operational (OP) and FS7/C2 parallel semioperational (FS7) experiments, verification Period I and Period II range from 0000 UTC 4 Aug to 1200 UTC 6 Oct 2019 and from 0000 UTC 22 Oct 2019 to 0000 UTC 1 Mar 2020, respectively (illustrated in Fig. 3 as well). However, FS7LowLvl and FS7GSIErr sensitivity experiments are only conducted for a 1-month period from 0000 UTC 16 Jan to 0000 UTC 16 Feb 2020 (within Period II). The GSI-default and CWB-estimated observation errors are described in section 3c; the QC settings are explained in section 3d.

View Table

The RO assimilation settings were then adjusted. In the FS7 experiment, two changes were made at 1200 UTC 6 October. First, the observation error profiles of all RO bending angle data were changed to those we estimated using the Desroziers method, as described in section 3c (long-dashed red and blue lines in Fig. 2a; Table 1). Second, a height threshold was imposed to reject all FS7/C2 RO data below 4-km impact height (i.e., the HT QC in section 3d). Regarding the observation error adjustment, we did not re-estimate new error profiles using the FS7/C2 data themselves at this stage. This decision was justified by the fact that the innovation statistics of the FS7/C2 data were found to be very similar to those of the other RO data (previously used for the observation error estimation study), which will be shown later in section 4d. The second change was decided based on some speculation at that time about a suspected negative impact from the low-level RO data, which was later proven to be wrong; nevertheless, the parallel semioperational experiment had been conducted with such settings. The investigation on the impact of this low-level height threshold will be later presented in section 6a. It is noted that the new observation errors in OP were effective only after 0000 UTC 22 October for some operational reason; therefore, the second verification period, Period II, was defined from the time, evaluating the data impact with the revised assimilation settings. This verification period was concluded at 0000 UTC 1 March 2020.

It is understood that the design of this parallel experiment, partly driven by operational implementation needs, may not be ideal to answer some scientific questions, such as the pure impact of the RO observation error tuning. However, the merit of the parallel experiment is to determine whether or not the new FS7/C2 data can again bring positive impact to the operational NWP similar to the existing RO datasets, and to understand how large the impact is quantitatively. Additional scientific questions will be investigated through a few retrospective experiments, which will be discussed in section 6.

a. Data distribution

The distribution of all GNSS RO data assimilated in the FS7 experiment is illustrated in Fig. 4, taking 1200 UTC 24 September 2019 as an example. The FS7/C2 data are plotted in blue and red colors, split for the GPS and GLONASS signals; all the other RO data assimilated in CWBGFS are plotted in green color. As described in section 2, the six FS7/C2 satellites were first deployed at very close orbits at launch and later transferred to equally spaced orbits. Therefore, at the moment shown in Fig. 4 the FS7/C2 data collected from the six satellites were not uniformly distributed in the longitude, and the distribution was becoming more uniform with time.

View Full Size

An exemplary snapshot (1200 UTC 24 Sep 2019) of the distribution of the GNSS RO data assimilated in the FS7 experiment. FS7/C2 RO data from GPS signals, FS7/C2 RO data from GLONASS signals, and other GNSS RO data used in the experiment are denoted by blue, red, and green marks, respectively. Note that RO profiles are not exactly upright, so each of them is projected to a short path in the horizontal map.

Citation: Monthly Weather Review 149, 7; 10.1175/MWR-D-20-0267.1

• Download Figure
• Download figure as PowerPoint slide

b. Quality control rejection rate

Given the rather complicated QC procedure for RO data in the GSI, as described in section 3d, it is worth first looking at how the QC functions with the FS7/C2 data. Figure 5 shows the rejection rate by each category of the QC criteria, BD, SR, and GC, for the FS7/C2 data binned by their impact height, in a period from 0000 UTC 11 September to 1800 UTC 17 September 2019. In this period, the 4-km height criterion has not been imposed (Period I; section 3e) so the QC for the HT category is not shown in the figure. The left panel shows the entire height range of the bending angle profile data, and the right panel is the zoom-in plot for the 0–10-km range. Also shown are the absolute numbers of the bending angle data before QC (black lines; upper axis) and after all QC steps (i.e., assimilated; orange lines; upper axis). It is noted again that the impact height is generally higher than the occultation height by roughly 2 km for the near-surface data, so the lowest few nonempty bins in Fig. 5b are essentially for the RO data whose occultation occurs very close to the surface. It is shown that BD QC first rejects data beyond the lower and upper boundaries of the model, and then SR QC plays an important role for the low-level data below 3.5-km impact height. After that, the GC QC, including the traditional 10-time observation error check and the statistical gross error check (section 3d), works to reject some data (about 0%–10%) in a wider height range, although the midlevel (10–35 km) data are least rejected, which is consistent with the well-known fact that the midlevel RO data have the highest quality (e.g., Kuo et al. 2004). Overall, the rejection rate is reasonably low, and its characteristics with respect to height are consistent with past studies (Cucurull et al. 2013), indicating the good quality of the FS7/C2 data and the correctness of our experimental setup.

View Full Size

Rejection rates (%; lower axis) of the FS7/C2 bending angle data in the FS7 experiment by each category of the QC criteria: out-of-model-boundary check (BD; red bins), superrefraction check (SR; blue bins), and gross error check (GC; green bins). Also shown are the total data counts before QC (black lines; upper axis) and after all QC steps (orange lines; upper axis). The statistics are computed for (a) 0–50-km impact height range with 1-km bins and (b) 0–10-km impact height range with 200-m bins, within the period from 0000 UTC 11 Sep to 1800 UTC 17 Sep 2019 (in Period I).

Citation: Monthly Weather Review 149, 7; 10.1175/MWR-D-20-0267.1

• Download Figure
• Download figure as PowerPoint slide

c. Innovation statistics

With the same period from the FS7 experiment, the root-mean-square differences (RMSD) and mean biases of the observation-minus-background (O–B) and observation-minus-analysis (O–A) values, or so-called “innovations,” for the FS7/C2 data are shown in Fig. 6. The unit is also the normalized bending angle (by the average observation values) similar to that in Fig. 2. This vertical pattern is also consistent with past studies (e.g., Cucurull et al. 2007, 2013; Rennie 2010), showing the largest relative O–B values near the lower (<8-km impact height) and upper (>40-km impact height) boundaries and a smaller peak around the tropopause (about 18-km impact height). The upper-troposphere and lower-stratosphere data have the highest quality, with small RMSDs and nearly zero biases in the innovations. The QC procedure helps reduce the RMSDs of O–B by rejecting bad data (black lines versus blue lines in Fig. 6a), especially for data at low and high levels. Another small but noticeable gap between these before-QC and after-QC lines is observed at about 20–22 km, which is likely related to the GPS L2P band issue mentioned in section 2. Here, it demonstrates that the issue may not be really problematic to the use of the FS7/C2 data in data assimilation, provided that an effective QC procedure is implemented. The O–A values (red lines) are reasonably smaller than the O–B values (blue lines) for all levels, indicating that the data assimilation analysis is normally performed. Regarding the biases, the data are almost bias free at most vertical levels, which is known to be an important advantage of the RO data (Cucurull et al. 2014), whereas a rather significant negative bias is seen at very low levels below 4-km impact height (or 2-km occultation height). Although this may also be attributed to the model bias, other verification studies with different verifying truths also show similar negative bias characteristics at these low levels (Schreiner et al. 2020), so it is believed to be a true bias in the FS7/C2 data. It is known that the low-level bias is not uncommon for other RO datasets, and this would be one of the important issues that needs to be addressed in data processing.

View Full Size

Observation-minus-background (O–B) and observation-minus-analysis (O–A) statistics for the FS7/C2 bending angle data in the FS7 experiment. (a) The root-mean-square differences (RMSD; solid lines) and mean biases (dashed lines) of O–B before QC (black lines), O–B after QC (blue lines), and O–A after QC (red lines) with respect to impact height. (b) As in (a), but for O–B after QC for the GPS-signal data (red lines) and the GLONASS-signal data (blue lines). The period is the same as that in Fig. 5, and the values are normalized by the average observation values in this period for each height interval.

Citation: Monthly Weather Review 149, 7; 10.1175/MWR-D-20-0267.1

• Download Figure
• Download figure as PowerPoint slide

The O–B statistics are also calculated with the data from the GPS and GLONASS signals separately (Fig. 6b) to confirm whether there is any difference in quality between these two data sources. Results show that, from the data assimilation point of view, almost no distinguishable differences can be found in the produced innovations between these two data sources at impact heights below 32 km. Above this level, the GLONASS data show larger O–B values (by as much as about 15%) than the GPS data. This result is also consistent with other verification studies (Schreiner et al. 2020).

d. Comparison with FORMOSAT-3/COSMIC and other RO data

Since one of the main goals of the FS7/C2 mission is to achieve higher quality of the low-level RO data through its more sensitive receiver antenna compared to FS3/C1 and all other existing RO missions, it is important to actually compare their errors. Therefore, in Fig. 7, we compute the O–B innovation statistics similar to those in Fig. 6 but for the FS3/C1 (thick red lines) and FS7/C2 (thick gray lines) data within 50°N–50°S latitudes in a one-month period in October 2019. It was mentioned before that the FS3/C1 data amount had been significantly reduced and not stable in recent times because of the aging of the satellites, so there were actually not too many choices of periods for making this comparison. Fortunately, this October 2019 period contained enough FS3/C1 data from its last alive (the sixth) satellite.

View Full Size

As in Fig. 6, but for the root-mean-square differences (RMSD; solid lines) and mean biases (dashed lines) of observation-minus-background (O–B) for FS3/C1 (thick red lines) and FS7/C2 (thick gray lines) RO data within 50°N–50°S latitudes. The period for the statistics is from 0000 UTC 1 Oct to 1800 UTC 31 Oct 2019, and the values are all normalized by the average observation values derived in Fig. 2. Also shown are the RMSD and bias of the O–B values for FS3/C1 (thin red lines), MetOp-A/B (thin orange lines), TerraSAR-X (thin green lines), and TanDEM-X (thin blue lines) in the same period as if they were having the same latitudinal distribution of data density as the FS7/C2 data. Data counts are shown in the legend.

Citation: Monthly Weather Review 149, 7; 10.1175/MWR-D-20-0267.1

• Download Figure
• Download figure as PowerPoint slide

Results show that FS7/C2 bending angle data have, however, larger O–B values compared to the FS3/C1 data on simple arithmetic average in most height levels (thick gray versus red lines in Fig. 7). This is actually not unexpected because the latitudinal distributions of FS3/C1 and FS7/C2 data densities are very different, and thus due to the latitude dependence of the RO observation errors, a simple average is not fair even if the same (but wide) latitudinal region (50°N–50°S) is chosen. To eliminate this problem, the RMSDs and biases of O–B values are further computed in latitude–height space (Fig. 8), which then shows very similar patterns and magnitudes of the errors and biases between these two datasets. The errors are generally larger in tropical areas compared to the subtropical areas at the same height. In another way, we compute the RMSDs and biases of the FS3/C1 O–B values as if they were having the same latitudinal distribution of data density as the FS7/C2 data, superposed in Fig. 7 in thin red lines; namely, it becomes a weighted average adjusting the latitudinal density distribution of the FS3/C1 data to be the same as that of the FS7/C2 data. These “adjusted” O–B innovation statistics clearly reveal that, from the assimilation perspective in NWP, the error and bias characteristics of the FS7/C2 bending angle data are indeed highly similar to the FS3/C1 data in all height levels if the latitude dependence of the data density is considered (thin red versus thick gray lines in Fig. 7). For completeness, the O–B statistics of other satellites’ RO data assimilated in the experiment [i.e., MetOp-A/B (orange), TerraSAR-X (green), and TanDEM-X (blue)], as if they were having the same latitudinal distribution of data density as the FS7/C2 data, are also computed and shown in Fig. 7. It is found that all of them exhibit very similar error and bias profiles, except for the TerraSAR-X and TanDEM-X data above about 37-km impact height, where they show slightly larger RMSDs of O–B values than other satellites’ data. The similarity of the innovation statistics among the FS7/C2 data and other existing RO data implies that re-estimating the new RO observation errors using the FS7/C2 data (fit with the same piecewise quadratic functions) would not make much difference from the one obtained in the preparatory study (section 3c), so we decided to directly implement the new observation errors (estimated without the FS7/C2 data) in Period II.

View Full Size

(a),(b) Mean biases and (c),(d) RMSDs of observation-minus-background (O–B) for (left) FS3/C1 and (right) FS7/C2 RO data, computed in latitude–height space. Data are binned in 5° latitude and 500-m intervals for the statistics. The period is the same as that in Fig. 7.

Citation: Monthly Weather Review 149, 7; 10.1175/MWR-D-20-0267.1

• Download Figure
• Download figure as PowerPoint slide

Importantly, we should note that these results do not conclude that the data from FS7/C2 are not superior to those from its ancestor, FS3/C1. The superiority of the FS7/C2 data can actually be shown in an upstream stage of data retrieval processes (e.g., Schreiner et al. 2020). We interpret these highly similar O–B statistics for the FS3/C1 and FS7/C2 data as that the other error sources different from the instrument errors but also contributing to the O–B values, such as the errors and biases in the model background and the representativeness errors, may be much larger than the differences in errors between the two datasets. Therefore, the O–B innovation statistics of these two datasets become similar. This suggests that using the same observation errors as those for the FS3/C1 RO data or other existing RO data would be a reasonable approach for an initial FS7/C2 data assimilation study, provided that the RO observation error specification is already latitude dependent. It does not mean that the higher quality of the FS7/C2 measurements such as its higher signal-to-noise ratio is totally useless in the assimilation, but it needs to be more carefully elaborated in future studies.

a. Impact on global forecast skills

This section presents the main results from the FS7 parallel experiment in comparison to OP, showing the impact of the FS7/C2 data assimilation on NWP forecast skills. For a quick look, Fig. 9 shows the time series (each 0000 UTC) of root-mean-square errors (RMSE) of the 5-day forecast of 500-hPa tropical temperature in both the OP and FS7 experiments during the 7-month study period. It is clear that the FS7/C2 assimilation remarkably improves this midlevel tropical temperature, starting from the very early stage of the mission and throughout the entire experimental period. This figure should be viewed alongside Figs. 1 and 3 for further information. First, the small or even zero-impact period in November 2019 in Fig. 9 coincides well with the data-sparse period (Fig. 1) when only two or three FS7/C2 satellites were intermittently transmitting observation data, which indicates that the data amount and the completeness of the satellite constellation are critical factors. Second, as shown in Fig. 3, the assimilation settings changed in the middle of the study period: Compared with the settings in Period I (before 6 October 2019), which were the GSI default, larger observation errors estimated based on the preparatory study (section 3c) and complete removal of the low-level (<4-km impact height) data were applied in Period II (after 22 October 2019). Figure 9 seems to indicate that the Period II settings would be somewhat inferior to the Period I settings as the improvement in FS7 over OP looks generally smaller in Period II, even not taking into account the November 2019 period. However, considering that all the variables, such as the data availability and quality and the seasons, are different in these two periods, we cannot conclude the optimality of the observation errors and QC settings simply from this parallel semioperational experiment. Further investigation on the sensitivities of these settings will be presented in section 6.

View Full Size

The time series of root-mean-square errors (RMSE) of 5-day forecast of 500-hPa temperature (K) in the tropics (20°S–20°N) in the OP (black lines), FS7 (red lines), and FS7LowLvl (blue lines) experiments, verified against the NCEP GFS analysis, during the 7-month study period. Thin lines represent values at every 0000 UTC, and thick lines are their 9-day running means.

Citation: Monthly Weather Review 149, 7; 10.1175/MWR-D-20-0267.1

• Download Figure
• Download figure as PowerPoint slide

Figure 10 shows a more comprehensive forecast verification in the so-called “scorecards,” displaying the statistical significances of the differences between the FS7 and OP experiments in different variables, pressure levels, verification areas, and forecast lead times. Here, the NCEP GFS analysis is used as the verifying truth. All these verifications are performed using the global model verification system provided by the NCEP Environmental Modeling Center (EMC) to CWB. It is shown that in both Periods I and II, the (statistically significant) positive impacts dominate. The most dominant positive impacts are found in the tropics (20°N–20°S), in the RMSEs of almost all variables (i.e., geopotential height, vector wind, and temperature), height levels, and all 1–7-day forecast lead times. This result is consistent with our expectation that FS7/C2 satellites should provide their greatest benefits in the tropical region because of their orbital design. In the extratropical regions of the Northern and Southern Hemispheres (20°–80°N/S) and in global average, the improvement in the RMSE is still evident, but those statistically significant positive impacts can only be found in more limited verification indices, namely, lower-stratospheric geopotential heights (for all forecast times) and temperature and winds for shorter forecast times (within 3 days). For biases, the results are a bit mixed, but positive impacts are the majority among various verification indices. The lower-stratospheric geopotential height biases and midtropospheric temperature biases are particularly improved, whereas the tropospheric wind biases and the lower-tropospheric geopotential height, especially in the Southern Hemisphere in Period II, show degradation. We note that it is usually trickier to analyze the impact on biases because they are usually much smaller than the errors, more sensitive to the bias in the forecast model, and thus more sensitive to the verifying truth if it comes from a different model. To ensure a correct interpretation of the results, the scorecards are also plotted for verification against the self-analysis (Fig. 11). The self-analysis verification shows roughly similar impacts in RMSEs but less negative impacts in biases; for example, the tropospheric wind biases and the lower-tropospheric geopotential height in the Southern Hemisphere in Period II are largely removed. Therefore, we believe that the increased biases seen in the former scorecards (i.e., verified against the NCEP GFS analysis) could possibly be related to the model biases, either in the NCEP GFS model or, more likely, in the CWBGFS model, and that these issues are less likely to indicate a problem of the FS7/C2 data assimilation. Nevertheless, for RMSEs, the stratospheric winds and temperature show some degradation in the self-analysis verification, unlike the mostly positive RMSE scores when verified against the NCEP GFS analysis. These issues are to be double-checked with the observation-based verification.

View Full Size

Scorecards showing the statistical significances of the differences between the FS7 and OP experiments in (a) Period I and (b) Period II in terms of anomaly correlations, RMSEs, and biases of different variables, pressure levels, verification areas, and forecast lead times, verified against the NCEP GFS analysis. Only the forecasts initialized at 0000 UTC are included. The Northern Hemisphere (N. Hemisphere), Southern Hemisphere (S. Hemisphere), and tropics are defined by 20°–80°N, 20°–80°S, and 20°N–20°S, respectively. Green (red) boxes denote that FS7 is better (worse) than OP, with the size of the triangles representing the significance levels (see legend); gray boxes denote no statistical significance; and blue shaded boxes denote the verification indices are not computed.

Citation: Monthly Weather Review 149, 7; 10.1175/MWR-D-20-0267.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

As in Fig. 10, but for scorecards verified against the self-analysis.

Citation: Monthly Weather Review 149, 7; 10.1175/MWR-D-20-0267.1

• Download Figure
• Download figure as PowerPoint slide

Figure 12 shows the verification against radiosonde observations for 6-h forecasts (i.e., background in each data assimilation cycle), averaged over the period from 0000 UTC 16 January to 1800 UTC 15 February 2020 for the tropics (20°N–20°S) and Southern Hemisphere (20°–80°S). These radiosonde observations are equally assimilated in both the OP and FS7 experiments, so the differences here reflect the impact of FS7/C2 RO assimilation through the cycling effect. It is shown that the FS7/C2 RO assimilation clearly improves the RMSEs and biases of the fit to radiosonde temperature over the tropical lower stratosphere and upper troposphere (Fig. 12b). For tropical winds, the improvement is also clearly observed near the tropopause to the lower stratosphere (Fig. 12a), indicating that the impact of the RO data that is more direct on the thermodynamic fields can also be suitably conveyed to the dynamical fields through the cross-variable background error covariance or the model dynamics. A positive impact on the tropical moisture field is also seen (up to 3% reduction in the specific humidity RMSE in midtroposphere; Fig. 12c), which may be critical for improving forecast of some tropical weather. Moreover, a notable positive impact is found in the South Hemisphere u wind (Fig. 12d) near the jet level. For the rest of variables and regions, the impacts are not as obvious (therefore, the figures for the Northern Hemisphere and the globe are not shown). It is emphasized that in this radiosonde-based verification, no obvious degradations in RMSEs and biases are found in all variables and verification areas; therefore, we are more convinced that the FS7/C2 RO assimilation can lead to an overall improvement in the model forecast. The few degradation signs in the model-based verifications are more likely related to the model bias issue. We also note that these radiosonde-based verification results are quite consistent with ECMWF’s recent results for the FS7/C2 data assimilation (Healy 2020; Ruston and Healy 2021), where they also emphasized the improvement in the tropics, for temperature, winds, and particularly tropospheric humidity.

View Full Size

RMSEs (solid lines) and biases (dashed lines) of (a),(d) u wind (m s⁻¹); (b),(e) temperature (K); and (c),(f) specific humidity (g kg⁻¹) of 6-h model forecasts in OP (blue lines) and FS7 (red lines), verified against radiosonde observations with respect to various height levels. Also shown are the relative changes (%) in the RMSEs resulted from the assimilation of the FS7/C2 RO data (FS7–OP; long-dashed green lines). The verification period is from 0000 UTC 16 Jan to 1800 UTC 15 Feb 2020, and the verification areas shown are (top) the tropics and (bottom) Southern Hemisphere defined as those in Fig. 10.

Citation: Monthly Weather Review 149, 7; 10.1175/MWR-D-20-0267.1

• Download Figure
• Download figure as PowerPoint slide

In all, the model- and observation-based verifications clearly indicate that the FS7/C2 data are valuable to data assimilation in NWP, regardless of the GSI-default settings for existing RO data assimilation or some variations from these default settings.

b. Impact on typhoon track forecasts

In addition to the impacts on general synoptic fields presented in section 5a, it is also valuable to verify the impacts on typhoon track forecasts. The average track forecast errors for western North Pacific typhoons during the study period are shown in Fig. 13, separately computed for 10 typhoons occurring in Period I and 8 typhoons occurring in Period II. Assimilation of the additional FS7/C2 RO data on average improves the typhoon track forecast after 3.5-day forecast lead time in Period I (Fig. 13a), but it degrades the track forecast after 2.5-day forecast lead time in Period II (Fig. 13b). However, in Period II, due to the seasonality of typhoon activity, six of the eight typhoons actually occurred during the zero-impact November 2019 period (Fig. 9) as mentioned in section 5a (which is linked to the dramatic drop in the amount of the FS7/C2 RO data at that time; Fig. 1), and all of the eight typhoons occurred before the end of December 2019 when the FS7/C2 data impact remained relatively small (Fig. 9). Meanwhile, in terms of statistical testing, most of the differences in typhoon track forecast errors in this study are not statistically significant due to the limited sample size. In addition, inspired by Chen et al. (2015), we decompose the track errors into the along-track (dashed lines in Fig. 13a) and cross-track (dotted lines in Fig. 13a) errors in Period I, but the results show that the track forecast improvement is not particularly preferential to any of the components, which is different from the results of Chen et al. (2015) that the improvement is dominated by the cross-track errors.

View Full Size

Average track forecast errors (km) vs forecast lead times (h) for western North Pacific typhoons during the experiment (a) Period I (10 typhoons in total) and (b) Period II (8 typhoons in total). OP and FS7 are shown in blue and orange, respectively, and their differences are shown in black dots with 95% confidence intervals shown in bars (right axis). The numbers of forecast cases (of different initial times) are printed below the lower axis. Also shown in (a) are the along-track (dashed lines) and cross-track (dotted lines) errors for both the experiments in Period I. The CWB’s official typhoon track data are used for verification.

Citation: Monthly Weather Review 149, 7; 10.1175/MWR-D-20-0267.1

• Download Figure
• Download figure as PowerPoint slide

Theoretically, if the FS7/C2 RO data assimilation can improve the synoptic fields, it should potentially be able to improve the typhoon track forecast; however, our current results are inconclusive, which may be due to simply an insufficient number of cases or other more sophisticated reasons. To obtain more insight, several possibilities could be considered, such as increasing the sample size or choosing few interesting cases to perform a more detailed case study. However, the contents of the current study are restricted to mainly show the general impact on the synoptic fields. An in-depth investigation on the typhoon forecast could potentially be a separate study.

c. Ensemble forecast sensitivity to observation impact

Sections 5a and 5b presented various forecast verifications to investigate the impact of the FS7/C2 RO data by comparing the results of the FS7 and OP experiments, and generally positive impacts were found. However, this regular approach cannot be used to separately attribute these impacts to each observing system assimilated in the experiments (e.g., each single GNSS RO satellite). The forecast sensitivity to observation impact (FSOI; Langland and Baker 2004) and its ensemble variant, ensemble forecast sensitivity to observation impact (EFSOI; Kalnay et al. 2012), which does not require the adjoint of the dynamical model, are useful techniques to achieve this demand without the need to reconduct many data-denial assimilation experiments. Details for the EFSOI formulation can be found in Kalnay et al. (2012), Ota et al. (2013), and Lien et al. (2018). In this study, a forecast length of 6 h is chosen for the evaluation time to compute the EFSOI, and the moist total energy norm as described in Ota et al. (2013) is used.

Figure 14 shows the EFSOI estimated observation impacts of each GNSS RO satellite in Period I in the FS7 experiment. Looking first at the average EFSOI for each observation datum (Figs. 14b,d), we find that all GNSS RO data assimilated in the CWBGFS system (i.e., MetOp-A/B, TerraSAR-X, TanDEM-X, FS3/C1, and FS7/C2) can contribute to similar magnitudes of forecast error reductions in the per-observation aspect. In specific, the MetOp-A/B data show a relatively smaller average impact (about 60% of other GNSS RO data), which may be partly attributed to the strict 8-km height threshold QC (section 3d) that is only applied to these data. By contrast, TerraSAR-X and TanDEM-X show a relatively larger average impact, which may be linked to the fact that the TerraSAR-X and TanDEM-X RO data exhibit more distinct spatial distributions compared to other major RO datasets (i.e., FS7/C2 data here; figure not shown); thus, the information contents per observation datum could be larger. Importantly, the FS7/C2 data accomplish a roughly similar per-observation impact to other existing RO datasets in these EFSOI diagnostics. The impacts of data from the GPS and GLONASS signals are also comparable, echoing the finding in section 4c that the innovation statistics of these two data types are very similar. Consequently, by multiplying the observation numbers, the average total EFSOI is shown for each GNSS RO satellite (Fig. 14a) and each satellite mission (Fig. 14c).¹ Straightforwardly due to the large amount of data from the entire FS7/C2 constellation, the average total EFSOI of all FS7/C2 RO data is impressive, much greater than the sum of the EFSOI of all other GNSS RO data available to CWBGFS in this period (Fig. 14c). It is noted that the larger total impact for the sixth FS7/C2 satellite (FS7GPS_06/FS7GLN_06) also results from its relatively large data amount in this initial operation period (which should be a transient characteristic), considering that the average per-observation EFSOI for each FS7/C2 satellite does not differ much.

View Full Size

EFSOI estimation of the GNSS RO observation impacts in Period I in the FS7 experiment. (a) The estimated reduction of 6-h forecast errors for one cycle, measured by the moist total energy norm (J kg⁻¹), by assimilating GNSS RO data from each single RO satellite, averaged over 248 cycles in Period I. The average observation counts in one cycle are printed in the right axis. The FS7/C2 data impact is further separately computed for the data from GPS signals and from GLONASS signals for each satellite. Only one (the sixth) of FS3/C1’s six satellites was functioning to deliver RO data during this period. (b) As in (a), but for the average error reductions per observation datum, which are equal to the values in (a) divided by the average observation counts in one cycle. (c),(d) As in (a) and (b), but for the total and per-observation impacts for each satellite mission, respectively (i.e., all the six FS7/C2 satellites are counted as a whole).

Citation: Monthly Weather Review 149, 7; 10.1175/MWR-D-20-0267.1

• Download Figure
• Download figure as PowerPoint slide

a. Impact of the use of low-level data

The bottom half of Table 2 lists the configurations for these additional experiments conducted for the one-month period within Period II. First of all, during this period, the FS7 experiment conducted in near–real time used the CWB-estimated observation errors for the RO data and applied a HT (height threshold) QC to reject all the data below 4-km impact height (section 3e; Fig. 3). To investigate the sensitivity to the 4-km HT QC, the “FS7LowLvl” experiment is conducted by switching off this QC, otherwise the same as FS7. This means that the GSI will try to assimilate all-level FS7/C2 RO data down to the surface. However, although the data are then not brutally rejected solely due to their heights, the other QC steps (i.e., BD, SR, GC) described in section 3d are still in effect, which do reject significant percentages of RO data below the 4-km impact height (e.g., Fig. 5). The performance of these QC steps would critically determine the actual assimilation impact since there are certainly some problematic RO data near the surface when used for assimilation (e.g., Cucurull 2015).

Figure 15a shows the scorecard comparing the forecast skills between the FS7LowLvl and FS7 experiments (verified against the NCEP GFS analysis), with green color representing that FS7LowLvl is better than FS7; namely, the use of the data below 4-km impact height is beneficial. The results are satisfactory: the artificial height threshold is unnecessary; the other more meaningful QC steps in GSI can already function well to deal with all the low-level FS7/C2 RO data. Specifically, the positive impacts by recovering these low-level data in the assimilation are dominant in the tropical region, consistent with the previous results that the tropical region gains the largest benefits by assimilating the FS7 data (Fig. 10). Even surprisingly, although the difference between these two experiments is merely the use of the data below the 4-km impact height (or 2-km occultation height as explained before), the positive impacts extend upward to the entire troposphere and even the lower stratosphere in the RMSEs of tropical temperature and winds (Fig. 16; also seen in the scorecard in Fig. 15a). In addition, an alternative verification against the self-analysis confirms a very similar impact (figure not shown). It is therefore curious to know how much of the reduction in FS7/C2 impact in Period II comes from the rejection of observations below 4-km impact height, which is answered with the blue lines in Fig. 9 showing the result of FS7LowLvl. The inclusion of the FS7/C2 low-level data in the assimilation (with GSI QC) pushes the forecast performance of the tropical 500-hPa temperature from the red lines to the blue lines in this one-month period.

View Full Size

Scorecards as those in Fig. 10 (verified against the NCEP GFS analysis), but for (a) the differences between the FS7LowLvl and FS7 experiments in a 1-month period from 16 Jan to 16 Feb 2020. Green (red) boxes denote that FS7LowLvl is better (worse) than FS7, with the size of the triangles representing the significance levels (see legend). (b) The differences between the FS7LowLvl and FS7GSIErr experiments in the same 1-month period. Green (red) boxes denote that FS7LowLvl is better (worse) than FS7GSIErr.

Citation: Monthly Weather Review 149, 7; 10.1175/MWR-D-20-0267.1

• Download Figure
• Download figure as PowerPoint slide

View Full Size

Differences in the RMSEs of (a) tropical (20°N–20°S) temperature (K) and (b) tropical vector winds (m s⁻¹) between the FS7LowLvl and FS7 experiments, shown in a cross section of pressure levels and forecast lead times. Greenish (reddish) colors mean that FS7LowLvl is more (less) accurate than FS7. The verification period is the same as that in Fig. 15.

Citation: Monthly Weather Review 149, 7; 10.1175/MWR-D-20-0267.1

• Download Figure
• Download figure as PowerPoint slide

In all, these results indicate that the current QC processes in GSI, long developed through a series of work (Cucurull et al. 2007, 2013; Cucurull 2015), remain useful for assimilating the FS7/C2 low-level data with the CWBGFS system. This finding can be reasonably understood by the fact that the error characteristics of the FS7/C2 data, viewed from global data assimilation, are largely similar to the other existing RO datasets (section 4d), for which the current QC algorithms were developed. However, we emphasize that it does not rule out the possibility for further improvements, such as relaxing some of the thresholds to use more low-level data given that the FS7/C2 low-level data are believed to have higher quality than the other datasets as shown by the validation study (Schreiner et al. 2020).

b. Impact of observation error tuning and further discussion

To investigate the pure impact of the CWB-estimated bending angle observation errors (obtained by the Desroziers method; see section 3c), the “FS7GSIErr” experiment is conducted using the GSI-default observation errors (Table 2); the 4-km HT QC is not used either. Note that all these settings in FS7GSIErr are exactly the same as those used in Period I of the FS7 parallel experiment, but now the experiment is conducted for the one-month period in Period II. Therefore, a relevant comparison can be made between the FS7LowLvl (using CWB-estimated RO observation errors) and FS7GSIErr (using GSI-default RO observation errors) experiments. The first glimpse at this comparison in the scorecard (Fig. 15b) seems to reinforce our previous impression from Fig. 9 that assimilation using the CWB-estimated observation errors leads to a somewhat inferior result. This is contradictory to our finding in the preparatory study conducted before the launch of FS7/C2 (section 3c) where the CWB-estimated observation errors were shown to be beneficial. However, alternative verifications with various verifying truths disagree with each other. Verified against the self-analysis, the experiment with the CWB-estimated observation errors displays overall better forecast than that with the GSI-default errors in the scorecard (figure not shown). Investigation on the actual values of the RMSEs and biases in these two experiments reveals that their differences are actually small (figure not shown), so the choice of the verifying truth is able to flip the results, which is likely related to the model bias problem discussed before. Additionally, a radiosonde observation-based verification (figure not shown) confirms that these two experiments exhibit similar short-term (6 h) forecast errors, where the experiment with the CWB-estimated errors (FS7LowLvl) generally performs slightly better for 6-h wind forecast but slightly worse for 6-h temperature forecast at upper troposphere.

At this point, it should be clarified that the self-analysis was used to verify the results in our preparatory study. Therefore, we think that the different choices of verifying truths would be the main reason for the opposite impacts obtained in the preparatory study and in the current FS7/C2 assimilation study when verified with the NCEP GFS analysis. The point that the preparatory study was not conducted with the FS7/C2 RO data is unlikely to be the main reason for the problem, because the error and bias characteristics of the FS7/C2 RO data (from the aspect of global data assimilation) have been shown to be roughly similar to the other RO data (section 4d). From all the above results and discussion, an appropriate interpretation would be that the degrees of optimality of these two observation errors (i.e., GSI-default and CWB-estimated), though their values differ by about twice, are actually similar. At the same time, we suspect that neither these two observation errors are really optimal, since there is still a large room for tuning in between. We leave this issue for future study.

In addition, we would like to mention NCEP’s results on this issue. Recently, they similarly estimated much larger RO bending angle observation errors compared with the GSI-default values using the Desroziers method with their current (FV3GFS-based) operational Global Forecast System (Shao et al. 2020). They further experimentally found that using the newly estimated observation errors improved the assimilation impact of the FS7/C2 RO data (Shao et al. 2020). Considering these NCEP’s findings, we are certain that the GSI-default RO observation errors, which were tuned for the NCEP’s system in the past, are not optimal even for the NCEP’s current operational system. Therefore, we again believe that careful tuning of the observation errors would be essential to optimize the FS7/C2 data assimilation with the CWBGFS-GSI system at CWB. Bowler (2020) recently conducted a comprehensive study on the RO observation error specification, providing some new insights that may be helpful to improve the RO data assimilation.