1. Introduction
Assimilation of all-sky satellite-measured observations, including cloud-affected radiance, are still quite limited, though usage of clear-sky satellite observations has been widely spread and has greatly improved the accuracy of numerical weather predictions. Geer and Bauer (2011, hereafter GB11) attributed part of the difficulty in assimilating cloudy-affected microwave radiances to their boundedness and heteroscedasticity, as well as strong nonlinear relationship to atmospheric states. These characteristics can severely degrade the impacts of data assimilation, by causing systematic errors, bias, representative errors, and errors in modeling convective-scale phenomena through forecasting models as well as microscale characteristics of clouds through forwarding models (Pires et al. 2010). Further difficulty in all-sky assimilation lies in the discontinuity between clear- and cloudy-sky radiances. Predicting the exact location of clouds is challenging because of either practical or intrinsic limits of atmospheric predictability (Lorenz 1969, 1982, 1996; Zhang et al. 2002, 2003, 2007). Practical predictability refers to the limit of prediction given the current realistic uncertainties in either the initial conditions or forecast model while intrinsic predictability refers to the fundamental limit in the atmospheric prediction with a nearly perfect model and nearly prefect initial conditions (Lorenz 1996; Melhauser and Zhang 2012; Tao and Zhang 2015; Sun and Zhang 2016). Large departures between the clear-sky surface temperature and cloud-top temperature can occur far more frequently than predicted by assuming Gaussianity in the background and observation error covariance. Many researchers reported that mismatches of the sky type (i.e., clear or cloudy) between the first guesses and observations often leads to a fat-tail distribution in the first-guess departures (e.g., GB11; Tavolato and Isaksen 2015).
Nevertheless, there are great potentials of assimilating cloudy-radiance observations for numerical weather prediction. For example, Otkin (2010, 2012) showed that direct assimilation of infrared cloudy radiances can improve prediction of convective-scale severe storms through strict quality control. Tavolato and Isaksen (2015) assessed the methodology to give relatively smaller weight to the observations with larger first-guess departures, and/or by inflating the observation errors. Though large observation errors may artificially suppress the analysis increments, they showed that the incremental accumulation of those small impacts can be large enough to make a distinguishable improvement from not using the cloudy radiances at all.
In this study, we assess the impacts of an adaptive observation error inflation (AOEI) technique for assimilating all-sky satellite brightness temperatures with the ensemble-based data assimilation framework first introduced in Zhang et al. (2016, hereafter ZMC16). Since the inner-core regions of tropical cyclones are mostly covered by clouds, existing studies on assimilation of satellite radiances for tropical cyclones have been mostly focused on the clear-sky observations away from the tropical cyclone cores (e.g., Wang et al. 2015; Zou et al. 2013, 2015). The impacts of AOEI were tested for the matured phase of Hurricane Karl (2010). This article is organized as follows. Model and experimental designs are described in section 2. The AOEI method is formulated and discussed in section 3. The performance of AOEI in comparison with existing methodologies is given in section 4. Conclusions are provided in section 5.
2. Model configuration and experimental design
As in ZMC16, we used the ensemble Kalman filter (EnKF) data assimilation system (Zhang et al. 2009, 2011; Weng and Zhang 2012, 2016) developed at The Pennsylvania State University (PSU), which is built around the Weather Research and Forecasting (WRF) Model (version 3.6.1) (Skamarock et al. 2008) and the Community Radiative Transfer Model (CRTM). This CRTM-integrated framework enables us to directly assimilate multiple channel brightness temperatures with high temporal and spatial resolution into PSU WRF-EnKF. As in ZMC16, there are three two-way nested model domains with grid spacings of 27, 9, and 3 km, respectively (refer to supplemental Fig. 1 of ZMC16 for domain configuration; the finest 3-km domain is shown and examined in all subsequent figures). All the domains used 61 levels with the model top at 50 hPa in a stretched vertical grid. For model physics, we utilized the WRF single-moment 6-class mixed-phase microphysics scheme (WSM6; Hong and Lim 2006), the Yonsei University planetary boundary scheme (Hong et al. 2006), the Tiedtke cumulous parameterization scheme (Tiedtke 1989), and the Rapid Radiative Transfer Model (RRTM) longwave and shortwave radiation schemes (Iacono et al. 2008). Cumulous parameterization was only applied to the coarsest (27 km) domain. Since nearly all convective activities associated with Hurricane Karl was within the finest (3 km) domain, the choice of whether or not using cumulous parameterization on the intermediate resolution (9 km) domain should not affect the results. CRTM is a rapid forward model to calculate brightness temperatures for the satellite data assimilation (Han et al. 2006, 2007; Weng 2007). The simulated brightness temperatures were computed with the successive order of interaction (SOI) forward solver (Heidinger et al. 2006) using the OPTRAN code from CRTM. The standard profiles for the tropical region defaulted in CRTM were used above the model top.
We conducted the perfect-model observing system simulation experiments (OSSEs) with the same 60 initial ensemble perturbations and the reference truth Karl simulation as the perfect-model OSSEs of ZMC16 (see configurations above). Hurricane Karl progressed westward across the Bay of Campeche before making landfall on 17 September. We computed brightness temperatures for hypothetical future GOES-R (now GOES-16) Advanced Baseline Imager [water vapor channels: channels 8, 9, and 10, wavelengths are 6.19, 6.95, and 7.34 μm, respectively; and window channel: channel 14, wavelength is 11.2 μm, refer to Schmit et al. (2005)] every 10 min. Synthetic observations were created by combining those simulated water vapor brightness temperatures (BTs) and assumed Gaussian-distributed universal, constant observation error with a standard deviation of 3 K, which includes both instrumental and representativeness errors. Those synthetic BT observations were thinned with the same channel selection method as employed in ZMC16, which retains for assimilation clear-sky radiances in all three water vapor channels (8, 9, and 10) in the clear-sky region and only the cloudy-sky radiances in channel 8. In addition to BTs, synthetic tropical cyclone minimum sea level pressure (SLP) observations available every hour were derived from the truth simulation with an assumed observation error of 3 hPa. The experiment that assimilated BT with AOEI and minimum SLP is hereafter called AOEI.
Slightly different from ZMC16, which employed a radius of influence of 30 km for hydrometeors and 200 km for other variables, this study utilizes the successive covariance localization (SCL) method proposed by Zhang et al. (2009), which allows better capturing convective-scale structures with a smaller covariance localization radius while updating vortex-scale structure with a larger covariance localization distance. Here the brightness temperatures are assimilated every 12 km by 12 km with a 30-km radius of influence and every 18 km by 18 km with a 300-km radius of influence. No vertical covariance localization is used in this study and ZMC16, while the covariance relaxation method of Zhang et al. (2004) with coefficient = 0.5 is applied. The assimilation was initialized at 2200 UTC 16 September and continued until 0600 UTC 17 September 2010 with brightness temperatures assimilated every 10 min and minimum SLP assimilated every hour. Despite using different radii of influence via SCL, the AOEI experiment performed very similarly to the experiment of ZMC16 that showed great potentials in assimilating all-sky radiance observations. The current study focuses on the methodology and effectiveness of using AOEI.
3. The adaptive observation error inflation (AOEI) algorithm
a. Error distribution
Figure 1c shows an example of the spatial distribution of the diagonal term (variance) of the ensemble-estimated forecast error covariances for the simulated brightness temperatures if observed by GOES-16 ABI channel 8 (6.19 μm) for Hurricane Karl at 2200 UTC 16 September (the verifying truth at this same time is shown in Fig. 1a just before the start of the initial EnKF assimilation of brightness temperature). The flow dependency of error distribution can be evidenced from Fig. 1b, which shows the squares of the first-guess departure (or innovation = observation − prior): the regions with larger departures, such as the rainbands in the northwest quadrant of the domain, are corresponding to regions with large background error variances (Fig. 1c). Figure 1d shows that the difference between the observed innovation square (
(a) Simulated brightness temperature from GOES-16 ABI channel 8, (b) the innovation square (
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
(a) Simulated brightness temperature from GOES-16 ABI channel 8, (b) the innovation square (
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
(a) Simulated brightness temperature from GOES-16 ABI channel 8, (b) the innovation square (
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
b. Observation error with a nonlinear observation operator
Figure 2 shows the ensemble distribution of the simulated BTs for GOES-16 ABI channel 8 (6.19 μm), column-integrated total hydrometeors, column-integrated total water vapor, and 10-m meridional wind at the selected locations marked in Fig. 1. The thermodynamic variables of hydrometeors and water vapor are more sensitive to the infrared brightness temperature measurements than the 10-m wind speed. The four points (A–D) sampled exemplify for the hurricane outside-of-core, eye, primary rainband, and eyewall regions, respectively. At point A, both the observation and the prior were categorized as in the clear-sky area. The truth is well within one standard deviation from the ensemble mean of BTs and the model state variables at point A (Figs. 2a–d). At the edge of the primary rainband where the ensemble members tend to diverge, the ensemble forecasted BTs in point B displayed a bimodal distribution with one peak typical of (low) cloudy-sky radiance and the other peak typical of (high) clear-sky-regime BT while the ensemble mean is close to the observation (Fig. 2e). Nevertheless, the true amount of water vapor mixing ratio and 10-m meridional wind are away from the range of values simulated by many ensemble members (Figs. 2g,h). Point C represents the hurricane eye (Figs. 2i–l), where the ensemble members are often forecasted as cloudy sky because of the difficulty in precisely capturing the location of the cloud-free (or cloudless) hurricane center in the truth simulation. Correspondingly, many of ensemble members forecasted a much lower BT with a larger amount of hydrometeors than the observation (Fig. 2e). However, though not perfect, the distributions of other model state variables did encompass the truth well approximately within their respective 1σ distribution ranges. Within the eyewall region (point D) where both model and observation tend to simulate strong clouds, the BT was better captured by the ensemble (Fig. 2m) compared to the primary rainband or eye region, but the ensemble distribution of model state variables in the eyewall are not necessarily consistent with BT (Figs. 2n–p). In particular, the true value of hydrometeors was considerably larger than the ensemble mean (Fig. 2n).
Prior ensemble distributions of the (a),(e),(i),(m) brightness temperature; (b),(f),(j),(n) column-integrated hydrometeors; (c),(g),(k),(o) column-integrated vapor mixing ratio; and (d),(h),(l),(p) the 10-m surface meridional wind. The sky-blue box shows the ensemble distribution, the blue line shows the estimated first-guess distribution based on Gaussian assumption, and the red star shows the verifying truth, for points A, B, C, and D marked in Fig. 1.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
Prior ensemble distributions of the (a),(e),(i),(m) brightness temperature; (b),(f),(j),(n) column-integrated hydrometeors; (c),(g),(k),(o) column-integrated vapor mixing ratio; and (d),(h),(l),(p) the 10-m surface meridional wind. The sky-blue box shows the ensemble distribution, the blue line shows the estimated first-guess distribution based on Gaussian assumption, and the red star shows the verifying truth, for points A, B, C, and D marked in Fig. 1.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
Prior ensemble distributions of the (a),(e),(i),(m) brightness temperature; (b),(f),(j),(n) column-integrated hydrometeors; (c),(g),(k),(o) column-integrated vapor mixing ratio; and (d),(h),(l),(p) the 10-m surface meridional wind. The sky-blue box shows the ensemble distribution, the blue line shows the estimated first-guess distribution based on Gaussian assumption, and the red star shows the verifying truth, for points A, B, C, and D marked in Fig. 1.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
A common practice in treating the inconsistency between the innovation and the prior ensemble spread is to inflate the ensemble covariance. However, the covariance inflation for the prior ensemble state variances may not necessarily inflate properly the ensemble spread in observation space when the observation operators are more nonlinear, which can be further complicated by the rank deficiency due to a limited ensemble size. Because large departures in the observation space do not necessarily correlate with large errors in the model state space, it might be advantageous to treat such uncertainties as part of the observation error for the lack of representativeness, rather than as part of the background error. It is clear that these representative errors may exist even if the forecast model is perfect, without considering errors in either the forecast model (WRF) and/or the radiance forward model (CRTM).
c. Adaptive estimation of observation error variance (AOEI)
More generally, observation error 
AOEI is intended to suppress erroneous analysis increments that are induced by the representative and sampling errors, which may also allow gradual transition between clear- and cloudy-sky regions. For example, the large analysis increment in SLP for the hypothetical BT observation in section 3b (with a large first-guess departure) will be limited to only 0.3 hPa with AOEI (instead of 15 hPa without AOEI). This much-reduced analysis increment is likely to maintain better balance, regardless of whether the forecasted sky conditions (clear or cloudy) match with the observation. Although it is expected and unavoidable that sometimes the use of AOEI may artificially suppress the “valid” analysis increments, this disadvantage may be partially compensated by assimilating large volumes of satellite radiance observations that will be available from GOES-16.
d. The alternative Geer–Bauer observation error inflation (GBOEI) method
To further demonstrate the effectiveness of AOEI, we compared the EnKF analysis using the newly proposed technique to those using two other types of observation error representation. The first is to assume observation error was constant (
(a),(c),(e) The Geer–Bauer observational error models for the BTs in GOES-16 ABI (a) channel 8, (c) channel 9, and (e) channel 10 used in experiment GBOEI. The standard deviation of first-guess departures is shown by a red solid line, which is modeled as a function of symmetric cloud effect parameter 
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
(a),(c),(e) The Geer–Bauer observational error models for the BTs in GOES-16 ABI (a) channel 8, (c) channel 9, and (e) channel 10 used in experiment GBOEI. The standard deviation of first-guess departures is shown by a red solid line, which is modeled as a function of symmetric cloud effect parameter
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
(a),(c),(e) The Geer–Bauer observational error models for the BTs in GOES-16 ABI (a) channel 8, (c) channel 9, and (e) channel 10 used in experiment GBOEI. The standard deviation of first-guess departures is shown by a red solid line, which is modeled as a function of symmetric cloud effect parameter
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
Figures 3b, 3d, and 3f compare distributions of the first-guess departures among AOEI, GBOEI, and no-OEI computed from each of the 60 ensemble members, all of which are normalized by the square root of the sum of the prior ensemble variance plus the observation errors assumed in each of the respective experiments. Both AOEI and GBOEI approached a more Gaussian-shaped distribution than by no-OEI. GBOEI fits better to the Gaussian distribution for large normalized first-guess departures (with an absolute value greater than 3) while AOEI overall has a better fit for relatively small normalized first-guess departures (approximately −2 to 2).
Figure 4 compares the spatial distribution of inflated observation errors 
The inflated observation error (K) for GOES-16 ABI channel 8 from (a),(c),(e) AOEI and (b),(d),(f) GBOEI, contour shows symmetric cloud effects parameter 
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
The inflated observation error (K) for GOES-16 ABI channel 8 from (a),(c),(e) AOEI and (b),(d),(f) GBOEI, contour shows symmetric cloud effects parameter
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
The inflated observation error (K) for GOES-16 ABI channel 8 from (a),(c),(e) AOEI and (b),(d),(f) GBOEI, contour shows symmetric cloud effects parameter
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
4. Potential impacts of AOEI
a. EnKF assimilation of all-sky radiance with different observation error representation
The truth versus posterior-simulated brightness temperatures of GOES-16 ABI channel 14 (11.2 μm), 10-m wind speed, water vapor mixing ratio, and SLP at 0, 3, and 6 h with the EnKF assimilation of the BTs and minimum SLP are shown (see Figs. 6–8) by applying three different representations of observation error: AOEI, GBOEI, and no-OEI. As an independent verification (i.e., these observations are not assimilated), we utilized simulated BT from channel 14, which is sensitive to cloud distribution. Experiment AOEI is almost the same as the perfect-model experiment, which assimilated both BTs and minimum SLP in ZMC16 except for using the SCL method for radius of influence (refer to section 2 and ZMC16). The impact of assimilating all-sky GOES-16 radiances has been clearly demonstrated in ZMC16 in terms of the accuracy of both the EnKF analysis and forecasts in comparison to experiment that only assimilated minimum SLP but not brightness temperatures. Here we compare the effectiveness and performance of the EnKF using two different observation error inflation methods (AOEI and GBOEI) as well as the one without observation error inflation (no-OEI).
Right after the initial assimilation cycle at 2200 UTC 16 September, the EnKF-analyzed BT field captured well the main characteristics of Hurricane Karl in the truth simulation, including both the primary eyewall and the outer rainbands (Figs. 5a–d). Continuous assimilations of BTs from channels 8–10 further improved the representation of the storm (Figs. 5f–h,j–l) and compared well with the verifying truth: by 0400 UTC 17 September, the EnKF analysis captured well the detailed structures of the clear region of the hurricane eye, the primary rainbands, as well as most of the individual convective clouds in the outer rainbands. While the analyzed BTs are overall similar among all three OSSEs (with flow-dependent adaptive observation error inflation), the analyzed 10-m wind speed and water vapor mixing ratio with AOEI and GBOEI are considerably smaller and more smoothed than no-OEI, though the overall patterns including the maximum in the eyewall and the highly asymmetric secondary maximum associated with the primary rainband are similar (Figs. 6 and 7).
The simulated brightness temperatures of GOES-16 ABI channel 14 at (a)–(d) 2200 UTC 16 Sep, (e)–(h) 0100 UTC 17 Sep, and (i)–(l) 0400 UTC 17 Sep from the (a),(e),(i) verifying truth; (b),(f),(j) OSSE with AOEI; (c),(g),(k) OSSE with GBOEI; and (d),(h),(l) OSSE with no-OEI.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
The simulated brightness temperatures of GOES-16 ABI channel 14 at (a)–(d) 2200 UTC 16 Sep, (e)–(h) 0100 UTC 17 Sep, and (i)–(l) 0400 UTC 17 Sep from the (a),(e),(i) verifying truth; (b),(f),(j) OSSE with AOEI; (c),(g),(k) OSSE with GBOEI; and (d),(h),(l) OSSE with no-OEI.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
The simulated brightness temperatures of GOES-16 ABI channel 14 at (a)–(d) 2200 UTC 16 Sep, (e)–(h) 0100 UTC 17 Sep, and (i)–(l) 0400 UTC 17 Sep from the (a),(e),(i) verifying truth; (b),(f),(j) OSSE with AOEI; (c),(g),(k) OSSE with GBOEI; and (d),(h),(l) OSSE with no-OEI.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
As in Fig. 5, but for the 10-m surface wind speed (color shading) and sea level pressure (contoured every 5 hPa).
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
As in Fig. 5, but for the 10-m surface wind speed (color shading) and sea level pressure (contoured every 5 hPa).
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
As in Fig. 5, but for the 10-m surface wind speed (color shading) and sea level pressure (contoured every 5 hPa).
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
As in Fig. 5, but for the vertically integrated water vapor mixing ratio (color shading).
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
As in Fig. 5, but for the vertically integrated water vapor mixing ratio (color shading).
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
As in Fig. 5, but for the vertically integrated water vapor mixing ratio (color shading).
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
b. Vertical error structure with different observation error representations
To further investigate the effectiveness of AOEI, we compare the impacts of using the three different representations of observation error in assimilating of a single cloudy radiance observations from GOES-16 ABI channel 8 on the EnKF analysis. The vertical distributions of the analysis error (at the observation grid) after each assimilation of the observations (points B and C marked in Fig. 1, respectively) that have a relatively large first-guess departure are shown in Fig. 8 as an example. The values for 
Vertical distribution of the prior error (gray shading), and posterior error for the EnKF analyses assimilating only one observation on each grid using AOEI (blue), GBOEI (red), and no-OEI (green), compared with different observation error representations at (a)–(c) point B and (d)–(f) point C marked in Fig. 1.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
Vertical distribution of the prior error (gray shading), and posterior error for the EnKF analyses assimilating only one observation on each grid using AOEI (blue), GBOEI (red), and no-OEI (green), compared with different observation error representations at (a)–(c) point B and (d)–(f) point C marked in Fig. 1.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
Vertical distribution of the prior error (gray shading), and posterior error for the EnKF analyses assimilating only one observation on each grid using AOEI (blue), GBOEI (red), and no-OEI (green), compared with different observation error representations at (a)–(c) point B and (d)–(f) point C marked in Fig. 1.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
Values (K) for 
At point B, because the ensemble spread is comparable to the first-guess departure, its observation error was slightly inflated using AOEI (and remains the same as in no-OEI). A smaller observation error (more accurate) and a larger first-guess departure will lead to larger analysis increments; with GBOEI, the analysis increments are much reduced and the analysis error remained large with the observation error inflation based on the simulation-derived climatology that mimics the approach in GB11 (Figs. 8a–c).
At point C, since the first-guess departure is much larger than the ensemble spread, the observation error is significantly inflated in AOEI, and thus the analysis increments with no-OEI, which are spuriously large for meridional wind, are effectively suppressed. The observation error is also inflated in GBOEI though to a lesser extent and thus the analysis increments remained considerably larger than those in AOEI but significantly smaller than those in no-OEI (Figs. 8d–f). Comparing with the verifying truth, these proof-of-concept single-observation assimilation experiments of the cloudy radiance observations at these two points exemplify the potential advantages of using AOEI over GBOEI and no-OEI given its use of situation-dependent observation error inflation. On the other hand, the impacts of a truly accurate observation and accurate correlation will likely be much reduced if the observation error is artificially inflated.
The relevance of these single-observation assimilation experiments is demonstrated through Fig. 9, which shows the comparison of the vertical distribution of the horizontal domain and temporally averaged root-mean-square error (RMSE) for the EnKF analysis of selected variables for each of the three OSSEs assimilating the all-sky radiances but with different representation of observation error. Overall, the RMSE both with AOEI and GBOEI are noticeably smaller than no-OEI (Figs. 9a–e), except for the hydrometeors, which is more directly connected to observed variables and, thus, is likely to have less representativeness error (Fig. 9f). The RMSEs with AOEI and GBOEI are mostly similar, but AOEI (GBOEI) slightly better analyzed lower (upper) troposphere. Comparison of the cloudy-affected and other indirectly affected variables further supports the potential effectiveness of inflating observation errors to deal with the representativeness error.
Vertical distribution of the square root of the domain-averaged EnKF analysis errors averaged over all hourly analysis times with AOEI (blue), GBOEI (red), and no-OEI (green) during 2200 UTC 16 Sep–0400 UTC 17 Sep for (a) zonal wind, (b) meridional wind, (c) temperature, (d) vertical wind, (e) mixing ratio, and (f) all hydrometeors.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
Vertical distribution of the square root of the domain-averaged EnKF analysis errors averaged over all hourly analysis times with AOEI (blue), GBOEI (red), and no-OEI (green) during 2200 UTC 16 Sep–0400 UTC 17 Sep for (a) zonal wind, (b) meridional wind, (c) temperature, (d) vertical wind, (e) mixing ratio, and (f) all hydrometeors.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
Vertical distribution of the square root of the domain-averaged EnKF analysis errors averaged over all hourly analysis times with AOEI (blue), GBOEI (red), and no-OEI (green) during 2200 UTC 16 Sep–0400 UTC 17 Sep for (a) zonal wind, (b) meridional wind, (c) temperature, (d) vertical wind, (e) mixing ratio, and (f) all hydrometeors.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
c. Temporal error growth with different observation error representations
More quantitatively, Fig. 10 shows the comparison of the time evolutions of the domain-averaged RMSE of the prior estimate and EnKF analysis of selected variables. The difference in RMSEs among three all-sky OSSEs are small for cloud-affected variables (Figs. 10f,h,i) but large for other indirectly affected variables (Figs. 10a–e,g), which is consistent with Figs. 5–7 that compared brightness temperatures, 10-m wind speed, water vapor mixing ratio, and SLP. The RMSEs of in particular indirectly affected variables with AOEI are slightly smaller than those in GBOEI, possibly by pulling more information from observations through overall smaller observation errors while suppressing potentially “erroneous” analysis increments through larger observation errors in localized area as shown in Fig. 4. Both AOEI and GBOEI are considerably smaller than no-OEI likely by limiting the analysis increments where there are large representative errors. Moreover, at most of the assimilation time, the EnKF updates with AOEI and GBOEI improved all variables from prior to analysis except for water vapor mixing ratio (which is likely due to more small-scale clouds influences and thus more representative error in the RMSE), while those with no-OEI often worsened the analysis of indirectly affected variables over prior, suggesting the accuracy of the sampling correlation between the BT observation and water vapor may be low especially under clouds while there are large representativeness errors in either BTs and/or water vapor fields.
Temporal evolution of domain-averaged EnKF RMSEs with AOEI (blue), GBOEI (red), no-OEI (green) and simultaneous estimation method (black) for (a) zonal wind, (b) meridional wind, (c) temperature, (d) vertical wind, (e) water vapor mixing ratio, (f) sum of mixing ratio for all hydrometeors, (g) sea level pressure, (h) brightness temperature of GOES-16 ABI channel 14, and (i) brightness temperature of GOES-16 ABI channel 8.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
Temporal evolution of domain-averaged EnKF RMSEs with AOEI (blue), GBOEI (red), no-OEI (green) and simultaneous estimation method (black) for (a) zonal wind, (b) meridional wind, (c) temperature, (d) vertical wind, (e) water vapor mixing ratio, (f) sum of mixing ratio for all hydrometeors, (g) sea level pressure, (h) brightness temperature of GOES-16 ABI channel 14, and (i) brightness temperature of GOES-16 ABI channel 8.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
Temporal evolution of domain-averaged EnKF RMSEs with AOEI (blue), GBOEI (red), no-OEI (green) and simultaneous estimation method (black) for (a) zonal wind, (b) meridional wind, (c) temperature, (d) vertical wind, (e) water vapor mixing ratio, (f) sum of mixing ratio for all hydrometeors, (g) sea level pressure, (h) brightness temperature of GOES-16 ABI channel 14, and (i) brightness temperature of GOES-16 ABI channel 8.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
To further explore the impact of AOEI, we also conducted another experiment using the method proposed by Li et al. (2009) that simultaneously estimates both the observation error and a multiplicative inflation factor. Following Li et al. (2009), the range of inflation factor was set between 0.9 and 1.2. After each assimilation cycle, we estimated the observation error and the inflation factor for BTs, both of which are assumed to be uniform within the entire domain 3 (without any consideration of heteroscedasticity). These estimated values will be applied for the subsequent assimilation cycle. The temporal evolution of RMSEs for this additional experiment are also shown in Fig. 10 as black lines (denoted as “simultaneous_est”). RMSEs with the Li et al.’s simultaneous estimation method were similar to those from the no-OEI experiment, but were noticeably larger than both the AOEI and GBOEI experiments, highlighting the merit of modeling heteroscedasticity and representativeness error by either GBOEI or AOEI.
To demonstrate the influence of small-scale structures and their associated representative error, we conducted two-dimensional (2D) Fourier decomposition on the prior estimate and EnKF analysis of the horizontally detrended selected variables to divide Fig. 10 into scales with 2D horizontal wavelengths larger than 300 km (Fig. 11) and smaller than 300 km (Fig. 12). AOEI showed some advantage over GBOEI in updating cloudy-affected variables at larger scales (Figs. 11f,h,i), possibly because AOEI modeled smaller observation error variance for most of the domain as shown in Fig. 4, while GBOEI better analyzed smaller-scale structures (Figs. 12f,h,i), with relatively smaller observation error variances in the regions where AOEI locally has large observation error inflation. The RMSEs are overall similar among three OSSEs for other indirectly affected variables in large scales (Figs. 11a–e,g), but distinguishably smaller for AOEI and GBOEI compared to no-OEI for indirectly affected variables in small scales (Figs. 12a–e,g), in particular for temperature and the mixing ratio. Moreover, the analysis RMSEs most of the time became larger than the prior for no-OEI, but they are noticeably suppressed for both AOEI and GBOEI. Results shown in Figs. 10–12 further suggested the potentials of AOEI to better control the erroneous analysis increments through assimilating brightness temperatures, which are likely due to representative error and sampling issues mainly in small-scale structures. Figure 4 further indicates the potential advantages of AOEI over GBOEI, both of which will lead to much smaller RMSEs than no-OEI for variables that do not directly affect or are not directly affected by the radiance being assimilated.
As in Fig. 10, but for the horizontal scales larger than 300 km.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
As in Fig. 10, but for the horizontal scales larger than 300 km.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
As in Fig. 10, but for the horizontal scales larger than 300 km.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
As in Fig. 10, but for the horizontal scales smaller than 300 km.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
As in Fig. 10, but for the horizontal scales smaller than 300 km.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
As in Fig. 10, but for the horizontal scales smaller than 300 km.
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
d. Dynamical imbalance
To further estimate the potential imbalance induced by EnKF data assimilation cycle, Fig. 13 compared the second derivative of surface pressure at each time step during the first 1-h integration of the deterministic forecasts from EnKF analysis mean. The domain root-mean-square (RMS) of 
Root of domain-averaged square of 
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
Root of domain-averaged square of
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
Root of domain-averaged square of
Citation: Monthly Weather Review 145, 3; 10.1175/MWR-D-16-0257.1
5. Conclusions
An empirical flow-dependent adaptive observation error inflation (AOEI) method is proposed in this study for assimilating all-sky satellite data under the EnKF framework. We assessed the potential usage of ensemble-based flow-dependent covariance structures on the estimation of adaptive observation error. The observation error was inflated if the square of the innovation (first-guess departure) exceeds the summation of background variance and original observation error variance. We have shown that the satellite radiances can have much larger representative error than the instrumental noises, either due to the strong nonlinearity in the observation operator and/or in the sampling error because of limited ensemble size. It can also be due to the strong nonlinearity and variability, and limited predictability, for the moist atmosphere where the radiances are affected by clouds. Those large representative errors are shown to exist even under a perfect-model scenario without any model error. The AOEI method is designed to limit unrepresentative innovations, inaccurate sampling correlation, and the associated large but potentially erroneous analysis increments. The potential impacts of AOEI were investigated through a series of convection-permitting perfect-model observing system simulation experiments using the PSU WRF-EnKF system, in comparison with experiments using an alternative empirical observation error inflation method proposed by Geer and Bauer (2011) as well as the experiments with a constant radiance observation error everywhere. Both methods with observation error inflation have shown to perform similarly and effectively in suppressing potentially large, erroneous analysis increments that will help to mitigate unphysical flow imbalance induced by the EnKF analysis, comparted to experiment using a constant observation error without inflation. There may be some potential advantages of the newly proposed AOEI method over the Geer–Bauer method in better updating some state variables at large scales and in better maintaining flow balance likely because of its flow- and situation-dependent observation error modeling. The AOEI method is also easier and more readily applied in the ensemble data assimilation framework that does not need to have climatological error distribution statistics. Preliminary results using the above methods to assimilate real-world observations (not shown) are also promising, and are consistent with results from the perfect-model OSSEs presented in this study. Future work will further examine the effectiveness of the proposed AOEI method for real-data observations that have more complex sources of errors likely not being sufficiently represented in this study.
One of the key assumptions made in this study is the ability of the ensemble to accurately represent the background uncertainty (spread) and correlations. Since the ensemble spread is also used here to estimate the flow-dependent observation error, an insufficient ensemble spread due to model error, limited ensemble size, and/or filter-divergence, may lead to weighting the observation inappropriately small and the underuse of otherwise more accurate observations. The evaluation of the amount of ensemble spread and correlations, as well as the methodology to improve them would be required in future study that includes, but is not limited to, the adaptive estimation of inflation factor (e.g., Li et al. 2009; Ying and Zhang 2015), variable covariance localization (e.g., Zhang et al. 2009; Zhen and Zhang 2014), more realistic and complicated errors in the forecast and forward models, to make further advances in fully utilized all-sky satellite radiances. More systematic evaluations of the proposed adaptive method, its variants, and sensitivity, in comparison with alternative methods, will be performed in the future for a large number of cases, and for different platforms of satellite measurements.
Acknowledgments
This research is partially supported by NSF Grant 1305798, ONR Grant N000140910526, and NASA Grants NNX16AD84G and NNX15AQ51G. Review comments by two anonymous reviewers were beneficial and greatly appreciated. MM was also supported by Japan’s Funai Overseas Scholarship of the Funai Foundation for Information Technology. We benefited from discussions with Eugene Clothiaux, Yonghui Weng, Michael Ying, Scott Sieron, Jason Otkin, Eugenia Kalnay, Fuzhong Weng, Alan Geer, and many others. Computing was provided by the Texas Advanced Computing Center (TACC). All data presented are stored and can be accessed through the TACC data archive.
REFERENCES
Dee, D. P., 1995: Online estimation of error covariance parameters for atmospheric data assimilation. Mon. Wea. Rev., 123, 1128–1145, doi:10.1175/1520-0493(1995)123<1128:OLEOEC>2.0.CO;2.
Desroziers, G., L. Berre, B. Chapnik, and P. Poli, 2005: Diagnosis of observation, background and analysis-error statistics in observation space. Quart. J. Roy. Meteor. Soc., 131, 3385–3396, doi:10.1256/qj.05.108.
Evensen, G., 2003: The ensemble Kalman filter: Theoretical formulation and practical implementation. Ocean Dyn., 53, 343–367, doi:10.1007/s10236-003-0036-9.
Geer, A. J., and P. Bauer, 2011: Observation errors in all-sky data assimilation. Quart. J. Roy. Meteor. Soc., 137, 2024–2037, doi:10.1002/qj.830.
Greybush, S. J., E. Kalnay, T. Miyoshi, K. Ide, and B. R. Hunt, 2011: Balance and ensemble Kalman filter localization techniques. Mon. Wea. Rev., 139, 511–522, doi:10.1175/2010MWR3328.1.
Han, Y., P. van Delst, Q. Liu, F. Weng, B. Yan, R. Treadon, and J. Derber, 2006: JCSDA Community Radiative Transfer Model (CRTM)—Version 1. NOAA Tech. Rep. NESDIS 122, 40 pp. [Available online at http://docs.lib.noaa.gov/noaa_documents/NESDIS/TR_NESDIS/TR_NESDIS_122.pdf.]
Han, Y., F. Weng, Q. Liu, and P. van Delst, 2007: A fast radiative transfer model for SSMIS upper atmosphere sounding channels. J. Geophys. Res., 112, D11121, doi:10.1029/2006JD008208.
Harnisch, F., M. Weissmann, and Á. Periáñez, 2016: Error model for the assimilation of cloud-affected infrared satellite observations in an ensemble data assimilation system. Quart. J. Roy. Meteor. Soc., 142, 1797–1808, doi:10.1002/qj.2776.
Heidinger, A., C. O’Dell, R. Bennartz, and T. Greenwald, 2006: The successive-order-of-interaction radiative transfer model. Part I: Model development. J. Appl. Meteor. Climatol., 45, 1388–1402, doi:10.1175/JAM2387.1.
Hong, S.-Y., and J. Lim, 2006: The WRF single-moment 6-class microphysics scheme (WSM6). J. Korean Meteor. Soc., 42 (2), 129–151.
Hong, S.-Y., Y. Noh, and J. Dudhia, 2006: A new vertical diffusion package with an explicit treatment of entrainment processes. Mon. Wea. Rev., 134, 2318–2341, doi:10.1175/MWR3199.1.
Houtekamer, P. L., and H. L. Mitchell, 2005: Ensemble Kalman filtering. Quart. J. Roy. Meteor. Soc., 131, 3269–3289, doi:10.1256/qj.05.135.
Iacono, M. J., J. S. Delamere, E. J. Mlawer, M. W. Shephard, S. A. Clough, and W. D. Collins, 2008: Radiative forcing by long-lived greenhouse gases: Calculations with the AER radiative transfer models. J. Geophys. Res., 113, D13103, doi:10.1029/2008JD009944.
Li, H., E. Kalnay, and T. Miyoshi, 2009: Simultaneous estimation of covariance inflation and observation errors within an ensemble Kalman filter. Quart. J. Roy. Meteor. Soc., 135, 523–533, doi:10.1002/qj.371.
Lorenz, E. N., 1969: Atmospheric predictability as revealed by naturally occurring analogues. J. Atmos. Sci., 26, 636–646, doi:10.1175/1520-0469(1969)26<636:APARBN>2.0.CO;2.
Lorenz, E. N., 1982: Atmospheric predictability experiments with a large numerical model. Tellus, 34A, 505–513, doi:10.1111/j.2153-3490.1982.tb01839.x.
Lorenz, E. N., 1996: Predictability—A problem partly solved. Proc. Seminar on Predictability, Reading, United Kingdom, ECMWF, 1–18.
Melhauser, C., and F. Zhang, 2012: Practical and intrinsic predictability of severe and convective weather at the mesoscales. J. Atmos. Sci., 69, 3350–3371, doi:10.1175/JAS-D-11-0315.1.
Okamoto, K., A. P. McNally, and W. Bell, 2014: Progress towards the assimilation of all-sky infrared radiances: An evaluation of cloud effects. Quart. J. Roy. Meteor. Soc., 140, 1603–1614, doi:10.1002/qj.2242.
Otkin, J. A., 2010: Clear and cloudy sky infrared brightness temperature assimilation using an ensemble Kalman filter. J. Geophys. Res., 115, D19207, doi:10.1029/2009JD013759.
Otkin, J. A., 2012: Assimilation of water vapor sensitive infrared brightness temperature observations during a high impact weather event. J. Geophys. Res., 117, D19203, doi:10.1029/2012JD017568.
Parrish, D. F., and J. C. Derber, 1992: The National Meteorological Center’s Spectral Statistical-Interpolation Analysis System. Mon. Wea. Rev., 120, 1747–1763, doi:10.1175/1520-0493(1992)120<1747:TNMCSS>2.0.CO;2.
Pires, C. A., O. Talagrand, and M. Bocquet, 2010: Diagnosis and impacts of non-Gaussianity of innovations in data assimilation. Physica D, 239, 1701–1717, doi:10.1016/j.physd.2010.05.006.
Poterjoy, J., and F. Zhang, 2014: Intercomparison and coupling of ensemble and four-dimensional variational data assimilation methods for the analysis and forecasting of Hurricane Karl (2010). Mon. Wea. Rev., 142, 3347–3364, doi:10.1175/MWR-D-13-00394.1.
Schmit, T. J., M. M. Gunshor, W. P. Menzel, J. J. Gurka, J. Li, and S. Bachmeier, 2005: Introducing the next-generation advanced baseline imager on GOES-R. Bull. Amer. Meteor. Soc., 86, 1079–1096, doi:10.1175/BAMS-86-8-1079.
Skamarock, W. C., and Coauthors, 2008: A description of the Advanced Research WRF version 3. NCAR Tech. Note NCAR/TN-475+STR, 113 pp., doi:10.5065/D68S4MVH.
Sun, Y. Q., and F. Zhang, 2016: Intrinsic versus practical limits of atmospheric predictability and the significance of the butterfly effect. J. Atmos. Sci., 73, 1419–1438, doi:10.1175/JAS-D-15-0142.1.
Tao, D., and F. Zhang, 2015: Effects of vertical wind shear on the predictability of tropical cyclones: Practical versus intrinsic limit. J. Adv. Model. Earth Syst., 7, 1534–1553, doi:10.1002/2015MS000474.
Tavolato, C., and L. Isaksen, 2015: On the use of a Huber norm for observation quality control in the ECMWF 4D-Var. Quart. J. Roy. Meteor. Soc., 141, 1514–1527, doi:10.1002/qj.2440.
Tiedtke, M., 1989: A comprehensive mass flux scheme for cumulus parameterization in large-scale models. Mon. Wea. Rev., 117, 1779–1800, doi:10.1175/1520-0493(1989)117<1779:ACMFSF>2.0.CO;2.
Wang, P., and Coauthors, 2015: Assimilation of thermodynamic information from advanced infrared sounders under partially cloudy skies for regional NWP. J. Geophys. Res. Atmos., 120, 5469–5484, doi:10.1002/2014JD022976.
Weng, F., 2007: Advances in radiative transfer modeling in support of satellite data assimilation. J. Atmos. Sci., 64, 3799–3807, doi:10.1175/2007JAS2112.1.
Weng, Y., and F. Zhang, 2012: Assimilating airborne Doppler radar observations with an ensemble Kalman filter for convection-permitting hurricane initialization and prediction: Katrina (2005). Mon. Wea. Rev., 140, 841–859, doi:10.1175/2011MWR3602.1.
Weng, Y., and F. Zhang, 2016: Advances in convection-permitting tropical cyclone analysis and prediction through EnKF assimilation of reconnaissance aircraft observations. J. Meteor. Soc. Japan, 94, 345–358, doi:10.2151/jmsj.2016-018.
Ying, Y., and F. Zhang, 2015: An adaptive covariance relaxation method for ensemble data assimilation. Quart. J. Roy. Meteor. Soc., 141, 2898–2906, doi:10.1002/qj.2576.
Zhang, F., C. Snyder, and R. Rotunno, 2002: Mesoscale predictability of the “Surprise” snowstorm of 24–25 January 2000. Mon. Wea. Rev., 130, 1617–1632, doi:10.1175/1520-0493(2002)130<1617:MPOTSS>2.0.CO;2.
Zhang, F., C. Snyder, and R. Rotunno, 2003: Effects of moist convection on mesoscale predictability. J. Atmos. Sci., 60, 1173–1185, doi:10.1175/1520-0469(2003)060<1173:EOMCOM>2.0.CO;2.
Zhang, F., C. Snyder, and J. Sun, 2004: Impacts of initial estimate and observation availability on convective-scale data assimilation with an ensemble Kalman filter. Mon. Wea. Rev., 132, 1238–1253, doi:10.1175/1520-0493(2004)132<1238:IOIEAO>2.0.CO;2.
Zhang, F., N. Bei, R. Rotunno, C. Snyder, and C. C. Epifanio, 2007: Mesoscale predictability of moist baroclinic waves: Convection-permitting experiments and multistage error growth dynamics. J. Atmos. Sci., 64, 3579–3594, doi:10.1175/JAS4028.1.
Zhang, F., Y. Weng, J. A. Sippel, Z. Meng, and C. H. Bishop, 2009: Cloud-resolving hurricane initialization and prediction through assimilation of Doppler radar observations with an ensemble Kalman filter. Mon. Wea. Rev., 137, 2105–2125, doi:10.1175/2009MWR2645.1.
Zhang, F., Y. Weng, J. F. Gamache, and F. D. Marks, 2011: Performance of convection-permitting hurricane initialization and prediction during 2008–2010 with ensemble data assimilation of inner-core airborne Doppler radar observations. Geophys. Res. Lett., 38, L15810, doi:10.1029/2011GL048469.
Zhang, F., M. Minamide, and E. E. Clothiaux, 2016: Potential impacts of assimilating all-sky infrared satellite radiances from GOES-R on convection-permitting analysis and prediction of tropical cyclones. Geophys. Res. Lett., 43, 2954–2963, doi:10.1002/2016GL068468.
Zhen, Y., and F. Zhang, 2014: A probabilistic approach to adaptive covariance localization for serial ensemble square root filters. Mon. Wea. Rev., 142, 4499–4518, doi:10.1175/MWR-D-13-00390.1.
Zou, X., F. Weng, B. Zhang, L. Lin, Z. Qin, and V. Tallapragada, 2013: Impacts of assimilation of ATMS data in HWRF on track and intensity forecasts of 2012 four landfall hurricanes. J. Geophys. Res. Atmos., 118, 11 558–11 576, doi:10.1002/2013JD020405.
Zou, X., Z. Qin, and Y. Zheng, 2015: Improved tropical storm forecasts with GOES-13/15 imager radiance assimilation and asymmetric vortex initialization in HWRF. Mon. Wea. Rev., 143, 2485–2505, doi:10.1175/MWR-D-14-00223.1.
