1. Introduction
Data assimilation (DA) methods produce the best estimate of the current state of a dynamical system by combining the background and observations with an “optimal” weight. The optimal weight, denoted
Reflecting the crucial importance of correctly specifying
Another relatively new approach, which has not been covered by Buehner (2010), originates from the adjoint sensitivity studies. The ground-breaking work by Langland and Baker (2004) introduced forecast sensitivity to observations (FSO), a technique that allows us to estimate how much each of the assimilated observations reduced or increased the forecast errors measured with some quadratic norm, without having to perform the expensive observing system experiments (OSEs). Daescu (2008) generalized the FSO technique and gave a formulation for forecast sensitivity to the background error covariance (
While being a powerful diagnostic tool, the applicability of the adjoint sensitivity methods such as FSO and Daescu’s methods had been somewhat limited because these approaches require the adjoint of the forecast model, which is difficult to develop and/or maintain. For FSO, this limitation has been recently resolved by adapting it to EnKF (Liu and Kalnay 2008; Li et al. 2010; Kalnay et al. 2012). The most recent formulation of the ensemble-based FSO, or EFSO, proposed by Kalnay et al. (2012), has been successfully implemented into a quasi-operational global EnKF system of the National Centers for Environmental Prediction (NCEP; Ota et al. 2013), a German convective-scale regional EnKF DA system (Sommer and Weissmann 2014), and JMA’s global DA system.
The objective of this paper is to show that it is possible, by combining the derivations of EFSO in Kalnay et al. (2012) and the
2. EFSR formulation
In this section we introduce our EFSR formulation, building upon the derivations of Daescu (2008) and Kalnay et al. (2012).
a. Forecast sensitivity to each element of 

1) General formulation



























2) Adjoint-based evaluation of Eq. (4) within a 4D-Var























3) Ensemble-based implementation (EFSR)

















The computational cost required to evaluate Eq. (9) or (10) is not very expensive. Denoting the dimension of the system’s state vector and the number of observations by Nstate and Nobs, respectively, an explicit evaluation of Eq. (9) requires only ~Nstate × K operations (for multiplying the vector
For convenience, we call the forecast sensitivity to observation error covariance matrix FSR (short for forecast sensitivity to
We emphasize that, unlike other diagnostic methods for optimality of
b. Sensitivity to scaling factors










c. Sensitivity to the covariance inflation factor














3. Toy-model experiments using the Lorenz ’96 model
a. Model and DA system



As the DA system, we adopt the local ensemble transform Kalman filter (LETKF; Hunt et al. 2007) with member size K = 40. Since the member size is equal to the dimension of the state space, there is no need for covariance localization in our experiments. To avoid filter divergence, however, we applied multiplicative covariance inflation (Anderson 2001) with a constant inflation parameter


b. Experimental design
First, we produced the nature (or “truth”) by running the forecast model Eq. (17) from an initial condition randomly generated from the uniform distribution in [0, 1]. The nature run is integrated from time t = 0 to 730 (which corresponds to 10 yr in dimensional time), generating truth for 14 600 cycles.
The initial background ensemble at time t = 0 is generated by picking up 40 truth states at 40 randomly chosen distinctive dates. Each DA experiment is run for 14 600 cycles (10 yr) and the first 1460 cycles (1 yr), regarded as a spinup period, are excluded from verification.
To examine the ability of AFSR and EFSR to detect the misspecification of observation error variances
The true and prescribed observation error variances for the experiments performed using the Lorenz ’96 model.
c. The SPIKE experiment
The SPIKE experiment is inspired by Liu and Kalnay (2008) and Kalnay et al. (2012), who examined the capacity of EFSO to capture the negative impacts from the observations at the 11th grid point that have larger observation errors than the others. In this experiment, all observations but the one at the 11th grid point have the error variance 0.22, while at the 11th grid point, it is 0.82. In the incorrect-
We first examine the analysis errors with respect to the truth to ensure that the system did not suffer from any malfunction (a “filter divergence” in particular). Figure 1a shows the root-mean-square errors (RMSEs) of analysis verified against the truth averaged over the 9 yr for the correct-
(a) Analysis errors verified against the truth for the SPIKE experiment displayed as a function of grid number. The filled circles and open squares show the analysis errors, respectively, for the incorrect-
Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1
We now examine the FSR diagnosed by ensemble- and adjoint-based methods. Figure 1b shows the EFSR-based forecast sensitivity to scaling factors of

This raises one concern: the FSR methods may not give a reliable diagnostic if accurate and inaccurate observations are located close to each other. This concern is addressed in the next experiment.The sensitivity gradient
, being a partial derivative, tells us how, for each index i, a small displacement in from unity would change the forecast error if the prescribed error variances for other observations are kept unchanged. Thus, if there is an observation that makes the forecast worse, then we can make the forecast better by giving higher credence to the adjacent, more accurate observations.
d. The STAGGERED experiment
The STAGGERED experiment is designed to assess whether the FSR diagnostics are robust to cases where observations with different magnitudes of error are located close to each other. The true observation error variances are 0.12 and 0.32, respectively, for odd- and even-numbered grid points. In the incorrect-
Figure 2a shows the analysis RMSE verified against the truth for the STAGGERED experiment. Both incorrect-
As in Fig. 1, but for the STAGGERED experiment.
Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1
The EFSR-based forecast sensitivity to scaling factors of
e. Adaptive online estimation of 
and inflation factor guided by EFSR

EFSR diagnostics allows us to estimate the gradient of the scalar forecast error
This algorithm does not update the observation error variance scaling factors
We implemented the above adaptive algorithm and experimented with many choices of
The results from the experiment with
Results from an adaptive online
Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1
Different choices of the parameters
4. Experiments with a real DA system: System description and experimental setup
We implemented and tested EFSR on the NCEP’s quasi-operational global NWP system designed to test the new proactive quality control (Hotta et al. 2017). The system is based on the operational suite that had been operational until January 2015 but with the reduced horizontal resolutions of T254 for the deterministic runs and T126 for the ensemble (as opposed to the operational T574 and T254). In this two-way interactive hybrid DA system, the variational Gridpoint Statistical Interpolation analysis system (GSI) incorporates flow-dependent background ensemble covariance from the EnKF first-guess perturbations to produce a deterministic analysis, and the EnKF analysis ensemble is recentered on the deterministic analysis thus produced. The weights given to the static and ensemble parts of the covariance in the hybrid GSI are 25% and 75%, respectively. The EnKF part of the DA system, which, in the operational suite, is the serial ensemble square root filter (EnSRF) of Whitaker and Hamill (2002), is replaced with the LETKF. The ensemble size is 80 and both localization and inflation are applied to the covariance. We note that the localization and inflation parameters used in this experiment are tuned for the operational higher-resolution system and thus may be suboptimal for our system. Nevertheless, our system worked well and without any problems.
We remark that in a hybrid DA system, the Kalman gain assumed in EFSR and EFSO computation differs from what is actually used in the hybrid analysis. Because of this inconsistency, EFSR and EFSO may not correctly estimate the sensitivities on
The experiment is performed with 6-hourly cycles for the 31-day period from 8 January to 8 February 2012, with a 7-day spinup period from 1 to 7 January 2012. All observations that were assimilated in the operational system during this period are also assimilated in our experiments. Observations are grouped into different types as in Ota et al. (2013) and Hotta et al. (2017). The EFSO impacts and EFSR-based forecast sensitivities to
5. Experiments with a real DA system: Results
Figure 4 shows the EFSR-based forecast sensitivities to
Forecast sensitivity to observation error scaling factors for different types of observations measured with the (a),(b) moist or (c),(d) dry total energy norms, at (a),(c) 6- or (b),(d) 24-h lead time averaged over the 31-day period. The units are J kg−1. The observation types whose error variances are scaled in the
Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1
We can also observe from Fig. 4 that, among all the observation types, aircraft, radiosonde, and Advanced Microwave Sounding Unit A (AMSU-A) exhibit higher sensitivities than the others, and that MODIS winds show negative sensitivity. This feature is consistently seen in any combination of the lead times and the error norms.
To assess the validity of the EFSR diagnostics described above, we performed an
If our EFSR diagnostics are valid, the use of the new
Time series of (line a, thick black solid line) the error of the 24-h forecast from analysis
Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1
The fact that no statistically significant differences were found between the two cycled experiments may seem disappointing, but this was in fact an expected outcome because the estimated 24-h forecast improvement by the scaling of
As in Fig. 4b, but computed for the data from the
Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1
Consistent with the insignificant differences in the forecast error reductions (
Forecast sensitivity to observation impact before (top bars) and after (bottom bars) the observation error variance scaling, evaluated for different types of observations measured with the moist total energy norm and 24-h lead time averaged over the 31-day period. The error bars on the top bars represent the confidence intervals at the 95% level computed by a t test for the differences among the paired data (see text for details). The units are J kg−1. As in Fig. 4, the observation types whose error variances are scaled in the
Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1
From Fig. 7 we can observe the following features. First, the EFSO impacts from aircraft, radiosonde, and AMSU-A all were statistically significantly increased by reducing their
6. Conclusions
The observation error covariance matrix
The main focus of this paper was on verifying the validity of the EFSR formulation. As such, our tuning effort to improve the operational system was only preliminary. In our experiment with the NCEP’s quasi-operational system, we examined the
One limitation of FSR diagnostics is that they only suggest whether we should reduce or increase each component of
In this study, we focused on the forecast sensitivity to observation error variances (i.e., diagonal elements of
Perhaps our EFSR will prove to be most useful when a new observing system is introduced to a DA system. In such a situation, in addition to assigning optimal
Finally, we note that an adjoint-based sensitivity methodology has been recently extended by Shaw and Daescu (2017) to weak-constraint 4D-Var formulation, allowing for the estimation of forecast sensitivity with respect to model-error bias and covariance specification; Shaw and Daescu (2017) also devised an online tuning procedure for these parameters based on the sensitivity guidance. In parallel with weak-constraint 4D-Var, EnKF formulations that explicitly account for model error biases have been proposed (e.g., Baek et al. 2006; Li et al. 2009b), and it will be an interesting future direction to explore the ensemble-based estimation of forecast sensitivity to prescribed parameters related to model errors within the framework of such model-error-aware variants of EnKF.
Acknowledgments
This work is inspired by Dr. Dacian Daescu’s talk delivered in 2013 at a seminar of the Weather and Chaos Group of University of Maryland. The manuscript grew out from DH’s Ph.D. dissertation, which was supported by Japan Meteorological Agency (JMA) through the Japanese Government Long-term Overseas Fellowship Program. The experiments with the NCEP’s quasi-operational system are performed on the Joint Center for Satellite Data Assimilation’s (JCSDA) Supercomputer for Satellite Simulations and Data Assimilation Studies [the “S4 supercomputer”; Boukabara et al. (2016)]. We express our gratitude to Dr. Sid Boukabara for his kind support and to Dr. Jim Jung for his guidance on using the Global Forecast System (GFS) and the GSI on the S4. We thank three anonymous reviews for their insightful comments that significantly improved the manuscript. This work was partially supported by a NOAA/NESDIS/JPSS Proving Ground (PG) and a Risk Reduction (RR) CICS grant (NA14NES4320003) and a JSPS KAKENHI Grant-in-Aid for Research Activity Start-Up (17H07352).
REFERENCES
Anderson, J. L., 2001: An ensemble adjustment Kalman filter for data assimilation. Mon. Wea. Rev., 129, 2884–2903, doi:10.1175/1520-0493(2001)129<2884:AEAKFF>2.0.CO;2.
Baek, S.-J., B. R. Hunt, E. Kalnay, E. Ott, and I. Szunyogh, 2006: Local ensemble Kalman filtering in the presence of model bias. Tellus, 58A, 293–306, doi:10.1111/j.1600-0870.2006.00178.x.
Bishop, C. H., and D. Hodyss, 2009a: Ensemble covariances adaptively localized with ECO-RAP. Part 1: Tests on simple error models. Tellus, 61A, 84–96, doi:10.1111/j.1600-0870.2008.00371.x.
Bishop, C. H., and D. Hodyss, 2009b: Ensemble covariances adaptively localized with ECO-RAP. Part 2: A strategy for the atmosphere. Tellus, 61A, 97–111, doi:10.1111/j.1600-0870.2008.00372.x.
Bormann, N., M. Bonavita, R. Dragani, R. Eresmaa, M. Matricardi, and A. McNally, 2016: Enhancing the impact of IASI observations through an updated observation-error covariance matrix. Quart. J. Roy. Meteor. Soc., 142, 1767–1780, doi:10.1002/qj.2774.
Boukabara, S. A., and Coauthors, 2016: S4: An O2R/R2O infrastructure for optimizing satellite data utilization in NOAA numerical modeling systems: A step toward bridging the gap between research and operations. Bull. Amer. Meteor. Soc., 97, 2359–2378, doi:10.1175/BAMS-D-14-00188.1.
Buehner, M., 2010: Error statistics in data assimilation: Estimation and modelling. Data Assimilation, W. Lahoz, B. Khattatov, and R. Menard, Eds., Springer, 93–112.
Buehner, M., R. McTaggart-Cowan, and S. Heilliette, 2017: An ensemble Kalman filter for numerical weather prediction based on variational data assimilation: VarEnKF. Mon. Wea. Rev., 145, 617–635, doi:10.1175/MWR-D-16-0106.1.
Cardinali, C., 2009: Monitoring the observation impact on the short-range forecast. Quart. J. Roy. Meteor. Soc., 135, 239–250, doi:10.1002/qj.366.
Cardinali, C., and S. Healy, 2014: Impact of GPS radio occultation measurements in the ECMWF system using adjoint-based diagnostics. Quart. J. Roy. Meteor. Soc., 140, 2315–2320, doi:10.1002/qj.2300.
Daescu, D. N., 2008: On the sensitivity equations of four-dimensional variational (4D-Var) data assimilation. Mon. Wea. Rev., 136, 3050–3065, doi:10.1175/2007MWR2382.1.
Daescu, D. N., and R. Todling, 2010: Adjoint sensitivity of the model forecast to data assimilation system error covariance parameters. Quart. J. Roy. Meteor. Soc., 136, 2000–2012, doi:10.1002/qj.693.
Daescu, D. N., and R. H. Langland, 2013: Error covariance sensitivity and impact estimation with adjoint 4D-Var: Theoretical aspects and first applications to NAVDAS-AR. Quart. J. Roy. Meteor. Soc., 139, 226–241, doi:10.1002/qj.1943.
Daescu, D. N., and R. H. Langland, 2017: Toward new applications of the adjoint sensitivity tools in data assimilation. Data Assimilation for Atmospheric, Oceanic and Hydrologic Applications, S. K. Park and L. Xu, Eds., Vol. III, Springer, 361–382.
Desroziers, G., L. Berre, B. Chapnik, and P. Poli, 2005: Diagnosis of observation, background and analysis-error statistics in observation space. Quart. J. Roy. Meteor. Soc., 131, 3385–3396, doi:10.1256/qj.05.108.
Ehrendorfer, M., R. M. Errico, and K. D. Raeder, 1999: Singular-vector perturbation growth in a primitive equation model with moist physics. J. Atmos. Sci., 56, 1627–1648, doi:10.1175/1520-0469(1999)056<1627:SVPGIA>2.0.CO;2.
Gasperoni, N. A., and X. Wang, 2015: Adaptive localization for the ensemble-based observation impact estimate using regression confidence factors. Mon. Wea. Rev., 143, 1981–2000, doi:10.1175/MWR-D-14-00272.1.
Hollingsworth, A., and P. Lönnberg, 1986: The statistical structure of short-range forecast errors as determined from radiosonde data. Part I: The wind field. Tellus, 38A, 111–136, doi:10.3402/tellusa.v38i2.11707.
Hoover, B. T., and R. H. Langland, 2017: Forecast and observation impact experiments in the Navy Global Environmental Model with assimilation of ECWMF analysis data in the global domain. J. Meteor. Soc. Japan, 95, 2017-023, https://doi.org/10.2151/jmsj.2017-023.
Hotta, D., T.-C. Chen, E. Kalnay, Y. Ota, and T. Miyoshi, 2017: Proactive QC: A fully flow-dependent quality control scheme based on EFSO. Mon. Wea. Rev., 145, 3331–3354, doi:10.1175/MWR-D-16-0290.1.
Hunt, B. R., E. J. Kostelich, and I. Szunyogh, 2007: Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filter. Physica D, 230, 112–126, doi:10.1016/j.physd.2006.11.008.
Ishibashi, T., 2010: Optimization of error covariance matrices and estimation of observation data impact in the JMA global 4D-Var system. Research Activities in Atmospheric and Oceanic Modelling, J. Côté, Ed., CAS/JSC Working Group on Numerical Experimentation Rep. 40, https://www.wcrp-climate.org/WGNE/BlueBook/2010/individual-articles/01_Ishibashi_Toshiyuki_ishibashi_optimization_WGNE2010_submit.pdf.
Kalnay, E., Y. Ota, T. Miyoshi, and J. Liu, 2012: A simpler formulation of forecast sensitivity to observations: Application to ensemble Kalman filters. Tellus, 64A, 18462, doi:10.3402/tellusa.v64i0.18462.
Kim, M., H.-M. Kim, J.-W. Kim, S.-M. Kim, C. S. Velden, and B. T. Hoover, 2017: Effect of enhanced satellite-derived atmospheric motion vectors on numerical weather prediction in East Asia using an adjoint-based observation impact method. Wea. Forecasting, 32, 579–594, doi:10.1175/WAF-D-16-0061.1.
Langland, R. H., and N. L. Baker, 2004: Estimation of observation impact using the NRL atmospheric variational data assimilation adjoint system. Tellus, 56A, 189–201, doi:10.3402/tellusa.v56i3.14413.
Li, H., E. Kalnay, and T. Miyoshi, 2009a: Simultaneous estimation of covariance inflation and observation errors within an ensemble Kalman filter. Quart. J. Roy. Meteor. Soc., 135, 523–533, doi:10.1002/qj.371.
Li, H., E. Kalnay, T. Miyoshi, and C. M. Danforth, 2009b: Accounting for model errors in ensemble data assimilation. Mon. Wea. Rev., 137, 3407–3419, doi:10.1175/2009MWR2766.1.
Li, H., J. Liu, and E. Kalnay, 2010: Correction of ‘Estimating observation impact without adjoint model in an ensemble Kalman filter.’ Quart. J. Roy. Meteor. Soc., 136, 1652–1654, doi:10.1002/qj.658.
Lien, G.-Y., D. Hotta, E. Kalnay, and T. Miyoshi, 2017: Accelerating assimilation development for new observing systems using EFSO. Nonlinear Processes Geophys., https://doi.org/10.5194/npg-2017-45, in press.
Liu, J., and E. Kalnay, 2008: Estimating observation impact without adjoint model in an ensemble Kalman filter. Quart. J. Roy. Meteor. Soc., 134, 1327–1335, doi:10.1002/qj.280.
Lorenz, E. N., 1996: Predictability: A problem partly solved. Proc. Seminar on Predictability, Vol. 1, Reading, United Kingdom, ECMWF, 1–18.
Lorenz, E. N., and K. A. Emanuel, 1998: Optimal sites for supplementary weather observations: Simulation with a small model. J. Atmos. Sci., 55, 399–414, https://doi.org/10.1175/1520-0469(1998)055<0399:OSFSWO>2.0.CO;2.
Lupu, C., C. Cardinali, and A. P. McNally, 2015: Adjoint-based forecast sensitivity applied to observation-error variance tuning. Quart. J. Roy. Meteor. Soc., 141, 3157–3165, doi:10.1002/qj.2599.
Miyoshi, T., E. Kalnay, and H. Li, 2013: Estimating and including observation-error correlations in data assimilation. Inverse Probl. Sci. Eng., 21, 387–398, doi:10.1080/17415977.2012.712527.
Ota, Y., J. C. Derber, T. Miyoshi, and E. Kalnay, 2013: Ensemble-based observation impact estimates using the NCEP GFS. Tellus, 65A, 20 038, doi:10.3402/tellusa.v65i0.20038.
Shaw, J. A., and D. N. Daescu, 2017: Sensitivity of the model error parameter specification in weak-constraint four-dimensional variational data assimilation. J. Comput. Phys., 343, 115–129, doi:10.1016/j.jcp.2017.04.050.
Sommer, M., and M. Weissmann, 2014: Observation impact in a convective-scale localized ensemble transform Kalman filter. Quart. J. Roy. Meteor. Soc., 140, 2672–2679, doi:10.1002/qj.2343.
Talagrand, O., 1999: A posteriori evaluation and verification of the analysis and assimilation algorithms. Proc. Workshop on Diagnosis of Data Assimilation Systems, Reading, United Kingdom, ECMWF, 17–28.
Trémolet, Y., 2008: Computation of observation sensitivity and observation impact in incremental variational data assimilation. Tellus, 60A, 964–978, doi:10.1111/j.1600-0870.2008.00349.x.
Weston, P. P., W. Bell, and J. R. Eyre, 2014: Accounting for correlated error in the assimilation of high-resolution sounder data. Quart. J. Roy. Meteor. Soc., 140, 2420–2429, doi:10.1002/qj.2306.
Whitaker, J. S., and T. M. Hamill, 2002: Ensemble data assimilation without perturbed observations. Mon. Wea. Rev., 130, 1913–1924, doi:10.1175/1520-0493(2002)130<1913:EDAWPO>2.0.CO;2.
Zhu, Y., and R. Gelaro, 2008: Observation sensitivity calculations using the adjoint of the Gridpoint Statistical Interpolation (GSI). Mon. Wea. Rev., 136, 335–351, doi:10.1175/MWR3525.1.