## 1. Introduction

Data assimilation (DA) methods produce the best estimate of the current state of a dynamical system by combining the background and observations with an “optimal” weight. The optimal weight, denoted

Reflecting the crucial importance of correctly specifying *O* − *B*), which contain contributions from both *O* − *A*) in addition to *O* − *B* to check the optimality of the currently prescribed covariances. These approaches have been applied to many systems and data and have proven to be useful, but each has its limitations because no single assumption is applicable to every situation (Buehner 2010).

Another relatively new approach, which has not been covered by Buehner (2010), originates from the adjoint sensitivity studies. The ground-breaking work by Langland and Baker (2004) introduced forecast sensitivity to observations (FSO), a technique that allows us to estimate how much each of the assimilated observations reduced or increased the forecast errors measured with some quadratic norm, without having to perform the expensive observing system experiments (OSEs). Daescu (2008) generalized the FSO technique and gave a formulation for forecast sensitivity to the background error covariance (

While being a powerful diagnostic tool, the applicability of the adjoint sensitivity methods such as FSO and Daescu’s methods had been somewhat limited because these approaches require the adjoint of the forecast model, which is difficult to develop and/or maintain. For FSO, this limitation has been recently resolved by adapting it to EnKF (Liu and Kalnay 2008; Li et al. 2010; Kalnay et al. 2012). The most recent formulation of the ensemble-based FSO, or EFSO, proposed by Kalnay et al. (2012), has been successfully implemented into a quasi-operational global EnKF system of the National Centers for Environmental Prediction (NCEP; Ota et al. 2013), a German convective-scale regional EnKF DA system (Sommer and Weissmann 2014), and JMA’s global DA system.

The objective of this paper is to show that it is possible, by combining the derivations of EFSO in Kalnay et al. (2012) and the

## 2. EFSR formulation

In this section we introduce our EFSR formulation, building upon the derivations of Daescu (2008) and Kalnay et al. (2012).

### a. Forecast sensitivity to each element of

#### 1) General formulation

*t*-hour forecast would change by small variations in the observation error covariance matrix from

*t*, and

*i*,

*j*) element of

*O*−

*A*residual, with

*H*linearized around the background model state

*t*linearized around the analysis trajectory.

#### 2) Adjoint-based evaluation of Eq. (4) within a 4D-Var

^{9}× 10

^{6}elements as of 2017), so that it can never be explicitly stored on memory. Also, given the complexity of the DA code, writing its adjoint line by line, as was done by Zhu and Gelaro (2008), is a demanding task. Within the context of FSO calculations, a practical algorithm has been proposed that multiplies a vector by

*O*−

*B*innovation and

**v**. Then, by applying the same expression

*J*

_{o}term in the cost function minimized in the incremental 4D-Var algorithm can be reorganized as

*t*to 0 from the “initial” conditions

**u**into the 4D-Var algorithm Eq. (7) in place of

*J*

_{o}term and its gradient

#### 3) Ensemble-based implementation (EFSR)

*K*is the ensemble size,

*i*th member analysis and

*H*is nonlinear,

*t*with

*i*th member

*t*-hour forecast from time 0 and

*t*hours to obtain

The computational cost required to evaluate Eq. (9) or (10) is not very expensive. Denoting the dimension of the system’s state vector and the number of observations by *N*_{state} and *N*_{obs}, respectively, an explicit evaluation of Eq. (9) requires only ~*N*_{state} × *K* operations (for multiplying the vector *N*_{obs} × *K* operations [for multiplying the resultant (*K* − 1) × 1 vector by the matrix *i*), by first computing the contribution from the *l*th component of the state vector as *l* from 1 to *N*_{state}, which requires ~*N*_{state} × *K* operations. This computation is repeated for *i* = 1, …, *N*_{obs}, amounting to a total of ~*N*_{obs} × *N*_{state} × *K* operations. This is more expensive than the case of Eq. (9) without localization by a factor of *N*_{obs}, but is still less expensive compared to the EnKF assimilation. In practice, the most expensive part of computing EFSR is generating extended-range (*t* hour) ensemble forecasts to obtain

For convenience, we call the forecast sensitivity to observation error covariance matrix FSR (short for forecast sensitivity to

We emphasize that, unlike other diagnostic methods for optimality of

### b. Sensitivity to scaling factors

*I*subgroups

*restriction*operator

*O*−

*A*residuals multiplied by the forecast sensitivity gradient to the corresponding observations summed up over all observations in that subgroup (with the sign flipped).

### c. Sensitivity to the covariance inflation factor

## 3. Toy-model experiments using the Lorenz ’96 model

### a. Model and DA system

*N*-dimensional ODE system defined by

*N*= 40 and

*F*= 8.0. Kalnay et al. (2012) and Liu and Kalnay (2008) used different values of

*F*for the nature run and DA cycles to simulate model errors, but here we use the same parameter

*F*= 8.0 for both the nature run and the forecast (an “identical twin” setting). The forecast model Eq. (17) is integrated by the standard fourth-order Runge–Kutta scheme with time step

As the DA system, we adopt the local ensemble transform Kalman filter (LETKF; Hunt et al. 2007) with member size *K* = 40. Since the member size is equal to the dimension of the state space, there is no need for covariance localization in our experiments. To avoid filter divergence, however, we applied multiplicative covariance inflation (Anderson 2001) with a constant inflation parameter

*j*th grid point, the observations are generated for every analysis time by adding independent Gaussian pseudorandom numbers with variance

### b. Experimental design

First, we produced the nature (or “truth”) by running the forecast model Eq. (17) from an initial condition randomly generated from the uniform distribution in [0, 1]. The nature run is integrated from time *t* = 0 to 730 (which corresponds to 10 yr in dimensional time), generating truth for 14 600 cycles.

The initial background ensemble at time *t* = 0 is generated by picking up 40 truth states at 40 randomly chosen distinctive dates. Each DA experiment is run for 14 600 cycles (10 yr) and the first 1460 cycles (1 yr), regarded as a spinup period, are excluded from verification.

To examine the ability of AFSR and EFSR to detect the misspecification of observation error variances

The true and prescribed observation error variances for the experiments performed using the Lorenz ’96 model.

### c. The SPIKE experiment

The SPIKE experiment is inspired by Liu and Kalnay (2008) and Kalnay et al. (2012), who examined the capacity of EFSO to capture the negative impacts from the observations at the 11th grid point that have larger observation errors than the others. In this experiment, all observations but the one at the 11th grid point have the error variance 0.2^{2}, while at the 11th grid point, it is 0.8^{2}. In the incorrect-^{2}. With this experiment, we intend to see whether the AFSR or EFSR can detect the misspecification of the error variance at the 11th grid point to provide useful guidance on how to correct it. We also examine whether the FSR diagnostics do not signal “false alarms” when the specification of

We first examine the analysis errors with respect to the truth to ensure that the system did not suffer from any malfunction (a “filter divergence” in particular). Figure 1a shows the root-mean-square errors (RMSEs) of analysis verified against the truth averaged over the 9 yr for the correct-

We now examine the FSR diagnosed by ensemble- and adjoint-based methods. Figure 1b shows the EFSR-based forecast sensitivity to scaling factors of

This raises one concern: the FSR methods may not give a reliable diagnostic if accurate and inaccurate observations are located close to each other. This concern is addressed in the next experiment.The sensitivity gradient

, being a partial derivative, tells us how, for each index i, a small displacement infrom unity would change the forecast error if the prescribed error variances for other observations are kept unchanged. Thus, if there is an observation that makes the forecast worse, then we can make the forecast better by giving higher credence to the adjacent, more accurate observations.

### d. The STAGGERED experiment

The STAGGERED experiment is designed to assess whether the FSR diagnostics are robust to cases where observations with different magnitudes of error are located close to each other. The true observation error variances are 0.1^{2} and 0.3^{2}, respectively, for odd- and even-numbered grid points. In the incorrect-^{2}; we should thus reduce–increase the error variances at odd–even grid points. The design of the STAGGERED experiment is very similar to the one performed by Daescu and Todling (2010), who sought to validate their AFSR diagnostics. The precise setup is not identical, but the incorrect-

Figure 2a shows the analysis RMSE verified against the truth for the STAGGERED experiment. Both incorrect-

As in Fig. 1, but for the STAGGERED experiment.

Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1

As in Fig. 1, but for the STAGGERED experiment.

Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1

As in Fig. 1, but for the STAGGERED experiment.

Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1

The EFSR-based forecast sensitivity to scaling factors of ^{2}) are larger than their actual values (0.1^{2}) and the opposite is true for the even grids. Similar results are reported by Daescu and Todling (2010) for their DAS-1 experiment (their Fig. 1b). On the other hand, in the correct-

### e. Adaptive online estimation of and inflation factor guided by EFSR

EFSR diagnostics allows us to estimate the gradient of the scalar forecast error *t*-hour forecast from the analysis of *t* hours ago should also improve the current analysis. Such an adaptive tuning algorithm has already been proposed and proven to be successful by Shaw and Daescu (2017) within the context of adjoint-based sensitivity for model error bias and covariance parameters within weak-constraint 4D-Var. A difficulty in combining the sensitivity diagnostics and an iterative optimization scheme is in how best to determine the step size. A larger step size may achieve faster convergence but at the risk of overcorrection that may harm the analysis. In this sense, a smaller step size (i.e., slowly adjusting the

This algorithm does not update the observation error variance scaling factors

We implemented the above adaptive algorithm and experimented with many choices of

The results from the experiment with

Results from an adaptive online

Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1

Results from an adaptive online

Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1

Results from an adaptive online

Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1

Different choices of the parameters

## 4. Experiments with a real DA system: System description and experimental setup

We implemented and tested EFSR on the NCEP’s quasi-operational global NWP system designed to test the new proactive quality control (Hotta et al. 2017). The system is based on the operational suite that had been operational until January 2015 but with the reduced horizontal resolutions of T254 for the deterministic runs and T126 for the ensemble (as opposed to the operational T574 and T254). In this two-way interactive hybrid DA system, the variational Gridpoint Statistical Interpolation analysis system (GSI) incorporates flow-dependent background ensemble covariance from the EnKF first-guess perturbations to produce a deterministic analysis, and the EnKF analysis ensemble is recentered on the deterministic analysis thus produced. The weights given to the static and ensemble parts of the covariance in the hybrid GSI are 25% and 75%, respectively. The EnKF part of the DA system, which, in the operational suite, is the serial ensemble square root filter (EnSRF) of Whitaker and Hamill (2002), is replaced with the LETKF. The ensemble size is 80 and both localization and inflation are applied to the covariance. We note that the localization and inflation parameters used in this experiment are tuned for the operational higher-resolution system and thus may be suboptimal for our system. Nevertheless, our system worked well and without any problems.

We remark that in a hybrid DA system, the Kalman gain assumed in EFSR and EFSO computation differs from what is actually used in the hybrid analysis. Because of this inconsistency, EFSR and EFSO may not correctly estimate the sensitivities on

The experiment is performed with 6-hourly cycles for the 31-day period from 8 January to 8 February 2012, with a 7-day spinup period from 1 to 7 January 2012. All observations that were assimilated in the operational system during this period are also assimilated in our experiments. Observations are grouped into different types as in Ota et al. (2013) and Hotta et al. (2017). The EFSO impacts and EFSR-based forecast sensitivities to

## 5. Experiments with a real DA system: Results

Figure 4 shows the EFSR-based forecast sensitivities to

Forecast sensitivity to observation error scaling factors for different types of observations measured with the (a),(b) moist or (c),(d) dry total energy norms, at (a),(c) 6- or (b),(d) 24-h lead time averaged over the 31-day period. The units are J kg^{−1}. The observation types whose error variances are scaled in the ^{−1}, as indicated by the numbers below each bar.

Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1

Forecast sensitivity to observation error scaling factors for different types of observations measured with the (a),(b) moist or (c),(d) dry total energy norms, at (a),(c) 6- or (b),(d) 24-h lead time averaged over the 31-day period. The units are J kg^{−1}. The observation types whose error variances are scaled in the ^{−1}, as indicated by the numbers below each bar.

Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1

Forecast sensitivity to observation error scaling factors for different types of observations measured with the (a),(b) moist or (c),(d) dry total energy norms, at (a),(c) 6- or (b),(d) 24-h lead time averaged over the 31-day period. The units are J kg^{−1}. The observation types whose error variances are scaled in the ^{−1}, as indicated by the numbers below each bar.

Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1

We can also observe from Fig. 4 that, among all the observation types, aircraft, radiosonde, and Advanced Microwave Sounding Unit A (AMSU-A) exhibit higher sensitivities than the others, and that MODIS winds show negative sensitivity. This feature is consistently seen in any combination of the lead times and the error norms.

To assess the validity of the EFSR diagnostics described above, we performed an *infinitesimally small* perturbation to the

If our EFSR diagnostics are valid, the use of the new ^{−1}, which is much smaller than the standard deviation 0.092 J kg^{−1} of the paired difference (which gives *p* > 0.27, the lower bound estimated by assuming that all samples are independent). Similarly, no statistically significant changes were detected for the 30-h forecast errors initialized from the first guess [black and gray thin solid lines (lines c and d)] or the forecast error reduction by the assimilation of observations [black and gray dotted lines (lines e and f)].

Time series of (line a, thick black solid line) the error of the 24-h forecast from analysis

Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1

Time series of (line a, thick black solid line) the error of the 24-h forecast from analysis

Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1

Time series of (line a, thick black solid line) the error of the 24-h forecast from analysis

Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1

The fact that no statistically significant differences were found between the two cycled experiments may seem disappointing, but this was in fact an expected outcome because the estimated 24-h forecast improvement by the scaling of ^{−1}, while the standard deviation of the 24-h forecast error ^{−1}. This suggests that step sizes

As in Fig. 4b, but computed for the data from the ^{−1}, as indicated by the number below the corresponding bar.

Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1

As in Fig. 4b, but computed for the data from the ^{−1}, as indicated by the number below the corresponding bar.

Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1

As in Fig. 4b, but computed for the data from the ^{−1}, as indicated by the number below the corresponding bar.

Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1

Consistent with the insignificant differences in the forecast error reductions (^{−1}, respectively, with effective sample size at most 124, giving the lower bound of p value *p* > 0.11). Interestingly, however, the renewal of *t* test for paired differences.

Forecast sensitivity to observation impact before (top bars) and after (bottom bars) the observation error variance scaling, evaluated for different types of observations measured with the moist total energy norm and 24-h lead time averaged over the 31-day period. The error bars on the top bars represent the confidence intervals at the 95% level computed by a *t* test for the differences among the paired data (see text for details). The units are J kg^{−1}. As in Fig. 4, the observation types whose error variances are scaled in the

Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1

Forecast sensitivity to observation impact before (top bars) and after (bottom bars) the observation error variance scaling, evaluated for different types of observations measured with the moist total energy norm and 24-h lead time averaged over the 31-day period. The error bars on the top bars represent the confidence intervals at the 95% level computed by a *t* test for the differences among the paired data (see text for details). The units are J kg^{−1}. As in Fig. 4, the observation types whose error variances are scaled in the

Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1

Forecast sensitivity to observation impact before (top bars) and after (bottom bars) the observation error variance scaling, evaluated for different types of observations measured with the moist total energy norm and 24-h lead time averaged over the 31-day period. The error bars on the top bars represent the confidence intervals at the 95% level computed by a *t* test for the differences among the paired data (see text for details). The units are J kg^{−1}. As in Fig. 4, the observation types whose error variances are scaled in the

Citation: Monthly Weather Review 145, 12; 10.1175/MWR-D-17-0122.1

From Fig. 7 we can observe the following features. First, the EFSO impacts from aircraft, radiosonde, and AMSU-A all were statistically significantly increased by reducing their *decreases* rather than increases in FSO impacts since the error norm of the forecast from the background (

## 6. Conclusions

The observation error covariance matrix

The main focus of this paper was on verifying the validity of the EFSR formulation. As such, our tuning effort to improve the operational system was only preliminary. In our experiment with the NCEP’s quasi-operational system, we examined the

One limitation of FSR diagnostics is that they only suggest whether we should reduce or increase each component of

In this study, we focused on the forecast sensitivity to observation error variances (i.e., diagonal elements of

Perhaps our EFSR will prove to be most useful when a new observing system is introduced to a DA system. In such a situation, in addition to assigning optimal

Finally, we note that an adjoint-based sensitivity methodology has been recently extended by Shaw and Daescu (2017) to weak-constraint 4D-Var formulation, allowing for the estimation of forecast sensitivity with respect to model-error bias and covariance specification; Shaw and Daescu (2017) also devised an online tuning procedure for these parameters based on the sensitivity guidance. In parallel with weak-constraint 4D-Var, EnKF formulations that explicitly account for model error biases have been proposed (e.g., Baek et al. 2006; Li et al. 2009b), and it will be an interesting future direction to explore the ensemble-based estimation of forecast sensitivity to prescribed parameters related to model errors within the framework of such model-error-aware variants of EnKF.

## Acknowledgments

This work is inspired by Dr. Dacian Daescu’s talk delivered in 2013 at a seminar of the Weather and Chaos Group of University of Maryland. The manuscript grew out from DH’s Ph.D. dissertation, which was supported by Japan Meteorological Agency (JMA) through the Japanese Government Long-term Overseas Fellowship Program. The experiments with the NCEP’s quasi-operational system are performed on the Joint Center for Satellite Data Assimilation’s (JCSDA) Supercomputer for Satellite Simulations and Data Assimilation Studies [the “S4 supercomputer”; Boukabara et al. (2016)]. We express our gratitude to Dr. Sid Boukabara for his kind support and to Dr. Jim Jung for his guidance on using the Global Forecast System (GFS) and the GSI on the S4. We thank three anonymous reviews for their insightful comments that significantly improved the manuscript. This work was partially supported by a NOAA/NESDIS/JPSS Proving Ground (PG) and a Risk Reduction (RR) CICS grant (NA14NES4320003) and a JSPS KAKENHI Grant-in-Aid for Research Activity Start-Up (17H07352).

## REFERENCES

Anderson, J. L., 2001: An ensemble adjustment Kalman filter for data assimilation.

,*Mon. Wea. Rev.***129**, 2884–2903, doi:10.1175/1520-0493(2001)129<2884:AEAKFF>2.0.CO;2.Baek, S.-J., B. R. Hunt, E. Kalnay, E. Ott, and I. Szunyogh, 2006: Local ensemble Kalman filtering in the presence of model bias.

,*Tellus***58A**, 293–306, doi:10.1111/j.1600-0870.2006.00178.x.Bishop, C. H., and D. Hodyss, 2009a: Ensemble covariances adaptively localized with ECO-RAP. Part 1: Tests on simple error models.

,*Tellus***61A**, 84–96, doi:10.1111/j.1600-0870.2008.00371.x.Bishop, C. H., and D. Hodyss, 2009b: Ensemble covariances adaptively localized with ECO-RAP. Part 2: A strategy for the atmosphere.

,*Tellus***61A**, 97–111, doi:10.1111/j.1600-0870.2008.00372.x.Bormann, N., M. Bonavita, R. Dragani, R. Eresmaa, M. Matricardi, and A. McNally, 2016: Enhancing the impact of IASI observations through an updated observation-error covariance matrix.

,*Quart. J. Roy. Meteor. Soc.***142**, 1767–1780, doi:10.1002/qj.2774.Boukabara, S. A., and Coauthors, 2016: S4: An O2R/R2O infrastructure for optimizing satellite data utilization in NOAA numerical modeling systems: A step toward bridging the gap between research and operations.

,*Bull. Amer. Meteor. Soc.***97**, 2359–2378, doi:10.1175/BAMS-D-14-00188.1.Buehner, M., 2010: Error statistics in data assimilation: Estimation and modelling.

*Data Assimilation*, W. Lahoz, B. Khattatov, and R. Menard, Eds., Springer, 93–112.Buehner, M., R. McTaggart-Cowan, and S. Heilliette, 2017: An ensemble Kalman filter for numerical weather prediction based on variational data assimilation: VarEnKF.

,*Mon. Wea. Rev.***145**, 617–635, doi:10.1175/MWR-D-16-0106.1.Cardinali, C., 2009: Monitoring the observation impact on the short-range forecast.

,*Quart. J. Roy. Meteor. Soc.***135**, 239–250, doi:10.1002/qj.366.Cardinali, C., and S. Healy, 2014: Impact of GPS radio occultation measurements in the ECMWF system using adjoint-based diagnostics.

,*Quart. J. Roy. Meteor. Soc.***140**, 2315–2320, doi:10.1002/qj.2300.Daescu, D. N., 2008: On the sensitivity equations of four-dimensional variational (4D-Var) data assimilation.

,*Mon. Wea. Rev.***136**, 3050–3065, doi:10.1175/2007MWR2382.1.Daescu, D. N., and R. Todling, 2010: Adjoint sensitivity of the model forecast to data assimilation system error covariance parameters.

,*Quart. J. Roy. Meteor. Soc.***136**, 2000–2012, doi:10.1002/qj.693.Daescu, D. N., and R. H. Langland, 2013: Error covariance sensitivity and impact estimation with adjoint 4D-Var: Theoretical aspects and first applications to NAVDAS-AR.

,*Quart. J. Roy. Meteor. Soc.***139**, 226–241, doi:10.1002/qj.1943.Daescu, D. N., and R. H. Langland, 2017: Toward new applications of the adjoint sensitivity tools in data assimilation.

*Data Assimilation for Atmospheric, Oceanic and Hydrologic Applications*, S. K. Park and L. Xu, Eds., Vol. III, Springer, 361–382.Desroziers, G., L. Berre, B. Chapnik, and P. Poli, 2005: Diagnosis of observation, background and analysis-error statistics in observation space.

,*Quart. J. Roy. Meteor. Soc.***131**, 3385–3396, doi:10.1256/qj.05.108.Ehrendorfer, M., R. M. Errico, and K. D. Raeder, 1999: Singular-vector perturbation growth in a primitive equation model with moist physics.

,*J. Atmos. Sci.***56**, 1627–1648, doi:10.1175/1520-0469(1999)056<1627:SVPGIA>2.0.CO;2.Gasperoni, N. A., and X. Wang, 2015: Adaptive localization for the ensemble-based observation impact estimate using regression confidence factors.

,*Mon. Wea. Rev.***143**, 1981–2000, doi:10.1175/MWR-D-14-00272.1.Hollingsworth, A., and P. Lönnberg, 1986: The statistical structure of short-range forecast errors as determined from radiosonde data. Part I: The wind field.

,*Tellus***38A**, 111–136, doi:10.3402/tellusa.v38i2.11707.Hoover, B. T., and R. H. Langland, 2017: Forecast and observation impact experiments in the Navy Global Environmental Model with assimilation of ECWMF analysis data in the global domain.

,*J. Meteor. Soc. Japan***95**, 2017-023, https://doi.org/10.2151/jmsj.2017-023.Hotta, D., T.-C. Chen, E. Kalnay, Y. Ota, and T. Miyoshi, 2017: Proactive QC: A fully flow-dependent quality control scheme based on EFSO.

,*Mon. Wea. Rev.***145**, 3331–3354, doi:10.1175/MWR-D-16-0290.1.Hunt, B. R., E. J. Kostelich, and I. Szunyogh, 2007: Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filter.

,*Physica D***230**, 112–126, doi:10.1016/j.physd.2006.11.008.Ishibashi, T., 2010: Optimization of error covariance matrices and estimation of observation data impact in the JMA global 4D-Var system.

*Research Activities in Atmospheric and Oceanic Modelling*, J. Côté, Ed., CAS/JSC Working Group on Numerical Experimentation Rep. 40, https://www.wcrp-climate.org/WGNE/BlueBook/2010/individual-articles/01_Ishibashi_Toshiyuki_ishibashi_optimization_WGNE2010_submit.pdf.Kalnay, E., Y. Ota, T. Miyoshi, and J. Liu, 2012: A simpler formulation of forecast sensitivity to observations: Application to ensemble Kalman filters.

,*Tellus***64A**, 18462, doi:10.3402/tellusa.v64i0.18462.Kim, M., H.-M. Kim, J.-W. Kim, S.-M. Kim, C. S. Velden, and B. T. Hoover, 2017: Effect of enhanced satellite-derived atmospheric motion vectors on numerical weather prediction in East Asia using an adjoint-based observation impact method.

,*Wea. Forecasting***32**, 579–594, doi:10.1175/WAF-D-16-0061.1.Langland, R. H., and N. L. Baker, 2004: Estimation of observation impact using the NRL atmospheric variational data assimilation adjoint system.

,*Tellus***56A**, 189–201, doi:10.3402/tellusa.v56i3.14413.Li, H., E. Kalnay, and T. Miyoshi, 2009a: Simultaneous estimation of covariance inflation and observation errors within an ensemble Kalman filter.

,*Quart. J. Roy. Meteor. Soc.***135**, 523–533, doi:10.1002/qj.371.Li, H., E. Kalnay, T. Miyoshi, and C. M. Danforth, 2009b: Accounting for model errors in ensemble data assimilation.

,*Mon. Wea. Rev.***137**, 3407–3419, doi:10.1175/2009MWR2766.1.Li, H., J. Liu, and E. Kalnay, 2010: Correction of ‘Estimating observation impact without adjoint model in an ensemble Kalman filter.’

,*Quart. J. Roy. Meteor. Soc.***136**, 1652–1654, doi:10.1002/qj.658.Lien, G.-Y., D. Hotta, E. Kalnay, and T. Miyoshi, 2017: Accelerating assimilation development for new observing systems using EFSO.

, https://doi.org/10.5194/npg-2017-45, in press.*Nonlinear Processes Geophys.*Liu, J., and E. Kalnay, 2008: Estimating observation impact without adjoint model in an ensemble Kalman filter.

,*Quart. J. Roy. Meteor. Soc.***134**, 1327–1335, doi:10.1002/qj.280.Lorenz, E. N., 1996: Predictability: A problem partly solved.

*Proc. Seminar on Predictability*, Vol. 1, Reading, United Kingdom, ECMWF, 1–18.Lorenz, E. N., and K. A. Emanuel, 1998: Optimal sites for supplementary weather observations: Simulation with a small model.

,*J. Atmos. Sci.***55**, 399–414, https://doi.org/10.1175/1520-0469(1998)055<0399:OSFSWO>2.0.CO;2.Lupu, C., C. Cardinali, and A. P. McNally, 2015: Adjoint-based forecast sensitivity applied to observation-error variance tuning.

,*Quart. J. Roy. Meteor. Soc.***141**, 3157–3165, doi:10.1002/qj.2599.Miyoshi, T., E. Kalnay, and H. Li, 2013: Estimating and including observation-error correlations in data assimilation.

,*Inverse Probl. Sci. Eng.***21**, 387–398, doi:10.1080/17415977.2012.712527.Ota, Y., J. C. Derber, T. Miyoshi, and E. Kalnay, 2013: Ensemble-based observation impact estimates using the NCEP GFS.

,*Tellus***65A**, 20 038, doi:10.3402/tellusa.v65i0.20038.Shaw, J. A., and D. N. Daescu, 2017: Sensitivity of the model error parameter specification in weak-constraint four-dimensional variational data assimilation.

,*J. Comput. Phys.***343**, 115–129, doi:10.1016/j.jcp.2017.04.050.Sommer, M., and M. Weissmann, 2014: Observation impact in a convective-scale localized ensemble transform Kalman filter.

,*Quart. J. Roy. Meteor. Soc.***140**, 2672–2679, doi:10.1002/qj.2343.Talagrand, O., 1999: A posteriori evaluation and verification of the analysis and assimilation algorithms.

*Proc. Workshop on Diagnosis of Data Assimilation Systems*, Reading, United Kingdom, ECMWF, 17–28.Trémolet, Y., 2008: Computation of observation sensitivity and observation impact in incremental variational data assimilation.

,*Tellus***60A**, 964–978, doi:10.1111/j.1600-0870.2008.00349.x.Weston, P. P., W. Bell, and J. R. Eyre, 2014: Accounting for correlated error in the assimilation of high-resolution sounder data.

,*Quart. J. Roy. Meteor. Soc.***140**, 2420–2429, doi:10.1002/qj.2306.Whitaker, J. S., and T. M. Hamill, 2002: Ensemble data assimilation without perturbed observations.

,*Mon. Wea. Rev.***130**, 1913–1924, doi:10.1175/1520-0493(2002)130<1913:EDAWPO>2.0.CO;2.Zhu, Y., and R. Gelaro, 2008: Observation sensitivity calculations using the adjoint of the Gridpoint Statistical Interpolation (GSI).

,*Mon. Wea. Rev.***136**, 335–351, doi:10.1175/MWR3525.1.