1. Introduction
Modern data assimilation (DA) in operational numerical weather prediction (NWP) has evolved into complex systems that ingest high volumes of data of many classes, prohibiting the detailed evaluation of the contributions from small subsets of observations through the use of data denial experiments (observing system experiments). To address this problem, important studies have focused on a more efficient estimation of observational impact using forecast sensitivity to observations. Langland and Baker (2004) first developed the adjoint-based forecast sensitivity to observation (adjoint FSO). With the adjoint model, FSO propagates back (to observation time) future forecast error changes between two consecutive forecasts resulting from data assimilation and attributes those changes to each individual observation, allowing detailed observational impact estimation for the first time. Kalnay et al. (2012) developed an ensemble equivalent of the adjoint FSO (EFSO), replacing the adjoint model with the use of ensemble forecasts, and shed light on the estimation of observational impact without the need of an adjoint model. In addition, Buehner et al. (2018) formulated an ensemble–variational (EnVar) approach that computes the observational impact through minimization of a cost function as in the adjoint-based approach but uses the ensemble when the adjoint model is needed. To date, many operational centers have adopted at least one of the approaches of FSO for a better understanding of the complex DA systems (Zhu and Gelaro 2008; Cardinali 2009; Gelaro et al. 2010; Lorenc and Marriott 2014; Ota et al. 2013; Sommer and Weissmann 2014).
Several studies have investigated the application of (generic) FSO impacts. Lien et al. (2018) demonstrated with an example of precipitation assimilation that using long-term averaged noncycled EFSO impact as guidance can accelerate the development of data selection and quality control procedures for new observing systems. Kotsuki et al. (2017) found that among all the ordering methods tested on a serial ensemble square root filter (EnSRF), the analysis can be significantly improved by ordering the observations from detrimental to beneficial based on EFSO.
Proactive quality control (PQC; Ota et al. 2013; Hotta et al. 2017a) based on EFSO was proposed as a strategy for reducing dropouts in forecast skill (Kumar et al. 2017) through identification and rejection of detrimental observations that may be harmful to the forecast. Ota et al. (2013) showed using the Global Forecast System (GFS) from the National Centers for Environmental Prediction (NCEP) that denying the detrimental observations identified by EFSO with a 24-h verification lead time reduced forecast errors in several forecast skill dropout cases. Hotta et al. (2017a) successfully showed with 20 forecast skill dropout cases that the forecast errors can be reduced by data denial based on EFSO with lead time of only 6 h. Furthermore, it was proposed that PQC would be more beneficial for sequential (cycling) data assimilation and that it is affordable in NCEP operation with some shortcuts (Hotta et al. 2017a; Chen 2018). A major potential benefit in cycling PQC is that the improved forecast may serve as a better background and subsequently lead to improvement in the following analyses and forecasts. However, cycling PQC has not yet been thoroughly tested. Idealized simulation experiments in a controlled environment can provide insights on how to optimally set up cycling PQC for realistic models.
In this study, we implement EFSO and PQC on the simple Lorenz (1996) model with 40 variables coupled with the ensemble transform Kalman filter (ETKF; Bishop et al. 2001) for DA. This ideally controlled environment allows the separation of factors contributing to the errors in DA, EFSO, and PQC that are entangled together in realistic systems. Section 2 reviews briefly the ETKF and EFSO formulation and introduces the PQC algorithm together with several proposed PQC update methods. The experimental setup will be described in section 3. Section 4 shows the results of the PQC performance using various configurations as well as its sensitivity to a suboptimal DA system. We summarize the findings of the study in section 5.
2. Methodology
a. ETKF
b. EFSO formulation
EFSO is the ensemble version of the adjoint FSO that replaces the adjoint model with ensemble forecasts to propagate the forecast sensitivity back to the observation time and attribute them to each observation based on their contributions to analysis increment (AI).
The impact of each observation can then be obtained by decomposing the sum of the inner product of the innovation vector 
c. Proactive quality control algorithm
The fundamental concept of PQC is to utilize the EFSO impact as observational QC in each DA cycle for identifying detrimental observations. The analysis is then modified to avoid the impact of those detrimental observations. It should be noted that EFSO cannot be computed until the next analysis becomes available for forecast error verification. The PQC algorithm is summarized in Fig. 1. It inserts additional steps (verifying analysis for EFSO, EFSO computation, PQC analysis update, and the forecast from the updated analysis) into a standard DA cycle. A major focus of this study is to compare the performance of five possible PQC analysis updates defined as follows (summarized in Table 1):
PQC_H: This method avoids the influence of detrimental observations by deleting their corresponding columns from the observational operator
PQC_R: This method removes the influence of detrimental observations by increasing the corresponding observational error covariance
- PQC_K: This method was proposed in Ota et al. (2013) and Hotta et al. (2017a) as an approximation to PQC_H by computing the PQC correction to analysis using the exact same gain matrix
where
- PQC_BmO: Another approach, following the thought of not repeating the analysis step and the idea of serial EnKF, is to treat the original analysis as background and assimilate the innovation [observation minus background (OmB)] associated with detrimental observations again but with the opposite sign (BmO), thus canceling the influence of those observations. This can be expressed as follows:where
- PQC_AmO: This method is a variant of the PQC_BmO. The only difference between the two is the definition of the innovation. Innovation in PQC_BmO is defined as the observation minus the original background while it is observation minus original analysis in PQC_AmO:where
Flowchart of cycling proactive QC algorithm [adapted from Hotta et al. (2017a)].
Citation: Monthly Weather Review 147, 1; 10.1175/MWR-D-18-0138.1
Summary of PQC update methods.
The PQC methods differ mostly in computational requirement and the changes in 
3. Experimental setup
Each experiment is initialized from a randomly chosen state and spun up for 500 time steps, allowing the ensemble members to converge to the model attractor. For the control and PQC experiments, an additional 500-step spinup for DA is performed. Each experimental period is 5000-step long after spinup. A “truth” run without DA and PQC is performed in order to generate the observations and verify the performance of the experiments. Following Lorenz and Emanuel (1998), the observations are generated at 
For “flawed” (imperfect) observing system experiments, we modify the values of μ and R to make them inconsistent with the prescribed observational error covariance matrix 
We chose to perform ETKF with a perfect model and ensemble size of 40 members because we are interested in assessing the EFSO and PQC performance without the need for localization and inflation, which are designed to deal with insufficient ensemble sizes and model error in EnKF (Liu and Kalnay 2008). The performance of each experiment is evaluated by computing the root-mean-square difference between the ensemble mean forecast/analysis and the truth from the nature run. Note that EFSO (and hence PQC) are norm-dependent methods. In this study, the error norm 
4. Proactive QC
In section 4a, we examine the performance of noncycling PQC in which the improved forecast is not used as background for the following analysis, as in Hotta et al. (2017a). In section 4b, the first experimental cycling of PQC is performed. We compare all the PQC methods introduced in section 2, each using different mechanisms to avoid the impact of the detrimental observations. In section 4c, the sensitivity of PQC performance to the choice of EFSO lead time and the amount of rejected observations are investigated. Finally, the robustness of PQC is tested in section 4d by introducing different sources of imperfections in the DA system, all relevant to operational applications.
a. Noncycling PQC
In this subsection, we explore the performance of PQC in noncycling fashion as in Hotta et al. (2017a). One of the key highlights of the study was the design of the data denial strategy. The EFSO impact was used as a guide on the denying priority, but not every detrimental observation was rejected. Note that only slightly more than 50% of the observations are beneficial on average, as reported by every operational center and research group, and it has been generally believed that rejecting nearly 50% of the observations would lead to forecast degradation. Additionally, every observation, by the nature of DA, provides an extra piece of information that should be useful in estimating the true state of the trajectory. Given the advantages of this simple low-dimensional system, we are able to test the sensitivity of the PQC performance to the number of rejected observations, with great granularity. In Fig. 2, we show how the noncycling PQC_H improves or degrades the forecasts by varying the number of rejected observations ordered from the most detrimental to the most beneficial. In this case, all the observations are perfectly consistent with the prescribed error covariance 
Forecast RMSE from single cycle PQC_H as a function of the number of rejected observations. The colored lines represent the forecast error evolution from PQC analysis corresponding to increasing the number of rejected observations from red to blue. Different colors simply distinguish individual experiments with different numbers of rejected observations. The black lines mark the forecast error at 1, 10, 20, and 30 steps.
Citation: Monthly Weather Review 147, 1; 10.1175/MWR-D-18-0138.1
b. Cycling PQC: Methods
Figure 3 compares the performance of all proposed PQC methods, using six time steps as a function of rejecting percentages of observations. Note that the magnitude of observational impacts are different and the percentages of beneficial observations could vary from 30% to 70% from cycle to cycle. There could be fewer observations with large detrimental impact for some of the cycles and more for others. Hence, it is not desirable to reject the same amount of observations throughout each cycle. Here we construct a range of thresholds corresponding to the 0th–100th percentiles of EFSO impacts obtained from a control experiment of 5000 cycles (shown in Table 2). Then PQC rejects observations with an impact above the threshold, where the 10th percentile threshold rejects the top 10% of the most detrimental observations and the 90th percentile threshold keeps only the top 10% of the most beneficial observations. The lead time here is chosen to be six steps rather arbitrarily, and the sensitivity to the lead time will be examined later in section 4c. Since the Kalman gain 
Performance of 6-step PQC with all 5 methods in terms of (a) analysis RMSE and (b) 30-step forecast RMSE as a function of rejection percentage.
Citation: Monthly Weather Review 147, 1; 10.1175/MWR-D-18-0138.1
Rejecting thresholds based on EFSO impact (J kg−1) as a function of EFSO lead time and rejecting percentile.
Intuitively, the “flat bottom” feature of PQC_K (rather than the “check mark” shape of PQC_H and PQC_R) is more consistent with the estimated impact of the observations since the magnitude of the impacts between the 10th and 60th percentile is really small compared to that of those below 10th percentile. And hence the error reduction should be insensitive (flat bottom) to the rejection of those observations between the 10th and 60th percentile. This explains why the results are better for PQC_K than for PQC_H since PQC_K is more consistent with the nature of EFSO and the estimated impact. Note that EFSO is simply an ensemble-based linear mapping between forecast error changes and the observational innovations, which is in turn associated with AIs through the gain matrix 
From the covariance perspective, PQC_K has another advantage over PQC_H and PQC_R. By using the same 
Finally, PQC_BmO and PQC_AmO change 
By comparing the performance of PQC_K with other methods, we conclude that PQC_K rejects the detrimental part of AIs in unstable modes via observational space. In this regard, PQC can be viewed as being related to “key analysis errors” method that perturbs the initial condition based on the adjoint linear sensitivity to future forecast error (Klinker et al. 1998; Isaksen et al. 2005). However, there are fundamental differences between the two. EFSO (and hence PQC) linearly maps forecast error changes, which is the result of nonlinear model propagation of AIs, back to individual observations and the associated AIs, whereas the “key analysis error” identifies perturbations in the initial condition that could potentially reduce the future forecast error. Additionally, since the norm of PQC corrections is bounded by the difference between the background and the analysis, PQC is free from the additional tuning of the size of the correction required in key analysis errors.
To advance the understanding of PQC, we explore the relation between the spatial pattern of the AIs and the corresponding EFSO impact. An example from a typical cycle using 21-step PQC_K in a Lorenz (1996) with reduced dimension (with only 20 model grids for figure readability) is shown in Fig. 4. It is clear that the largest increments in (Fig. 4c) are associated with the large magnitudes in the error covariance (Fig. 4a). We further show that the most detrimental and beneficial directions are revealed when sorted with the EFSO impact (Fig. 4b). In Fig. 4d, the most detrimental increments (15th–19th) share a similar pattern in model grids, and the pattern is the same for the most beneficial increment (0th) but with an opposite sign. The example shows supportive evidence that PQC rejects observations associated with AIs in similar detrimental directions and controls the error growth in undesired directions.
(a) Analysis error covariance matrix from a typical cycle. (b) Sorted EFSO impact for all 40 assimilated observations. (c) Analysis increments on each model grid from each observation. (d) As in (c), but the order of the observations is sorted with EFSO impact from the most beneficial to the most detrimental to align with (b).
Citation: Monthly Weather Review 147, 1; 10.1175/MWR-D-18-0138.1
c. Cycling PQC: Sensitivity to EFSO lead times
In this section, we explore the sensitivity of cycling PQC_H, PQC_K, and PQC_AmO to the rejecting threshold and, more importantly, to the EFSO lead time. We refer to PQC based on t-step EFSO as t-step PQC hereafter for simplicity.
Beginning with PQC_H shown in Fig. 5, it seems to be quite difficult to conclude the relation between the PQC_H performance and the length of lead time simply from the analysis error reduction. It is also counterintuitive to find that 21-step PQC_H performs worse than that with only 6 steps. However, the dependence of the performance on forecast error reduction can be easily summarized as follows. The forecast quality increases with the lead times up to 16 steps and then remains the same with 21 steps. The result suggests the optimal choice of EFSO lead time settles between 16 and 21 steps, best describing the impact corresponding to the underlying dynamical evolution. Short lead-time PQC seemingly reduces the analysis or even short-term forecast error but may be irrelevant to long-term error growth. This speculation can be confirmed by comparing the performances of 6-step and 21-step PQC_H. It is clear that a considerable portion of error reduction by 6-step PQC_H lies within the stable subspace and decays over time. Additionally, the corrections of 21-step PQC for rejecting 10% of the observations does not reduce much of the total analysis error compared to other lead times, but it turns out that those are the most relevant to error growth in the long term, and rejecting them leads to huge forecast improvement, after 30 steps (Fig. 5b).
Comparison of cycling PQC_H performance using various EFSO lead time as a function of rejection percentage in terms of (a) analysis and (b) 30-step forecast analysis RMSE.
Citation: Monthly Weather Review 147, 1; 10.1175/MWR-D-18-0138.1
Now we show in Fig. 6 the sensitivity of the PQC_K performance to the lead time and the rejecting threshold. It is qualitatively consistent with the result of PQC_H, where the maximum forecast rather than analysis error reduction increases with lead time and saturates at 16–21 steps. There is a general feature in forecast error reduction shared among all lead times. The error drops dramatically when rejecting with 10th percentile followed by the flat bottom feature when increasing the rejecting percentile. Then the filter diverges when rejecting beyond a critical percentile. The only differences are the percentile where PQC_K leads to filter divergence and the magnitudes of error reduction. The critical threshold gradually approaches from the 80th percentile for 6 steps to the 50th percentile for 21 steps.
This dependence of PQC_K on the rejecting percentile is explained in Fig. 7, which is a schematic of EFSO computation in an one-dimensional model space with a group of unbiased observations centering around the truth and forecasts initiated from T = −6 and 
Schematic of EFSO computation in a 1D model. The black dashed curve represents the previous forecast trajectory initiated from 
Citation: Monthly Weather Review 147, 1; 10.1175/MWR-D-18-0138.1
With Fig. 8, we show the PQC_AmO performance sensitivity to the choice of lead time and rejecting percentile. It is clear that it performs comparably well or even better than PQC_K when rejecting a small number of observations. This result is due to the extra contraction on the ensemble spread (or reduction in covariance) by “assimilating additional observations” reflecting that PQC updated mean state is more accurate than the original mean. However, it is the overcontraction in the spread that leads to filter divergence when rejecting additional observations.
As in Fig. 5, but for PQC_AmO.
Citation: Monthly Weather Review 147, 1; 10.1175/MWR-D-18-0138.1
The reason why the error reduction saturates at longer lead times is the linear assumption of EFSO. The increasing nonlinearity in the errors with longer lead times degrades the linear mapping between observation time and forecast verification time. This issue can be revealed in the correlation between the actual forecast error changes caused by DA and the ensemble-explained counterpart in EFSO computation. Table 3 shows this correlation decreases as the verifying lead time increases, indicating that the linear mapping cannot keep track of the actual impact well in long lead times.
Correlation coefficients between the actual (due to DA) and the ensemble explained forecast error changes (total EFSO impact) as a function of verifying lead time (steps).
d. PQC with suboptimal DA systems
To remain relevant to applications in an operational environment, we would like to explore PQC in suboptimal conditions including imperfect model, flawed observing system, DA window, and various sizes of ensemble and observing network. For the rest of the paper, we will be using 6-step PQC, which also improves the quality of the forecast (though not as much as 21-step PQC), but is less computationally expensive.
In high-dimensional complex chaotic systems, we do not have the luxury of using a sufficiently large ensemble size because of the limited computational resources. It is important to examine the performance of PQC with different numbers of ensemble members. We tested a wide range of ensemble sizes (from 5 to 640), as shown in Fig. 9. The experiments with ensemble size less than 40 suffered, as expected, from filter divergence, since no localization was applied. Surprisingly, PQC is able to reduce the analysis error significantly even though the filter still diverges. With about 40 ensemble members, ETKF works well and PQC improves the quality of analysis as expected, whereas the analysis error shows a slight increase as the ensemble size doubles monotonically beyond 40. This is a characteristic unique to the family of ensemble square root type of filter and has been documented in the literature (e.g., Lawson and Hansen 2005; Ng et al. 2011). The prevailing explanation is the high probability of having ensemble outliers leading to ensemble clustering, which can be ameliorated by applying additional random rotation to the transform matrix, but it is not of our interest in this study. The PQC analysis error also increases with ensemble size, but is still smaller than the control error except for PQC_H with a size larger than 320.
PQC performance in terms of analysis RMSE as a function of ensemble size.
Citation: Monthly Weather Review 147, 1; 10.1175/MWR-D-18-0138.1
In a “realistic” application, the state is only partially observed, and hence it is important to show whether PQC also benefits the system even when the size of the observing network does not match the size of the model. For the extreme case where only 5 observations are available (12.5% observed), PQC_H seems to degrade both the analyses and the forecasts while PQC_K and PQC_AmO still improve the forecasts. This result again shows that PQC operates on the AIs rather than the observation itself since PQC_H degrades the system by changing the gain matrix 
Performance of PQC_H, PQC_K, and PQC_AmO in terms of (a),(c) analysis RMSE and (b),(d) 30-step forecast RMSE as a function of (top) the size of observing network and (bottom) the DA window.
Citation: Monthly Weather Review 147, 1; 10.1175/MWR-D-18-0138.1
So far, we have been using observations with errors consistent with 
Mean EFSO impact for each observation throughout 5000 cycles of experiment that assimilates flawed observations: (a) observation at 10th grid point with various random error larger than specified in observational error covariance matrix 
Citation: Monthly Weather Review 147, 1; 10.1175/MWR-D-18-0138.1
PQC_H, PQC_K, and PQC_AmO performance in terms of (left) analysis RMSE and (right) 30-step forecast RMSE in suboptimal DA system as a function of (top) random error larger than specified in observational error covariance matrix 
Citation: Monthly Weather Review 147, 1; 10.1175/MWR-D-18-0138.1
Besides flawed observations, model error is another source of error that deserves special attention. We examine the response of the control and PQC by setting the forcing term F to be slightly different from the nature run, which is both the verifying truth and where the observations were drawn from. Figures 12e and 12f visualize both the control and PQC error in analysis and forecast increases with the model error that will eventually lead to filter divergence. It is shown that PQC improvements are almost invariant with respect to the model error.
In this section, we have shown that PQC improves the quality of analysis and the forecast even in suboptimal DA system. The improvement is even larger when the imperfections are originated from flawed observations. In the biased observation case, it is shown that PQC provides extra stability and avoids filter divergence. Additionally, we found that PQC still improves the analysis and forecast even in the case of suboptimal configurations in DA as long as the quality of verifying analysis is still more accurate than that of the forecast.
5. Summary and discussion
In this study, we explore the characteristics of different PQC methods and the sensitivity of PQC to various factors using a simple Lorenz (1996) model with ETKF as the data assimilation scheme.
We examine the performance of PQC_H, PQC_R, PQC_K, PQC_BmO, and PQC_AmO using various configurations of different EFSO lead times and rejecting percentiles. We show that PQC_H and PQC_R are suboptimal and computationally more expensive than other methods for repeating the analysis process again. While the PQC_AmO and PQC_BmO can easily lead to filter divergence because of overconfidence, we found that PQC_K has the best performance because it rejects the analysis increments (AIs) contributed by detrimental observations without changing the original EFSO gain matrix. We find that in the Lorenz (1996) system, even in the absence of flawed observations, the PQC rejection of the 10% most detrimental observations significantly improves the forecasts. Between 10% and 50%, however, the rejection of more observations has little effect, and the forecasts deteriorate significantly beyond 50% rejection. The improvement of PQC increases with the forecast length, but it saturates around 16–20 time steps as a balance between capturing long-term error growth and the linear assumption in EFSO.
We believe the source of the PQC correction comes mainly from future information (the verifying analysis for EFSO). This smoother aspect of PQC may explain the large error reduction even in a perfect system, where no defects in observations, DA, and model exist. We also examine PQC performance for suboptimal DA setup with varying ensemble and observing network size, DA window, biased observation, random error inconsistent with 
For future directions, we would like to point out that we deliberately avoided the issues associated with localization accounting for insufficient ensemble sizes, dynamical systems with multiple time scales, and implementing PQC with serial-type ensemble filter (e.g., Houtekamer and Mitchell 2001; Kotsuki et al. 2017) and variational DA systems. It is possible that the rejecting percentile threshold needs to be evaluated for each grid point because of the localization. For multiple time-scale systems, it is clear that the optimal lead time should be long enough to capture the error growth in the time scale we are interested in, but a shorter-than-optimal EFSO lead time could still improve the system for the short-range forecast.
In addition, to implement PQC with a serial-type ensemble filter, the PQC_K method may not be optimal since the real contribution of each observation to the AI is determined by the intermediate gain matrix 
In summary, we have demonstrated the beneficial impact of applying PQC in Lorenz (1996) system. PQC improves the system in the presence of flawed observations, showing its potential of being a care-free and automated QC scheme on the fly with the DA system. More importantly, even in a flawless system, PQC still improves the quality of analysis and forecast by eliminating “detrimental” growing components of the analysis increment.
Acknowledgments
We thank the reviewers for valuable comments and suggestions. This work is based on T. Chen’s Ph. D. dissertation supported by a NESDIS/JPSS Proving Ground (PG) and a Risk Reduction (RR) CICS Grant.
REFERENCES
Bishop, C. H., B. J. Etherton, and S. J. Majumdar, 2001: Adaptive sampling with the ensemble transform Kalman filter. Part I: Theoretical aspects. Mon. Wea. Rev., 129, 420–436, https://doi.org/10.1175/1520-0493(2001)129<0420:ASWTET>2.0.CO;2.
Buehner, M., P. Du, and J. Bédard, 2018: A new approach for estimating the observation impact in ensemble–variational data assimilation. Mon. Wea. Rev., 146, 447–465, https://doi.org/10.1175/MWR-D-17-0252.1.
Cardinali, C., 2009: Monitoring the observation impact on the short-range forecast. Quart. J. Roy. Meteor. Soc., 135, 239–250, https://doi.org/10.1002/qj.366.
Chen, T.-C., 2018: Applications of ensemble forecast sensitivity to observations for improving numerical weather prediction. Ph.D. dissertation, University of Maryland, College Park, 180 pp., https://drum.lib.umd.edu/handle/1903/21165.
Gelaro, R., R. H. Langland, S. Pellerin, and R. Todling, 2010: The THORPEX Observation Impact Intercomparison Experiment. Mon. Wea. Rev., 138, 4009–4025, https://doi.org/10.1175/2010MWR3393.1.
Hotta, D., T.-C. Chen, E. Kalnay, Y. Ota, and T. Miyoshi, 2017a: Proactive QC: A fully flow-dependent quality control scheme based on EFSO. Mon. Wea. Rev., 145, 3331–3354, https://doi.org/10.1175/MWR-D-16-0290.1.
Hotta, D., E. Kalnay, Y. Ota, and T. Miyoshi, 2017b: EFSR: Ensemble forecast sensitivity to observation error covariance. Mon. Wea. Rev., 145, 5015–5031, https://doi.org/10.1175/MWR-D-17-0122.1.
Houtekamer, P. L., and H. L. Mitchell, 2001: A sequential ensemble Kalman filter for atmospheric data assimilation. Mon. Wea. Rev., 129, 123–137, https://doi.org/10.1175/1520-0493(2001)129<0123:ASEKFF>2.0.CO;2.
Isaksen, L., M. Fisher, E. Andersson, and J. Barkmeijer, 2005: The structure and realism of sensitivity perturbations and their interpretation as ‘Key Analysis Errors.’ Quart. J. Roy. Meteor. Soc., 131, 3053–3078, https://doi.org/10.1256/qj.04.99.
Kalnay, E., Y. Ota, T. Miyoshi, and J. Liu, 2012: A simpler formulation of forecast sensitivity to observations: Application to ensemble Kalman filters. Tellus, 64A, 18462, https://doi.org/10.3402/tellusa.v64i0.18462.
Klinker, E., F. Rabier, and R. Gelaro, 1998: Estimation of key analysis errors using the adjoint technique. Quart. J. Roy. Meteor. Soc., 124, 1909–1933, https://doi.org/10.1002/qj.49712455007.
Kotsuki, S., S. J. Greybush, and T. Miyoshi, 2017: Can we optimize the assimilation order in the serial ensemble Kalman filter? A study with the Lorenz-96 model. Mon. Wea. Rev., 145, 4977–4995, https://doi.org/10.1175/MWR-D-17-0094.1.
Kumar, V. K., A. Eichmann, J. Alpert, and S. Boukabara, 2017: Global forecast dropout prediction tool in support of the NCEP Model Evaluation Group (MEG)––Collaboration project between JCSDA/NESDIS & NWS. JCSDA Quart., 55, 5–13, https://doi.org/10.7289/V5V98648.
Langland, R. H., and N. L. Baker, 2004: Estimation of observation impact using the NRL atmospheric variational data assimilation adjoint system. Tellus, 56A, 189–201, https://doi.org/10.1111/j.1600-0870.2004.00056.x.
Lawson, W. G., and J. A. Hansen, 2005: Alignment error models and ensemble-based data assimilation. Mon. Wea. Rev., 133, 1687–1709, https://doi.org/10.1175/MWR2945.1.
Lien, G.-Y., D. Hotta, E. Kalnay, T. Miyoshi, and T.-C. Chen, 2018: Accelerating assimilation development for new observing systems using EFSO. Nonlinear Processes Geophys., 25, 129–143, https://doi.org/10.5194/npg-25-129-2018.
Liu, J., and E. Kalnay, 2008: Estimating observation impact without adjoint model in an ensemble Kalman filter. Quart. J. Roy. Meteor. Soc., 134, 1327–1335, https://doi.org/10.1002/qj.280.
Lorenc, A. C., and R. T. Marriott, 2014: Forecast sensitivity to observations in the Met Office global numerical weather prediction system. Quart. J. Roy. Meteor. Soc., 140, 209–224, https://doi.org/10.1002/qj.2122.
Lorenz, E., 1996: Predicability—A problem partly solved. Seminar on Predictability, Reading, United Kingdom, ECMWF, 1–18, https://www.ecmwf.int/en/elibrary/10829-predictability-problem-partly-solved.
Lorenz, E., and K. A. Emanuel, 1998: Optimal sites for supplementary weather observations: Simulation with a small model. J. Atmos. Sci., 55, 399–414, https://doi.org/10.1175/1520-0469(1998)055<0399:OSFSWO>2.0.CO;2.
Ng, G.-H. C., D. Mclaughlin, D. Entekhabi, and A. Ahanin, 2011: The role of model dynamics in ensemble Kalman filter performance for chaotic systems. Tellus, 63A, 958–977, https://doi.org/10.1111/j.1600-0870.2011.00539.x.
Ota, Y., J. C. Derber, E. Kalnay, and T. Miyoshi, 2013: Ensemble-based observation impact estimates using the NCEP GFS. Tellus, 65A, 20038, https://doi.org/10.3402/tellusa.v65i0.20038.
Sommer, M., and M. Weissmann, 2014: Observation impact in a convective-scale localized ensemble transform Kalman filter. Quart. J. Roy. Meteor. Soc., 140, 2672–2679, https://doi.org/10.1002/qj.2343.
Sommer, M., and M. Weissmann, 2016: Ensemble-based approximation of observation impact using an observation-based verification metric. Tellus, 68A, 27885, https://doi.org/10.3402/tellusa.v68.27885.
Zhu, Y., and R. Gelaro, 2008: Observation sensitivity calculations using the adjoint of the Gridpoint Statistical Interpolation (GSI) analysis system. Mon. Wea. Rev., 136, 335–351, https://doi.org/10.1175/MWR3525.1.
