1. Introduction
Sensitivity analysis is one of the important features of data assimilation algorithms because it allows the determination of how changes in the control variables of a numerical model (e.g., initial conditions) affect the target quantities (TQs) of interest (e.g., temperature in a certain domain) at the time of the model forecast. This type of analysis can be extended further by combining sensitivity estimates with information about the prior observational and background errors to assess the impact of new observations on the a posteriori errors of the TQs (e.g., Baker and Daley 2000).
In the past decade, many general circulation models have been supplied with their adjoint code. This development has enabled a particular type of sensitivity analysis based on the application of the adjoint models to obtain the derivatives of the TQs with respect to a variety of variables controlling the model solutions. To cite a few oceanographic examples of the “adjoint sensitivity” (AS) approach, Lee et al. (2001) estimated the sensitivity of the Indonesian Throughflow to remote wind forcing, Losch and Heimbach (2007) assessed the AS of model solutions to the bottom topography, while Moore et al. (2009) and Veneziani et al. (2009) conducted a comprehensive AS analysis in the California Current region.
The major limitation of the AS approach is the assumption that models can be effectively linearized in the vicinity of the optimal trajectory to compute sensitivities with respect to infinitesimal perturbations of the control variables over finite time intervals. However, this assumption does not work in many applications, especially in cases involving strongly nonlinear background states characterized by exponential growth of the infinitesimal perturbations. To cope with this issue, Hoteit et al. (2005, 2010) introduced stabilization of the adjoint model by artificially enhancing its dissipation to a level that suppressed the unstable modes.
Partly due to the above-mentioned limitation of the adjoint technique, ensemble methods of data assimilation have experienced rapid development in recent years. The ensemble approach is capable of better handling the quasi-chaotic model behavior induced by nonlinearities, does not require development and maintenance of the adjoint code, and appears to be consistent with the modern trend in computer technology of massive parallelization. The ensemble sensitivity (ES) analysis (e.g., Ancell and Hakim 2007; Torn and Hakim 2008) follows the basic principle of perturbing a model by the ensemble members and analyzing the respective responses of the TQs. In contrast to the AS, this approach is naturally restricted to the subspace spanned by the ensemble members, but the ensemble members tend to cover the dynamically significant manifold that accounts for the major part of the model variability (Farrell and Ioannou 2005).
Of certain interest are the methods merging the ensemble approach with traditional variational assimilation techniques (e.g., Cao et al. 2007; Liu et al. 2008; Yaremchuk et al. 2009; Clayton et al. 2013). In particular, Yaremchuk et al. (2009) demonstrated that the reduced-order four-dimensional variational data assimilation (R4DVAR) algorithm appears to be advantageous in handling nonlinear optimizations and is comparable in computational cost with the 4DVAR technique. The ES analysis has been considered in detail by a number of authors (e.g., Torn and Hakim 2009; Zack et al. 2010) in application to the ensemble methods that keep the ensemble size fixed in the process of sequential assimilation. The hybrid R4DVAR method considered in this paper produces a sequence of ensembles spanning the Krylov subspaces that account for the most dynamically significant error components and can be naturally used in the ES analysis of the optimized solution. A similar sequence (of search directions) is generated in the process of 4DVAR minimization that can be used for ES analysis of 4DVAR solutions.
In this study we compare the accuracy of the adjoint 4DVAR and the R4DVAR sensitivity analysis techniques using a quasigeostrophic model of intermediate complexity. To mimic real applications, when the true state is unknown and sensitivities are computed using linearization around the available optimal state, we adopt the following experimentation procedure. First, true states are generated from model integrations and observations are sampled from these true states for the data assimilation experiments (using R4DVAR and 4DVAR techniques) to obtain “optimal” states. The sensitivity experiments are then conducted with these optimal states serving as the model trajectories and results of the experiments are compared with each other using the sensitivities computed from the “true” model trajectories as benchmarks. Special attention is given to the experiments in the strongly nonlinear case characterized by instability of the tangent linear (TL) model.
The paper is organized as follows. In the next section we briefly provide an overview of the basic relationships of the linear sensitivity analysis, discuss the sensitivity concept in strongly nonlinear geophysical flows, and show that averaging sensitivity estimates over realizations of a turbulent flow can be approximated by a single AS estimate generated by a stabilized adjoint model. In section 3, the methodology of the numerical experiments with the 4DVAR and R4DVAR systems is described. Results of the experiments, which include consequent treatment of the linear, weakly nonlinear, and strongly nonlinear cases, are presented in section 4. The results are summarized and discussed in section 5.
2. Sensitivity analysis
a. Linear case
Consider now a (scalar) target quantity q of interest (e.g., transport through a certain section in the model domain averaged over a certain period of time) represented by a linear function q = {Qk, k = 1, … , N} of X: 
In real applications, the values of n and N are quite large (106–109) and the relationship (5) appears to be impractical due to the large size of the required ensemble. However, as has been already shown (Ancell and Hakim 2007; Torn and Hakim 2008), meaningful results can be obtained by inverting
In most applications, however, the background flow is characterized by strong nonlinearities, which cause the TL operator (and its adjoint) to have exponentially growing (unstable) modes. As a consequence, the AS analysis becomes impractical for forecast periods exceeding the e-folding times of the unstable modes.
b. Nonlinear case
It is noteworthy that the operator
Let us consider the case when the forecast time T is considerably larger than smallest TLI time scale, a not restrictive constraint in the majority of applications. In that case we can still apply the AS algorithm (3), although the result will be severely contaminated by the small-scale noise caused by TLIs. One may expect that averaging the AS estimates (3) over a suitable ensemble of the background states (i.e., over the ensemble of the matrices 
In the following sections, we compare the accuracy and efficiency of the AS and ES methods within the framework of twin-data experiments with the 4DVAR and R4DVAR data assimilation systems in a nonlinear QG model (Yaremchuk et al. 2009). The R4DVAR method has many features of the ensemble assimilation techniques, as it directly computes the cost function gradient (without the adjoint) for a sequence of subspaces spanned by the continuously updated ensemble members. Considering real applications, we adopt the following experimentation strategy: the AS and ES are estimated with respect to the optimized states obtained by, respectively, the 4DVAR and R4DVAR methods, and then these estimates are compared with the sensitivity estimates of the “true” state (which has been sparsely subsampled to obtain the above-mentioned optimal solutions).
3. Methodology
a. Experimental setting
Streamfunction (a) at the start and at the end of integration for the (b) linear, (c) WNL, and (d) NL true solutions. Locations of the TQs used in the sensitivity experiments are shown in (c). Asterisks in (b)–(d) show subsampling (“observation”) locations of the true solution.
Citation: Monthly Weather Review 142, 2; 10.1175/MWR-D-13-00123.1
Three true solutions have been used in the sensitivity experiments, all of them starting with the initial distribution of ψ shown in Fig. 1a. The first solution, purely linear, with ν = 300 m2 s−1 was obtained by removing the Jacobian in (11). The corresponding distribution of the streamfunction at the end of the integration is shown in Fig. 1b. The second, weakly nonlinear (WNL) solution had diffusion ν = 300 m2 s−1 comparable in magnitude with advection to ensure stability of the TL model (Fig. 1c). The third (nonlinear, NL) solution (Fig. 1d) was advection dominated with a grid-scale dissipation time of δ2/ν ~ 100 days (ν = 30 m2 s−1).
b. 4DVAR and R4DVAR optimization methods
To compare the accuracy of the AS and ES estimates, we simulated the entire procedures of their computation. At the first stage, an optimal solution is obtained either by using the adjoint model (4DVAR method) or by using an ensemble approach (R4DVAR). After that, these optimal solutions are employed to assess sensitivities in a similar way: using the adjoint code (3) and using the output of the R4DVAR optimization algorithm (5).
The 4DVAR optimization computed the cost function gradient with respect to the vector of control variables c = ζ(t0) using the adjoint code. The quasi-Newtoninan minimization algorithm of Gilbert and Lemarechal (1989) was used for descent.
c. AS and ES experiments
The diffusion coefficient in the WNL case was obtained after a series of experiments with the response of the adjoint model to the forcing by the operators of the TQs q1 and q2: In the NL case, the adjoint model exhibited exponential growth of the solutions with an approximate e-folding time τ of 5–7 days (Fig. 2, top curves). To stabilize the adjoint solution, an extra diffusion term with ν = 260 m2 s−1 was added to diffuse the adjoint variables.
Growth of the adjoint relative vorticity ω associated with the TQs q1 (shown in black) and q2 (gray). Bottom lines correspond to the adjoint model stabilized with the additional diffusion.
Citation: Monthly Weather Review 142, 2; 10.1175/MWR-D-13-00123.1
The ES experiments utilized the sequence of ensembles generated by the R4DVAR algorithm. The ensemble members were used to perturb the optimal solutions and compute the covariances p,
4. Results
a. Linear case
Figure 3 shows the AS and ES estimates in the linear case with β = 4 × 10−11 m−1 s−1. In such a simplified setting, solutions of the optimization problem (14) by the 4DVAR and R4DVAR methods are quite similar, with the approximation errors e = 0.19.
(a)–(c) Sensitivities in the linear case. (a) Correlation coefficients ρ with the AS distribution computed by averaging over the squared subdomain are shown for the ES estimates obtained with (b) 60 and (c) 4 ensemble updates. (d) The dependence of 1 − ρ on the number of ensemble updates nu used in the R4DVAR ES estimates.
Citation: Monthly Weather Review 142, 2; 10.1175/MWR-D-13-00123.1
The true sensitivity map obtained by the adjoint method (Fig. 3a) appears to be accurately approximated (ρ = 0.99) by the ES map computed by averaging over the 900 ensemble members (nu = 60; Fig. 3b). It is unlikely that such a level of accuracy can be achieved in real applications because it may require an ensemble size comparable with the number of the control variables. A more realistic estimate with only 60 ensemble members (nu = 4) still produces an acceptable approximation (Fig. 3c) to Fig. 3a, indicating that the TQ operator can be described by these members with a reasonable degree of accuracy. The corresponding sensitivity distribution (Fig. 3c) appears to be somewhat smoothed (cf. Fig. 3a) due to the lack of high-frequency harmonics in the leading ensemble members. Similar behavior was also observed by Ancell and Hakim (2007) in ES experiments with a 90-member ensemble Kalman filter applied to a regional Weather Research and Forecasting Model (WRF).
The accuracy of the approximation of the AS map by the ES estimates is shown in Fig. 3d as a function of the number of ensemble members used. In the linear case considered, only nu = 4 ensemble updates (60 members) are sufficient to achieve a correlation of ρ = 0.81 with the true sensitivity map shown in Fig. 3a.
From a computational point of view, performing nu = 4 ensemble updates (mnu = 60 perturbed model runs) of the R4DVAR algorithm is approximately equivalent to 28 iterations of the 4DVAR algorithm, which actually converged in 240 iterations, whereas the R4DVAR approach required 60 ensemble updates (1.6 times more CPU time) to obtain the pattern in Fig. 3b.
b. Nonlinear case
The results of assimilation in the nonlinear case (true solutions shown in Figs. 1c,d) were quite different. Similar to the linear case, the R4DVAR algorithm converged after 60–70 ensemble updates, whereas the 4DVAR method exhibited a loss of search direction after 80–120 iterations. As a result, the respective approximation errors e of the true solution were significantly larger in the 4DVAR case than in the R4DVAR case (e = 0.32–0.44 versus 0.21–0.28). On the other hand, the CPU time required for 4DVAR optimization was 2.5–4 times less than for R4DVAR. It should be noted, however, that after 8–10 ensemble updates, the R4DVAR approximation error was already compatible with the one obtained by the 4DVAR method.
In the sensitivity experiments, we considered two nonlinear cases: with the stable and unstable adjoint models. The results of the respective AS and ES runs allowed us to compare the pros and cons of both techniques and investigate the sensitivity in the NL case from a conceptual point of view.
1) Stable TL model
Figure 4 compares the q1 AS and ES estimates computed using linearizations with respect to the corresponding optimal solutions against the AS estimate computed in the vicinity of the true solution (Fig. 1c). Similar to the linear case, the ES map of the true solution (not shown) was almost identical (ρ = 0.99) to the AS map in Fig. 4a. The major differences between the sensitivity maps in panels Figs. 4b–d and Fig. 4a are due to the differences in the optimal states obtained by the 4DVAR and R4DVAR methods.
Sensitivities in the WNL case: (a) true sensitivity, (b) AS estimate, and (c),(d) ES estimates. Estimates in (b)–(d) were computed using model linearizations in the vicinity of the optimal states obtained by the 4DVAR method in (b) and the R4DVAR method with 60 and 20 ensemble updates in (c) and (d), respectively.
Citation: Monthly Weather Review 142, 2; 10.1175/MWR-D-13-00123.1
The approximation error of the true sensitivity by the ES method as a function of the number of ensemble updates is shown in Fig. 3d by the gray line. Compared to the linear case, the WNL ES estimates required twice as many ensemble updates (nu = 9) to achieve a reasonable correlation (ρ = 0.8) with the true sensitivity. At larger n, however, the difference in ρ between the linear and weakly nonlinear cases becomes virtually indistinguishable.
2) Unstable TL model
The case of the unstable TL model is the most interesting from the conceptual point of view. In this case, at any given time the TL operator has a number of exponentially growing modes whose structure is characterized by small-scale spatial variability. Numerically, this property of the TL model causes a considerable difference between the forecasts generated by the unperturbed and perturbed model solutions, which shows no linear dependence on the perturbations' magnitude. Although this difference is bounded by the conservation laws governing the dynamical system, normalization of the model's response by the magnitude of the perturbation produces unacceptably large sensitivity maps dominated by the grid-scale patterns (Fig. 5a). It is instructive that patterns of this type are produced by both the AS and ES algorithms, because the only difference between them is in the method of computing the derivatives with respect to the perturbations. The AS method employs analytical differentiation to obtain the code for multiplication by 
Sensitivity estimates in the NL case computed by linearization in the vicinity of the true state: (a) the AS estimate, (b) estimate obtained by averaging over the ensemble of AS estimates computed by linearization in the vicinity of the perturbed true states, (c) the smoothed AS estimate (a), and (d) the AS estimate with diffusive stabilization of the adjoint model.
Citation: Monthly Weather Review 142, 2; 10.1175/MWR-D-13-00123.1
As a consequence, one comes to the conclusion that model solutions at integration times exceeding the TLI time scale (Fig. 2) are nondifferentiable numerically. This phenomenon degrades the performance of the 4DVAR optimization algorithms in strongly nonlinear regimes and makes the linear sensitivity estimates worthless. The problem could be resolved by changing the concept of the sensitivity analysis in the NL regimes. Given the intrinsic uncertainty of an optimized solution, it is more reasonable to consider a “composite” AS map, obtained by averaging over the maps corresponding to linearizations with respect to the ensemble of solutions whose spread around the optimal one is consistent with the a posteriori knowledge of the above-mentioned uncertainty.
Figure 5b shows the result of averaging the AS estimates of the true solution over the ensemble of 200 model runs generated by adding white noise perturbations to the true initial distribution of the potential vorticity (the corresponding streamfunction is shown in Fig. 1a). To suppress eventual instabilities caused by violation of the Courant–Friedrichs–Lewy (CFL) condition, the perturbations were filtered with a cutoff length scale of two grid steps. The magnitude ξ of the perturbations (relative to the RMS magnitude of the true solution) was equal to the typical value of the approximation error e ~ 0.3. It is remarkable that the sensitivity map in Fig. 5b resembles the NL map in Fig. 4a, although the magnitude of the sensitivity maxima in Fig. 5b is 30%–50% larger than in Fig. 4a. In the following, we will refer to the ensemble-averaged sensitivity (EAS) map in Fig. 5b as the true sensitivity in the NL case.
Obtaining EAS maps of the type shown in Fig. 5b is unfeasible in practice because it requires many runs of the model and its adjoint. Therefore, numerically efficient approximations to such maps should be of interest. A straightforward method of smoothing the AS map in Fig. 5a appears to be inefficient. The largest correlation ρ = 0.34 with Fig. 5c is obtained with a smoothing scale l = 5.4δ and the respective sensitivity distribution (Fig. 5c) barely resembles Fig. 5b. An alternative approach is to compute the sensitivity using the adjoint model with enhanced dissipation (Hoteit et al. 2005). The resulting stabilized adjoint sensitivity (SAS) map (Fig. 5d) correlates well with Fig. 5b (ρ = 0.80) but has a somewhat smaller magnitude. Similarity between the maps in Figs. 5b,d supports the considerations in section 2b and justifies the stabilization method of Hoteit et al. (2005).
In real applications the true state is never known, so we consider further below the AS (ES) sensitivities with respect to the optimal states, obtained in the course of the 4DVAR (R4DVAR) optimizations. The R4DVAR analog of the SAS technique requires integration of the R4DVAR perturbation model with the enhanced dissipation to obtain a “stabilized version” of the correlation vector p, which is then used for computing an approximation to the EAS map via (5). In applications, such integration has to be performed in parallel with the R4DVAR optimization process. Figure 6b demonstrates the stabilized ES (SES) map obtained using such an approach. The sensitivity pattern is well correlated (ρ = 0.75) with the EAS map in Fig. 6a, despite utilization of the R4DVAR-optimized solution (e = 0.28) as a reference state for the perturbations. The SES map in Fig. 6b approximates the true sensitivity map in Fig. 6a much better than does the SAS maps generated using the 4DVAR- and stabilized 4DVAR-optimized solutions for linearization of the adjoint model, which demonstrate lower correlations (ρ = 0.50 and ρ = 0.54, respectively) with the true sensitivity map in Fig. 6a.
Sensitivity estimates in the NL case computed by linearizations in the vicinity of the optimized states obtained by (b) R4DVAR (SES method), (c) 4DVAR (SAS method), and (d) the stabilized 4DVAR (SAS method). (a) For comparison the true sensitivity is shown.
Citation: Monthly Weather Review 142, 2; 10.1175/MWR-D-13-00123.1
To compare the skill of the above-mentioned sensitivity computation techniques (EAS, SES, and SAS), a series of numerical experiments has been conducted with the three different TQs shown in Fig. 1c and described by (16) and (17). The results of the experiments are presented in Table 1. It is evident that both the SAS and SES methods tend to underestimate the sensitivity magnitude of the (computationally much more expensive) EAS method, but, nevertheless, provide a reasonable approximation to the overall pattern (cf. the two bottom numbers in the rightmost column with the top ones). The R4DVAR method appears to have slightly better skill than 4DVAR, but this is mostly due to the better approximation of the true solution provided by R4DVAR (e = 0.28 versus e = 0.44). The difference between the 4DVAR and stabilized 4DVAR (S4DVAR) assimilation results was virtually indistinguishable, both in terms of the errors e = 0.44–0.46 and the accuracy of the sensitivity maps.
RMS magnitude (numbers to the left within the cells) and correlation coefficient ρ (numbers to the right within the cell) of the EAS, SES, and SAS estimates (respectively, the top, middle, and bottom numbers within the cells) for the three TQs. Computations were performed using the true reference state (rows 1–3) and the reference states obtained with the stabilized 4DVAR (S4DVAR; rows 4–6) and R4DVAR (rows 7–9) methods. The magnitude of the sensitivity estimates is normalized by the respective EAS values.
We also explored the dependence of the EAS estimates on the amplitude of the ensemble perturbations (Fig. 7). The U-shaped curve is caused by two types of instabilities: on the one hand, perturbations cannot be too small due to the presence of growing modes in the TL model, and on the other hand, they are limited from above by the CFL condition. In our case, the true NL solution was characterized by the maximum CFL value of 0.6. As a consequence, imposing perturbations with the same amplitude as the solution caused a violation of the CFL stability criterion. In addition, imposing perturbations of such a large magnitude is questionable in itself, as the reference model dynamics can be lost in the background of the model's interactions with the perturbations.
Sensitivity magnitude estimated by the EAS method as a function of the relative perturbation amplitude ξ. Dashed line shows |S| computed by the SAS technique.
Citation: Monthly Weather Review 142, 2; 10.1175/MWR-D-13-00123.1
In a sense, the magnitude of the perturbations in the EAS method is clinched between the Scylla of the TL instability and the Charybdis imposed by the above-mentioned natural limitations. In the reported experiments, we used the value of ξ = 0.3 for computation of the true and R4DVAR-referenced sensitivities. It is remarkable, however, that the largest values of ρ for the 4DVAR- and S4DVAR-referenced EAS estimates was achieved at ξ = 0.45, a value consistent with the approximation errors of the respective optimized solutions.
c. Computational cost
ES estimation within the R4DVAR algorithm in the linear and WNL regimes requires just a small fraction of the CPU time required for data optimization because the perturbations of the TQs can be effectively computed during the perturbed model runs in the course of minimization of the cost function. Upon completion, the correlation vector p is easily computed together with the pseudoinversion of the perturbation matrix. With a relatively small (100–500) number of perturbations, these operations are not costly and produce ES estimates comparable in accuracy to the AS ones (Fig. 4).
In the NL case, the concept of sensitivity has to be transformed from the deterministic to the statistical framework. As a consequence, direct numerical sensitivity estimates appear to be computationally unfeasible and require efficient approximation methods. Extension to the R4DVAR case of the adjoint stabilization technique proposed by Hoteit et al. (2005) requires additional integration of the stabilized perturbation model, which can be executed in parallel with the ensemble runs of the R4DVAR optimization algorithm. The twofold increase of the computational requirements can be partly justified by the improved performance of the R4DVAR algorithm in the NL regimes.
5. Summary and discussion
In many geophysical problems, application of the 4DVAR data assimilation technique encounters difficulties associated with the emergence of exponentially growing modes of the TL operator in flow regimes characterized by nonlinear instabilities. A straightforward way to deal with the problem is to either increase the diffusion coefficient in the adjoint of the background numerical model (Hoteit et al. 2005; Zhang et al. 2011) or to reduce the duration of the data assimilation window to the smallest e-folding time scales of the instabilities (Xu and Daley 2000). The latter approach is acceptable in operational meteorology, which enjoys dense daily coverage by observations, but is rarely used in oceanography (Ngodock et al. 2009) where observations within the ocean interior are sparse.
Within the context of a sensitivity analysis, NL systems require a conceptual transition from a deterministic (adjoint based) to a probabilistic (ensemble based) methodology. In this study we have shown that AS estimates stabilized by enhanced dissipation are capable of providing a good approximation to the probabilistic sensitivities in the NL regimes.
Special attention has been paid to the comparison of the sensitivity estimates obtained with the ensemble-based R4DVAR data assimilation technique and with the 4DVAR method in both linear and nonlinear regimes. In the linear and WNL regimes, the R4DVAR ES estimates are similar in accuracy to the AS ones and do not require additional computations if the perturbations of the TQs are computed in the process of R4DVAR optimization. In the NL case, the ensemble analog of the stabilized AS technique is developed. The proposed stabilized ES method is similar in accuracy to the SAS approach, but requires a twofold increase in computer time due to the necessity of augmenting the R4DVAR optimization process with integration of the stabilized perturbation model.
The R4DVAR ES technique is similar to the one considered by Ancell and Hakim (2007) with the difference that the ensemble averages are not removed and the correlation vector p is always multiplied by the pseudoinverse of the ensemble matrix. A number of computations were performed with the removal of the ensemble averages and/or replacement of
Theoretical arguments (section 2b) supported by numerical experiments have shown the ability of the SES and SAS methods to effectively approximate the EAS sensitivity maps in the NL regime. Confirming this result with a real OGCM requires particular care in the definition of the amplitude of the ensemble perturbations (section 4b), which contain multivariate state vectors and may be affected by the higher-order nonlinearities present in the model.
Extending the R4DVAR ES technique to optimization (with respect to TQs) of the observational networks (observation targeting) appears to be a feasible task, because products of the ensemble members by the Hessian of the assimilation problem are among the outputs of the R4DVAR optimization process. A similar approach to observation targeting can be undertaken with the 4DVAR method, which generates a sequence of descent directions in the process of minimizing the cost function.
The linearization technique underlying the 4DVAR and AS analysis imposes natural limitations on the performance of the adjoint methods in NL regimes. This opens further prospects for the development of the ensemble approach in sensitivity analysis and optimization of observational networks.
Acknowledgments
This study was supported by ONR Program Element 0602435N as part of the project Adjointless 4DVAR for Navy Ocean Models and was made possible in part by a grant from BP/The Gulf of Mexico Research Initiative through the Consortium for Advanced Research on Transport of Hydrocarbon in the Environment (CARTHE). Helpful discussions with Prof. E. Benilov are also acknowledged.
APPENDIX
Averaging the Advective Propagator
The operator 
REFERENCES
Ancell, B., and G. J. Hakim, 2007: Comparing adjoint- and ensemble-sensitivity analysis with applications to observation targeting. Mon. Wea. Rev., 135, 4117–4134.
Baker, N., and R. Daley, 2000: Observation and background adjoint sensitivity in the adaptive observation-targeting problem. Quart. J. Roy. Meteor. Soc., 126, 1431–1454.
Cao, Y., J. Zhu, I. M. Navon, and Z. Luo, 2007: A reduced-order approach to four-dimensional variational data assimilation using proper orthogonal decomposition. Int. J. Numer. Methods Fluids, 53, 1571–1583.
Clayton, A. M., A. C. Lorenc, and D. M. Barker, 2013: Operational implementation of a hybrid ensemble/4D-Var global data assimilation system at the Met Office. Quart. J. Roy. Meteor. Soc., 139, 1445–1461, doi:10.1002/qj.2054.
Farrell, B. F., and P. J. Ioannou, 2005: Approximating optimal state estimation. Predictability of Weather and Climate, T. N. Palmer and R. Hagedorn, Eds., Cambridge University Press, 181–216.
Gilbert, J., and C. Lemarechal, 1989: Some numerical experiments with variable-storage quasi-Newton algorithms. Math. Programming, 45, 407–435.
Hoteit, I., B. Cornuelle, A. Köhl, and D. Stammer, 2005: Treating strong adjoint sensitivities in tropical eddy-permitting variational data assimilation. Quart. J. Roy. Meteor. Soc., 131, 3659–3682.
Hoteit, I., B. Cornuelle, and P. Heimbach, 2010: An eddy-permitting, dynamically consistent adjoint-based assimilation system for the tropical Pacific: Hindcast experiments in 2000. J. Geophys. Res., 115, C03001, doi:10.1029/2009JC005437.
Lee, T., B. Cheng, and R. Giering, 2001: Adjoint sensitivity of Indonesian Throughflow transport to wind stress: Application to interannual variability. Jet Propulsion Laboratory Publ. 01-11, 34 pp.
Liu, C., Q. Xiao, and B. Wang, 2008: An ensemble-based four-dimensional variational data assimilation scheme. Part I: Technical formulation and preliminary test. Mon. Wea. Rev., 136, 3363–3373.
Losch, M., and P. Heimbach, 2007: Adjoint sensitivity of an ocean general circulation model to bottom topography. J. Phys. Oceanogr., 37, 377–393.
Mirouse, I., and A. Weaver, 2010: Representation of correlation functions in variational data assimilation using an implicit diffusion operator. Quart. J. Roy. Meteor. Soc., 136, 1421–1443.
Moore, A. M., H. G. Arango, E. D. Lorenzo, A. J. Miller, and B. D. Cornuelle, 2009: An adjoint sensitivity analysis of the Southern California Current circulation and ecosystem. J. Phys. Oceanogr., 39, 702–720.
Ngodock, H., S. Smith, and G. Jacobs, 2009: Cycling the representer method with non-linear models. Data Assimilation for Atmospheric, Oceanic and Hydrological Applications, S. K. Park and L. Xu, Eds., Vol. 1, Springer, 321–340.
Purser, R. J., W. Wu, D. F. Parrish, and N. M. Roberts, 2003: Numerical aspects of the application of recursive filters to variational statistical analysis. Part II: Spatially inhomogeneous and anisotropic covariances. Mon. Wea. Rev., 131, 1536–1548.
Torn, R. D., and G. J. Hakim, 2008: Ensemble-based sensitivity analysis. Mon. Wea. Rev., 136, 663–677.
Torn, R. D., and G. J. Hakim, 2009: Initial condition sensitivity of western Pacific extratropical transitions determined using ensemble-based sensitivity analysis. Mon. Wea. Rev., 137, 3388–3406.
Veneziani, V., C. A. Edwards, and A. M. Moore, 2009: A central California coastal ocean modeling study: Adjoint sensitivity to local and remote forcing mechanisms. J. Geophys. Res., 114, C04020, doi:10.1029/2008JC004775.
Xu, L., and R. Daley, 2000: Towards a true 4-dimensional data assimilation algorithm: Application of a cycling representer algorithm to a simple transport problem. Tellus, 52A, 109–128.
Xu, Q., 2005: Representation of the inverse covariances by differential operators. Adv. Atmos. Sci., 22, 181–194.
Yaremchuk, M., D. Nechaev, and G. G. Panteleev, 2009: A method of successive corrections of the control subspace in the reduced-order variational data assimilation. Mon. Wea. Rev., 137, 2966–2978.
Yaremchuk, M., M. Carrier, S. Smith, and G. Jacobs, 2013: Background error correlation modeling with diffusion operators. Data Assimilation for Atmospheric, Oceanic and Hydrological Applications, S. K. Park and L. Xu, Eds., Vol. 2, Springer, 177–203.
Zack, J., E. Natenberg, S. Young, G. V. Knowe, K. Waight, J. Manobianco, and C. Kamath, 2010: Application of ensemble sensitivity analysis to observation targeting for short-term wind speed forecasting in the Washington–Oregon region. LLNL Tech. Rep. 458086, Lawrence Livermore National Laboratory, Livermore, CA, 65 pp.
Zhang, X., B. Cornuelle, and D. Roemmich, 2011: Adjoint sensitivity of the Niño-3 surface temperature to wind forcing. J. Climate, 24, 4480–4493.
