1. Introduction

In numerical weather prediction (NWP), necessity often arises to evaluate sensitivity of forecasts with respect to changes in initial conditions. A natural measure of such sensitivity is a gradient of some interesting aspects of forecasts with respect to initial conditions. This gradient-based sensitivity is called adjoint sensitivity and can be quantified by running adjoint models associated with numerical models from which forecasts are produced (Errico and Vukicevic 1992; Langland et al. 1995). Since adjoint models are essential in any four-dimensional variational (4DVAR) data assimilation systems, adjoint sensitivity can be estimated relatively easily if we have access to a 4DVAR system by utilizing the driving model and its adjoint code therein.

However, adjoint models are not always available in practice due to their high maintenance costs when NWP models are frequently updated with more sophisticated components. In such cases, adjoint sensitivity can still be estimated from ensembles of analyses and forecasts (Ancell and Hakim 2007; Hakim and Torn 2008), which are always available in any ensemble prediction system. This ensemble-based estimation of adjoint sensitivity can be reinterpreted as taking multiple linear regression coefficients between forecasts and initial conditions as a measure of sensitivity. Regression coefficients in this case are determined by two covariances: analysis error covariances $\mathsf{A}$ and cross covariances between forecasts and initial conditions. In practice, these covariances are approximated by sample covariances calculated from analysis and forecast perturbations.

The regression-based estimation of adjoint sensitivity requires inversion of $\mathsf{A}$, which is impractical since $\mathsf{A}$ is a very large matrix in NWP. In its particular application to observation targeting, Ancell and Hakim (2007) showed that the inverse of $\mathsf{A}$ need not be calculated explicitly. This is because the inverse of $\mathsf{A}$ that appears in the regression-based estimation of adjoint sensitivity cancels out with $\mathsf{A}$ that comes from the definition of the impact of targeted observations. Along with its derivation, they introduced a new quantity for sensitivity, called ensemble sensitivity, or more precisely, univariate ensemble sensitivity, which helps to simplify the mathematical expression. Ensemble sensitivity shares the same form as the regression-based expression of adjoint sensitivity but with $\mathsf{A}$ replaced by a diagonal matrix $\mathsf{D}$, which is derived from $\mathsf{A}$ by setting all off-diagonal elements of $\mathsf{A}$ to zeros. With this diagonal matrix, ensemble sensitivity can be reinterpreted as simple linear regression coefficients between forecasts and initial conditions. Note that, as demonstrated by Ancell and Hakim (2007), despite its designation as univariate sensitivity, its nature is in fact multivariate in the sense that a univariate ensemble sensitivity of a response function with respect to a single element of the initial state involves adjoint sensitivities to all elements of the initial state and the covariance relationship of this single element with all elements of the initial state.

The use of ensemble sensitivity has expanded beyond its original derivation as a convenient tool or an intermediate step in quantifying the impact of targeted observations and it has later been used independently in many sensitivity studies (Torn and Hakim 2008; Garcies and Homar 2009; Torn 2010; Chang et al. 2013; Hanley et al. 2013; Ito and Wu 2013; Bednarczyk and Ancell 2015; Brown and Hakim 2015; Hill et al. 2016; Yokota et al. 2016). The extensive use of ensemble sensitivity in sensitivity analysis can be attributed to its simplicity in formulation where the otherwise expensive inversion of $\mathsf{A}$ is no longer required. However, ensemble sensitivity is not the same as adjoint sensitivity in the context of ensemble sensitivity analysis, and its relevance in the context of sensitivity analysis needs a justification. Due to the resemblance of ensemble sensitivity and simple linear regression coefficients, one of a justification could be given by making use of the so-called diagonal approximation which assumes that $\mathsf{A}$ can be approximated by a diagonal matrix with the same diagonal elements as those of $\mathsf{A}$ (Hacker and Lei 2015; Limpert and Houston 2018). Under this assumption, adjoint sensitivity reduces to ensemble sensitivity. This justification is clearly inadequate in most situations in NWP where analysis errors at adjacent grid points are known to be highly correlated. Ancell and Hakim (2007) and later Ancell and Coleman (2022) have challenged such a controversial interpretation by showing that ensemble sensitivity is in fact a product of regression coefficients of individual state variables and adjoint sensitivity, thus being multivariate in nature. In other words, the diagonal matrix $\mathsf{D}$ is not an approximation, but is simply a part of the definition of ensemble sensitivity. However, this interpretation does not explain the reason underlying the crucial step of replacing $\mathsf{A}$ by its diagonal one, leading to the source of confusion in understanding the nature of ensemble sensitivity.

Along this vein, in order to get rid of the controversial diagonality assumption, Hacker and Lei (2015) came back to the understanding of the ensemble-based estimation of adjoint sensitivity as multiple linear regression coefficients. They avoided the inversion of $\mathsf{A}$ by reformulating the underlying regression problem as an underdetermined least squares problem, and identified the minimum-norm solution of this problem with ensemble sensitivity. An advantage of this approach is that we only need to invert a small matrix in ensemble space. Ensemble sensitivity resulting from this approach is called multivariate ensemble sensitivity to distinguish it from univariate ensemble sensitivity introduced by Ancell and Hakim (2007).

It is expected that multivariate ensemble sensitivity should be different from univariate ensemble sensitivity since the former does not rely on the diagonal approximation, as claimed and demonstrated in Hacker and Lei (2015) and Ren et al. (2019). Curiously, however, in their numerical experiments, Hacker and Lei (2015) found that changes of forecasts yielded by univariate and multivariate ensemble sensitivity with respect to one standard deviation changes of initial conditions are identical when localization is not applied (see Fig. 1a in their paper). The authors did not give any adequate explanations for such a coincidence. Note that localization is an ad hoc technique to mitigate the impact of sampling errors on sample covariances. Thus, in the limited case when we can access a large number of ensemble members so that localization can be turned off, those results imply an interesting dilemma: despite the difference in foundations of univariate and multivariate ensemble sensitivity, they have the same effect on changes of forecasts. In other words, we return to multivariate ensemble sensitivity to avoid the diagonality assumption, and finally we end up with univariate ensemble sensitivity again. How can we explain the effective work of univariate ensemble sensitivity given its controversial diagonality assumption?

In this paper, we show that it is not accidental that univariate and multivariate ensemble sensitivity yield the same changes of forecasts. Underlying this dilemma is a theory that establishes a robust foundation for ensemble sensitivity. In short, ensemble sensitivity is equivalent to an effective adjoint sensitivity that yields the same forecast response as adjoint sensitivity without a need of calculating perturbations at correlated points. This is justified by the proof that the same forecast response is obtained for any analysis error covariances. By showing that ensemble sensitivity is a rigorous concept, we can resolve confusion regarding the use of ensemble sensitivity in practice. Furthermore, we prove that ensemble sensitivities to all elements of the initial state form the most sensitive initial perturbation which maximizes changes of forecasts among all initial perturbations with the same magnitudes—a fact that, to the best knowledge of the authors, has been hitherto unrecognized in the literature.

This paper is organized as follows. Section 2 prepares a mathematical background for deploying a robust foundation of ensemble sensitivity. In section 3, we revisit the formerly proposed definition of sensitivity as regression coefficients and seek a new definition that also takes into account probability distributions of initial perturbations. Section 4 shows how ensemble sensitivity follows naturally from the established framework. Another important role of the proposed ensemble sensitivity is examined in section 5 in which ensemble sensitivity is proved to form the most sensitive analysis perturbation. Finally, section 6 summarizes the main findings of this study.

2. Background on ensemble sensitivity

Due to a limited number of ensemble members, $\mathsf{X}\mathsf{X}$^T is usually a low-rank matrix and consequently its inverse does not exist in practice. Ancell and Hakim (2007) proposed to circumvent these issues by replacing the sample covariance $\mathsf{X}\mathsf{X}$^T in (5) by a diagonal matrix $\mathsf{D}$ whose diagonal is the diagonal of $\mathsf{X}\mathsf{X}$^T, i.e., the analysis sample variances. The sensitivity that results from this procedure was termed ensemble sensitivity by Ancell and Hakim (2007). By replacing $\mathsf{X}\mathsf{X}$^T by $\mathsf{D}$, the matrix to be inverted in (5) becomes full rank and the inversion operation is now trivial. Ensemble sensitivity has been shown to be fundamentally different from adjoint sensitivity. However, its nature is usually justified by the so-called diagonal approximation (that $\mathsf{A}$ can be assumed diagonal), which obscures interpretation of the ensemble sensitivity thus defined. We will discuss this issue in more detail in section 4.

Thus, (7) shows that we can calculate adjoint sensitivity by using either the primal form ($\mathsf{X}\mathsf{X}$^T)$\mathsf{X}\mathsf{Y}$^T or the dual form $\mathsf{X}$($\mathsf{X}$^T$\mathsf{X}$)$\mathsf{Y}$^T. The advantage of the dual form is that we have only invert the small K × K matrix $\mathsf{X}$^T$\mathsf{X}$, which is much easier than to invert the very large n × n matrix $\mathsf{X}\mathsf{X}$^T in the primary form. However, the side effect is that localization is difficult to introduce into the dual form. Hacker and Lei (2015) proposed this form in estimating adjoint sensitivity using ensembles of analyses and forecasts. It is clear from (7) that the solution obtained in Hacker and Lei (2015) is indeed the regression coefficients represented in their dual form. Therefore, the ensemble-based adjoint sensitivity that they examined is identical to the ensemble-based adjoint sensitivity considered in Ancell and Hakim (2007). In the following sections, we mainly work with the primal form in mathematical treatment.

3. Standardized ensemble sensitivity

Without using the diagonality assumption due to its potential confusion, we will define ensemble sensitivity through the forecast response considering both linear transformations $[\partial J/\partial {\mathbf{x}}_0]$ and perturbations Δx₀. In contrast, the regression-based version of adjoint sensitivity given in (7) identifies sensitivity only with linear transformations between Δx₀ and ΔJ. Thus, while the former approaches sensitivity from the viewpoint of predictability, the latter approaches sensitivity from the viewpoint of dynamical systems. This former approach underlines the important role of the distribution of initial perturbations besides linear transformations since Δx₀ are not arbitrary and have to follow a certain probability distribution which in our problem is assumed to be a multivariate normal distribution with the mean 0 and the covariance $\mathsf{A}$.

4. A unified theory for ensemble sensitivity

There exist many probability distributions of Δx₀ that have the mean 0 and the covariance $\mathsf{A}$. To find a closed form for Δx_0i in (11), we assume that Δx₀ follow a multivariate normal distribution $\mathcal{N}(0,\mathsf{A})$. Our problem is how we determine Δx_0j(j ≠ i) given Δx_0i = std(Δx_0i). Ren et al. (2019) proposed that Δx_0j can be estimated from Δx_0i by performing simple linear regressions of Δx_0j on Δx_0i. Here, we show a more systematic way to assign appropriate values to Δx_0j(j ≠ i) whence Δx_0i is known. At first, note that we take Δx_0i = [0, …, 0, std(Δx_0i), 0, …, 0]^T when $\mathsf{A}$ is diagonal, i.e., all Δx_0j(j ≠ i) are set to zero. This shows that in the absence of correlations between Δx_0j and Δx_0i, we take the expectations of Δx_0j, which are zero, as their typical values. Therefore, this suggests that in the presence of correlations between Δx_0j and Δx_0i, the conditional expectations of Δx_0j, i.e., E[Δx_0j|Δx_0i = std(Δx_0i)], should be taken as their typical values.

As we have seen above, the ensemble sensitivity follows naturally from the definition of standardized ensemble sensitivity (11) and the multivariate normal distribution of Δx₀. The controversial diagonality assumption is irrelevant in construction of ensemble sensitivity. This diagonality assumption has been the source of confusion in understanding and applying ensemble sensitivity. First, given the inadequacy of neglecting spatial and interelement correlation in analysis perturbations, ensemble sensitivity has often been considered as a rough approximation of adjoint sensitivity, and the validity of its use has been only justified by its simplicity. Second, since the diagonality assumption is equivalent to assuming that all elements Δx_0i are independent, ensemble sensitivity has often been regarded to be univariate in nature, which has led to a common conception that sensitivity as represented by ensemble sensitivity is overestimated because significant contributions from other elements are ignored. The mathematical arguments in this section point out that ensemble sensitivity is multivariate in nature and is not just an approximation to adjoint sensitivity but rather is a distinct measure of sensitivity with a robust mathematical foundation.

In practice, both vectors s (standardized ensemble sensitivity) and std(ΔJ)s (normalized ensemble sensitivity) can be used as a measure of sensitivity. As shown by (16) s_i are simply correlations between the forecast response and initial conditions. As a result, it is relatively easy to estimate s_i and localization can even be imposed on s. Furthermore, because s_i are dimensionless unlike std(ΔJ)s_i, which are dimensioned, they have an advantage of allowing us to compare sensitivities of different forecast metrics. However, since std(ΔJ)s_i give changes of the forecast response when Δx_0i = std(Δx_0i), if forecast metrics are of the same type so that all std(ΔJ)s_i have the same unit, e.g., rainfall at different areas, it is better to use std(ΔJ)s_i. This is because two forecast metrics with the same correlations s_i = cor(Δx_0i, ΔJ) will show different responses if their standard deviations std(ΔJ) are different.

Equation (16) points out a surprising fact: standardized ensemble sensitivities are simply cross correlation coefficients cor(Δx₀, ΔJ) regardless of the form of $\mathsf{A}$. This clearly explains why Hacker and Lei (2015) observed similar estimates of changes in forecast responses with univariate and multivariate ensemble sensitivity when localization was not applied in their numerical experiments. Note that the final Δx_0i in their studies can be shown to take the same form as Δx_0i in (14), although they performed simple linear regressions of Δx_0j on Δx_0i to estimate Δx_0j. In contrast, when localization was applied, they observed significant differences in the estimates of ΔJ between univariate and multivariate ensemble sensitivity. In our mathematical arguments that led to (16), we relied on the true covariances and not their approximations, i.e., the sample covariances. Consequently, even if localization is applied, the estimated changes of forecasts should be similar between multivariate and univariate ensemble sensitivity as long as localization reduces differences between the sample and true covariances. Therefore, the differences of ΔJ that they observed are likely artifacts of the way localization was applied in their study. These authors relied on the dual form $\mathsf{Y}$($\mathsf{X}$^T$\mathsf{X}$)$\mathsf{X}$^T to estimate adjoint sensitivity, and it is difficult to introduce localization into this form as discussed in section 2. As a result, they could only apply localization to analysis perturbations Δx_0i. This inconsistency in calculating the regression coefficients and analysis perturbations might result in these apparent differences.

5. Ensemble sensitivity as the most sensitive perturbation

In this section, we show the relevance of ensemble sensitivity in another important problem related to the sensitivity problem. Instead of estimating how each state element will change the forecast response, we now examine which analysis perturbation among all possible ones with the same magnitudes will yield the largest change in the forecast response. Due to collaborative effect among different elements, this most sensitive analysis perturbation is not expected to be any individual analysis perturbations. As we shall see, intriguingly, ensemble sensitivity is closely involved in this optimization problem.

It is natural to continue with a question on the second most sensitive perturbation by taking the optimization problem further. Thus, we will find the analysis perturbation Δx₀ that maximizes the norm of ΔJ subject to the two constraints ${\Vert \mathrm{\Delta }{\mathbf{x}}_0\Vert }_{{\mathsf{A}}^{-1}}=\rho$ and ${⟨\mathrm{\Sigma }s,\hspace{0.17em}\mathrm{\Delta }{\mathbf{x}}_0⟩}_{{\mathsf{A}}^{-1}}=0$. Compared to the previous optimization problem, an additional constraint has been introduced which limits Δx₀ to the orthogonal complement of the first most sensitive perturbation $\mathrm{\Sigma }$s where orthogonality is defined with respect to the inner product ${⟨⟩}_{{\mathsf{A}}^{-1}}$. Noticing (22) and (23), this constraint can also be rewritten as ${⟨\mathbf{s},\hspace{0.17em}\mathrm{\Delta }{\tilde{x}}_0⟩}_{{\mathsf{C}}^{-1}}=0$. However, (23) has already shown that $\mathrm{\Delta }J=\text{std}(\mathrm{\Delta }J){⟨\mathbf{s},\hspace{0.17em}\mathrm{\Delta }{\tilde{x}}_0⟩}_{{\mathsf{C}}^{-1}}$. This entails that all vectors in the orthogonal complement of s yield ΔJ = 0, i.e., they have no impact on the forecast response. In other words, only vectors in the one-dimensional subspace spanned by $\mathrm{\Sigma }$s can effectively cause changes in the forecast metric.

6. Discussion and conclusions

In ensemble sensitivity analysis, adjoint sensitivity is estimated from the regression coefficients that are obtained by regressing changes in a forecast response on initial perturbations. Ancell and Hakim (2007) pointed out the existence of a new kind of sensitivity in the ensemble context that they called ensemble sensitivity, and ensemble sensitivity has been proved to be a very useful sensitivity measure in practice. While being proved successful, ensemble sensitivity leaves some confusion in its interpretation because its derivation involves an ad hoc operation that replaces the analysis error covariance $\mathsf{A}$ by a diagonal matrix $\mathsf{D}$ whose diagonal elements are taken from those of $\mathsf{A}$. This corresponds to making a so-called diagonal approximation which assumes that all elements of the initial perturbation are uncorrelated although the authors of ensemble sensitivity have challenged such a controversial interpretation by showing the multivariate nature of ensemble sensitivity. A few studies have tried to clear up this confusion by eliminating reliance on the diagonal approximation, leading to yet another measure of sensitivity called multivariate ensemble sensitivity, while leaving the old one with the name univariate ensemble sensitivity. Intriguingly, the two variants of ensemble sensitivity are found to give equivalent results when given with an initial perturbation each element of which has the magnitude of one standard deviation. The purpose of this study is to establish a mathematically rigorous framework that resolves the following three questions:

1. why is univariate ensemble sensitivity so effective in practice despite its (apparent) reliance on the diagonal approximation?

2. why multivariate and univariate ensemble sensitivity result in equivalent sensitivity?

3. what are potential applications of ensemble sensitivity to optimal initial conditions that affect the forecast metric?

The key to resolving the three questions above is to distinguish two kinds of sensitivity: (i) the gradient $[\partial J/\partial {\mathbf{x}}_0]$ of a forecast response J(x_t) with respect to an initial perturbation Δx₀, and (ii) a linear estimation of the change ΔJ in the forecast response given some typical values of an initial perturbation Δx₀. The sensitivity in the sense of (i) above provides the steepest ascent direction of the forecast response and is suitable when we approach the sensitivity problem from a dynamical systems perspective where our primary concern is the structure of the most sensitive initial perturbations. This approach, however, is not suitable when we approach the sensitivity problem from a statistical perspective: we know that elements of the initial perturbation are correlated and thus the initial perturbation Δx₀ cannot take an arbitrary shape, and this fact is totally ignored by the sensitivity (i). Comparison between sensitivities with respect to different meteorological variables is also inconvenient with the sensitivity (i) because the gradient is dimensioned, which means that sensitivities with respect to different variables have different physical dimensions, which are incomparable. Furthermore, and more importantly, the sensitivity (i) may show high sensitivity if the variables of interest have small standard deviations when in fact, they are weakly correlated with the forecast response, and vice versa.

To overcome these deficits and inconveniences, we approach ensemble sensitivity in the sense of (ii) above that accounts for contributions to ΔJ not only from the gradient of forecast response but also from the initial perturbation Δx₀. We have developed a mathematical background to precisely define such a sensitivity measure, which we term standardized ensemble sensitivity, and showed that it leads to the formerly proposed ensemble sensitivity without making any ad hoc assumption like the diagonal approximation, making it a sound and appealing choice as a sensitivity measure. Thus, the mathematical results provide rigorous explanations to the three questions raised above. First, it is found that the same change of the forecast response is obtained regardless of whether $\mathsf{A}$ is diagonal or not, which shows the equivalence of univariate and multivariate ensemble sensitivities. Second, the simple form of univariate ensemble sensitivity is a result of the fact that $\mathsf{A}$⁻¹ in the regression coefficients cancels out $\mathsf{A}$ in the individual analysis perturbations, and we misinterpret this fact as the effect of the diagonal approximation. Finally, as an interesting by-product, the theory reveals that the standardized sensitivity vector coincides with the most sensitive analysis perturbation in the sense of maximizing changes in the forecast response under the constraint of having the same norm.

It is worth emphasizing that the theory points out standardized ensemble sensitivity as an important and rigorous concept rather than ensemble sensitivity. This is because the vector containing standardized ensemble sensitivities carries in itself three important quantities at the same time: 1) standardized changes of the forecast response with one standard deviation changes of individual state variables; 2) correlations between the forecast response and individual state variables; and 3) the most sensitive analysis perturbation, which, for a quadratic response function, coincides with the leading ensemble singular vector. These different theoretical aspects provide a comprehensive picture of ensemble sensitivity.

The theoretical framework developed in this study clarifies the difference between the conventional, gradient-based sensitivity measure and the newly proposed impact-based sensitivity measure. While the former approaches the sensitivity problem from a dynamical systems perspective, the latter is better suited for assessment of sensitivity from a probabilistic perspective of predictability and causality. Future practitioners of ensemble sensitivity analysis are advised to clarify which aspect of sensitivity they intend to investigate before initiating analysis. We hope the theoretical elucidation given in this manuscript will be helpful in this regard.

Data availability statement.

No datasets were generated or analyzed during the current study.