On Controlling the Shape of the Cost Functional in Dynamic Data Assimilation: Guidelines for Placement of Observations and Application to Saltzman’s Model of Convection

1. Introduction

Judicious selection of the type (pressure, humidity, and wind, to name a few) and placement (the location within a given spatiotemporal domain of interest) of weather observations to advance understanding of a phenomenon and the concomitant practical ability to improve forecasts has a rich history in meteorology. Among the most celebrated use of weather observations to understand the ubiquitous extratropical cyclone was the fieldwork of Scandinavian meteorologists Jacob Bjerknes and Eric Palmén (Bjerknes and Palmén 1937). They organized the simultaneous release of 120 radiosondes from 11 European countries over a 2-day period that led to diagnosis of motions in the upper troposphere along a 3500-mi (5630-km) latitudinal cross section. Further, the study made it clear that the meridional gradient of the Coriolis parameter was an important parameter in the dynamics of upper waves. This revelation inspired Carl-Gustaf Rossby to derive his celebrated wave formula (Eliassen 1995).

In the spirit of the Bjerknes–Palmén study, but in our current day, the use of research aircraft to obtain thermodynamic and wind structure in the vicinity of Atlantic hurricanes through use of Omega dropwindsondes (ODWs) has gone far to improve the track forecasting of hurricanes (Burpee et al. 1996). Transmission of these data to operational prediction centers in real time permitted inclusion of these data into the operational models. The improvement in track forecasting as tested on numerous hurricanes was impressive—track error reductions from 16% to 30%. Also impressive was the hurricane forecasting experience of the Burpee team that led them to choose observation locations on the hurricane’s periphery as well as near the storm’s center where the thermodynamic and wind data were critically important as input to track forecasting. Majumdar (2016) and Majumdar et al. (2011) present an excellent review of “targeted observations,” which classified the Burpee et al. (1996) as an example of targeting based on synoptic reasoning—not only an appropriate label for this contribution but also for the earlier contribution by Bjerknes and Palmén (1937).

Starting from the mid-1990s there has been a growing body of work within the meteorological literature related to (i) the analysis of impact of initial condition and observation on a chosen scalar metric related to forecast quality (over a subdomain or the entire domain of interest) and (ii) targeted/adaptive observation aimed at finding the type of a new set of observations that will bring improvement to forecast quality, such as, reduction of analysis covariance, and (iii) when there is a large amount of correlated set of observations (as may happen with satellites or radars), there has been great interest in the so-called “thinning” and/or creating superobservations by computing, say, an average over a given subdomain and associating that observation with the centroid of the subdomain.

The techniques used in these areas may be broadly divided into two groups: either based on the adjoint approach rooted in the classical variational analysis or using one of the many ensemble implementations of Kalman filtering approach to data assimilation. While much of the literature concentrates on the choice of the suitable type of observations to improve the forecast quality, a few also consider distribution of observation to answer a similar goal.

Specifically, Berliner et al. (1999) examine the application of statistical experimental design (Cochran and Cox 1992) to adaptively decide on supplementary observations that would improve the analyses. Torn and Hakim (2008), Ancell and Hakim (2007, hereafter referred to as AH2007), and Hakim and Torn (2008) use ensemble methods for targeting observations. AH2007 also contains a comparison of the adjoint based and ensemble based approaches to observation targeting. Lorenz and Emanuel (1998), Langland et al. (1999), Langland and Baker (2004), Evans et al. (1998), Koch et al. (2018), Manohar et al. (2018), Kotsuki et al. (2019), Majumdar et al. (2011), and Majumdar (2016) contains extensive discussion of “thinning,” creating superobservations, targeted-observation strategies including adjoint sensitivity, forecast sensitivity to observations, ensemble transform filter method, and ensemble sensitivity. All these strategies are designed to improve analyses and ultimately reduce forecast error.

Our approach falls outside the strategies used in targeted/adaptive observations although the goal has strong overlap with those methodologies. It is assumed that we have observations of a given type represented by a vector Z in the observation space R^m. It is well known that when the model map $\mathsf{M}$ and/or the forward operator h (that relates the observation to the state) are nonlinear, the cost functional may exhibit multiple minima (Lewis et al. 2006, example 24.4.1 in chapter 24). In addition to these intrinsic multiple minima that are purely dependent on the given nonlinearities in $\mathsf{M}$ and h, and the covariance matrix of the observational noise, the shape of the cost functional in the control space (consisting of initial conditions and model parameters) is also critically dependent on distribution (location) of the observations in the state space. Our goal is to control the shape of the cost functional by analyzing the dependence of its gradient (with respect to the control) on the (spatiotemporal) location of the observations. Namely, force the magnitude of this gradient to be bounded away from zero outside the intrinsic local minimum.

This is achieved through a strategy that decides on observation placement in space and time but in the context of classical four-dimensional variational data assimilation (4D-Var) (LeDimet and Talagrand 1986; Lewis and Derber 1985). It is well known that the shape of the cost function as measured by its gradient in control space determines the marching of the control iterates toward a local minimum of the cost function. The iterates of the minimization algorithm can become marooned in regions where the gradient is very small. An open question is how to avoid these “flat” regions in control space by guaranteeing that the norm of the gradient of the cost function is bounded away from zero. While much of the earlier work deal with sensitivity of the cost functional only with respect to the initial conditions alone, our analysis simultaneously deals with initial conditions and parameters on the same footing.

This question is answered in two steps. First, we derive a linear transformation defined by a symmetric positive semidefinite (SPSD) Gramian¹ $\mathsf{G}={\overline{\mathsf{F}}}^{\text{T}}\overline{\mathsf{F}}$ that maps the error in the control to the gradient of the cost functional alias adjoint sensitivity or gradient,² where the component matrix $\overline{\mathsf{F}}$ is the product of 1) the forward sensitivity of the model solution, 2) the Jacobian of the forward operator and 3) the square root of the inverse of the covariance matrix of the observation noise. Refer to Fig. 1 for illustration. It is then shown that by placing observations where the square of the Frobenius norm of $\overline{\mathsf{F}}$ attains a maximum, we can indeed bound the norm of the adjoint gradient away from zero.

View Full Size

A pictorial view of the linear relation g = $\mathsf{G}$f between f = δc, the error in the initial in control c and g = −∇_cJ the negative of the gradient of the cost function J(c), where $\mathsf{G}={\overline{\mathsf{F}}}^{\text{T}}\overline{\mathsf{F}}$. The component $\widehat{\mathbf{g}}$ is the orthogonal projection of g along f. That is, $\widehat{\mathbf{g}}=\widehat{\mathbf{f}}\langle \widehat{\mathbf{f}},\mathbf{g}\rangle ={\mathsf{P}}_{\mathbf{f}}\mathbf{g}$, where $\widehat{\mathbf{f}}$ is the unit vector along f and $\mathsf{P}$_f is the orthogonal projection matrix along f.

Citation: Journal of the Atmospheric Sciences 77, 8; 10.1175/JAS-D-19-0329.1

• Download Figure
• Download figure as PowerPoint slide

In section 2, we provide a short summary of the forecast error correction problem by relating the adjoint sensitivity or adjoint gradient to the innovation or forecast error. By relating this forecast error to the model’s forward sensitivity, we derive our main results in section 3. The intrinsic relation that exists between the adjoint sensitivity and the control error is further analyzed in section 4. The impact of observation placement on controlling the shape of the cost functional is analyzed in section 5. Two examples are considered in section 6: an air–sea interaction problem in time alone, and Saltzman’s (1962) more complicated nonlinear model of convection in space and time. Both examples exhibit the methodology’s avoidance of flat gradients in control space. Section 7 contains recommendations for placement of observations and concluding remarks. Appendix A contains mathematical details that are key to the analysis in sections 3–5. The Gramian structure of $\mathsf{G}$ is developed in appendix B. In the short appendix C, we further illustrate the difference between our approach and a typical adjoint sensitivity analysis in AH2007.

2. Forecast error correction problem

d. Innovation/forecast error

Let z(k) be a single observation given at time k. The difference

$\begin{array}{l}\overline{\mathbf{e}}\left(k\right)=\mathbf{z}\left(k\right)-\mathbf{h}\left[\mathbf{x}\left(k\right)\right]\hfill \end{array}$(2.4)

is called the innovation or the forecast error. Refer to Fig. 2 for an illustration.

View Full Size

$\overline{\mathbf{x}}\left(\mathit{k}\right)$ is the true but unknown state x(k) of the model at time k starting from the true but unknown control $\overline{\mathbf{c}}={\left[{\overline{\mathbf{x}}}^{\text{T}}\left(0\right),{\overline{\mathit{\alpha }}}^{\text{T}}\right]}^{\text{T}}$. x(k) is the model forecast at time k starting from the erroneous initial control $\mathbf{c}={\left[\mathbf{x}\left(0\right),{\mathit{\alpha }}^{\text{T}}\right]}^{\text{T}}\ne \overline{\mathbf{c}}$. $\mathit{\delta }\mathbf{x}\left(k\right)=\overline{\mathbf{x}}\left(k\right)-\mathbf{x}\left(k\right)$ is the first variation in the model state at time k indicated by the first variation δc = [δx^T(0), δα^T]^T, where $\mathit{\delta }\mathbf{x}\left(0\right)=\overline{\mathbf{x}}\left(0\right)-\mathbf{x}\left(0\right)$ and $\mathit{\delta }\mathit{\alpha }=\overline{\mathit{\alpha }}-\mathit{\alpha }$. z(k) is the observation at time k that contains information on $\overline{\mathbf{x}}\left(k\right)$ through the relation $\mathbf{z}\left(\mathit{k}\right)=\mathbf{h}\left[\overline{\mathbf{x}}\left(\mathit{k}\right)\right]$. The forecast error $\begin{array}{l}\overline{\mathbf{e}}\left(k\right)=\mathbf{z}\left(k\right)-\mathbf{h}\left[\mathbf{x}\left(k\right)\right]\hfill \end{array}$.

Citation: Journal of the Atmospheric Sciences 77, 8; 10.1175/JAS-D-19-0329.1

• Download Figure
• Download figure as PowerPoint slide

3. Fine structure of adjoint sensitivity

Instead of determining the adjoint sensitivity through use of the tangent linear model as typically done in the standard 4D-Var mechanics (see Lewis et al. 2006, chapter 24), we derive an alternate expression for δx(k) in terms of the forecast sensitivity to control and the associated incremental change in control.

4. A tale of two vectors: δc and ∇_cJ—Further analysis

An immediate import of the above analysis is the deviation of the simple but key relation in (3.14) where the matrix ${\mathsf{F}}^{\text{T}}\overline{\mathsf{H}}\mathsf{F}$, that maps the control space R^(n+p) of unknown control error vector δc into the space R^(n+p) of the adjoint sensitivity ∇_cJ.

As a prelude to further analysis, let us further simplify the notation by defining the following equivalences:

Consequently, form (3.14) we obtain a fundamental relation

Remark 4.1: An estimate of the Hessian of J(c) at the minimum: It is well known that if $\mathsf{A}$ is an SPD matrix and b is a vector, then ∇$\mathsf{Q}$(x) = $\mathsf{A}$x − b if and only if $\mathsf{Q}$(x) = (1/2)x^T$\mathsf{A}$x − b^Tx + d for some arbitrary constant d (Meyer 2000; Lewis et al. 2006, chapter 10). From (4.1) and (4.2), we get ${\nabla }_{\mathbf{c}}J=-\mathsf{G}\mathbf{f}=-\mathsf{G}\left(\overline{\mathbf{c}}-\mathbf{c}\right)=\mathsf{G}\mathbf{c}-\mathbf{b}$ where $\mathbf{b}=\mathsf{G}\overline{\mathbf{c}}$. Combining these two facts, we can locally approximate J(c) around $\overline{\mathbf{c}}$ by a quadratic functional given by $\overline{J}\left(\mathbf{c}\right)=\left(1/2\right){\mathbf{c}}^{\text{T}}\mathsf{G}\mathbf{c}-{\mathbf{b}}^{\text{T}}\mathbf{c}$. Hence, the Hessian of $\overline{J}\left(\mathbf{c}\right)$ is given by ${\nabla }^{2}J\left(\mathbf{c}\right)\approx {\nabla }^2\overline{J}\left(\mathbf{c}\right)=\mathsf{G}={\mathsf{F}}^{\text{T}}\overline{\mathsf{H}}\mathsf{F}$, which by (3.18) is also the covariance of ∇_cJ. Clearly, this Hessian is independent of the observation and its condition number controls the shape of J(c).

The meaning and the power of this new relation in (4.2) is captured in Fig. 1 from which it is evident that we can express g as the sum two orthogonal components:

where $\widehat{\mathbf{g}}$ is the orthogonal projection of g along f and ${\widehat{\mathbf{g}}}^{\perp }=\mathbf{g}-\widehat{\mathbf{g}}$.

Thus, when we use g routinely in the minimization of cost functional in the 4D-Var method (Lewis et al. 2006, chapters 22–24), it is now clear from Fig. 1 that it is not g, but only its component $\widehat{\mathbf{g}}$ that is responsible for bringing the reduction in the value of the cost functional and hence the forecast error. An important question is: what factors affect the magnitude of $\widehat{\mathbf{g}}$? The 4D-Var framework is silent on this question. Consequently, we now introduce a new measure for the effectiveness of an observation.

Definition 4.1: An observation z(k) ∈ R^m at time k is said to be effective if the magnitude of the orthogonal projection $\Vert \widehat{\mathbf{g}}\Vert$ is large. If $\Vert \widehat{\mathbf{g}}\Vert$ is small, then the observation is said to be ineffective.

It turns out that $\Vert \widehat{\mathbf{g}}\Vert$ can be small due to two mutually exclusive reasons. First, taking the norm of both sides of (4.2) and using (4.3), it is immediate that

Hence, if the magnitude of the elements G_i,j, of the matrix $\mathsf{G}$ are simultaneously small, then $\Vert \mathsf{G}\Vert$ is small and hence $\Vert \widehat{\mathbf{g}}\Vert$ is small.

Stated in other words, if we place the observations in locations where the model is least sensitive, then from (4.4) it follows that the norm ||g|| is small in and around these locations the cost functional has flat patches in the control space R^n+p. Consequently, if $\Vert \widehat{\mathbf{g}}\Vert$ is small, then the gradient method will bring very little improvement to forecast quality.

There is yet another situation where $\Vert \widehat{\mathbf{g}}\Vert$ can be very small. This happens when the control error vector f lies either in the null space⁴ of $\mathsf{G}$ or very close to it. In such cases, g is either orthogonal or nearly orthogonal to f and so $\widehat{\mathbf{g}}$ is very small even if $\Vert \mathbf{g}\Vert$ is large.

It is against this backdrop, we describe a framework in appendix A that directly relates the spectral properties of the matrix $\mathsf{G}$ and its attendant impact on $\widehat{\mathbf{g}}$.

Since f is unknown but a constant vector, from (A.19) we get a basic relation:

From (A.20), we can express

which is the Frobenius norm of $\overline{\mathsf{F}}$.

Combining (4.5) and (4.6), it is now clear that we can maximize $\Vert \widehat{\mathbf{g}}\Vert$ by maximizing the Frobenius norm of the sensitivity matrix $\overline{\mathsf{F}}$ and this is independent of f.

In the following section 5, we systematically explore the conditions for maximizing $\Vert \widehat{\mathbf{g}}\Vert$ using (4.5) and (4.6).

5. Impact of observation placement

In exploring the properties of the projection $\widehat{\mathbf{g}}$ and consequently the shape of the cost functional $\mathsf{J}$(c) through dependence of its gradient on the forward sensitivities, we now move to analysis of expressions in (3.12)–(3.16).

• Case 1: x(0) is varied and α is held constant.

  In this case, δα = 0 and from (3.14)–(3.16) and appendix B,

  $\mathbf{g}=-{\nabla }_{\mathbf{x}\left(0\right)}J=\left[{\mathsf{U}}^{\text{T}}\overline{\mathsf{H}}\mathsf{U}\right]\delta \mathbf{x}\left(0\right)=\left[{\overline{\mathsf{U}}}^{\text{T}}\overline{\mathsf{U}}\right]\delta \mathbf{x}\left(0\right).$(5.1)

  • Case a: m ≥ n and $\text{Rank}\left({\mathsf{U}}^{\text{T}}\overline{\mathsf{H}}\mathsf{U}\right)=n$ (Meyer 2000). Then $\text{NULL}\left({\mathsf{U}}^{\text{T}}\overline{\mathsf{H}}\mathsf{U}\right)=\left\{\mathbf{0}\right\}$.

    From (4.5) and (4.6), it follows that

    $\begin{array}{l}{\displaystyle \frac{\Vert \widehat{\mathbf{g}}\Vert }{\Vert \mathbf{f}\Vert }}\ge {\min }_{y_i^2\ne 0}\left\{y_i^2\right\}{\displaystyle \sum _j{\Vert {\overline{\mathbf{U}}}_{\mathrm{*}\mathit{j}}\Vert }^2{\Vert {\overline{\mathbf{U}}}_{\mathrm{*}\mathit{j}}\Vert }^2}\hfill \end{array},$(5.2)

    where ${\overline{\mathbf{U}}}_{*j}$ is the jth column of $\overline{\mathsf{U}}$ and y = (y₁, y₂, …, y_n)^T ∈ Rⁿ are coordinates of δx(0) in the orthogonal basis formed by the eigenvectors of the Gramian ${\mathsf{U}}^{\text{T}}\overline{\mathsf{H}}\mathsf{U}$. In (5.2), while we do not have any control over the magnitude of $y_i^{\prime }s$, we can maximize the first factor on the right-hand side of (5.2) by placing the observations where the sum of the squares of the norms of the columns of $\overline{\mathsf{U}}$ is a maximum.

  • Case b: m < n and rank $\left({\mathsf{U}}^{\text{T}}\overline{\mathsf{H}}\mathsf{U}\right)=m$. Then $\text{NULL}\left({\mathsf{U}}^{\text{T}}\overline{\mathsf{H}}\mathsf{U}\right)$ has dimension (n × m). For any δc in this null space, g ≡ 0.

  Again from (4.5) and (4.6), it is immediate that

  $\begin{array}{l}{\displaystyle \frac{\Vert \widehat{\mathbf{g}}\Vert }{\Vert \mathbf{f}\Vert }}\ge {\min }_{y_i^2\ne 0}\left\{y_i^2\right\}{\displaystyle \sum _j{\Vert {\mathbf{U}}_{\mathit{*}j}\Vert }^2}\hfill \end{array}.$(5.3)

  The second factor on the right-hand side of (5.3) can be maximized by placing the observations where the sum of the squares of the norms of the first m columns of $\mathsf{U}$ is a maximum.

• Case 2: α is varied and x(0) is held constant.

  In this case δx(0) = 0 and from (3.14)–(3.16), $\mathbf{g}=-{\nabla }_{\mathit{\alpha }}J=\left[{\mathsf{V}}^{\text{T}}\overline{\mathsf{H}}\mathsf{V}\right]\delta \mathit{\alpha }$ where we recall from appendix B that ${\mathsf{V}}^{\text{T}}\overline{\mathsf{H}}\mathsf{V}={\overline{\mathsf{V}}}^{\text{T}}\overline{\mathsf{V}}\in R^{p\times p}$. Again, two cases arise depending on rank $\left({\mathbf{V}}^{\text{T}}\overline{\mathsf{H}}\mathbf{V}\right)=p$ or rank $\left({\mathsf{V}}^{\text{T}}\overline{\mathsf{H}}\mathsf{V}\right)<p$. In the latter case, the null space of ${\mathsf{V}}^{\text{T}}\overline{\mathsf{H}}\mathsf{V}$ has dimension greater than or equal to one and for any δc in this null space, g ≡ 0.

  From (4.5) and (4.6), it can be verified that

  $\begin{array}{l}{\displaystyle \frac{\Vert \widehat{\mathbf{g}}\Vert }{\Vert \mathbf{f}\Vert }}\ge {\min }_{y_i^2\ne 0}\left\{y_i^2\right\}{\displaystyle \sum _j{\Vert {\overline{\mathbf{V}}}_{\mathit{*}j}\Vert }^2}\hfill \end{array}.$(5.4)

  By placing the observations in the time interval where the sum of the squares of the norms of the matrix is a maximum, we can enforce or guarantee a nonzero lower bound on the norm of the adjoint gradient.

• Case 3: Both x(0) and α are varied.

  In this case, from (3.14)–(3.16), we get

  $\begin{array}{l}\mathbf{g}=-{\nabla }_{\mathbf{c}}\mathsf{J}=\left({\mathsf{F}}^{\text{T}}\overline{\mathsf{H}}\mathsf{F}\right)=\left({\mathsf{F}}^{\text{T}}\mathsf{F}\right)\delta \mathbf{c}\hfill \end{array}.$(5.5)
  From (4.5) and (4.6), we get

  $\begin{array}{l}{\displaystyle \frac{\Vert \widehat{\mathbf{g}}\Vert }{\Vert \mathbf{f}\Vert }}\ge {\min }_{y_i^2\ne 0}\left\{y_i^2\right\}\left({\displaystyle \sum _i{d}_i}\right)\hfill \end{array}.$(5.6)

  Hence, by placing observations in the time intervals containing the maximum of the sum of the square of the norms of the columns of ${\overline{\mathsf{F}}}^{\text{T}}$, we bound the norm of $\widehat{\mathbf{g}}$ as far away from zero as is feasible.