1. Introduction

The generalized inversion (GI) is the most flexible tool for combining information on the state of the ocean provided by physical models and observations (Egbert and Bennett 1996; Kivman 1997a). The approach explicitly takes into account uncertainties inherent in both model equations and measurements; this is achieved by the proper weighting of corresponding cost terms. An assumption that the model equations are perfect has been widely used in data assimilation. Though accounting for the model errors allows one to improve accuracy of the prediction (see, e.g., Evensen 1997; Gong et al. 1998), the question of what the model errors are remains an open scientific issue (Courtier 1997). The main goal of this paper is to show that as soon as we combine information from different sources into a unique solution, methods of the information theory are applicable and may be of help in selecting a number of error statistics parameters.

From a probabilistic standpoint on data assimilation (van Leeuwen and Evensen 1996), a system state ψ and measurement results d₁, . . . , d_M are viewed as random quantities from a set Ψ of admissible solutions and a data space D, respectively. Given a prior probability density function (PDF) p(ψ) and the data d = (d₁, . . . , d_M), with errors described by a conditional PDF p(d|ψ), we update our knowledge about ψ using the Bayes theorem

View Expanded

Then, the objective of data assimilation is to seek a maximum probable state ψ_i when d is observed.

The problem is that we know little about the prior p(ψ). As soon as ψ_i becomes heavily dependent on p(ψ), tuning statistical parameters of the prior raises considerable interest. A prerequisite in prescribing the prior is the mathematical expectation or the mean

View Expanded

In oceanography, the amount of data is typically much less than the number of unknown variables, and we must look for additional prior information to construct E(ψ). A useful way is to invoke a physical model for ψ; however, model uncertainties are rarely estimated a priori. Since errors in model equations are never observed directly, their statistical properties could be derived only from the data. To minimize the effects of measurement noise, the number of observations should be at least an order of magnitude larger than the dimension of the system state space, which is rarely the case. To close an inverse problem, we should adopt assumptions about the model uncertainties. Further, physical laws underlying models are usually expressed as differential equations. Initial and/or boundary conditions for them can come only from observations. Thus, even if the probability distribution of the model errors was known, it would not be enough for defining the prior. We have to guess the statistics of ψ at the initial moment and at the boundary of the modeled domain.

Though there have been some attempts to estimate systematic model errors (Dee and da Silva 1998; Griffith and Nichols 1996), in most studies, the errors in the model equations and initial and boundary conditions have been taken to be unbiased and Gaussian distributed. That is, in the linear case, the mean E(ψ) is assumed to be given by the model solution satisfying the initial and boundary data exactly (Bennett 1992). The initial and boundary conditions must be obtained from observations. Thus, the data obtained at the initial time and/or at the boundary of the domain are considered to be equal to the mean of the prior E(ψ), while the data assimilated inside the domain do not possess that exclusive property. It is a rather general situation that there are not enough observations at the intial moment or at the boundary; then, we should introduce bogus data to get a well-posed problem (Bennett and Miller 1991). We work with the two datasets that get different interpretations: the data at the initial time and/or at the boundary of the domain are considered to be equal to the mean E(ψ), while the data assimilated inside the domain are not. Partitioning the data into the two subsets is arbitrary to some extent; thus, essential asymmetry in the interpretation of these two sets is introduced.

Having few observations, we cannot estimate everything and should restrict our attention to tuning a certain number of statistical parameters, such as error variances and error decorrelation scales, while keeping other inputs fixed. Two methods have been widely used in data assimilation community, namely, maximum likelihood (ML; Dee 1995) and general cross-validation (GCV; Egbert et al. 1994). As the statistical theory suggests (Cramer 1954), if the true prior PDF is indeed of an assumed functional shape, then ML is the most powerful tool for estimating parameters of the prior PDF from the data in the sense of minimizing variances of the estimates. However, if the assumptions about p(ψ) are violated, this approach may be less effective than some other tuning criteria (Wahba 1985). Gao (1994) demonstrated that in a particular example, the ML substantially overestimated a parameter of smoothing (error decorrelation length scale) in comparison to the GCV. When even the mean E(ψ) is not known, the ML estimates of the statistical parameters are suspect.

Another approach to building the prior PDF may be based on the principle of maximum entropy (PME). The PME and Bayesian viewpoints complement each other (Jaynes 1988a). Indeed, the former assesses the prior probabilities in the most cautious way in the sense that the PDF obtained assumes nothing beyond what was actually known and used as input. Once the prior PDF is chosen, the Bayes method updates it when new data become available. It is worth emphasizing that probabilities derived from the PME should be considered as“degrees of belief” rather than “relative frequencies” (Jaynes 1988b); it has been precisely the interpretation of probabilities used by Tarantola (1987) in the statistical inverse problems theory. Thacker and Holloway (1990) pointed out that the PME might be of help in choosing the prior in oceanographic inverse problems.

We outline the fundamentals of the PME in section 2. Then, we show that in the linear case, any choice of the prior generates a measure in the space of possible outcomes, which may be viewed as a generalization of the probability measures. Thus, one is able to use the PME to select a certain number of statistical parameters (see section 3 for the theory and section 4 for a numerical example). Any method for tuning the statistical parameters substantially complicates data assimilation and is very difficult to employ in realistic large-dimensional problems. The maximum data cost criterion (MDC), which is easier to implement, is shown to be an approximation to the PME. Though the MDC established on heuristic grounds has worked well (Kivman 1997c; Kurapov and Kivman 1999), its statistical background is clarified only now (see section 5).

The cost functional generally includes certain smoothing terms, which are introduced to provide well-posedness. It was demonstrated by Gong et al. (1998) that they could be as important as the weight controlling the trade-off between the model and the data. We will show in section 6 how to tune a smoothing parameter in the scope of the PME and the MDC.

2. Maximum entropy formalism

What can one conclude about p(ψ|d) having a number of constraints on it? This question is as old as statistical mechanics, and an answer was provided by Boltzmann (1964) and Gibbs (1902). Namely, one should maximize entropy

View Expanded

subject to the constraints imposed. Here, μ(ψ) is the so-called noninformative PDF that expresses the state of“lowest” information about ψ (Tarantola 1987).

The concept of entropy originated about a century ago within the context of a fundamental but specific physical problem; it has recently found widespread application in dynamical systems theory, ergodicity theory, inverse problems, communication theory, numerical analysis, theory of functions, decision theory, etc., that is, to problems staying far from the classical thermodynamics. Thanks to Shannon (1948), all these applications were made possible only after recognizing the fact that entropy has much more wider content than had been previously thought. To be concise, Shannon proved that any measure of uncertainty in a probability distribution is proportional to the Boltzmann–Gibbs entropy if that measure is

(i)ii a continuous function of probabilities;

(ii)i a decreasing function of the number of possible events in the case when they are equally probable;and

(iii) consistent, that is, if there is more than one way of evaluating its value, every possible way yields the same result.

Thus, the PME is in fact a refined version of Bernoulli’s principle of insufficient reason for assigning probabilities to possible events in the light of all prior knowledge (see Thacker and Holloway 1990, and references therein). It should be emphasized that the PME is an extremely conservative procedure in the sense that its application yields a probability distribution that is as uncertain or “spread” as allowed by properties we wish the distribution to have and that possesses a minimal information content among all distributions satisfying the constraints. In other words, the maximum entropy distribution is free from any information additional to that we actually knew and used as input.

3. An entropy approach to selecting priors in weak constraint data assimilation

An additional assumption is that the discrete set (4), (5) is overdetermined. If it is not the case, one should augment the dataset with bogus data at boundary and/or initial points to provide well-posedness of weak constraint data assimilation (Bennett and Miller 1991). Thus, our assumptions are valid in all practical applications. If (4), (5) constitute the overdetermined set, we have to admit that at least a part of information about ψ contains errors. It is natural to assume that the errors inherent in the data and model equations have finite variances σ_d and σ_m, respectively, that is,

View Expanded

Here, the PDF p(ψ|d) is defined with respect to the standard Lebesgue measure Πdψ on Ψ, and ( · )_d and ( · )_r are inner products in the spaces D and R, respectively. They are not necessarily Euclidean inner products but may involve any positively defined matrices C_d and C_m:

View Expanded

The rationale behind (6) and (7) is that at first, our measurement instruments indeed have finite error variance; at second, the desired smoothness of physical fields may be expressed as the boundedness of a norm (usually a Hilbert norm) of uncertainties in (5). The matrices C_d and C_m are usually interpreted as covariances of the data and model errors. As it was mentioned in section 1, there is a lot of uncertainty in the assessment of the model error covariances, at least in the oceanographic context. The choice will always be a hypothesis, which is difficult, if ever possible, to verify a posteriori.

An exercise in the calculus of variations leads to the PDF

pψdCμψa_mAψf²_ra_dLψd²_d(9)

which maximizes (3) subject to (6), (7). Here,

View Expanded

and a_m, a_d are some positive weights appearing as Lagrangian multipliers. It must be emphasized that a_m ≠ (1/2)σ⁻²_m and a_d ≠ (1/2)σ⁻²_d. Together with a normalization constant C, they should be determined from (6), (7) and normalizability

View Expanded

If μ(ψ) is chosen as a constant, that is, we consider a priori all the states ψ as equally probable, the most likely state minimizes a cost function

Jψa_mAψf‖²_ra_dLψd‖²_d(12)

Hence, the PME leads to a Gaussian distribution if A and L are linear. Based on the PME, we obtain the cost functional commonly used in weak constraint data assimilation when information about higher moments is not available. Though, if we had possessed the information about the higher moments, we would have been able to account for it in the framework of the PME by imposing additional constraints on p(ψ|d).

Calculating the weights a_m and a_d from the error variances with (6), (7), (11) is a tough routine (Urban 1996). We will not be able to determine the weights unless both σ_d and σ_m are known. Luckily, we are rarely interested in the whole PDF and would be pleased to obtain a reliable estimate of its maximizer or the mean. To calculate it, we only need the ratio w = a_d/a_m. Surprisingly, if there is no model solution fitting the data exactly (i.e., the model and the data are in contradiction), the PME makes it possible to proceed a bit further.

How the principle works may be illustrated with the example of multiple measurements of a scalar random variable y (Ghill and Malanotte-Rizzoli 1991). If two measurements y₁ and y₂ are available, an estimate ỹ of y may be obtained by minimizing a cost function

Jya₁yy₁²a₂yy₂²(13)

with certain positive weights a_i, i = 1, 2. Then,

ỹα₁y₁α₂y₂(14)

where

View Expanded

Since

α₁α₂(16)

α₁, α₂ can be viewed as probabilities of the realizations y₁ and y₂. Thus, any choice of the weights in (13) generates a probabilistic measure dα with support in the set of the observed states. The mean with respect to this measure coincides with the minimizer (14) of the objective function (13). If we do not know the error variance of either measurement, the PME is of help. Maximizing the entropy

View Expanded

subject to (16) yields α₁ = α₂ and a₁ = a₂. That is, not having enough constraints to maximize the entropy over continuous PDFs, we “quantize” the system state space Ψ and pick up the maximum-entropy singular PDF.

Let us return to the multidimensional case. The information presented in (12) is excessive, and we will try to construct a singular measure over a quantized space, which would consist of the observed system states. Then, the PME would be a guide to choosing the weight ratio. At first, in the case of the linear model and observational operators, an inner product on Ψ may be defined as

ψ₁ψ₂a_mAψ₁Aψ_2ra_dLψ₁Lψ_2d(18)

which allows us to rewrite (9) as

pψdC₁‖ψ‖²_Ψa_mAψf_ra_dLψd_d(19)

with a new normalization constant C₁. The last two terms are bounded functionals in Ψ. According to the Riesz theorem (Yosida 1980), for any f ∈ R and d ∈ D there exist unique vectors ψ_A and ψ_L such that

View Expanded

for all ψ ∈ Ψ. Consequently, we may define operators A∗: R → Ψ and L∗: D → Ψ such that

Afψ_ALdψ_L(21)

Note that these operators are not adjoint to A and L, though they have very close meaning.

Now we need to define the entropy H(M) of the OVM that would meet (i)–(iii) (see section 2). A possible candidate is

View Expanded

where λ_i, 1 − λ_i are eigenvalues of the operators M_d and M_m, respectively. Since we have more freedom in the assessment of the OVM than an ordinary probability measure (PM), conditions (i)–(iii) are not enough to determine the entropy uniquely. However, if we wish to retain additiveness of Shannon’s entropy, a property intensively used in thermodynamics, (25) is the only choice. A rigorous mathematical proof of this fact is out of the scope of the present paper and will be given elsewhere. Thus, (25) can be considered as a natural generalization of (17).

Two properties of H(M) are easy to establish. First, H(M) is maximum when M_m = M_d = 0.5I. Second, if the mean eigenvalue,

View Expanded

is fixed, H(M) is maximum when λ_i = λ for all i. In the latter case, the OVM becomes the PM. Thus, OVMs generally contain more information than PMs. This conclusion is very natural, since the OVM appeared due to our knowledge of the inner structure of random events in the multidimensional space Ψ; by assigning ordinary probabilities to the events observed (ψ_m and ψ_d), we would ignore that structure.

Note that only nonzero and nonunity eigenvalues make a contribution to H(M), and there are not more than

N₊DM_m

such eigenvalues. Here, ker M_m denotes the null space of M_m. As each term under the summation symbol in (25) reaches its upper bound at λ_i = 0.5, it immediately follows that

View Expanded

We emphasize that the operators M_m and M_d depend not only on weights a_m, a_d and the operators A and L, but also on the specific form of the inner products ( · )_r and ( · )_d. Involved in the definitions of the operators A∗ and L∗, they reflect our a priori assumptions about the data and model error covariances, C_d and C_m. In the standard practice of data assimilation, we know the approximate C_d and have few ideas about what C_m looks like. Hence, we might try to maximize H(M) as a guide to selecting the priors. However, dimD ≪ dimR as a rule, and maximizing the entropy (25) will not determine C_m uniquely. In this instance, if we could estimate the relative weight w and the decorrelation length of the model errors, that would be quite a satisfactory result.

4. Case study: Weak constraint data assimilation into a 1D model for tides in Tatarsky Strait

a. Model

To test what the PME can give in practice, we have applied it to a weak constraint formulation of a linear channel model for the M₂ tide in Tatarsky Strait (Fig. 1). If the strait width W is much smaller than a characteristic wavelength and varies smoothly, the cross-channel velocity is negligible. Then, complex amplitudes of surface elevation ζ and alongstrait mass transport q averaged over the strait width are related by the continuity and by alongchannel momentum equations

View Expanded

Here, ω = 1.405 × 10⁻⁴ s⁻¹ is the frequency of the M₂ tide constituent considered, g is the gravity acceleration, h(x) is the mean depth across the strait, and x is the alongstrait coordinate. Set (27), (28) was reduced to a second-order equation for q:

View Expanded

The data d(x) available are complex amplitudes of the surface elevation at certain coastal points (see Fig. 1). They are related to the model variable q via the cross-strait momentum equation, which is reduced to geostrophy under the assumptions adopted (Toulany and Garrett 1984):

View Expanded

Here, f is the Coriolis parameter and ζ₊ and ζ₋ are complex amplitudes of the surface elevation at the west and east coast, respectively. If we assume surface elevation to vary linearly in the cross-strait direction and take approximately

View Expanded

Eqs. (27), (30), and (31) relate q to the data d(x),

View Expanded

where + (−) is held for the west (east) coast.

We assimilate the east coast data, while those located at the west side of the strait are left for verification of the inverse solution. The cost function J(q) is

JqJ_mwJ_d(33)

Here, the first term represents the mean-square misfit in the weak finite-element approximation of (29) (Kurapov and Kivman 1999), the second one refers to the square misfit in the data equation (32), and w is the weight to be tuned. The cost function is minimized by the conjugate gradient.

We would like the generalized inverse of our 1D model to reproduce features essential for weak constraint data assimilation into 2D and 3D models. One of them is ill-posedness of the problem in the continuum limit (see section 6). To allow for it, the dynamical errors are supposed to be uncorrelated. The same are assumed for the data errors. If we penalized the mean-square norm of the residuals referred to the model equations (27), (28), we would obtain ζ, q ∈ W¹₂ in the continuum limit, that is, ζ, q, ζ_X, q_x would belong to the space L₂ of square-integrable functions. Since every function from W¹₂ in the 1D case is continuous (Adams 1975), the ζ data at isolated points would generate observational functionals bounded in the continuum limit; the problem would be well-posed, even if a diagonal dynamical error covariance matrix was used. This circumstance would not allow us to test the ability of the entropy approach to recognize ill-possedness and tell us when smoothing is necessary. Therefore, to introduce ill-posedness, we have used the second-order equation (29) for the tidal transport q. Then, the observations are related to values of q_x at the isolated points via (32). We can guarantee q_x ∈ L₂ only, and the observational functionals are unbounded in the continuum limit.

5. Maximum data cost criterion

To compute the entropy, one needs a calculation of eigenvalues of the operator M_d or M_m, which is not feasible to obtain for large N. We would like to find an approximation to the entropy approach and to avoid explicit calculation of the entropy. It is easy to check the multiple measurements example discussed in section 2 so that the maximum-entropy measure dα yields the weight maximizing the cost a₁(ỹ − y₁)² when a₂ is fixed, and vice versa. Does the same fact hold for operator-valued measures appearing in multidimensional spaces? A relevant theorem is proven in appendix B. The MDC coincides with the PME under the assumption that the shape of the error covariance matrices allows the entropy to attain its upper bound (26) at some w. Fixing C_m and C_d a priori, we cannot guarantee that the upper bound of the entropy will be attained. However, we may still select w in accordance with the MDC criterion, considering it as an approximation to the PME.

Let us return to the case study. The MDC predicts weights w_mdc = 15 and w_mdc = 5 at grids 20 and 38, correspondingly. Though these weights are underestimated in comparison with the maximum-entropy weights w_me = 45 and w_me = 22, the MDC solutions are still of good quality (see Fig. 2). At grid 75, the MDC weight is already an order of magnitude less than the PME weight (w_mdc = 1 against w_me = 10). As we know from above, a sort of smoothing should be applied at this resolution. Further grid refinement leads to splitting the extremum of the data cost curve. Hence, the necessary condition for the MDC to coincide with the PME is violated (see appendix B).

6. Smoothing

What lies behind difficulties rising at the fine resolution? Since the model errors are assumed to be uncorrelated, the Hilbert structure in Ψ, introduced by the inner product (18), does not guarantee smoothness of ψ in the continuum limit. Although the inverse solution is bounded at any discrete grid, it is very sensitive to the data noise and to the spatial resolution if the grid becomes too fine. As an example, if the data are values of model variables at isolated points, the inverse solution has pathological behavior at the data sites with holes and spikes in their vicinity, and it does not converge to a continuous function as numerical lattice tends to zero (Bennett and McIntosh 1982). Kivman (1997b) has demonstrated that ill-posedness is not just a mathematical technicality; refining the grid can in fact degrade the inverse solution compared with that obtained at a coarser numerical lattice. In principle, well-posedness may be restored by reducing the relative data weight with grid refinement. However, this scheme relies on the assumption that the continuous model equations are error free, which is difficult to judge.

Smoothing at the fine resolution allows us to account for the model uncertainties that are not caused by discretization. Several strategies have been put forward to smooth the solution to the generalized inverse. Bennett and McIntosh (1982) used spatially variable dynamical weights with suitable singularities at the data sites. Provost and Salmon (1986) augmented the cost functional by a penalty on higher derivatives of the solution. Dynamical error covariances with nonzero off-diagonal elements were adopted in Egbert et al. (1994). Instead of penalizing the L₂ norm of the model equation residuals, Kivman (1996) imposed a penalty on higher norms.

The theory of partial differential equations says that if the inverse solution satisfies an elliptic equation, it is continuous everywhere except, possibly, at the data sites. Hence, smoothing is necessary only in the data neighborhood, not in the whole computational domain. Based on this fact, we propose to “spread” the data in order to regularize the problem. The procedure consists in the insertion of bogus data of the same value as the original datum in the vicinity of each data point. It is possible to show that the inverse solution converges to a smooth function in the continuum limit if the radius of spreading is fixed.

From the physical standpoint, the decorrelation length of the field of interst is nonzero, and thus the values that the field takes in the vicinity of the data site is close to what is actually observed. Thus, inserting such bogus data around the measurement location within an area of strong correlations is plausible. Furthermore, we have more ideas about the decorrelation length of physical fields than about that of the model errors.

Because the number of observations M is always finite, the norm generated by (18) becomes only a seminorm at some resolution in 2D and 3D applications. That is, 〈ψ, ψ〉 = 0 does not necessarily mean that ψ = 0. Consequently, we will have to close the inverse problem. When we consider an open system exchanging the energy with its surroundings, the number M_o of numerical grid nodes at the open boundary may exceed the amount of the data there, and maximizing (19) becomes an ill-posed problem, even if M > M_o. A standard regularization routine reduces to spreading the data along the open boundary (interpolating between available data). Here, we propose to apply this regularization methodology to the interior observations.

How can we embed a procedure for tuning the smoothing parameter into the MDC? We may try to determine a threshold spatial resolution where the general inverse starts working badly and additional regularization is already necessary. Smoothing makes the inverse solution more tolerant to the inputs. However, the solution must be insensitive to a smoothing parameter. The optimal weight w_mdc estimated by the MDC can play the role of a sensitivity measure of the inverse solution to the spreading length. If the data cost curve changes dramatically after the data are spread to the closest neighbors of the actual data locations, the spreading is not necessary at all.

Does the sensitivity study point to the same threshold resolution as that estimated by the PME (N = 75)? Yes, it does. As Fig. 3 depicts, spreading the data over adjacent nodes results in a dramatic increase of the MDC weight for N = 20. At grid N = 38, spreading the data causes the appearance of the second maximum. For N = 75, the MDC weight does not change if we spread the data just over two neighboring nodes, while further spreading drastically affects the MDC weight. An interesting point is that the optimal spreading length obtained at the two finer grids (N = 75, 149) remains the same in physical units. Thus, the smoothing length scale may be tuned at the coarser grid and used to obtain a solution at the finer resolution.

7. Summary

The principle of maximum entropy (PME) is a rigorous scheme for estimating weights and smoothing parameters involved in weak constraint data assimilation. Though the operator-valued measure generated by the prior seems to be an extravagant mathematical object, measures of that type arise naturally in the quantum probability theory. They appear in there instead of classical probability distributions because of the incomplete observability of quantum systems (Kholevo 1972). We face quite the same situation in data assimilation. The data are not enough to estimate even the mean E(ψ), let alone the full statistics. In this instance, we invoke a hypothesis about the distribution of the model errors, which are unobservable; thus, the data space and the space of the model errors are generally nonisomorphic. Only in the simplest example of multiple measurements of one variable do we have Ψ = D = R; thus the classical probability measure dα can be used.

Application of the PME to weak constraint data assimilation in the 1D tide model has allowed us to obtain the inverse solution of good quality. In addition, the PME has been able to point at the threshold spatial resolution where the generalized inverse already requires smoothing.

The PME is difficult to employ in large-dimensional problems. In this instance, the maximum data cost criterion (MDC) is a feasible alternative to the PME. The MDC was used earlier as a heuristic rule for estimating the relative data weight in weak constraint data assimilation (Kivman 1997c; Kurapov and Kivman 1999). Connection between the MDC and the PME sheds light on reasons for efficiency of the former. Namely, the MDC can be viewed as an approximation to the PME when the curve ψ(w) ⊂ Ψ is close to a straight segment. It is also important that the MDC is computationally simpler than the cross-validation. Moreover, if the data are from different sources (say, those from pressure gauges and current meters; in situ and satellite altimetry measurements), the method can be applied to multiple weights tuning (Kurapov and Kivman 1999).

In the case study, the MDC underestimated the data weight in comparison with the maximum entropy value. Nevertheless, the inverse solution was fairly good. We should admit that the MDC still requires large computational efforts because the data cost value can be determined only after the inverse solution was obtained. Luckily, the weight can be and should be estimated at the coarse spatial resolution where no smoothing is required. Then we transfer it onto a finer grid.

The method also points to the fine grids where smoothing is necessary. We spread the data to provide well posedness. The spreading length obtained by the sensitivity criterion is in agreement with the PME value. It is worth noting that this smoothing parameter corresponds to the decorrelation length of the tidal fields and, consequently, is easier to be evaluated a priori than the decorrelation length of the model errors. In our numerical example, the quality of the inverse solution did not appear to degrade with grid refinement, even if the smoothing routine was not applied. A reason may lie in that the data are of high quality and stay close to each other if compared with the model length scale. To check this hypothesis, we have unrealistically shallowed the channel and observed that smoothing was actually necessary.