• Anderson, J. L., 2007: Exploring the need for localization in ensemble data assimilation using a hierarchical ensemble filter. Physica D, 230, 99111, https://doi.org/10.1016/j.physd.2006.02.011.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bishop, C. H., and E. A. Satterfield, 2013: Hidden error variance theory. Part I: Exposition and analytic model. Mon. Wea. Rev., 141, 14541468, https://doi.org/10.1175/MWR-D-12-00118.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bishop, C. H., E. A. Satterfield, and K. T. Shanley, 2013: Hidden error variance theory. Part II: An instrument that reveals hidden error variance distributions from ensemble forecasts and observations. Mon. Wea. Rev., 141, 14691483, https://doi.org/10.1175/MWR-D-12-00119.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bocquet, M., 2011: Ensemble Kalman filtering without the intrinsic need for inflation. Nonlinear Processes Geophys., 18, 735750, https://doi.org/10.5194/npg-18-735-2011.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Burgers, G., P. Jan van Leeuwen, and G. Evensen, 1998: Analysis scheme in the ensemble Kalman filter. Mon. Wea. Rev., 126, 17191724, https://doi.org/10.1175/1520-0493(1998)126<1719:ASITEK>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Clayton, A. M., A. C. Lorenc, and D. M. Barker, 2013: Operational implementation of a hybrid ensemble/4D-Var global data assimilation system at the Met Office. Quart. J. Roy. Meteor. Soc., 139, 14451461, https://doi.org/10.1002/qj.2054.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Desroziers, G., L. Berre, B. Chapnik, and P. Poli, 2005: Diagnosis of observation, background and analysis-error statistics in observation space. Quart. J. Roy. Meteor. Soc., 131, 33853396, https://doi.org/10.1256/qj.05.108.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Etherton, B. J., and C. H. Bishop, 2004: Resilience of hybrid ensemble/3DVAR analysis schemes to model error and ensemble covariance error. Mon. Wea. Rev., 132, 10651080, https://doi.org/10.1175/1520-0493(2004)132<1065:ROHDAS>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Furrer, R., and T. Bengtsson, 2007: Estimation of high-dimensional prior and posterior covariance matrices in Kalman filter variants. J. Multivar. Anal., 98, 227255, https://doi.org/10.1016/j.jmva.2006.08.003.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gaspari, G., and S. E. Cohn, 1999: Construction of correlation functions in two and three dimensions. Quart. J. Roy. Meteor. Soc., 125, 723757, https://doi.org/10.1002/qj.49712555417.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., and C. Snyder, 2000: A hybrid ensemble Kalman filter–3D variational analysis scheme. Mon. Wea. Rev., 128, 29052919, https://doi.org/10.1175/1520-0493(2000)128<2905:AHEKFV>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., J. S. Whitaker, and C. Snyder, 2001: Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter. Mon. Wea. Rev., 129, 27762790, https://doi.org/10.1175/1520-0493(2001)129<2776:DDFOBE>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hodyss, D., W. F. Campbell, and J. S. Whitaker, 2016: Observation-dependent posterior inflation for the ensemble Kalman filter. Mon. Wea. Rev., 144, 26672684, https://doi.org/10.1175/MWR-D-15-0329.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hollingsworth, A., and P. Lönnberg, 1986: The statistical structure of short-range forecast errors as determined from radiosonde data. Part I: The wind field. Tellus, 38A, 111136, https://doi.org/10.1111/j.1600-0870.1986.tb00460.x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kuhl, D. D., T. E. Rosmond, C. H. Bishop, J. McLay, and N. L. Baker, 2013: Comparison of hybrid ensemble/4DVar and 4DVar within the NAVDAS-AR data assimilation framework. Mon. Wea. Rev., 141, 27402758, https://doi.org/10.1175/MWR-D-12-00182.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lorenc, A. C., 2003: The potential of the ensemble Kalman filter for NWP—A comparison with 4D-Var. Quart. J. Roy. Meteor. Soc., 129, 31833203, https://doi.org/10.1256/qj.02.132.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lorenz, E. N., 1996: Predictability: A problem solved. Proc. Seminar on Predictability, Shinfield Park, Reading, United Kingdom, ECMWF, 18 pp.

  • Lorenz, E. N., 2005: Designing chaotic models. J. Atmos. Sci., 62, 15741587, https://doi.org/10.1175/JAS3430.1.

  • Ménétrier, B., and T. Auligné, 2015: Optimized localization and hybridization to filter ensemble-based covariances. Mon. Wea. Rev., 143, 39313947, https://doi.org/10.1175/MWR-D-15-0057.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ménétrier, B., T. Montmerle, Y. Michel, and L. Berre, 2015: Linear filtering of sample covariances for ensemble-based data assimilation. Part I: Optimality criteria and application to variance filtering and covariance localization. Mon. Wea. Rev., 143, 16221643, https://doi.org/10.1175/MWR-D-14-00157.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sacher, W., and P. Bartello, 2008: Sampling errors in ensemble Kalman filtering. Part I: Theory. Mon. Wea. Rev., 136, 30353049, https://doi.org/10.1175/2007MWR2323.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wang, X., D. Barker, C. Snyder, and T. M. Hamill, 2008a: A hybrid ETKF–3DVAR data assimilation scheme for the WRF Model. Part I: Observing system simulation experiment. Mon. Wea. Rev., 136, 51165131, https://doi.org/10.1175/2008MWR2444.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wang, X., D. Barker, C. Snyder, and T. M. Hamill, 2008b: A hybrid ETKF–3DVAR data assimilation scheme for the WRF Model. Part II: Real observation experiments. Mon. Wea. Rev., 136, 51325147, https://doi.org/10.1175/2008MWR2445.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wang, X., D. Parrish, D. Kleist, and J. Whitaker, 2013: GSI 3DVar-based ensemble–variational hybrid data assimilation for NCEP Global Forecast System: Single-resolution experiments. Mon. Wea. Rev., 141, 40984117, https://doi.org/10.1175/MWR-D-12-00141.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Yaremchuk, M., D. Nechaev, and C. Pan, 2011: A hybrid background error covariance model for assimilating glider data into a coastal ocean model. Mon. Wea. Rev., 139, 18791890, https://doi.org/10.1175/2011MWR3510.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • View in gallery
    Fig. 1.

    The (left) prior and (right) posterior with ensemble variance values of (a),(b) 0.1, (c),(d) 1, and (e),(f) 2. The black line in the right panels shows the state estimate from a Kalman filter that makes use of the hybrid variance. All experiments assume , , αt = 3, and βt = 2.

  • View in gallery
    Fig. 2.

    True variance binned as a function of sample variance (red line) and linear regression of the true variance onto the sample variance (blue line) for cases in which the true variance is drawn from (a) an inverse-gamma distribution and (b) a uniform distribution.

  • View in gallery
    Fig. 3.

    As in Fig. 1, but for the case in which the true variances are drawn by a uniform distribution defined by , where a = 0.1, b = 5.8, and u is a random variable drawn from the uniform distribution on 0–1. Also shown in the bottom panel is the linear regression (dashed black line) expected true variance as a function of the sample variance (red line), which is now a curve.

  • View in gallery
    Fig. 4.

    The ratio of ensemble variance to error variance shown as a function of weight given to the ensemble variance in the hybrid system. Shown are the prior ratio (dashed blue line), inflated prior ratio (solid blue line), hybridized ratio (red dashed line), and posterior ratio (solid cyan line) for (top) Ne = 5 and (bottom) Ne = 10. Lines indicate the mean of iterations 4–9, and error bars show the 95% confidence interval. All experiments use an entirely ensemble based static.

  • View in gallery
    Fig. 5.

    Binned spread–skill diagrams for the (left) inflated prior ensemble and (right) hybridized ensemble for (a),(d) Ne = 5 member ensemble with linear regression-based hybrid, (b),(e) Ne = 5 member ensemble with cubic regression-based hybrid, and (c),(f) Ne = 10 member ensemble with linear regression–based hybrid. The perfect 1-to-1 line is shown in red. The blue dots show the bin mean variances, and the dashed blue line shows the regression fit.

  • View in gallery
    Fig. 6.

    The ratio of ensemble variance to error variance by iteration. Shown are the prior ratio (dashed blue line), inflated prior ratio (solid blue line), and hybridized ratio (red dashed line) for the (top) Ne = 5 linear regression–based hybrid, (middle) Ne = 5 cubic regression–based hybrid, and (bottom) Ne = 10 linear regression–based hybrid. All experiments use an entirely ensemble based static.

  • View in gallery
    Fig. 7.

    RMSE as a function of ensemble weighting for (top) Ne = 5 and (bottom) Ne = 10. Also shown are the RMSE based on a linear regression (magenta) and cubic regression (red; shown only for Ne = 5). Lines indicate the mean of iterations 4–9, and error bars show the 95% confidence interval. All experiments use an entirely ensemble based static.

  • View in gallery
    Fig. 8.

    RMSE as a function of ensemble weighting for Ne = 5. Shown for observation error variance (top) decreased by a factor of 100 and (bottom) increased by a factor of 100. Also shown are the RMSE based on a linear regression (magenta) and cubic regression (red). Lines indicate the mean of iterations 5–10, and error bars show the 95% confidence interval. All experiments use an entirely ensemble based static.

  • View in gallery
    Fig. 9.

    As in Fig. 8, but for an observation density of (top) 80% and (bottom) 50%. In both cases, the middle grid points are observed.

  • View in gallery
    Fig. 10.

    As in Fig. 8, but for (top) changing the forcing in the nature run from f = 8 to f = 10 and (bottom) including state-dependent model error (in this case Gaussian random noise with variance q = 0.05 is added when ensemble states are greater than a threshold of 7).

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 264 129 9
PDF Downloads 205 101 13

Observation-Informed Generalized Hybrid Error Covariance Models

Elizabeth A. SatterfieldNaval Research Laboratory, Monterey, California

Search for other papers by Elizabeth A. Satterfield in
Current site
Google Scholar
PubMed
Close
,
Daniel HodyssNaval Research Laboratory, Monterey, California

Search for other papers by Daniel Hodyss in
Current site
Google Scholar
PubMed
Close
,
David D. KuhlNaval Research Laboratory, Washington, D.C.

Search for other papers by David D. Kuhl in
Current site
Google Scholar
PubMed
Close
, and
Craig H. BishopNaval Research Laboratory, Monterey, California

Search for other papers by Craig H. Bishop in
Current site
Google Scholar
PubMed
Close
Full access

Abstract

Because of imperfections in ensemble data assimilation schemes, one cannot assume that the ensemble-derived covariance matrix is equal to the true error covariance matrix. Here, we describe a simple and intuitively compelling method to fit calibration functions of the ensemble sample variance to the mean of the distribution of true error variances, given an ensemble estimate. We demonstrate that the use of such calibration functions is consistent with theory showing that, when sampling error in the prior variance estimate is considered, the gain that minimizes the posterior error variance uses the expected true prior variance, given an ensemble sample variance. Once the calibration function has been fitted, it can be combined with ensemble-based and climatologically based error correlation information to obtain a generalized hybrid error covariance model. When the calibration function is chosen to be a linear function of the ensemble variance, the generalized hybrid error covariance model is the widely used linear hybrid consisting of a weighted sum of a climatological and an ensemble-based forecast error covariance matrix. However, when the calibration function is chosen to be, say, a cubic function of the ensemble sample variance, the generalized hybrid error covariance model is a nonlinear function of the ensemble estimate. We consider idealized univariate data assimilation and multivariate cycling ensemble data assimilation to demonstrate that the generalized hybrid error covariance model closely approximates the optimal weights found through computationally expensive tuning in the linear case and, in the nonlinear case, outperforms any plausible linear model.

For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Elizabeth Satterfield, elizabeth.satterfield@nrlmry.navy.mil

Abstract

Because of imperfections in ensemble data assimilation schemes, one cannot assume that the ensemble-derived covariance matrix is equal to the true error covariance matrix. Here, we describe a simple and intuitively compelling method to fit calibration functions of the ensemble sample variance to the mean of the distribution of true error variances, given an ensemble estimate. We demonstrate that the use of such calibration functions is consistent with theory showing that, when sampling error in the prior variance estimate is considered, the gain that minimizes the posterior error variance uses the expected true prior variance, given an ensemble sample variance. Once the calibration function has been fitted, it can be combined with ensemble-based and climatologically based error correlation information to obtain a generalized hybrid error covariance model. When the calibration function is chosen to be a linear function of the ensemble variance, the generalized hybrid error covariance model is the widely used linear hybrid consisting of a weighted sum of a climatological and an ensemble-based forecast error covariance matrix. However, when the calibration function is chosen to be, say, a cubic function of the ensemble sample variance, the generalized hybrid error covariance model is a nonlinear function of the ensemble estimate. We consider idealized univariate data assimilation and multivariate cycling ensemble data assimilation to demonstrate that the generalized hybrid error covariance model closely approximates the optimal weights found through computationally expensive tuning in the linear case and, in the nonlinear case, outperforms any plausible linear model.

For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Elizabeth Satterfield, elizabeth.satterfield@nrlmry.navy.mil

1. Introduction

In a chaotic system like the atmosphere, the true error covariance of short-term forecasts is highly flow dependent. Limited ensemble size and imperfections in methods used for initial condition and model error perturbations mean that ensemble covariances are inevitably inaccurate. Hence, there is a distribution of true covariances, given an imperfect ensemble covariance. The imperfections of ensemble-based estimates of the forecast error covariance are well recognized. Researchers such as Anderson (2007), Furrer and Bengtsson (2007), Sacher and Bartello (2008), Bocquet (2011), and Ménétrier et al. (2015) have suggested methods to ameliorate the effect of these imperfections on the analysis correction and/or the covariance of the prior ensemble produced by ensemble data assimilation schemes. Bishop and Satterfield (2013) and Bishop et al. (2013) showed that if 1) the climatological probability density function (PDF) of true error variances takes the form of an inverse-gamma PDF and 2) the PDF of ensemble variances, given a true error variance, is given by a gamma PDF, then the PDF of true error variances, given an ensemble variance, is also an inverse-gamma PDF. In this case, the mean or minimum error variance estimate of the true error variance is a weighted linear combination of the mean climatological error variance and an imperfect ensemble variance. In other words, it is a linear hybrid.

Many practical tests (Etherton and Bishop 2004; Wang et al. 2008a,b, 2013; Yaremchuk et al. 2011; Clayton et al. 2013; Kuhl et al. 2013; Ménétrier and Auligné 2015) suggest that hybrid error covariance models (Hamill and Snyder 2000; Lorenc 2003) yield superior data assimilation performance to either just localized ensemble covariances (Hamill et al. 2001) or quasi-static climatological error covariance models. Ménétrier and Auligné (2015) outline a variational approach to optimize localization and hybridization simultaneously and further show that localized–hybridized covariances are more accurate than their localized-only counterparts, whatever the static covariance matrix specified. Their method was based on considering sampling error only and adjusting parameters to minimize the difference with the covariance matrix obtained in the limit of a large ensemble size.

A primary aim of this paper is to investigate the extent to which optimal hybridization parameters can be derived from an archive of (observation minus forecast, ensemble variance) pairs generated by a single long run of a hybrid data assimilation scheme. Here, we derive a simple and intuitively compelling formula to obtain the mean of this distribution of true error variances, given an ensemble variance. Such an approach is attractive because of its simplicity and flexibility, benefits from the use of observational information, and has the potential to greatly reduce the amount of time required to tune hybrid error covariance models.

The rest of this paper is organized as follows. Section 2 provides the detailed Bayesian development required to rigorously understand how hybrid covariance models relate to data assimilation. The application and verification of this theory in two univariate model examples are presented in section 3. The development and application of a method applicable to numerical models are presented with a simple model case using the “Lorenz ’96” model in section 4. The conclusions are presented in section 5.

2. Theory

The goal of ensemble data assimilation is to obtain a posterior distribution for the state variable x, given the ensemble-based estimate of the true prior variance and the observation y of the state x. To this end, we will first define the Bayesian formulation for this problem. Next, we will expand on the work of Bishop and Satterfield (2013) to define the prior variance conditioned on the ensemble variance and show that the mean of that distribution results in a hybrid covariance model. Last, we show the relationship between this result and the optimal Kalman gain.

a. Bayesian formulation

We would like to perform data assimilation in the situation in which the sample estimate of the prior variance is known but the true prior variance is unknown. To this end, we formulate the problem using the language of Bayesian statistics. Using the chain rule of probability, we write the joint PDF governing the relevant variables in two different ways:
e1
We emphasize that the densities in (1) use the standard notation that the conditioning on prior observations from previous cycles is suppressed.
In solving for in (1), we obtain a formula for the posterior distribution for the state variable x, namely,
e2
Because we do not know the true prior variance , we must marginalize with respect to that random variable to make use of this theory. In other words, what we really need is . To this end, we note that
e3
which simplifies to
e4
Note further that another application of the chain rule for the prior reveals that
e5
By using (5) in (4), we obtain
e6
where the prior is , the new observation likelihood is , and the sought-after posterior is .

b. Application to sampling error

To proceed, we assume that there is no model error and that the only issue with the ensemble variance estimate is sampling statistics. In this case, the observation likelihood in (6) simplifies to
e7
By contrast, note that, when there is model error (such as representation error) and the state variable x is no longer a random draw from the true distribution, the simplification in (7) is not generally valid. In any event, when (7) holds, then (6) simplifies to
e8
which shows that the impact on the state variable from the sampling error only comes in through the prior. In this section, we restrict attention to the case in which the true prior for the state variable is Gaussian:
e9
where is the true prior mean. As discussed in many places (e.g., Hodyss et al. 2016), when the true prior for the state variable is Gaussian and is given by (8), sampling statistics have the property that the sample variance may be written as
e10
where a is a random variable that is with Ne − 1 degrees of freedom and Ne is the sample size.
Given that the ensemble provides an imperfect estimate of the true prior error variance, we would like to know the distribution of true variances, given an ensemble estimate . As discussed by Bishop and Satterfield (2013), when is a (gamma) PDF and the climatological PDF of true error variances is an inverse-gamma PDF (with shape and scale parameters αt and βt), this implies that is also inverse gamma distributed. Under these assumptions, we have
e11
where Γ is the gamma function and α and β are the shape and scale parameters:
e12
e13
Consistent with Bishop and Satterfield (2013), this implies that the mean true prior variance conditioned on the ensemble variance is
e14
where
e15
e16
e17
Hence, the mean of the true prior variance conditioned on the ensemble variance is a linear function of the ensemble variance under the assumptions of this section. Therefore, an estimate of the true variance is a “hybrid” that linearly weights the ensemble variance and the climatological variance. We note that our choice of an inverse gamma pdf for the climatological distribution of true error variances and a likelihood results in a linear hybrid form consistent with (14) for the optimal error variance estimate. We note that Bocquet (2011) presents a very similar idea. In Bocquet (2011), accounting for sampling error is achieved by computing the prior PDF conditioned on the ensemble. This is consistent with (2), where we then marginalize with respect to the unknown true prior distribution of variances. In our case, instead of solving variationally, we show (see the appendix) that when sampling error is considered, the minimum posterior variance is achieved when the Kalman gain uses the mean of that conditional distribution (which is a known linear function in the gamma–inverse-gamma case). In what follows, we explore the relationship of mean true prior variances (14) to regression analysis and to data assimilation.

c. Relationship to regression analysis

It is well known that the mean of a conditional density [i.e., (14)] can be estimated using the techniques of regression. To that end, note that a linear regression model for the mean true prior variance, given an ensemble variance, would take the following form:
e18
where
e19
e20
e21
with cov(x, y) = 〈(x − 〈x〉)(y − 〈y〉)〉 and var(x) = 〈(x − 〈x〉)2〉. If we use (10), (19), (20), and (21) along with the known properties of the inverse-gamma distribution, we may show that we = G and wc = 1 − G, which proves that the hybrid variance model in (14) is consistent with linear regression under the assumptions of section 2a. We demonstrate below that, when the climatological distribution is no longer inverse gamma, this relationship between G and we is broken.
In practice, we do not have available samples of to use in a regression; therefore, we use squared innovations because it can be shown that
e22
where the innovation is written as . The relationship defined in (22) can be derived by noting that y = x + εo and , where εo is a random variable with mean 0 and variance and ζ is a random variable with mean 0 and variance 1. Using these definitions for y and x in the innovation d allows the proof for (22) to be completed.

d. Relationship to data assimilation

For applications to data assimilation, what we really want to know is how the prior state x relates to the ensemble variance rather than the true variance. Note that in general this new prior is determined from
e23
For the special case of an inverse-gamma climatological PDF of true error variances and a χ2 PDF of ensemble variances, given a true error variance, one can show that the structure of this prior is a kind of t distribution of the form
e24
The t distribution in (24) is known to have fatter tails than a Gaussian and therefore shows that the uncertainty in the state x for small ensemble size is a symmetric, but non-Gaussian, distribution with fat tails.
A Kalman filter state estimate must make use of the variance of the prior. Again, we may calculate the variance of (24) to obtain
e25
Because (25) is equal to (14), we know that the prior variance about the true prior mean is also a “hybrid” that weights the ensemble prior variance and the climatological variance to find an estimate of the true prior variance. While we have now explicitly shown that the variance of the prior state x is equal to the mean of the distribution of true prior variances, given a sample variance [e.g., (14)] for the specific case of sampling error for a Gaussian true prior [e.g., (9)], it is possible to show that this result holds for all priors by simply using (23) in (25) directly. Hence, the prior variance of a state variable x when conditioned on the ensemble variance is always determined by the mean of the distribution of true variances, given a sample variance .
To formally relate this new form of prior to the Kalman filter state estimate, we need to minimize the average squared errors for the state variable x, and we emphasize that we do this under the assumptions of section 2b. To this end, we note that the average squared error in a Kalman state estimate is
e26
where
e27
We perform the standard operations to derive the Kalman gain Ge that minimizes (26). Please see the appendix for the derivation. Briefly, by taking the derivative of (26) with respect to Ge, setting the result to zero, and subsequently solving for Ge, we obtain
e28
Hence, we have now proven that the gain that minimizes the posterior variance uses the expected true prior variance, given an ensemble variance, as in (14). We note that this proof can be easily extended in a couple of ways. First, if we continue to make the assumption that the state estimate (27) makes use of the true prior mean, then one obtains (28) even for non-Gaussian true priors [i.e., replace (9) with a non-Gaussian PDF]. Second, if we continue with the Gaussian assumption in (9) but replace the true prior mean with the sample prior mean in (27), we obtain a gain of the same form as (28) but for which the expected true prior variance, given an ensemble variance , is simply inflated by a factor 1 + 1/Ne to account for the additional uncertainty from sampling error in the prior mean. However, because it can be shown that the sampling error in the prior mean and the sampling error in the prior variance are correlated for non-Gaussian (skewed) priors, extending the proof to non-Gaussian true priors (9), while also including sampling error in the prior mean in (27), is nontrivial and left to future work.

To illustrate the behavior of this new form of Kalman state estimate, we provide an example of data assimilation for the model of section 2b. We use a single scalar assimilation of a generic problem, where the typical assumptions are satisfied. The observations will be taken to have been drawn from a Gaussian observation likelihood with an observation error variance equal to 1. We plot (8) as a function of the observation y for three different values of sample variance in Fig. 1. The parameters are set as , αt = 3, βt = 2, and Ne = 5. These values imply that the mean of the climatological PDF of prior error variance is . Using these parameters, we plot the prior PDF in Fig. 1 for equal to 0.1, 1, and 2, respectively. Along with the Bayesian posterior given by (8), in Fig. 1 we also plot (black line) the state estimate from a Kalman filter that makes use of the hybrid variance estimator in (14) and, equivalently, (18). This shows that the use of the hybrid variance estimator correctly aligns the Kalman state estimate along the true posterior mean in the high-probability region of the posterior PDF. Note, however, that for large values of the innovation (or the observation), the curvature of the posterior mean leads to large errors in the Kalman state estimate. At these extreme values of the innovation, the Kalman state estimate is too close to the prior mean and not close enough to the observed value: it does not draw the analysis close enough to the observation. This is a direct consequence of the fact that the Kalman filter mistakenly assumes that the forecast error variance is precisely known; it is not, and this uncertainty implies that the chances of the truth lying many standard deviations from the prior mean is increased.

Fig. 1.
Fig. 1.

The (left) prior and (right) posterior with ensemble variance values of (a),(b) 0.1, (c),(d) 1, and (e),(f) 2. The black line in the right panels shows the state estimate from a Kalman filter that makes use of the hybrid variance. All experiments assume , , αt = 3, and βt = 2.

Citation: Monthly Weather Review 146, 11; 10.1175/MWR-D-18-0016.1

3. Univariate model experiments

a. Inverse gamma

We may numerically verify the theory of section 2 for the case in which the climatological distribution of true variances is an inverse-gamma PDF and the PDF of ensemble variance given a true variance is a gamma PDF by letting the true state be a result of the stochastic process given by
e29
where εf is a draw from a normal Gaussian PDF with mean 0 and variance , with prior mean equal to 1. The observations will be taken to have a Gaussian observation likelihood with an observation error variance equal to 1. We employ 107 trials. For each trial of this experiment, a sample truth will be created using (29) and random noise will be added to create an observation. The true forecast error variances will be drawn from an inverse-gamma PDF with αt = 3, βt = 2, and Ne = 5. The theory above [(15) and (16)] states that for this problem setup the weights are equal (i.e., we = wc = 0.5). We continue with the assumptions of this section and use the true prior mean; we will illustrate the modifications required when using the sample prior mean in the next section when using the Lorenz ’96 model. Note that the use of the true prior mean here implies that prior inflation is not required because both the sample variance and the innovations are unbiased. To determine the linear hybrid regression model, we employ (18) by drawing ensembles from (29), calculating the sample variance, and regressing the sample variance against the true variance as discussed in section 2c. Of course, in a realistic application, we will not have available the true variance to regress; therefore, in the next section we discuss how to perform this regression using innovations. For now, however, we continue with the framework of section 2c to clearly verify the theory. The result of this regression finds G = 0.501, which verifies that regression finds the appropriate weights. Figure 2a plots this regression line as well as the true variances binned as a function of sample variances. The similarity of these two lines verifies (14) because it shows that the mean really is a straight line with the slope and intercept determined by we and wc, respectively.
Fig. 2.
Fig. 2.

True variance binned as a function of sample variance (red line) and linear regression of the true variance onto the sample variance (blue line) for cases in which the true variance is drawn from (a) an inverse-gamma distribution and (b) a uniform distribution.

Citation: Monthly Weather Review 146, 11; 10.1175/MWR-D-18-0016.1

For each trial of this experiment, we calculate three state estimates and measure their quality in terms of posterior mean square error (MSE). The first state estimate is to simply use the sample variance in the Kalman state estimate (i.e., the traditional ensemble Kalman filter) using we = 1 and wc = 0, which obtains a posterior MSE of 0.50. The second state estimate uses (27) with the hybrid variance model determined from (18), which obtains a posterior MSE of 0.472. The last state estimate we examine is determined by “brute force” tuning the weights of the hybrid covariance model to find the configuration with the smallest posterior MSE. Employing this procedure leads to we = 0.497 and wc = 0.503 with a posterior MSE of 0.472, in agreement with the linear hybrid variance model. While the improvement over simply using the sample variance is small, it is nonetheless clear that the linear hybrid model is superior and that the weights determined from regression are in fact the ones that minimize the posterior MSE.

b. Uniform

The goal of this section is to show what happens when the climatological distribution of true variances differs from an inverse-gamma distribution. We make use of the same framework as section 2d. The only change we make is to create the random variable for the true variances using the following:
e30
where a = 0.1, b = 5.8, and u is a random variable drawn from the uniform distribution on 0 to 1. We illustrate the structure of the prior and posterior distributions in this case in Fig. 3. Also, we again determine the linear hybrid regression model by drawing ensembles from (29), calculating the sample variance, and regressing the sample variance against the true variance. The linear regression line is plotted in Fig. 2b. Also shown in Fig. 2b is the expected true variance as a function of the sample variance (red line), which is now a curve rather than a straight line, as in the previous section. This means that the linear estimate of the expected true variance by the linear hybrid is, in this case, incorrect.
Fig. 3.
Fig. 3.

As in Fig. 1, but for the case in which the true variances are drawn by a uniform distribution defined by , where a = 0.1, b = 5.8, and u is a random variable drawn from the uniform distribution on 0–1. Also shown in the bottom panel is the linear regression (dashed black line) expected true variance as a function of the sample variance (red line), which is now a curve.

Citation: Monthly Weather Review 146, 11; 10.1175/MWR-D-18-0016.1

To further illustrate this potential problem in the estimation, we again report the posterior MSE for three different experiments. For the first experiment, we simply use the sample variances that we obtain from an ensemble using we = 1 and wc = 0, which obtains a posterior MSE of 0.796. Second, we employ regression to determine the weights for the hybrid variance model and find G = 0.322, which when used as a hybrid variance model in a Kalman filter obtains a posterior MSE of 0.734. Last, we again employ brute-force tuning of the weights of the hybrid covariance model to find the configuration with the smallest posterior MSE. Employing this procedure leads to we = 0.653 and wc = 0.437 with a posterior MSE of 0.729. Last, we could employ (27) with gain (28), for which we define from the red curve in Fig. 2b. This configuration obtains a posterior MSE of 0.722 and, consistent with the theory of section 2d, provides the state estimate with the smallest posterior MSE.

Note that if regression had determined the correct weight on the ensemble variance then we would find that G was equal to the we obtained from brute-force tuning experiments. This is not the case for the climatological distribution of variances given by (30). Experiments (not shown here) with a variety of climatological distributions whose tails were narrower than the inverse-gamma distribution all found that G < we. Hence, we believe that this behavior is general and therefore that replacing the traditional linear hybrid variance model that cannot possibly account for this curvature with a generalized hybrid variance model that better accounts for this curvature will result in superior data assimilation performance.

4. Lorenz ’96 model experiments

In this section, we explore the case in which, as in a real-world application, we do not know the climatological distribution of true error variances. To simplify things, we can control the observation error variance , observation density, and influence of model error. We will show that the we value required to minimize the analysis error variance in a hybrid data assimilation system is underestimated by linear regression and that the true prior variance can be estimated as a higher-order polynomial function of the ensemble variance, consistent with section 2. We assess the optimal weighting of flow-dependent and static covariances using an implementation of the perturbed observations form of the ensemble Kalman filter on a 10-variable version of the Lorenz ’96 model [Lorenz (1996); also see model 1 of Lorenz (2005)]. In what follows, we provide a brief description of this analysis/forecast system.

a. Description of model and data assimilation

We utilize the Lorenz model with a forcing term (F = 8). We use a time step of Δt = 0.05 and the fourth-order Runge–Kutta time integration scheme, while allowing for model spinup by disregarding the initial 240 time steps. We define the true state as a draw from this previously generated nature run. We note that the assimilation experiments are performed using the same setting as in the nature run. We calculate the background-error covariance from the ensemble mean as follows:
e31
where b denotes an M × Ne matrix of ensemble perturbations,
e32
where M is the model dimension and Ne is the number of ensembles. Observations are created by perturbing the “truth” with observation error drawn from a zero-mean Gaussian distribution with variance :
e33
For the following experiments, we observe every grid point at each time step.
We assimilate observations using a stochastic update approach (Burgers et al. 1998) as follows:
e34
where denotes the observation operator (which in this case is simply the identity matrix). We define
e35
We update the analysis ensemble following
e36
where refers to the observation , defined in (33), perturbed with a random observation error. The analysis is defined as the ensemble mean:
e37

The ensemble forecasts are cycled to create the background-error covariance matrix for the next time step. Note that, because the covariance matrix of the xbi is not equal to the hybrid covariance model, the resulting ensemble xai does not actually satisfy the posterior covariance equation in (34). To maintain consistency with standard ensemble generation methods, we nevertheless will use (36) and leave the derivation of an ensemble generation method that eliminates this issue to future work.

b. Localization

To avoid spurious correlations at longer range, we localize the ensemble-based covariance matrix by a Schur product with a correlation function as defined in Gaspari and Cohn (1999):
eq1
where z represents the Euclidean distance between grid points in physical space and c is defined such that the correlation reduces from 1.0 to 0 at a distance of 2c. For the experiments that follow, we define c = 3 on the basis of coarse tuning for the five-member ensemble case. All tuning experiments were integrated for Nt = 100 000 time steps and used a 100%-ensemble-based error covariance matrix. The ensemble variance was coarsely tuned to match the mean squared error in the background forecast. We note that for the Ne = 10 ensemble case the minimum RMSE was found when no localization was used. However, we chose to retain the c = 3 for consistency and to prevent divergence for the 100%-ensemble case.

c. Experimental design and results

We define the hybrid background-error covariance matrix following
e38
where the correlation matrices are defined following
e39
and where the matrices denote the diagonal matrices of variances. The diagonal matrix ens can be interpreted as the weight given to the flow-dependent covariances. When ens is defined by a linear function of the ensemble variances (e.g., ens = weens), (38) can be written as a traditional linear hybrid, namely,
e40
We can extend this notation to the case in which the relationship between the true error variance and the sample variance is modeled by some nonlinear function, such as a higher-degree polynomial (as in Fig. 2b) , where each element of is the nth power of the corresponding element in the diagonal matrix of variances .

For the ideal linear case, the climatological average of the ensemble variance and the climatological error variance are both approximately equal to the climatological average mean squared error . In such a case, we expect that wc = 1 − we, since in this case the coefficients are simply acting to weight the flow-dependent and static components rather than weight and also inflate the variance to account for unrepresented uncertainty.

In what follows, the static component of the background-error covariance matrix is defined by collecting forecast errors from a 100%-ensemble-based system for M = 10 and taking the expectation (denoted 〈.〉) over Nt = 100 000 time steps:
e41
e42
Our experiments proceed as follows:
  1. To start our experiments, we first run a fully ensemble-based (α = 1) experiment, which is cycled for 100 000 time steps (after disregarding the first 100 time steps to allow for spinup). This experiment also serves to generate a static . This experiment uses a standard prior inflation (of ρ = 1.2).

  2. We iterate this 100%-ensemble-based experiment 10 times, each time adjusting the prior ensemble variance inflation as follows. The prior ensemble variance is inflated, multiplicatively, following . For each of the 10 iterations associated, we also iterate on prior multiplicative inflation values:
    e43
    where is the ensemble variance, given by
    e44
    We then adjust the inflation factor for the next iteration following
    e45
    To prevent the inflation factor from adjusting too rapidly, we cap the overall change in inflation parameters at 10%. Since the error statistics are sensitive to ensemble inflation, we write out a new on each iteration, such that the used for the hybridized experiments is obtained on the 10th iteration. Although the static error covariance matrix will generally be overvariant for hybridized cases, this is a more realistic model, given that NWP centers usually do not update their static covariance models when a hybrid system is used. We also acknowledge that the optimal linear combination may be one in which we + wc ≠ 1; however, for the sake of computational expense and modeling realistic tuning, we simply search for the we value that gives the minimum posterior RMSE constrained to the one-dimensional search space by the condition we + wc = 1.
  3. We then perform standard hybrid integrations following (40) for each value of we decreasing from 0.9 to 0 at 0.1 increments. This gives us 10 experiments for hybrid weighting (we) values ranging from 1 to 0. For each of these 10 experiments, run for 100 000 time steps, we use the 100%- ensemble-based static but adjust the ensemble variances as described in (2), such that .

After applying steps 1–3, we have a set of experiments, for each alpha value, with a static climatological error covariance matrix (consistent with the we = 1 case) and a flow-dependent error covariance matrix with appropriate dispersion. Figure 4 shows that for the Ne = 5 case the inflated prior ensemble is near-optimally inflated, with the ratio falling along the y = 1 line. The hybridized variance is perfect for the 100%-ensemble case (since a 100%-ensemble-based static is used) and overinflated when error variances are reduced through hybridization, where 0.2 < we < 1. The posterior is undervariant for all cases in which the ensemble weighting is nonzero, where sampling error plays a role. For the Ne = 10 case, the optimal performance is found when more weight is given to the ensemble. For we ≥ 0.5, the prior and hybridized variances have a nearly optimal ratio (nearly equal to 1). For the larger ensemble size, sampling error is reduced and the posterior is only slightly undervariant.
Fig. 4.
Fig. 4.

The ratio of ensemble variance to error variance shown as a function of weight given to the ensemble variance in the hybrid system. Shown are the prior ratio (dashed blue line), inflated prior ratio (solid blue line), hybridized ratio (red dashed line), and posterior ratio (solid cyan line) for (top) Ne = 5 and (bottom) Ne = 10. Lines indicate the mean of iterations 4–9, and error bars show the 95% confidence interval. All experiments use an entirely ensemble based static.

Citation: Monthly Weather Review 146, 11; 10.1175/MWR-D-18-0016.1

We now turn our attention to deriving optimal static and flow-dependent weights in a computationally efficient manner. We perform a linear regression using the inflated ensemble variance as a predictor of the squared error. In this case, the truth is known, so we can compute the actual forecast error (even though we will have sampling error in the mean for these small ensemble sizes). In an operational setting, one would compute the forecast error variances following
e46
where d denotes the debiased innovation yoxf. The observation error variance would need to be estimated using a method (e.g., Desroziers et al. 2005; Hollingsworth and Lönnberg 1986).
The slope term of the linear regression now gives a weight for the ensemble variance wens, and the weighting for the static term wclim is determined such that
e47
where tr() denotes the trace of the matrix and gclim is the intercept term of the regression. We note that, from the definition in (47), wclim could act to inflate or attenuate the static covariances , such that they equal the portion of the true error variance that does not vary as a function of ensemble variance. Figure 5 shows binned spread–skill diagrams with 10 bins. A perfectly dispersed ensemble would fall on the 1-to-1 line (solid red line); in such a case, we expect the optimal hybrid covariance to equal the ensemble-based covariance. If the ensemble has no skill, we expect the regression line to be horizontal, in which case the optimal hybrid covariance is purely static. We tend to see that the relationship between the ensemble variance and the error variance is curved rather than linear for this model, indicating that a higher-degree polynomial may provide a better fit. Figure 5d shows the linear regression-based hybridized variances, whereas Fig. 5e shows hybridized variances based on a cubic regression. The bottom panels of Fig. 5 show the Ne = 10 member ensemble case. The 10-member ensemble case also indicated a curved spread–skill relationship; however, after several iterations, negative leading coefficients were found. This behavior indicates that the polynomial fit would produce decreasing variances as the ensemble variance became large, which could result in negative variances. In such cases, there are many ways to avoid decreasing or negative variances—for example, holding the fitted regression constant at the point that it begins to decrease (which is done in section 4d). However, for the sake of brevity, in this section we will simply show the linear model for the 10-member ensemble case. Figure 6 shows the ratio of ensemble variances to error variances by iteration for the Ne = 5 linear regression–based hybrid (top panel), Ne = 5 cubic regression–based hybrid (middle panel), and Ne = 10 linear regression–based hybrid (bottom panel). Although there is some fluctuation in how quickly the optimal ratio is achieved, all experiments show prior ratios ranging between 0.8 and 1.1.
Fig. 5.
Fig. 5.

Binned spread–skill diagrams for the (left) inflated prior ensemble and (right) hybridized ensemble for (a),(d) Ne = 5 member ensemble with linear regression-based hybrid, (b),(e) Ne = 5 member ensemble with cubic regression-based hybrid, and (c),(f) Ne = 10 member ensemble with linear regression–based hybrid. The perfect 1-to-1 line is shown in red. The blue dots show the bin mean variances, and the dashed blue line shows the regression fit.

Citation: Monthly Weather Review 146, 11; 10.1175/MWR-D-18-0016.1

Fig. 6.
Fig. 6.

The ratio of ensemble variance to error variance by iteration. Shown are the prior ratio (dashed blue line), inflated prior ratio (solid blue line), and hybridized ratio (red dashed line) for the (top) Ne = 5 linear regression–based hybrid, (middle) Ne = 5 cubic regression–based hybrid, and (bottom) Ne = 10 linear regression–based hybrid. All experiments use an entirely ensemble based static.

Citation: Monthly Weather Review 146, 11; 10.1175/MWR-D-18-0016.1

Figure 7 shows the RMSE as a function of ensemble weighting for Ne = 5 (top) and Ne = 10 (bottom). Also shown are the RMSEs based on regression. For the Ne = 5 brute-force tuning experiment, we get a minimum RMSE when we = 0.7. The linear regression indicates 0.32 < we < 0.45; therefore, because the spread–skill relationship is curved, the ensemble weights for a linear model are underestimated in this case. [We note that the magenta curve (linear regression) and the blue curve (tuning) are not equal for that range of weights since the y intercept associated with the magenta curve is not forced to equal (1 − we)clim, as it would be if the static were the correct static for those ensemble weights.] The cubic regression-based hybrid shown in red gives a lower RMSE than any of the linear models, as would be expected based on the spread–skill plots shown in Fig. 5. For the Ne = 10 case, the linear model also underestimates the optimal weights of we = 0.9, which is again expected, from Fig. 5c.

Fig. 7.
Fig. 7.

RMSE as a function of ensemble weighting for (top) Ne = 5 and (bottom) Ne = 10. Also shown are the RMSE based on a linear regression (magenta) and cubic regression (red; shown only for Ne = 5). Lines indicate the mean of iterations 4–9, and error bars show the 95% confidence interval. All experiments use an entirely ensemble based static.

Citation: Monthly Weather Review 146, 11; 10.1175/MWR-D-18-0016.1

d. Impact of observation error, observing network density, and model error

As indicated in the beginning of this section, one of the motivations in using a simplified model is the ability to control parameters such as the observation error variance , observation density, and model error. In the following, we investigate the impact of changes to these parameters on the results shown in section 4c. These experiments help to test the limits of performance of these methods and to identify potential issues with implementation in an operational system. We proceed following the experiment design described in section 4c, but here focus only on the Ne = 5 case. In addition, since in many cases we are incorporating more severe errors, we found that we had to adjust the cubic fit to prevent decreasing with large ensemble values, as negative leading coefficients sometimes occurred. This was done by determining the maximum value of the function and holding that value constant as ensemble variance increased. Additionally, we set a lower bound to the cubic fit defined by the smaller of the minima of the inflated ensemble variance from the previous iteration or 0.0001. The upper bound was defined by a 10% increase of the maximum of the fitted function from the previous iteration.

We first investigate the influence of changing the observation error. Figure 8 shows the RMSE as a function of ensemble weighting for the Ne = 5 case. The top panel shows the case of observation error variance decreased by a factor of 100, and the bottom panel shows the case of the observation error variance being increased by a factor of 100. In this experiment, as in Fig. 7, all points are observed. In the top panel of Fig. 8, we see that although the error has been reduced by a factor of 10, the optimal weighting of the ensemble found by brute-force tuning is very similar to that in Fig. 7. Here, the regression again shows that the spread–skill relationship is curved, and the cubic regression again shows improved performance when compared with the linear regression. When we increase the observation error variance by a factor of 100 (bottom panel), we find that the model background is less accurate and lower weight should be given to the ensemble. In this case, the minimum RMSE is found when the weight given to the ensemble is we = 0.4. We also find very similar performance between the cubic and linear regressions. This behavior is consistent with the spread–skill relationship becoming more horizontal, in the case of a less accurate ensemble.

Fig. 8.
Fig. 8.

RMSE as a function of ensemble weighting for Ne = 5. Shown for observation error variance (top) decreased by a factor of 100 and (bottom) increased by a factor of 100. Also shown are the RMSE based on a linear regression (magenta) and cubic regression (red). Lines indicate the mean of iterations 5–10, and error bars show the 95% confidence interval. All experiments use an entirely ensemble based static.

Citation: Monthly Weather Review 146, 11; 10.1175/MWR-D-18-0016.1

We now investigate the impact of changing the observing density, but still defining as in the baseline experiments shown in Fig. 7. We compare different observation densities, always observing the center points, so as to mimic observing differences over land versus ocean. Figure 9 shows RMSE as a function of ensemble weighting for the cases of 80% of the grid points observed (top) and 50% of the grid points observed. In these cases, has location dependence, so determining the weighting for the static term wclim on the basis of (47) is not consistent with brute-force tuning. Instead, we modify (47) to take the trace over only the observed points. In other words, we fit both the linear and cubic functions based on the observed locations and apply those functions at all locations, whether observed or unobserved. We note that in such cases improved performance could potentially be obtained by weighting unobserved points differently; however, this approach would require computationally expensive tuning and is outside the scope of this study. Figure 9 shows that when the observing density is reduced to 80% (top panel), we find the minimum posterior RMSE through brute-force tuning (computed over all grid points, whether observed or unobserved) we ≈ 0.8. In this case, the ensemble and observations are still very accurate and the cubic fit offers some benefit over the linear fit. However, when the observation density is further reduced to 50%, the linear and cubic fits converge, with a minimum RMSE found when the ensemble weighting is we ≈ 0.7.

Fig. 9.
Fig. 9.

As in Fig. 8, but for an observation density of (top) 80% and (bottom) 50%. In both cases, the middle grid points are observed.

Citation: Monthly Weather Review 146, 11; 10.1175/MWR-D-18-0016.1

Next, we turn our attention to the impact of model error. We simulate model error by forcing the nature run, which provides the true state for verification and is perturbed to create observations, to be determined by a different model. This simulation is achieved by changing the forcing term F = 10 for the nature run but keeping the standard forcing (F = 8) for the experiment. The ensemble only accounts for this simulated model error through tuning multiplicative inflation. The results are shown in the top panel of Fig. 10. Again, consistent with a less accurate ensemble, the minimum RMSE found through brute-force tuning is achieved when the ensemble weighting is we ≈ 0.6, and little difference is seen between the cubic and linear fits. Last, we consider state-dependent error by adding a model error with variance drawn from a Gaussian with zero mean and variance q = 0.05 (1/10 of the observation error) to each ensemble member at each grid point where a value greater than a threshold of 7 is achieved (the typical range of values for this experimental setup is [−15, 15]). In this case, the state-dependent model error changes both the ensemble mean and the variance. The RMSE is plotted as a function of weighting given to the ensemble covariance on the bottom panel of Fig. 10. Here, the optimal weighting given to the ensemble found through brute-force tuning is we ≈ 0.6; however, the difference between the cubic and linear fits is still significant.

Fig. 10.
Fig. 10.

As in Fig. 8, but for (top) changing the forcing in the nature run from f = 8 to f = 10 and (bottom) including state-dependent model error (in this case Gaussian random noise with variance q = 0.05 is added when ensemble states are greater than a threshold of 7).

Citation: Monthly Weather Review 146, 11; 10.1175/MWR-D-18-0016.1

In all of the cases considered in this section, the cubic fit has either equally performed or outperformed the optimal weights found through brute-force tuning. It is also found that the degree to which the cubic outperforms the best linear model is based on the curvature of the spread–skill relationship, which depends on the performance of the ensemble. While the above results indicated that fitting to a higher-degree polynomial is most important when the ensemble variance more accurately tracks the true error variance, Fig. 5 indicated that, as the ensemble size increases, the spread–skill relationship becomes more linear. These two results together indicate that there may be a range in which a higher-order polynomial fit is preferable. It is also worth noting that, in many of the cases in which the cubic outperforms the linear regression, there is a more optimal linear fit, shown through brute-force tuning, than that estimated by regression.

5. Conclusions

In this study, we built on the findings of Bishop and Satterfield (2013) to show that the expected true prior variance, given an ensemble sample variance, is consistent with linear regression for the case in which the climatological distribution of true error variances is described by an inverse gamma; the distribution of ensemble variances, given a true error variance, is a gamma PDF; and the distribution of the state, given a true forecast error variance, is Gaussian. Theory was developed to confirm that, when only the sampling error in the ensemble variance was considered, the gain that minimizes the posterior error variance uses the expected true prior variance, given an ensemble sample variance. We demonstrated that, after a single run of a fully ensemble data assimilation scheme, one can use regression to obtain a model of optimal hybrid variance. For the idealized univariate data assimilation and multivariate cycling ensemble data assimilation considered here, it was found that, when the relationship between the ensemble variance and true error variance is linear, linear regression closely approximates the optimal weights found through the simple, but computationally expensive, process of testing every plausible combination of weights. For the case in which the relationship between the ensemble variance and true error variance is nonlinear, we introduce a generalized hybrid model defined by higher-order polynomial regression and demonstrated that such a scheme outperformed any plausible linear hybrid model. It is also found that the degree to which higher-order polynomial regression outperforms the best linear model is dependent on the performance of the ensemble. Typically, fitting to a higher-degree polynomial is most important when the ensemble variance more accurately tracks the true error variance.

The focus of this work has been on hybrid formulations that form a hybridized error covariance matrix (e.g., Hamill and Snyder 2000) and for filters that assimilate all available observations at once. Additional work would be needed to implement serial filters. The theory presented here has only been applied to variances, and additional work is needed to account for correlations, although a similar method could be applied. In addition, it would be of interest to examine how to optimally form a static error covariance matrix and update such a matrix with updates to hybrid coefficients. These issues, as well as application of this theory to the Navy Global Environmental Model (NAVGEM), are the subject of future work.

Acknowledgments

The authors thank the three anonymous reviewers, whose suggestions helped us to improve the presentation of our results. This research is supported by the Office of Naval Research (ONR) through the NRL Base Program PE 0601153N.

APPENDIX

Derivation of the Kalman Gain

We follow the assumptions of section 2b, such that we write the true posterior mean in this case as
ea1
ea2
and the one with sampling error will be written as
ea3
We emphasize that, consistent with section 2b, we have assumed for simplicity that the only sampling error comes in through the gain and the prior variance. Note that we have not assumed a form for Ge; it has been left completely arbitrary. It is the goal of this section to derive its optimal representation.
We perform the standard steps to minimize the posterior variance. The posterior variance about (A3) in this case is
ea4
Using (A1)(A3) in (A4), we obtain
ea5
where the true posterior variance is . The first term on the right-hand side is the true posterior error variance averaged over the climatological distribution of true variances. Let us call this , and therefore we have
ea6
The integral on the right decomposes as
ea7
Using (A7) in (A6), we obtain
ea8
Next, we take the derivative of (A8) with respect to Ge and set the result to zero:
ea9
We then solve (A9) for the sought-after result, namely,
ea10

REFERENCES

  • Anderson, J. L., 2007: Exploring the need for localization in ensemble data assimilation using a hierarchical ensemble filter. Physica D, 230, 99111, https://doi.org/10.1016/j.physd.2006.02.011.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bishop, C. H., and E. A. Satterfield, 2013: Hidden error variance theory. Part I: Exposition and analytic model. Mon. Wea. Rev., 141, 14541468, https://doi.org/10.1175/MWR-D-12-00118.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bishop, C. H., E. A. Satterfield, and K. T. Shanley, 2013: Hidden error variance theory. Part II: An instrument that reveals hidden error variance distributions from ensemble forecasts and observations. Mon. Wea. Rev., 141, 14691483, https://doi.org/10.1175/MWR-D-12-00119.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bocquet, M., 2011: Ensemble Kalman filtering without the intrinsic need for inflation. Nonlinear Processes Geophys., 18, 735750, https://doi.org/10.5194/npg-18-735-2011.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Burgers, G., P. Jan van Leeuwen, and G. Evensen, 1998: Analysis scheme in the ensemble Kalman filter. Mon. Wea. Rev., 126, 17191724, https://doi.org/10.1175/1520-0493(1998)126<1719:ASITEK>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Clayton, A. M., A. C. Lorenc, and D. M. Barker, 2013: Operational implementation of a hybrid ensemble/4D-Var global data assimilation system at the Met Office. Quart. J. Roy. Meteor. Soc., 139, 14451461, https://doi.org/10.1002/qj.2054.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Desroziers, G., L. Berre, B. Chapnik, and P. Poli, 2005: Diagnosis of observation, background and analysis-error statistics in observation space. Quart. J. Roy. Meteor. Soc., 131, 33853396, https://doi.org/10.1256/qj.05.108.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Etherton, B. J., and C. H. Bishop, 2004: Resilience of hybrid ensemble/3DVAR analysis schemes to model error and ensemble covariance error. Mon. Wea. Rev., 132, 10651080, https://doi.org/10.1175/1520-0493(2004)132<1065:ROHDAS>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Furrer, R., and T. Bengtsson, 2007: Estimation of high-dimensional prior and posterior covariance matrices in Kalman filter variants. J. Multivar. Anal., 98, 227255, https://doi.org/10.1016/j.jmva.2006.08.003.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gaspari, G., and S. E. Cohn, 1999: Construction of correlation functions in two and three dimensions. Quart. J. Roy. Meteor. Soc., 125, 723757, https://doi.org/10.1002/qj.49712555417.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., and C. Snyder, 2000: A hybrid ensemble Kalman filter–3D variational analysis scheme. Mon. Wea. Rev., 128, 29052919, https://doi.org/10.1175/1520-0493(2000)128<2905:AHEKFV>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., J. S. Whitaker, and C. Snyder, 2001: Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter. Mon. Wea. Rev., 129, 27762790, https://doi.org/10.1175/1520-0493(2001)129<2776:DDFOBE>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hodyss, D., W. F. Campbell, and J. S. Whitaker, 2016: Observation-dependent posterior inflation for the ensemble Kalman filter. Mon. Wea. Rev., 144, 26672684, https://doi.org/10.1175/MWR-D-15-0329.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hollingsworth, A., and P. Lönnberg, 1986: The statistical structure of short-range forecast errors as determined from radiosonde data. Part I: The wind field. Tellus, 38A, 111136, https://doi.org/10.1111/j.1600-0870.1986.tb00460.x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kuhl, D. D., T. E. Rosmond, C. H. Bishop, J. McLay, and N. L. Baker, 2013: Comparison of hybrid ensemble/4DVar and 4DVar within the NAVDAS-AR data assimilation framework. Mon. Wea. Rev., 141, 27402758, https://doi.org/10.1175/MWR-D-12-00182.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lorenc, A. C., 2003: The potential of the ensemble Kalman filter for NWP—A comparison with 4D-Var. Quart. J. Roy. Meteor. Soc., 129, 31833203, https://doi.org/10.1256/qj.02.132.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lorenz, E. N., 1996: Predictability: A problem solved. Proc. Seminar on Predictability, Shinfield Park, Reading, United Kingdom, ECMWF, 18 pp.

  • Lorenz, E. N., 2005: Designing chaotic models. J. Atmos. Sci., 62, 15741587, https://doi.org/10.1175/JAS3430.1.

  • Ménétrier, B., and T. Auligné, 2015: Optimized localization and hybridization to filter ensemble-based covariances. Mon. Wea. Rev., 143, 39313947, https://doi.org/10.1175/MWR-D-15-0057.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ménétrier, B., T. Montmerle, Y. Michel, and L. Berre, 2015: Linear filtering of sample covariances for ensemble-based data assimilation. Part I: Optimality criteria and application to variance filtering and covariance localization. Mon. Wea. Rev., 143, 16221643, https://doi.org/10.1175/MWR-D-14-00157.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Sacher, W., and P. Bartello, 2008: Sampling errors in ensemble Kalman filtering. Part I: Theory. Mon. Wea. Rev., 136, 30353049, https://doi.org/10.1175/2007MWR2323.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wang, X., D. Barker, C. Snyder, and T. M. Hamill, 2008a: A hybrid ETKF–3DVAR data assimilation scheme for the WRF Model. Part I: Observing system simulation experiment. Mon. Wea. Rev., 136, 51165131, https://doi.org/10.1175/2008MWR2444.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wang, X., D. Barker, C. Snyder, and T. M. Hamill, 2008b: A hybrid ETKF–3DVAR data assimilation scheme for the WRF Model. Part II: Real observation experiments. Mon. Wea. Rev., 136, 51325147, https://doi.org/10.1175/2008MWR2445.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wang, X., D. Parrish, D. Kleist, and J. Whitaker, 2013: GSI 3DVar-based ensemble–variational hybrid data assimilation for NCEP Global Forecast System: Single-resolution experiments. Mon. Wea. Rev., 141, 40984117, https://doi.org/10.1175/MWR-D-12-00141.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Yaremchuk, M., D. Nechaev, and C. Pan, 2011: A hybrid background error covariance model for assimilating glider data into a coastal ocean model. Mon. Wea. Rev., 139, 18791890, https://doi.org/10.1175/2011MWR3510.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
Save