• Atlas, R., 1997: Atmospheric observations and experiments to assess their usefulness in data assimilation. J. Meteor. Soc. Japan, 75, 111130.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bocquet, M., and et al. , 2015: Data assimilation in atmospheric chemistry models: Current status and future prospects for coupled chemistry meteorology models. Atmos. Chem. Phys., 15, 53255358, doi:10.5194/acp-15-5325-2015.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Chai, T., G. R. Carmichael, Y. Tang, A. Sandu, A. Heckel, A. Richter, and J. P. Burrows, 2009: Regional NOx emission inversion through a four-dimensional variational approach using SCIAMACHY tropospheric NO2 column observations. Atmos. Environ., 43, 50465055, doi:10.1016/j.atmosenv.2009.06.052.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Clemitshaw, K., 2004: A review of instrumentation and measurement techniques for ground-based and airborne field studies of gas-phase tropospheric chemistry. Crit. Rev. Environ. Sci. Technol., 34, 1108, doi:10.1080/10643380490265117.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ebert, E. E., 2008: Fuzzy verification of high-resolution gridded forecasts: A review and proposed framework. Meteor. Appl., 15, 5164, doi:10.1002/met.25.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Elbern, H., and H. Schmidt, 2001: Ozone episode analysis by four-dimensional variational chemistry data assimilation. J. Geophys. Res., 106, 35693590, doi:10.1029/2000JD900448.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Elbern, H., A. Strunk, H. Schmidt, and O. Talagrand, 2007: Emission rate and chemical state estimation by 4-dimensional variational inversion. Atmos. Chem. Phys., 7, 37493769, doi:10.5194/acp-7-3749-2007.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fu, G., H. X. Lin, A. W. Heemink, A. J. Segers, S. Lu, and T. Palsson, 2015: Assimilating aircraft-based measurements to improve forecast accuracy of volcanic ash transport. Atmos. Environ., 115, 170184, doi:10.1016/j.atmosenv.2015.05.061.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fu, G., A. Heemink, S. Lu, A. Segers, K. Weber, and H.-X. Lin, 2016: Model-based aviation advice on distal volcanic ash clouds by assimilating aircraft in situ measurements. Atmos. Chem. Phys., 16, 91899200, doi:10.5194/acp-16-9189-2016.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fu, G., F. Prata, H. X. Lin, A. Heemink, A. Segers, and S. Lu, 2017: Data assimilation for volcanic ash plumes using a satellite observational operator: A case study on the 2010 Eyjafjallajökull volcanic eruption. Atmos. Chem. Phys., 17, 11871205, doi:10.5194/acp-17-1187-2017.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gilleland, E., D. Ahijevych, B. G. Brown, B. Casati, and E. E. Ebert, 2009: Intercomparison of spatial forecast verification methods. Wea. Forecasting, 24, 14161430, doi:10.1175/2009WAF2222269.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gilleland, E., D. A. Ahijevych, B. G. Brown, and E. E. Ebert, 2010: Verifying forecasts spatially. Bull. Amer. Meteor. Soc., 91, 13651373, doi:10.1175/2010BAMS2819.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Haben, S. A., A. S. Lawless, and N. K. Nichols, 2011a: Conditioning and preconditioning of the variational data assimilation problem. Comput. Fluids, 46, 252256, doi:10.1016/j.compfluid.2010.11.025.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Haben, S. A., A. S. Lawless, and N. K. Nichols, 2011b: Conditioning of incremental variational data assimilation, with application to the Met Office system. Tellus, 63A, 782792, doi:10.1111/j.1600-0870.2011.00527.x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Huneeus, N., F. Chevallier, and O. Boucher, 2012: Estimating aerosol emissions by assimilating observed aerosol optical depth in a global aerosol model. Atmos. Chem. Phys., 12, 45854606, doi:10.5194/acp-12-4585-2012.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jacquez, J. A., and P. Greif, 1985: Numerical parameter identifiability and estimability: Integrating identifiability, estimability, and optimal sampling design. Math. Biosci., 77, 201227, doi:10.1016/0025-5564(85)90098-7.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kawabata, T., H. Iwai, H. Seko, Y. Shoji, K. Saito, S. Ishii, and K. Mizutani, 2014: Cloud-resolving 4D-Var assimilation of Doppler wind lidar data on a meso-gamma-scale convective system. Mon. Wea. Rev., 142, 44844498, doi:10.1175/MWR-D-13-00362.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lahoz, W., B. Khattatov, and R. Menard, 2010: Data Assimilation: Making Sense of Observations. 1st ed. Springer-Verlag, 718 pp., doi:10.1007/978-3-540-74703-1.

    • Crossref
    • Export Citation
  • Lamsal, L. N., and et al. , 2011: Application of satellite observations for timely updates to global anthropogenic NOx emission inventories. Geophys. Res. Lett., 38, L05810, doi:10.1029/2010GL046476.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lorenc, A. C., N. E. Bowler, A. M. Clayton, S. R. Pring, and D. Fairbairn, 2015: Comparison of hybrid-4DEnVar and hybrid-4DVar data assimilation methods for global NWP. Mon. Wea. Rev., 143, 212229, doi:10.1175/MWR-D-14-00195.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lu, S., H. X. Lin, A. W. Heemink, G. Fu, and A. J. Segers, 2016a: Estimation of volcanic ash emissions using trajectory-based 4D-Var data assimilation. Mon. Wea. Rev., 144, 575589, doi:10.1175/MWR-D-15-0194.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lu, S., H. X. Lin, A. W. Heemink, A. J. Segers, and G. Fu, 2016b: Estimation of volcanic ash emissions through assimilating satellite data and ground-based observations. J. Geophys. Res. Atmos., 121, 10 97110 994, doi:10.1002/2016JD025131.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • McMurry, P., 2000: A review of atmospheric aerosol measurements. Atmos. Environ., 34, 19591999, doi:10.1016/S1352-2310(99)00455-0.

  • Meirink, J. F., P. Bergamaschi, and M. C. Krol, 2008: Four-dimensional variational data assimilation for inverse modelling of atmospheric methane emissions: Method and comparison with synthesis inversion. Atmos. Chem. Phys., 8, 63416353, doi:10.5194/acp-8-6341-2008.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mittermaier, M., and N. Roberts, 2010: Intercomparison of spatial forecast verification methods: Identifying skillful spatial scales using the fractions skill score. Wea. Forecasting, 25, 343354, doi:10.1175/2009WAF2222260.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Paulino, C., and C. de Bragança Pereira, 1994: On identifiability of parametric statistical models. J. Ital. Stat. Soc., 3, 125151, doi:10.1007/BF02589044.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Prata, A. J., and A. T. Prata, 2012: Eyjafjallajökull volcanic ash concentrations determined using Spin Enhanced Visible and Infrared Imager measurements. J. Geophys. Res., 117, D00U23, doi:10.1029/2011jd016800.

    • Search Google Scholar
    • Export Citation
  • Robert, C., S. Durbiano, E. Blayo, J. Verron, J. Blum, and F. X. Le Dimet, 2005: A reduced-order strategy for 4D-Var data assimilation. J. Mar. Syst., 57, 7082, doi:10.1016/j.jmarsys.2005.04.003.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rothenberg, T. J., 1971: Identification in parametric models. Econometrica, 39, 577591, doi:10.2307/1913267.

  • Talagrand, O., and P. Courtier, 1987: Variational assimilation of meteorological observations with the adjoint vorticity equation. I: Theory. Quart. J. Roy. Meteor. Soc., 113, 13111328, doi:10.1002/qj.49711347812.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wang, Y., and et al. , 2014: Assimilation of lidar signals: Application to aerosol forecasting in the western Mediterranean basin. Atmos. Chem. Phys., 14, 12 03112 053, doi:10.5194/acp-14-12031-2014.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Webley, P. W., T. Steensen, M. Stuefer, G. Grell, S. Freitas, and M. Pavolonis, 2012: Analyzing the Eyjafjallajökull 2010 eruption using satellite remote sensing, lidar and WRF-Chem dispersion and tracking model. J. Geophys. Res., 117, D00U26, doi:10.1029/2011JD016817.

    • Search Google Scholar
    • Export Citation
  • View in gallery

    Process of gradient-based approaches.

  • View in gallery

    (a) Simulation and assimilation domain of Iceland. (b) Columns of the volcanic ash cloud at 1900 UTC 14 Apr 2010.

  • View in gallery

    Estimation results of emission rates with (a) complete observations and (b) synthetic satellite observations.

  • View in gallery

    Normalized gradients generated by perturbations in a single input parameter at the (a) sixth and (b) seventh layers above the summit, with a 1-h assimilation window using Std4DVar.

  • View in gallery

    Criteria values of Std4DVar vs Trj4DVar.

  • View in gallery

    Comparison of estimation results with a 6-h assimilation window using Std4DVar vs using Trj4DVar.

  • View in gallery

    Normalized gradients generated by perturbations in a single input parameter at the (a) fifth and (b) seventh layers, with a 6-h-assimilation window using Trj4DVar.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 19 19 1
PDF Downloads 15 15 0

Evaluation Criteria on the Design for Assimilating Remote Sensing Data Using Variational Approaches

View More View Less
  • 1 Delft Institute of Applied Mathematics, Delft University of Technology, Delft, Netherlands
  • | 2 Department of Climate, Air and Sustainability, TNO, Utrecht, Netherlands
  • | 3 Delft Institute of Applied Mathematics, Delft University of Technology, Delft, Netherlands
© Get Permissions
Full access

Abstract

Remote sensing, as a powerful tool for monitoring atmospheric phenomena, has been playing an increasingly important role in inverse modeling. Remote sensing instruments measure quantities that often combine several state variables as one. This creates very strong correlations between the state variables that share the same observation variable. This may cause numerical problems resulting in a low convergence rate or inaccurate estimates in gradient-based variational assimilation if improper error statistics are used. In this paper, two criteria or scoring rules are proposed to quantify the numerical robustness of assimilating a specific set of remote sensing observations and to quantify the reliability of the estimates of the parameters. The criteria are derived by analyzing how the correlations are created via shared observation data and how they may influence the process of variational data assimilation. Experimental tests are conducted and show a good level of agreement with theory. The results illustrate the capability of the criteria to indicate the reliability of the assimilation process. Both criteria can be used with observing system simulation experiments (OSSEs) and in combination with other verification scores.

Denotes content that is immediately available upon publication as open access.

© 2017 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author e-mail: Sha Lu, s.lu-1@tudelft.nl

Abstract

Remote sensing, as a powerful tool for monitoring atmospheric phenomena, has been playing an increasingly important role in inverse modeling. Remote sensing instruments measure quantities that often combine several state variables as one. This creates very strong correlations between the state variables that share the same observation variable. This may cause numerical problems resulting in a low convergence rate or inaccurate estimates in gradient-based variational assimilation if improper error statistics are used. In this paper, two criteria or scoring rules are proposed to quantify the numerical robustness of assimilating a specific set of remote sensing observations and to quantify the reliability of the estimates of the parameters. The criteria are derived by analyzing how the correlations are created via shared observation data and how they may influence the process of variational data assimilation. Experimental tests are conducted and show a good level of agreement with theory. The results illustrate the capability of the criteria to indicate the reliability of the assimilation process. Both criteria can be used with observing system simulation experiments (OSSEs) and in combination with other verification scores.

Denotes content that is immediately available upon publication as open access.

© 2017 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author e-mail: Sha Lu, s.lu-1@tudelft.nl

1. Introduction

During the past three decades, the assimilation of atmospheric observations as an aid in improving forecasts of air quality and in constructing reanalyses of past weather and climate change has gained growing interest (Talagrand and Courtier 1987; Elbern and Schmidt 2001; Elbern et al. 2007; Fu et al. 2015). The available observations consist of a mixture of in situ, visual, and remotely sensed observations of temperature, wind velocity, pressure, humidity, and clouds (McMurry 2000; Clemitshaw 2004; Lahoz et al. 2010). Remote sensing makes it possible to collect data from dangerous or inaccessible areas, and meteorological satellites provide an indispensable supplement to the conventional meteorological observing system. Due to their ability of acquiring data in traditionally data-poor regions of the oceans, the stratosphere, and the Southern Hemisphere, as well as the high horizontal resolution, satellite observations have played an increasingly important role in atmospheric studies (Bocquet et al. 2015; Fu et al. 2017). Numerous experiments have been conducted in order to make good use of the satellite data in operational numerical weather forecasting or to improve the analysis and understanding of atmospheric phenomena and dynamics.

The assimilation of satellite data and other remote sensing data using variational approaches has been successfully applied to various atmospheric problems, such as efforts to improve initial model states and estimates of the emissions of natural or anthropogenic pollutants. For instance, Chai et al. (2009) and Lamsal et al. (2011) estimated regional or global nitrogen oxides (NOx) emission inventories with satellite nitrous oxide (NO2) column observations. Besides the estimation of a single emission species, Huneeus et al. (2012) demonstrated the simultaneous estimate of global emissions of multiple gaseous and aerosol species including dust, sea salt, black carbon (BC), organic carbon (OC), and sulfur dioxide (SO2) by assimilating daily MODIS total and fine-mode aerosol optical depth (AOD). Kawabata et al. (2014) used the nonhydrostatic 4D-Var assimilation system to assimilate the Doppler wind lidar (DWL) data to forecast the heavy rainfall event of 5 July 2010 in Japan.

The remote sensing instruments including satellite, lidar, and radar acquire information without physical contact with the object (the state) by detecting the electromagnetic radiation, solar radiation, or microwave radiation. The retrieval algorithm of the detections, such as satellite-retrieved AOD data (Prata and Prata 2012) and lidar backscatter coefficients (Wang et al. 2014), usually requires the combination or integration of multiple state variables and subsequently sensor-induced correlations (SICs) are introduced between the states that share the same combined observation data. These SICs may have a negative impact on the performance of the parameter estimation method when erroneous or improper specification of error statistics or of the prior information is used. Alternative 4D-Var approaches that lead to better-conditioned estimation problems should be used to remedy this. For example, Lu et al. (2016a) demonstrated that using the 4D-Var method with a standard form of the cost function to estimate the vertical profile of the volcanic ash emission rate from the satellite ash column data could result in undesired estimates. The standard cost function for parameter estimation computes the sum of the squared deviations of the analysis values from the observations weighted by the accuracy of the observations, plus the sum of the squared deviations of the estimated parameters and the background parameters weighted by the accuracy of the background information [as in Meirink et al. (2008), their Eq. (2)]. The above problem was solved by a using trajectory-based 4D-Var (Trj4DVar) approach with a reformulated cost function.

Observing system simulation experiments (OSSEs) are an important tool for evaluating the potential impact of proposed observing systems, as well as for evaluating trade-offs in the observing system design, as well as in developing and assessing improved methodologies for assimilating new observations (Atlas 1997). These OSSEs are used in combination with scoring rules and verification skills, which usually measure the deviations between forecasts and the observation values, hits, misses, and false alarms (Mittermaier and Roberts 2010; Gilleland et al. 2009; Ebert 2008; Gilleland et al. 2010). In addition, there are many tools for observability or identifiability analysis that can be used to diagnose whether the parameters can be identified from a given set of observations (Paulino and de Bragança Pereira 1994; Rothenberg 1971; Jacquez and Greif 1985). However, these methods focus more on the usefulness of the observations and are incapable of determining the numerical robustness of the estimation procedure. The condition number of the Hessian is able to indicate the numerical performance of the gradient-based variational assimilation approach (Haben et al. 2011a,b). However, the computation of this condition number is computationally very expensive, especially when the number of state variables is large.

In this paper, two criteria are proposed to quantify the numerical effects of the SICs on the assimilation process. The criteria are simple and practical to implement for a rough evaluation of the numerical performance of assimilating a certain type of observations for a given application. They were inspired by the previous work of Lu et al. (2016a), and were originally developed to evaluate the performance of two 4D-Var approaches with different specifications of error statistics in assimilating satellite column data to estimate a vertically distributed emission. It is found that the two criteria can be also used for other applications using remote sensing data or other integrated data.

2. The evaluation methodology

a. Preliminary knowledge about variational data assimilation

Consider a discrete dynamic model given by
e1
where the subscript k represents the time step . The model state vector and its corresponding dynamics operator are and , respectively; is the static parameter vector including the model parameters, inputs, and initial conditions, which need to be estimated in this case.
The background or the first guess of the parameters are assumed to differ from the true parameters by stochastic perturbations:
e2
where .
Observations at time are defined by
e3
where is the observation operator that projects the state space into observation space and is the observation uncertainty.
The 4D-Var approach minimizes the cost function J that measures the weighted sum of the squares of distances to the background parameters and to the observations obtained over a time interval :
e4
where represents the simulated observations. In this paper we focus on the impact of observations on the update. Therefore, the development of the evaluation criteria requires only the observation term .
The minimization usually requires the gradient of the cost function with respect to the parameters:
e5
where and are the tangent linear model and its adjoint, respectively, corresponding to observation operator .
In addition,
e6
is the sensitivity of the states with respect to the parameters, and is its transpose.

b. Sensor-induced correlations and their impact

Remote sensing observations, along with some other types of observations, measure quantities whose computation involves multiple state variables or parameters. SICs are created between those variables that share a common type of observation data, and may have a negative numerical effect on the assimilation process or lead to ill-conditioned numerical problems. The impact of observations on the assimilation and the mechanism of a gradient-based algorithm is illustrated by Fig. 1.

Fig. 1.
Fig. 1.

Process of gradient-based approaches.

Citation: Monthly Weather Review 145, 6; 10.1175/MWR-D-16-0289.1

The difference between the true parameters and the background (first guess) parameters will result in a difference between the true state fields and the background state fields . The perturbations are determined based on the model sensitivity behavior:
e7
where is given by Eq. (6).
However, or even usually cannot be observed directly by remote sensing instruments for computing or . To estimate , we will make use of the difference between the true observations and the simulated observations . First, the observed difference between states is computed by distributing to the states according to the observational error statistics and the adjoint operational operator by
e8
where is termed the observed sensitivity representing the sensitivity of states through observations with respect to the parameters. The parameters will then be updated according to model dynamics and the observed perturbation of states as
e9
This mechanism can be interpreted by rewriting the gradient in Eq. (5):
e10
If the actual perturbation of states is used for the update of the parameters , this update () is not affected by SICs. The formulation of the corresponding gradient is given as follows:
e11
We can view as the gradient of a cost function when using a “complete observation operator” , which observes the complete states:
e12
In a physical sense only, reflects the model sensitivity behavior. The impact of SICs is implied by how much the observed model sensitivity differs from the physical model sensitivity, where the former is reflected by and the latter is reflected by .

c. Evaluation criteria

Two criteria are now presented to test how much the SICs will influence the assimilation process in a negative way, which can also be considered as an indication of the usefulness of the data on a given DA system.

1) Criterion 1 (FIM criterion)

The distance between the normalized Hessian or the normalized Fisher information matrix (FIM) generated with complete observation and remote sensing observation:
e13
where the norm is the Frobenius norm with or the norm with for a matrix , and for a linear model, while and for a nonlinear model or other specifications of error statistics (formulation of cost function) with and are the cost functions formed by complete observation and remote sensing observation, respectively.

The DA process influenced only by the physical dynamics or the model sensitivity behavior is reflected by , while reflects the performance of DA as a result of combining the observation operator and model dynamics. This criterion provides global information on to what degree the SICs change the sensitivity behavior that is used for updating the parameters (gradient) over the iterations. The FIM criterion is practical for those cases where the FIM (Hessian) or its approximation is easy to compute.

2) Criterion 2 (gradient criterion)

The distance between the normalized model gradient and the normalized observed gradient:
e14
or
e15
where the norm is the Euclidean norm with for a vector . The model gradient is and is the observed gradient, as defined in Eqs. (11) and (5) for a standard cost function, respectively. Or and with and are defined as in FIM Criterion for other formulations of the cost function.

Criterion (14) provides local and detailed information that measures the impact of SICs on the quality of gradient as well as on the convergence performance. A large value of this criterion indicates that a poor gradient is obtained using the observations. We can perturb one parameter or one state variable and compute the criterion value, which indicates whether the observation is capable of estimating the perturbed parameter. In general, perturbations can also be performed on a set of closely related parameters or states. Criterion (15) calculates in this case the mean of the differences between the two normalized gradients generated by a number of random perturbations of parameters or states, and this provides global information about whether the gradients can well represent the model’s sensitivity behavior. Criteria (14) and (15) can be applied to cases where the adjoint model is available but the Hessian is difficult to obtain.

The values of the two criteria range from 0 to 2. A small value (say less than 0.1) implies a good observation operator, which almost preserves the characteristics of the model dynamics. The two criteria will give large values that serve as a warning when SICs are created by using remote sensing data or other integrated data and will lead to ill-conditioned assimilation processes. Theoretically, a bad situation occurs when the gradient is unable to distinguish the perturbed parameters and gives an equal update on each parameter; in such a case, the result leads to a gradient criterion value of with . An even worse scenario occurs when larger updates are given on the unperturbed parameters than on the perturbed ones, which results in even larger criterion values. Tests suggest that FIM criterion values larger than 0.9 or gradient criterion values larger than 1.0 indicate a very ineffective assimilation process for the gradient-based method. Values less than 0.6 turned out to be acceptable for our case of volcanic ash.

Criteria (13) and (15) both provide a global assessment of the numerical robustness of the assimilation process and of the reliability of the forecast after assimilation. The values of both criteria change with observation operators (observation position and observation type). Criterion (13) is invariant to the perturbed variables and more robust than criterion (15). Criteria (14) and (15) can potentially be used as a diagnostic tool to detect which parameters are correlated via observations but not physically and how this will affect the assimilation outcomes. This approach provides a means for better analyzing the sensitivity behavior and developing a more effective alternative method for the use of certain types of observations.

Note that the background term of the cost function plays an important role in the performance of the 4D-Var approach in order to distinguish different variable in the analysis increments and as a regularization term. However, in the derivation of the criteria we ignore the background term. That is because first this study focuses on exploring the impact of remote sensing observations and other integrated observations on the DA process. Second, gradient criteria (14) and (15) are calculated from the gradient of cost function (4) at , so the background term has no effect on the gradient. The impact of using a different background can still be tested using the criteria, since the information from the background is implicitly included in the observation term by the use of model-simulated observation . As well, the impact of different perturbed parameters is implicitly included in of .

d. Example: Trajectory-based 4D-Var approach

In this section, trajectory-based 4D-Var (Trj4DVar) will be briefly introduced and the procedure for determining its corresponding criteria will be described; this approach will be used in the case study in the next section.

Trj4DVar seeks an optimal linear combination of trajectories generated with different emissions to fit the observation data coupled with a priori information, by minimizing a reformulated 4D-Var cost function.

We assume that the vector of parameters is in a parameter space spanned by the perturbed parameter sets () and can be represented in the following form:
e16
where is the weight of in the sum. If p is large, the parameter space can be approximated by a smaller space spanned by a smaller number of perturbed parameters. Therefore, the determination of corresponds to estimating .
The simulated observations in cost function (4) can be approximated by
e17
where are reference trajectories computed using background parameters and are trajectories associated with perturbation of parameters in the neighborhood of .
Therefore, the coefficients can be computed by minimizing a reformulation of the cost function (4) given by
e18
The gradient of in cost function (18) with respect to is computed by
e19
where . The Hessian can be similarly obtained as
e20
Note that Eq. (20) corresponds to an approximate, but not an exact, Hessian whenever is nonlinear. The counterparts of the gradient and Hessian for can be obtained by substituting the complete observation operator (12) for in the computation of trajectories.

Note that the formulation of Trj4DVar is similar to model-order reduced 4D-Var (MOR-4D-Var) methods (Robert et al. 2005), or the family of four-dimensional (4D) ensemble–variational data assimilation (4DEnVar) methods (Lorenc et al. 2015). The objective of the MOR-4D-Var approach is to seek a low-rank approximation of the model to reduce the computational effort of 4D-Var, and that of 4DEnVar is to obtain a low-rank and flow-dependent representation of the background error statistics. Therefore, the sampling of the snapshots in MOR-4D-Var or ensembles in 4DEnVar is usually randomly distributed. However, the objective of Trj4DVar is to solve the ill-conditioned problem caused by the lack of information or the improper use of the SICs in this case. The generation of the trajectories in Trj4DVar requires the knowledge of the characteristics of the model sensitivity.

3. Case study

We illustrate uses of both criteria for the evaluation on the effectiveness of the assimilation process and for the design of the assimilation system. The criteria are tested using a case where SICs typically influence the assimilation process negatively when using a 4D-Var approach with a standard type of cost function. It was explicitly pointed out by Lu et al. (2016a) that using satellite ash column data can result in inaccurate estimates of volcanic ash emissions. Therefore, twin experiments are conducted based on a volcanic ash estimation problem.

a. Experimental setup

Twin experiments are carried out to estimate the emission rates of volcanic ash by assimilating synthetic observations. A 3D aerosol transport model of the Iceland area (Fig. 2a) is used to simulate the Eyjafjallajökull volcanic activity during 14–19 April 2010, with a temporal resolution of 15 min and a spatial resolution of 0.25° × 0.25°. For simplicity, the transport model includes only advection and diffusion processes for which the adjoint model is available. Wind fields are obtained from 3-hourly meteorological data from the European Centre for Medium-Range Weather Forecasts (ECMWF), which is interpolated to hourly resolution. Figure 2b is an illustration of a volcanic ash cloud simulated by the model.

Fig. 2.
Fig. 2.

(a) Simulation and assimilation domain of Iceland. (b) Columns of the volcanic ash cloud at 1900 UTC 14 Apr 2010.

Citation: Monthly Weather Review 145, 6; 10.1175/MWR-D-16-0289.1

The emission information from the first few days of the explosive eruption is taken from Webley et al. (2012) and is shown in Table 1. The eruption is described in terms of parameters such as the total emission rate and the plume height, which are assumed to be constant during an emission episode of several hours. The “true” emission rates in the vertical layers are generated using a Poisson distribution according to the emission information shown in Table 1. The “background”/first-guess emission is calculated with an underestimated total emission rate of the true emission and a correspondingly lower plume height.

Table 1.

Input parameters for the 14–19 Apr 2010 period of activity at Eyjafjallajökull, taken from Webley et al. (2012).

Table 1.

The synthetic observations are generated hourly by running the model with the true emissions. The complete observations are 3D state fields. The satellite-like observations are ash columns generated by weighted summations of ash loads along columns (kg m−2) given as
e21
where is the observation variable at the ith pixel, is the ash density at the ith pixel and the lth vertical layer, and is the height of the grid cell where is located.

Note that sedimentation is important for modeling volcanic ash in real life (Fu et al. 2016). However, currently there is no adjoint for the sedimentation process in the model [Long Term Ozone Simulation–European Ozone Simulation (LOTOS-EUROS)] used in this paper. In addition, the omission of sedimentation will not significantly change the performance of the assimilation approach on this model. This is because the influences of this process are most impactful on the amount of the ash concentrations, but not on the flow pattern that strongly affects the numerical performance of the assimilation process.

b. Twin experiments using standard 4D-Var

Twin experiments are conducted with both complete observations and column-integrated observations. The results are shown in Figs. 3a and 3b, respectively. In Fig. 3a it can be seen that the “estimated” result perfectly matches the truth, which implies that the model is physically well conditioned and emission rates can be well estimated according to the model dynamics. However, in Fig. 3b, with 1-hourly assimilation of column-integrated observations, the estimated emission rates (denoted by est_1h) are increased by the same amount in all layers without recognizing the vertical profile in the truth. The injection layer, with the maximum of the emission rate, cannot be correctly determined. Similar results are obtained with longer assimilation windows (3 and 6 h). This shows that it is very ineffective to estimate volcanic ash emission rates using satellite data, and that this finding is not caused by the model but by the type of observations used.

Fig. 3.
Fig. 3.

Estimation results of emission rates with (a) complete observations and (b) synthetic satellite observations.

Citation: Monthly Weather Review 145, 6; 10.1175/MWR-D-16-0289.1

Now we investigate the problem using the gradient criteria. First, perturbations on a single state variable are carried out, and the model gradient and observed gradient are computed with complete observations and column-wise observations, respectively. The gradients are sensitivities of the perturbed state with respect to the parameters (emission rates). Four single-state-perturbation experiments are performed. The first two states are located at horizontal positions shown by meas1 and meas2 in Fig. 2a and marked by red asterisks, in the sixth layer above the summit, denoted as and , respectively. The other two are located at the same pixels as the first two, but now in the seventh layer above the summit, denoted by and accordingly. The horizontal locations of the perturbed states are chosen such that they are downwind and close to the summit and thus carry more information about the parameters than those located upwind or farther away. The vertical layers are chosen to be the injection layers of the truth and the “background” where the injection height is located, since states at those two layers play important roles in this parameter estimation process. The gradient criterion results given in Table 2 show the values are all larger than 1. This implies that the observed sensitivity behavior is not able to represent the model dynamics.

Table 2.

Gradient criterion values computed from four single-state perturbation experiments.

Table 2.

Then, perturbations of a single parameter are carried out, and the model gradient and the observed gradient are computed. The perturbed parameters are selected to be the inputs at the sixth and seventh layers—the injection layers in the background and truth, respectively. The gradient criterion results in Table 3 all show large values around 1. This implies that SICs have considerable influence on the numerical process for the update of the parameters, and the perturbed parameters cannot be determined accurately using this kind of error statistics. This is also reflected by the estimation results in Fig. 3b, where the injection layer cannot be identified by assimilating the ash columns using a standard 4D-Var approach.

Table 3.

Gradient criterion values computed from two single-input-perturbation experiments.

Table 3.

To diagnose how the SICs affect the sensitivity behavior, the normalized model gradients and observed gradients of the parameter perturbation experiments are shown in Fig. 4. We can see that as a result of the model dynamics, an input variable is sensitive to its own perturbation and slightly sensitive to the inputs in the near layers. SICs are introduced by using column-integrated data, making a single input variable almost equally sensitive to all inputs and even slightly more sensitive to the variable in other layers.

Fig. 4.
Fig. 4.

Normalized gradients generated by perturbations in a single input parameter at the (a) sixth and (b) seventh layers above the summit, with a 1-h assimilation window using Std4DVar.

Citation: Monthly Weather Review 145, 6; 10.1175/MWR-D-16-0289.1

c. Twin experiments using trajectory-based 4D-Var

Based on the sensitivity analysis in section 3b, Trj4DVar (see section 2d) should be applied to perturb the emission rate in each layer one by one and then to compute the corresponding trajectories to obtain a better estimate using the ash column data. In this experiment, we will demonstrate how the two criteria are used for the configuration of the assimilation system and for a sensitivity analysis to better understand the estimation results.

The FIM criterion and the gradient criterion in Eq. (15) are applied for the selection between Trj4DVar and standard 4D-Var (Std4DVar), as well as the selection of a proper assimilation window. The criteria values are shown in Fig. 5, where std represents Std4DVar, trj represents Trj4DVar, FIM represents the FIM criterion, and grd represents the gradient criterion. It can be seen that both approaches result in criteria values that decrease with larger assimilation windows. Using Std4DVar, this decease becomes smaller. This indicates that enlarging the assimilation window will introduce fewer improvements in the estimates. This result is consistent with the experimental results in section 3b. On the other hand, the criteria values obtained using Trj4DVar are clearly smaller and they decrease faster than those obtained using Std4DVar. Based on the diagnosis of the criteria results, we can see that Trj4DVar is a better choice for this application.

Fig. 5.
Fig. 5.

Criteria values of Std4DVar vs Trj4DVar.

Citation: Monthly Weather Review 145, 6; 10.1175/MWR-D-16-0289.1

Assimilation windows larger than 3 h lead to criteria values that are acceptable (<0.6) for our case mentioned in section 2c. The 6-h assimilation window produces the smallest values and thus is the best option. Therefore, the assimilation is conducted using a 6-h window to test the performance of Trj4DVar. Figure 6 shows a comparison between the estimation results using Std4DVar and those using Trj4DVar. Both approaches are carried out using the same prior information and synthetic observations. The vertical profile of the estimate is significantly improved using Trj4DVar. The injection layer is correctly determined in the seventh layer. However, the emission rates in the seventh through ninth layers are almost the same. Parameter-perturbation experiments are conducted to illustrate the reason behind this. The normalized gradients of individually perturbing the sixth- and seventh-layer inputs are illustrated in Fig. 7. We can observe that the seventh- through ninth-layer inputs are equally correlated. It is because meteorological patterns in the seventh through ninth layers above the summit are similar and changes occurring in any of the three layers are not distinguishable.

Fig. 6.
Fig. 6.

Comparison of estimation results with a 6-h assimilation window using Std4DVar vs using Trj4DVar.

Citation: Monthly Weather Review 145, 6; 10.1175/MWR-D-16-0289.1

Fig. 7.
Fig. 7.

Normalized gradients generated by perturbations in a single input parameter at the (a) fifth and (b) seventh layers, with a 6-h-assimilation window using Trj4DVar.

Citation: Monthly Weather Review 145, 6; 10.1175/MWR-D-16-0289.1

Note that this study aims at evaluating the numerical aspects or the robustness of applying the given observation operator to a specific configuration (including a statistic choice for the method) of an assimilation system, not the observability of a specific dataset (real data). Actually, the two criteria can be used as an indication for the quality of the performance when using real data. This can be confirmed by the field data experiment in Lu et al. (2016b), which shows that a 6-h assimilation window leads to the optimal and robust assimilation results.

4. Conclusions and discussion

In this study two criteria were presented to evaluate the numerical performance of gradient-based parameter estimation algorithms for a given type of remote sensing observations. The first criterion (FIM) was constructed to provide global information on how numerically robust an assimilation process is and how accurate the assimilation results will be. The second criterion can provide local and detailed information about sensitivity behavior. This can be used to diagnose what went wrong when poor estimates were obtained. Twin experiments were carried out to validate the criteria and to illustrate how the criteria can be applied in practice for multiple purposes.

These two criteria indicate the estimation quality and the forecast quality after assimilation. They can be used for the design and configuration of an assimilation system that will benefit the most from a given dataset. Configurations include the selection of data when a huge amount of data is obtained, the selection of the assimilation algorithm, and the configuration of the assimilation system such as the length of the assimilation window. Furthermore, the criteria are also recommended as a diagnostic tool for sensitivity analyses, which provides the possibility of seeking alternative methods when the use of the traditional methods is problematic as a result of improper statistical choices for making use of the SICs.

It should be noted that the two criteria are necessary but not sufficient conditions for quantifying the numerical robustness of the procedure for assimilating the remote sensing data. The benefits are that they are simple to implement and the results can be easy to understand when they are used as diagnostic tools. They can be used in OSSEs (twin experiments) where “complete observations” exist. Twin experiments performed in this paper could be regarded as OSSEs. For real data, the two criteria can be used in combination with other verification scores for forecasts. For instance, the criteria can be performed first to test the potential impact of assimilating a new type of observation on a certain application; then, verification scores can be used to quantify the quality of forecasts after assimilating real data, and, finally, the criteria can be used as a diagnostic tool for sensitivity analysis if poor results are found when assimilating real data.

Acknowledgments

We are very grateful to the editor and reviewers for their reviews and insightful comments.

REFERENCES

  • Atlas, R., 1997: Atmospheric observations and experiments to assess their usefulness in data assimilation. J. Meteor. Soc. Japan, 75, 111130.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bocquet, M., and et al. , 2015: Data assimilation in atmospheric chemistry models: Current status and future prospects for coupled chemistry meteorology models. Atmos. Chem. Phys., 15, 53255358, doi:10.5194/acp-15-5325-2015.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Chai, T., G. R. Carmichael, Y. Tang, A. Sandu, A. Heckel, A. Richter, and J. P. Burrows, 2009: Regional NOx emission inversion through a four-dimensional variational approach using SCIAMACHY tropospheric NO2 column observations. Atmos. Environ., 43, 50465055, doi:10.1016/j.atmosenv.2009.06.052.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Clemitshaw, K., 2004: A review of instrumentation and measurement techniques for ground-based and airborne field studies of gas-phase tropospheric chemistry. Crit. Rev. Environ. Sci. Technol., 34, 1108, doi:10.1080/10643380490265117.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Ebert, E. E., 2008: Fuzzy verification of high-resolution gridded forecasts: A review and proposed framework. Meteor. Appl., 15, 5164, doi:10.1002/met.25.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Elbern, H., and H. Schmidt, 2001: Ozone episode analysis by four-dimensional variational chemistry data assimilation. J. Geophys. Res., 106, 35693590, doi:10.1029/2000JD900448.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Elbern, H., A. Strunk, H. Schmidt, and O. Talagrand, 2007: Emission rate and chemical state estimation by 4-dimensional variational inversion. Atmos. Chem. Phys., 7, 37493769, doi:10.5194/acp-7-3749-2007.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fu, G., H. X. Lin, A. W. Heemink, A. J. Segers, S. Lu, and T. Palsson, 2015: Assimilating aircraft-based measurements to improve forecast accuracy of volcanic ash transport. Atmos. Environ., 115, 170184, doi:10.1016/j.atmosenv.2015.05.061.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fu, G., A. Heemink, S. Lu, A. Segers, K. Weber, and H.-X. Lin, 2016: Model-based aviation advice on distal volcanic ash clouds by assimilating aircraft in situ measurements. Atmos. Chem. Phys., 16, 91899200, doi:10.5194/acp-16-9189-2016.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fu, G., F. Prata, H. X. Lin, A. Heemink, A. Segers, and S. Lu, 2017: Data assimilation for volcanic ash plumes using a satellite observational operator: A case study on the 2010 Eyjafjallajökull volcanic eruption. Atmos. Chem. Phys., 17, 11871205, doi:10.5194/acp-17-1187-2017.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gilleland, E., D. Ahijevych, B. G. Brown, B. Casati, and E. E. Ebert, 2009: Intercomparison of spatial forecast verification methods. Wea. Forecasting, 24, 14161430, doi:10.1175/2009WAF2222269.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gilleland, E., D. A. Ahijevych, B. G. Brown, and E. E. Ebert, 2010: Verifying forecasts spatially. Bull. Amer. Meteor. Soc., 91, 13651373, doi:10.1175/2010BAMS2819.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Haben, S. A., A. S. Lawless, and N. K. Nichols, 2011a: Conditioning and preconditioning of the variational data assimilation problem. Comput. Fluids, 46, 252256, doi:10.1016/j.compfluid.2010.11.025.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Haben, S. A., A. S. Lawless, and N. K. Nichols, 2011b: Conditioning of incremental variational data assimilation, with application to the Met Office system. Tellus, 63A, 782792, doi:10.1111/j.1600-0870.2011.00527.x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Huneeus, N., F. Chevallier, and O. Boucher, 2012: Estimating aerosol emissions by assimilating observed aerosol optical depth in a global aerosol model. Atmos. Chem. Phys., 12, 45854606, doi:10.5194/acp-12-4585-2012.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Jacquez, J. A., and P. Greif, 1985: Numerical parameter identifiability and estimability: Integrating identifiability, estimability, and optimal sampling design. Math. Biosci., 77, 201227, doi:10.1016/0025-5564(85)90098-7.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kawabata, T., H. Iwai, H. Seko, Y. Shoji, K. Saito, S. Ishii, and K. Mizutani, 2014: Cloud-resolving 4D-Var assimilation of Doppler wind lidar data on a meso-gamma-scale convective system. Mon. Wea. Rev., 142, 44844498, doi:10.1175/MWR-D-13-00362.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lahoz, W., B. Khattatov, and R. Menard, 2010: Data Assimilation: Making Sense of Observations. 1st ed. Springer-Verlag, 718 pp., doi:10.1007/978-3-540-74703-1.

    • Crossref
    • Export Citation
  • Lamsal, L. N., and et al. , 2011: Application of satellite observations for timely updates to global anthropogenic NOx emission inventories. Geophys. Res. Lett., 38, L05810, doi:10.1029/2010GL046476.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lorenc, A. C., N. E. Bowler, A. M. Clayton, S. R. Pring, and D. Fairbairn, 2015: Comparison of hybrid-4DEnVar and hybrid-4DVar data assimilation methods for global NWP. Mon. Wea. Rev., 143, 212229, doi:10.1175/MWR-D-14-00195.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lu, S., H. X. Lin, A. W. Heemink, G. Fu, and A. J. Segers, 2016a: Estimation of volcanic ash emissions using trajectory-based 4D-Var data assimilation. Mon. Wea. Rev., 144, 575589, doi:10.1175/MWR-D-15-0194.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lu, S., H. X. Lin, A. W. Heemink, A. J. Segers, and G. Fu, 2016b: Estimation of volcanic ash emissions through assimilating satellite data and ground-based observations. J. Geophys. Res. Atmos., 121, 10 97110 994, doi:10.1002/2016JD025131.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • McMurry, P., 2000: A review of atmospheric aerosol measurements. Atmos. Environ., 34, 19591999, doi:10.1016/S1352-2310(99)00455-0.

  • Meirink, J. F., P. Bergamaschi, and M. C. Krol, 2008: Four-dimensional variational data assimilation for inverse modelling of atmospheric methane emissions: Method and comparison with synthesis inversion. Atmos. Chem. Phys., 8, 63416353, doi:10.5194/acp-8-6341-2008.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mittermaier, M., and N. Roberts, 2010: Intercomparison of spatial forecast verification methods: Identifying skillful spatial scales using the fractions skill score. Wea. Forecasting, 25, 343354, doi:10.1175/2009WAF2222260.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Paulino, C., and C. de Bragança Pereira, 1994: On identifiability of parametric statistical models. J. Ital. Stat. Soc., 3, 125151, doi:10.1007/BF02589044.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Prata, A. J., and A. T. Prata, 2012: Eyjafjallajökull volcanic ash concentrations determined using Spin Enhanced Visible and Infrared Imager measurements. J. Geophys. Res., 117, D00U23, doi:10.1029/2011jd016800.

    • Search Google Scholar
    • Export Citation
  • Robert, C., S. Durbiano, E. Blayo, J. Verron, J. Blum, and F. X. Le Dimet, 2005: A reduced-order strategy for 4D-Var data assimilation. J. Mar. Syst., 57, 7082, doi:10.1016/j.jmarsys.2005.04.003.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Rothenberg, T. J., 1971: Identification in parametric models. Econometrica, 39, 577591, doi:10.2307/1913267.

  • Talagrand, O., and P. Courtier, 1987: Variational assimilation of meteorological observations with the adjoint vorticity equation. I: Theory. Quart. J. Roy. Meteor. Soc., 113, 13111328, doi:10.1002/qj.49711347812.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wang, Y., and et al. , 2014: Assimilation of lidar signals: Application to aerosol forecasting in the western Mediterranean basin. Atmos. Chem. Phys., 14, 12 03112 053, doi:10.5194/acp-14-12031-2014.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Webley, P. W., T. Steensen, M. Stuefer, G. Grell, S. Freitas, and M. Pavolonis, 2012: Analyzing the Eyjafjallajökull 2010 eruption using satellite remote sensing, lidar and WRF-Chem dispersion and tracking model. J. Geophys. Res., 117, D00U26, doi:10.1029/2011JD016817.

    • Search Google Scholar
    • Export Citation
Save