a. En4DVAR basic algorithm

Variational data assimilation systems typically use the incremental approach (Courtier et al. 1994) and preconditioning technique (Gilbert and Lemarechal 1989). The cost function in control variable w space is

View Expanded

where w is the control variable, I is the total number of time levels on which observations are available, 𝗛 is the tangent linear observation operator, 𝗠 is the tangent linear forecast model, and 𝗢 is the observation error covariance. The innovations at different times (with subscript i) are calculated using

View Expanded

where 𝘅_b is the background state matrix, in which each column represents one member of the background state vector, H is the observation operator, M is the forecast model, and y is the observation vector. The precondition matrix, 𝗨, is defined by

View Expanded

The final analysis is obtained from

View Expanded

The idea behind En4DVAR is that the preconditioning matrix in the incremental approach of 4DVAR is replaced with a perturbation matrix. The columns of the perturbation matrix, 𝗫′_b, are the normalized deviations from the ensemble mean and are estimated by N ensemble members, namely,

View Expanded

The background error covariance 𝗕 is approximated by

View Expanded

The En4DVAR cost function in control variable space is defined by

View Expanded

To avoid tangent linear and adjoint models in calculating the gradient of cost function, we transform the perturbation matrix to observation space via

View Expanded

The gradient of the cost function is then calculated using

View Expanded

After minimization iteration, the optimal analysis 𝘅_a can be obtained from

View Expanded

The dimension of control vector w in En4DVAR is the same as the ensemble size instead of the analysis vector dimension in 4DVAR. The computational cost of En4DVAR is much less than 4DVAR because the ensemble size is far less than the analysis vector dimension. However, as discussed in section 1, reducing the dimension of control vector w causes the problem to be undetermined and analysis noise is produced. We use the localization technique to solve this problem.

b. Horizontal and vertical localization in En4DVAR

To reduce sampling errors caused by a finite number of ensemble members, Houtekamer and Mitchell (2001) employed the Schur operator (Gaspari and Cohn 1999) in EnKF, whereas Lorenc (2003) and Buehner (2005) used the Schur operator in the ensemble-based variational scheme. In En4DVAR, we introduce an EOF decomposed correlation function operator to modify the perturbation matrix 𝗫′_b. This is an approach similar to the spatial localization of Buehner (2005). Using the mathematical proofs in appendix A, we obtain the modified perturbation 𝗣′_b, which is defined by

View Expanded

In (11), the subscripts h and υ are horizontal and vertical indices, respectively. Subscript number 1 means that the first column replaces all columns of matrix. Here 𝗘 contains all of the eigenvectors and λ is a diagonal matrix with its diagonal elements corresponding to eigenvalues obtained from an EOF decomposition of the correlation function 𝗖:

View Expanded

In defining 𝗖 in (12), we use the compactly supported second-order autoregressive function as horizontal correlation model (Liu and Rabier 2003):

View Expanded

where s is the spherical separation in degree between two data points, and s₀ and s₁ are the correlation scale and the cutoff distance beyond which the correlation becomes zero. Following Zhang et al. (2004), we perform vertical localization using the correlation function,

View Expanded

where Δ log p is the distance between two vertical levels in log p space, and p is pressure.

The new perturbation matrix in (11) in En4DVAR is equivalent to the modified 𝗕 matrix by correlation function [e.g., the Schur product operator, in the EnKF (see appendix A)]:

View Expanded

The correlation function using (13) makes the correlation equal to zero beyond the cutoff distance. It can smooth the correlation within the cutoff distance as well. After localization, the perturbation matrix 𝗫′_b in (7) is replaced by 𝗣′_b, and the new cost function can be rewritten as

View Expanded

If all the eigenvectors and eigenvalues are used in (11), 𝗣′_b will be a matrix with n rows and n × n × N columns (n is the analysis vector dimension). The dimension of the control vector will be increased to n × n × N. But if a few leading modes can represent the correlation function very well, the dimension of the control vector is reduced to r_h × r_υ × N, in which r_h is the number of the horizontal eigenvector truncation mode and r_υ is the number of vertical eigenvector truncation mode. This reduction of dimension can greatly reduce the En4DVAR computational cost in real-dimension data assimilation.

c. Analysis time tuning in En4DVAR

In EnKF analysis, the perturbation matrix is calculated at the analysis time. The analysis noise comes from the spatially spurious correlation between the two grid points. The further the two sets of perturbations between different grid points, the noisier the analysis (Houtekamer and Mitchell 1998). In contrast to EnKF, En4DVAR needs not only the perturbation matrix at analysis time, but also all perturbation matrices at observation times. Therefore, the analysis noise can come from both the spatially spurious correlations and the temporally spurious correlations in the perturbations.

Adding the time subscript index to (7), the cost function can be rewritten as

View Expanded

For simplicity, we use the En4DVAR analysis with only a single time observation to illustrate the problem. With only one time observation, the mathematical expression for the analysis derived from (17) becomes

View Expanded

If the observation and analysis times are identical, (18) is the same as the equation in the EnKF analysis and the 𝗫′_b0(𝗛₁𝗠_0∼1𝗫′_b0)^T term becomes 𝗫′_b0(𝗛₀𝗫′_b0)^T. At large distances from observation points, the correlation element between 𝗫′_b0 and 𝗛₀𝗫′_b0 should be close to zero. However, some spurious correlations may appear due to the limited ensemble members used in calculating the perturbation matrix. These spurious correlations will make the analysis noisy and contaminate the analysis quality. Similarly, when the observation time is significantly different from the analysis time, 𝗫′_b0(𝗛₁𝗠_0∼1𝗫′_b0)^T should be close to a zero matrix because the correlation between 𝗫′_b0 and 𝗛₁𝗠_0∼1𝗫′_b0 is very small. Limited ensemble members could lead to temporal analysis noise resulting from the observation times departing significantly from the analysis time.

To reduce temporal analysis noise, the En4DVAR cost function is modified to allow an analysis time as close as possible to the time of most observations, instead of the beginning time of the data assimilation window. The cost function can be reformulated as

View Expanded

In (19), the subscript α indicates any time within assimilation window. If the analysis time is not at the beginning of assimilation window, the backward tangent linear model and forward adjoint model are needed for calculating the gradient of 4DVAR. Appendix B shows that the perturbation matrix calculation in (19) is the same as in (17). Therefore, En4DVAR can be implemented at any analysis time within the assimilation window and does not require the use of the backward tangent linear model and the forward adjoint model.

a. Synoptic overview

During 24–25 January 2000, a major winter storm, termed the “Blizzard of 2000” (Zupanski et al. 2002), affected most of the east coast of the United States. Heavy rain, mixed with snow and freezing rain fell in North and South Carolina, producing an enormous natural hazard. In the Washington, D.C., area, the heavy snow blocked most roads and causing business closures and significant disruption of commerce. The operational model failed to predict strong convection and precipitation for the snowstorm case.

A broad synoptic trough moved over the eastern United States during 24–25 January 2000. At 1200 UTC 24 January, an associated surface low appeared over north Florida. The surface low then moved near the Washington, D.C., area along the eastern coast during the next 24 h. After 0000 UTC 25 January, the convection developed rapidly and heavy snowfalls were produced from the Carolinas through the Washington, D.C., area. The minimum sea level pressure associated with the cyclone showed a rapid drop of 22 hPa from 1200 UTC 24 January to 1200 UTC 25 January (Zhang et al. 2002). This case has been analyzed in conjunction with a variety of topics including mesoscale predictability (Zhang et al. 2002), dynamics and structure (Zhang 2005), 4DVAR experiments (Zupanski et al. 2002), and EnKF performance for mesoscale data assimilation (Zhang et al. 2006; Meng and Zhang 2007).

b. Experimental design

The ARW-WRF (Skamarock et al. 2005) was used for this study. All experiments were conducted over a grid mesh of 94 × 94 with grid spacing of 27 km. The domain covers the eastern half of the United States. In the vertical, there are 27 η layers, and the model top is 50 hPa. The physical processes in all experiments include the Rapid Radiative Transfer Model (RRTM) based on Mlawer et al. (1997) for longwave radiation, the Dudhia (1989) shortwave radiation scheme, the Yonsei University (YSU) PBL scheme (Hong et al. 2006), and the Kain–Fritsch (new Eta) scheme (Kain 2004). The National Centers for Environmental Prediction (NCEP) Global Forecast System (GFS) analysis provided the first-guess fields and boundary conditions.

The control run (CTRL) is integrated for 42 h and is initialized using the first-guess fields at 1800 UTC 23 January 2000. Random perturbations are added to the first guess of CTRL to produce 37 ensemble members. The ensemble member that compares most favorably to observations in terms of the location and strength of the cyclone is chosen as the truth simulation, while the other members constitute the reference forecast ensemble. The random perturbations are derived from the background error covariance of the WRF 3DVAR data assimilation system using an approach similar to the initial ensemble generating method in Houtekamer and Mitchell (2005). Therefore, the perturbations are consistent with the background error covariance defined by the WRF 3DVAR data assimilation. We also perturbed boundary conditions using the Data Assimilation Research Test bed (DART) system (Anderson 2001, 2003).

Figures 1a–c show the sea level pressure (SLP) and 1000-hPa wind vectors from the truth simulation at 12-h intervals from 1200 UTC 24 January to 1200 UTC 25 January 2000. The corresponding potential vorticity (PV), geopotential heights, and wind vectors at 300 hPa are shown in Figs. 1d–f. At 1200 UTC 24 January (Fig. 1a), the location and intensity of the simulated cyclone are similar to the analysis in Zupanski et al. (2002). But 12 hours later (Fig. 1b), the strength of the simulated cyclone was a little weaker than the GFS analysis (Zupanski et al. 2002). At 1200 UTC 25 January (Fig. 1c), the intensity of the simulated cyclone is over 8 hPa higher than in the analysis, and its location is east of the analyzed cyclone location.

To conduct En4DVAR OSSE, we generated the simulated observations from 0900 UTC 24 January to 1200 UTC 25 January 2000 from the truth simulation with random normal perturbations containing variances identical to the observation errors defined in WRF-VAR. Four types of observations [sounding, surface observation, satellite-retrieved winds, and Quick Scatterometer (QuikSCAT) winds] were tested in this study. The simulated observations are located in the site of real observations but their values are replaced by the simulated values. Following the EnKF implementation in Houtekamer and Mitchell (1998), these observations are perturbed first, and then assimilated to update the ensemble forecast fields in the En4DVAR cycling procedure.

Figure 2 shows the temporal distributions of different observations. We designed an assimilation window of 6 h for each update analysis, which resulted in four windows used in the En4DVAR cycling for the OSSEs from 0900 UTC 24 to 1200 UTC 25 January. Figure 3 describes the flowchart of En4DVAR procedure used in this OSSE. The deterministic forecast is updated by observations and the ensemble forecast is updated by perturbed observations, which provides the perturbation matrices for En4DVAR. We chose u and υ wind components, temperature, and humidity as the En4DVAR analysis variables. We use the deterministic forecast as background field in the En4DVAR. The ensemble mean is used in calculating the perturbation matrix. One of the main purposes of this study is to examine the effect of the flow-dependent 𝗕 matrix in the 4DVAR. If ensemble mean is used as the background field, it is hard to know whether the improvement in the En4DVAR analysis comes from the flow-dependent 𝗕 matrix or the ensemble forecast.

c. Test of the En4DVAR localization technique

To perform localization, we introduced the correlation function operator in the 𝗕 matrix. Since the effects of observations propagate in time, ideally, a flow-dependent localization should be used. However, the correlation function operator we currently have in the WRF En4DVAR does not have this feature because of the complexity of implementation and computational costs. Because the perturbation matrix 𝗫′_b is estimated by the model ensemble forecast and the correlation operator is fixed in time, the analysis may lose some information from observations stretched far from the analysis time. We will investigate this problem and develop the new perturbation matrix 𝗣′_b, which contains flow-dependent location in the future.

When the correlation function operator is applied to En4DVAR, the control vectors are enlarged and the computational cost becomes very expensive. To reduce the computational cost, EOFs are used to decompose the correlation function and limited truncation modes are selected. Table 1 shows the cumulative relative RMSE corresponding to different horizontal and vertical truncation modes, which is defined as

View Expanded

where G is the cumulative relative RMSE, m is the selected truncation mode number, m_t is the total truncation modes number, and λ is the eigenvalue. In this study, we selected 36 truncation modes in the horizontal and 10 in the vertical. As shown in Table 1, both the horizontal and vertical cumulative relative RMSEs are over 90% with these selections. Although the truncation makes the 𝗕 matrix not full rank, the rank is still comparable to the number of observations in this experiment.

We employed a single observation test to examine the En4DVAR localization technique in this study. A single temperature observation at 33.5°N, 78.4°W and 850 hPa is assimilated using WRF 3DVAR, En4DVAR without localization (En4DVAR-NL), and En4DVAR with localization (En4DVAR-L). Resulting wind vector and temperature increments at 1000 hPa are shown in Fig. 4. The temperature analysis increment in WRF 3DVAR (Fig. 4a) indicates a homogeneous and isotropic structure that the 𝗕 matrix of WRF 3DVAR has. Because of the balance constraint of quasigeostrophy in the variable transform of WRF 3DVAR, the wind analysis increments show quasigeostrophic characteristics in the single temperature observation assimilation experiment. On the other hand, Figs. 4b,c demonstrate the flow-dependent 𝗕 matrix structure in the En4DVAR. The temperature analysis increments in En4DVAR extend along the eastern coast, corresponding to the flow direction (Fig. 1a) and isotherms (not shown). In contrast to the variable transform in WRF 3DVAR, the relations among different physical variables in the En4DVAR depend on ensemble statistics. The analysis increments in the En4DVAR (Figs. 4b,c) show characteristics presented by ensemble statistics.

If no localization is performed in the En4DVAR (Fig. 4b), a lot of increment noise is found because of sampling errors. This has been discussed in section 1. The amplitude of such noise can be comparable to the increment signal at observation locations. Using localization by the EOF decomposed correlation function operator, the noise almost disappears but the increment signal from the observation is still maintained (Fig. 4c). Figure 5 shows the vertical profiles of the temperature analysis increments at the observation location obtained using En4DVAR-NL and En4DVAR-L. At high levels, there is an obvious spurious analysis increment if the localization is not applied, but the noise is filtered out after localization. Although noise can be filtered out by the localization technique, the analysis increment signal is also a little reduced compared to the one without localization because limited modes are used. However, the major analysis feature is retained since over 90% of the signal is still maintained.

d. Analysis time tuning in En4DVAR

To illustrate the impact of temporal sampling errors in the En4DVAR, we calculate the horizontal mean absolute correlation coefficient of the background perturbation at the observation level (the 8th η level) at different forecast times (Fig. 6). It is shown that the perturbation correlation between different times decreases as the forecast time increases. This means that an observation far from the analysis time will have little impact on the analysis. To evaluate the correlation sensitivity to ensemble size, the correlation coefficients with different ensemble sizes are also calculated. It shows that the correlation difference between different ensemble sizes increases with the time. That is, a smaller correlation needs more samples to be estimated, while observation information far from the analysis time needs more ensembles to be extracted.

Comparing the correlation coefficients among different variables, the time scale of the humidity error temporal correlation is the shortest; while that of the winds is the longest. This indicates that the humidity analysis is more sensitive to temporal sampling errors in the En4DVAR. In the unstable flow region, the perturbations usually develop rapidly so that the temporal correlations among perturbations are less. Figure 6, which shows the temporal correlation coefficient among perturbations at the same single observation location in Fig. 4 near the cyclone, illustrates this point. The scale length of temporal correlation is less than that in the horizontal mean. In particular, note that the humidity perturbation correlation coefficient is less than 0.2 after t = 2 h. This indicates that the humidity observation will not provide significant information to the analysis if the observation time differs from the analysis time by 2 h or more.

To evaluate the sensitivity of En4DVAR to analysis time, we choose 0900, 1200, and 1500 UTC 24 January as En4DVAR analysis times, and denote these three experiments as En4D-09, En4D-12, and En4D-15, respectively. The RMSEs obtained from integrating the En4D-09 and En4D-12 analysis fields to 1500 UTC 24 January are compared in Fig. 7. Due to sampling errors, the En4DVAR analyses are very sensitive to the analysis time even within the 6-h assimilation window. Comparing all RMSEs, it is obvious that En4D-12 provides the best analysis because its analysis time is set closest to the majority of observations; and its temporal analysis noise is smallest. Except for humidity, the RMSEs in En4D-09 are larger than in En4D-15 (Figs. 7a–c) because En4D-09 sampling errors quickly increase in a 6-h forecast. The spatial and temporal correlation scales in humidity variable errors are usually smaller than in other variables. Therefore, the humidity analysis is more sensitive to sampling errors. Because the humidity observations are at 0900 (surface observation) and 1200 UTC (sounding) 24 January, the humidity analysis in En4D-15 is obviously affected by temporal sampling errors and produces an even worse analysis than the one in En4D-09 (Fig. 7d).

e. En4DVAR OSSE results

First, we examine the minimization efficiency of WRF En4DVAR. Figure 8 shows the En4DVAR cost function and its gradient for the first analysis time (1200 UTC 24 January). In this experiment, the En4DVAR cost function reaches the WRF-VAR minimization convergence standard after 38 iterations. It indicates that the minimization of WRF En4DVAR converges efficiently. Table 2 compares A − O (analysis minus observation) with B − O (background minus observation) at analysis times, indicating a closer agreement of analysis fields than background fields with observations.

Figure 9 shows the CTRL forecast errors (forecast minus truth) at 1200 UTC 24 January and 0000 and 1200 UTC 25 January (similar to Fig. 1, but showing the errors between the forecast and the truth). Before the winter storm occurs, the forecast error has small amplitude even after the 18-h forecast from 1800 UTC 23 (Figs. 9a,d). When the storm develops, the forecast error rapidly increases near the storm region (Figs. 9b,e). With the En4DVAR OSSE configuration described in section 2 the analysis error in Fig. 10 is similar to the forecast error in Fig. 9. It is obvious that the En4DVAR analysis errors are less than the background errors. The greatest analysis errors of En4DVAR are mainly over the western Atlantic because the background error develops most rapidly near the cyclone, and the limited synoptic observations in the region are not enough to correct the error.

Figure 11 shows the vertical profiles of domain-averaged RMSEs in the 6-h forecasts by CTRL and En4DVAR, as well as in the En4DVAR analyses at 1200 UTC 24 January, 0000 UTC 25 January, and 1200 UTC 25 January. The RMSE in CTRL increases several times after 12-h forecast from 1200 UTC 24 January with the error developing rapidly after the onset of the winter storm. After 0000 UTC 25 January, most errors have reached saturation and no longer increase. But the errors of V-wind component and humidity in CTRL at lower levels continue to increase because the V-wind component and humidity still change significantly after 0000 UTC 25 January (Figs. 1b,c). In contrast to CTRL, the forecast error in En4DVAR decreases after the first analysis and maintains small amplitude during 24-h forecast–analysis cycling. Overall, the En4DVAR analysis error is always smaller than the background error. However, the error reduction in the En4DVAR analysis compared with background decreases in higher levels than in lower levels. This is because only rawinsonde observations exist at high levels and observations from lower levels have marginal impacts on the high-level analysis. Figure 12 shows the domain-averaged analysis bias in the CTRL and En4DVAR experiments from 1200 UTC 24 January to 1200 UTC 25 January. The biases of winds in CTRL (Figs. 12a,b) are mainly in the upper levels (around the15th level) and the low levels (below the 5th level), which is similar to RMSE vertical profile in Fig. 11. It suggests that CTRL has a systematic error in predicting the upper-level trough and the cyclone. But after En4DVAR analysis, the upper-level biases are reduced significantly and the low-level biases are near zero. We suspect that the observations of QuikSCAT winds have a positive impact on the En4DVAR analysis.

f. Comparison of En4DVAR and En3DVAR cycling

En4DVAR is an ensemble-based retrospective algorithm. If no sampling errors are considered, the En4DVAR analysis fits to the optimal trajectory. Unlike the ensemble-based sequential algorithm, it does not necessarily fit observations at individual points. The temporal sampling errors can affect En4DVAR analysis as discussed previously in this paper. To compare the ensemble-based sequential and ensemble-based retrospective algorithms, we designed an En3DVAR cycling experiment that uses the same configuration of En4DVAR but assimilates only one-time observation and cycles all the observations in the assimilation window.

Figure 13 shows the time variations of the domain-averaged RMSE in the analyses of CTRL, En3DVAR cycling, and En4DVAR. It is clear that the error in CTRL rises suddenly after 1800 UTC 24 January and becomes stable after 2100 UTC 24 January, but the humidity error still increases slowly until 0600 UTC 25 January. CTRL hardly predicts the cyclone development from 1200 UTC 24 January resulting in the most significant errors near the cyclone location (Fig. 9). When En3DVAR or En4DVAR analysis cycling is applied, the forecast–analysis errors are far less than those of CTRL. Meanwhile, it is found that the overall performance of En4DVAR is better than that of En3DVAR, indicating that the ensemble-based retrospective algorithm is more robust than the ensemble-based sequential algorithm. The time variation of forecast–analysis spread statistics in En4DVAR is also plotted in Fig. 13. During the first two cycles, it is obvious that the perturbation spread is larger than the RMSE. After the third cycle, however, the perturbation spread is a little less than the RMSE, which means the filter divergence also exists in En4DVAR. The filter divergence seems more serious in the humidity analysis (Fig. 13d) because the humidity perturbations are confined to small amplitude in our experiments to prevent oversaturation or negative humidity. This suggests that a better humidity perturbation method should be explored in the future. Another reason for the humidity analysis divergence is that humidity spatial and temporal scales are smaller and more sensitive to sampling errors. Adding the inflation factor (Anderson and Anderson 1999) can relax the filter divergence problem. Figure 14 shows the vertical profiles of the domain-averaged RMSE in the En3DVAR and En4DVAR analyses at 1200 UTC 24 January, 0000 UTC 25 January, and 1200 UTC 25 January. At most altitudes, the RMSEs in En4DVAR are less than those in En3DVAR. However, sometimes the RMSE of the En4DVAR analysis are larger than those of the En3DVAR analysis at low levels because the scale length of the temporal error correlation is smaller, and En4DVAR analysis is seriously affected by the temporal sampling errors at low levels. The humidity analysis in En4DVAR is worse than that in En3DVAR in the first analysis. As discussed in section 3d, the humidity analysis is more sensitive to temporal sampling errors. After several forecast–analysis cycles, the humidity RMSE in the En4DVAR forecast–analysis is comparable to that of En3DVAR because the analysis of other variables in En4DVAR is better than for En3DVAR. Therefore, the final En4DVAR humidity analysis is also improved. Since both En4DVAR and En3DVAR use the flow-dependent 𝗕 matrix, we do not expect En4DVAR to result in a significant improvement over En3DVAR for the designed OSSEs in this study. First, the analysis of En4DVAR is affected by temporal sampling errors, which does not exist in En3DVAR. Second, as discussed in section 4c, the fixed localization is used in our experiment. Third, most observations are at the analysis time of En4DVAR, especially sounding observations; this reduces the benefits of En4DVAR compared with En3DVAR. However, we believe the results of En4DVAR can be further improved compared to those of En3DVAR if a flow-dependent localization technique is adopted and more nonsynoptic observations are used during assimilation cycles.