A direct piece-by-piece data assimilation targeting strategy through observing system simulation experiments was used to examine the quality of the target area for forecast metrics with different nonlinearities in a mesoscale convective vortex–associated rainfall event from both a deterministic and probabilistic perspective.
The target area was determined based on the impact of assimilating synthetic wind-profiler observations, piece by piece, on the forecast error of strongly nonlinear rainfall and weakly nonlinear total energy around the initial vortex center. The quality of the target area in terms of its effectiveness and variability was examined for members of a reasonable ensemble. Apparently different target areas were found for different members, even for those with very small differences for both forecast metrics, with a larger variability observed for rainfall than for total energy. This result indicated that target areas estimated in deterministic scenarios are likely unreliable.
Probabilistic target areas were created by averaging data-impact index values over the ensemble. Significant differences were also observed in the ensemble-based target areas for rainfall and total energy. For total energy, assimilating data in an inaccurate target area could decrease the forecast error at a similar magnitude as that in the target area. For rainfall, however, much less error reduction was obtained, the magnitude of which was almost comparable to the no-data-assimilation experiment. Overall, the results of this study suggest that designing a particular observation plan based on an estimated target area could be unnecessary for total energy and useless for rainfall, given the difficulty involved in accurately determining a target area in an operational setting.
Targeted observation, or adaptive observation, is a process that estimates areas where assimilated additional observations would substantially reduce forecast error. Targeted observation has been an active research topic since it was first proposed (Snyder 1996). Targeting strategies can be roughly grouped into two categories, including adjoint sensitivity or singular vector (e.g., Buizza and Palmer 1995; Langland and Rohaly 1996; Pu et al. 1997; Palmer et al. 1998; Gelaro et al. 2002; Coutinho et al. 2004; Wu et al. 2007b), and ensemble-based methods (e.g., Bishop and Toth 1999; Anderson 2001; Bishop et al. 2001; Majumdar et al. 2002a,b; Hamill and Snyder 2002; Ancell and Hakim 2007; Liu and Kalnay 2008). In addition to all the above linear-based methods, Mu et al. (2003) proposed a conditional nonlinear optimum perturbation (CNOP) method.
Shortly after the theory of targeted observation was proposed, a series of field campaigns were launched for midlatitude weather, such as the Fronts and Atlantic Storm Track Experiments (FASTEX; Joly et al. 1997) in 1997, the North Pacific Experiment (NORPEX; Langland 1999) in 1998, the Winter Storm Reconnaissance Program (WSR; Szunyogh et al. 2000) since 1999, and the North Atlantic The Observing System Research and Predictability Experiment (THORPEX) Regional Campaign (NA-TReC; Mansfield et al. 2005) in 2003. In addition, tropical cyclone targeting field programs were launched, such as the Dropwindsonde Observations for Typhoon Surveillance near the Taiwan Region (DOTSTAR; Wu et al. 2005) since 2003, the THORPEX Pacific Asian Regional Campaign (T-PARC; Aberson 2011) in 2008, and the North Atlantic Hurricane Surveillance program (Aberson 2003) since 1998. On average, the short-range forecast error has been reduced by 10% over regional verification areas, with some cases showing reductions of up to 50%. Forecast skill has been improved in about 70% of cases when additional data have been assimilated into identified target areas (Langland 2005). These numbers certainly show the positive contribution to reducing forecast error made by targeted observations. However, the improvements in forecast skill demonstrated in field experiments have been much lower than expected and more and more verification studies are showing insignificant or neutral impact of targeted observations (e.g., Aberson 2002; Wu et al. 2007a; Chou et al. 2011; Hamill et al. 2013). Consequently, it is important to revisit the limitations and issues involved in targeted observation.
To make targeted observation successful, three conditions have to be satisfied (Langland 2005): 1) the target area must be accurate, 2) the target area must be adequately sampled, and 3) the assimilation of targeted observations must be able to reduce the forecast error of the chosen metric. The accuracy of the target area, target area sampling, quality of the background field, observation type and representativeness, assimilation technique, and the numerical model can all influence the impact of targeted observation (Aberson 2002, 2003, 2008; Reynolds and Rosmond 2003; Langland 2005). Among these factors, the quality of the target area is one of the most important, and should be assessed by its effectiveness and variability. A target area can be regarded as “effective” if the error of the concerned forecast metric is decreased by a significantly larger magnitude through assimilating additional observations in the target area rather than anywhere else. In addition, from a probabilistic point of view, the variability of the target area obtained in equally possible situations needs to be low.
The quality of the target area is closely related to the skill of targeting strategies. A major issue in current targeting approaches that may compromise the quality of the target area is that all current targeting strategies except CNOP involve significant linear assumptions. However, an atmospheric model is a nonlinear system. Different degrees of linear assumption involved in targeting strategies may change the structure, size, or location of target areas, thus increasing its variability. For examples, Mu et al. (2009) found substantial differences when they compared the target areas produced by CNOP and first singular vectors (FSVs) for typhoon cases. A large variability in target area was observed among different linear-based targeting strategies (e.g., Wu et al. 2009; Ancell and Hakim 2007). Even for a single linear-based targeted observation strategy, the location and size of the target area could be very sensitive to the model resolution (Ancell and Mass 2006), model physics, and basic-state trajectory (Ancell and Mass 2008).
The variability of the target area would become even larger for a forecast metric with stronger nonlinearity relative to a weakly nonlinear forecast metric that has been used for almost all current targeting strategies. Synoptic-scale metrics may remain quasi-linear longer than at smaller scales (Reynolds and Rosmond 2003). Majumdar et al. (2002b) found that the total energy singular vector and ensemble transform Kalman filter approaches produce substantial differences in the target area for smaller-scale target features. Xie et al. (2013), with accumulated rainfall as the forecast metric for Typhoon Morakot (2009), clearly demonstrated the limitation of using ensemble-based sensitivity analysis in locating the target area, due to the strong nonlinearity in the governing dynamics of the typhoon. Consequently, the quality of the target area for forecast metrics with different nonlinearity needs to be examined using a strategy that fully accounts for the features of nonlinear error growth.
Another issue of all current strategies is that they estimate the analysis and forecast variance instead of the forecast error (Mu 2013). However, identifying an area that can decrease the forecast error variance the most does not necessarily result in the largest error reduction of the forecast metric concerned. It could be more intuitive and direct if the target area can be detected based on the ultimate goal of targeted observation (i.e., minimum forecast error) instead of minimum forecast variance.
In the present reported study, we used an error-based targeting strategy that did not possess a linear-error growth assumption to assess the quality of the target area for forecast metrics with different nonlinearity from both a deterministic and probabilistic perspective for a heavy-rain event associated with a mesoscale convective vortex (MCV) in south China in June 2008. The targeting strategy used was the one most commonly used for assessing targeted observation; namely, data assimilation. We used “brute force” assimilation of different observations, piece by piece, through an observing system simulation experiment (OSSE) to quantify their fully nonlinear impact on the forecast metric and identify the target area where high data-impact index values were located.
This piece-by-piece data assimilation targeting method (PBPDA) has several benefits: 1) it does not require any linear assumption, 2) it is error—rather than variance—based, 3) any type and resolution of targeted data can be examined through the OSSE method, and 4) the targeting strategy is totally consistent with the data assimilation procedure (i.e., the detection of the target area is totally consistent with its evaluation, which has been regarded as important in targeted observations; Berliner et al. 1999; Langland 2005). Although the method might be difficult to use operationally due to its reliance on future observational data, our expectation was that it would be a good approach to examine the theoretical concept of targeted observation. We used an ensemble to examine the quality of the target areas for the forecast metrics with different nonlinearity among different members in the deterministic approach, and the quality of the ensemble-based target area from the statistical perspective.
The remainder of the paper is structured as follows. A synoptic overview of the MCV case is given in section 2. Section 3 introduces the PBPDA targeting strategy, the OSSE procedures, and the forecast metric calculation. The results of the control experiment (i.e., without data assimilation) are reported in section 4. Section 5 discusses the quality of the target area determined through PBPDA for strongly nonlinear rainfall and weakly nonlinear total energy (TE). And finally, a summary and discussion are presented in section 6.
2. Case overview
The weather event we studied was a torrential rain event in Guangxi Province, China (Fig. 1), on 12 June 2008, which was associated with an MCV that originated from Sichuan Province in southwest China. This kind of MCV is usually referred to as a “southwest vortex” (Tao and Ding 1981), which is a mesoscale cyclonic vortex (with a horizontal radius of 300–500 km) that forms in the lower troposphere over southwest China. The formation of a southwest vortex is defined by the appearance of a mesoscale cyclonic circulation and a closed isobar at 700 hPa (Lu and Chen 1993). Southwest vortices are usually caused by the interaction between the terrain of the Tibetan Plateau and the atmospheric circulation. They usually dissipate where they form, seldom moving away from their origin (Duan 2006). That said, some southwest vortices do move out to the east, and usually cause extensive and severe heavy rain along their tracks (Tao and Ding 1981; Lu and Chen 1993; Chen et al. 2003; Fu et al. 2010). Eastward-moving MCVs have emerged as key weather systems that cause torrential rain in summer over east China.
The southwest vortex in this study formed in south Sichuan Province at 0000 UTC 11 June 2008 (Fig. 1). It first moved to the southeast and then turned to the northeast. During its lifetime, the vortex center (the cross in Fig. 2) remained in front of or near to the base of a 500-hPa geopotential trough followed by a 500-hPa temperature trough (Fig. 2), which is a favorable environment for the development of upward motion. With the eastward movement of a short wave at 500 hPa from the Tibetan Plateau (Figs. 2a,b), a low pressure system with closed contours of geopotential height first appeared at 700 hPa over southern Sichuan Province at 1800 UTC 10 June (Fig. 3b). The streamline field showed a convergence center near the low pressure center with increased relative vertical vorticity, but no apparent cyclonic circulation at this moment at 700 hPa. The cyclonic circulation became apparent at 0000 UTC 11 June together with a closed isobar at 700 hPa (Fig. 3c), which featured the formation of the vortex. The vortex then intensified, expanded, matured at 1800 UTC 12 June over northern Guangxi Province (Figs. 3d,e), and dissipated over Jiangxi Province after 0000 UTC 13 June.
The MCV lasted for 2 days and caused severe damage to Guangxi Province. Almost half of Guangxi Province was covered by a 24-h rainfall event of >100 mm [shading in Fig. 1; the observed rainfall data were provided by the China Meteorological Administration (CMA)]. The 24-h accumulated rainfall during 0000 UTC 12 June–0000 UTC 13 June 2008 (hereafter referred to as RAIN24) reached as much as 250 mm in northern Guangxi (Fig. 1). Around 1.6 million people in 35 counties of Guangxi Province were affected by this torrential rain event (Fu et al. 2010).
a. The PBPDA method for targeted observation
A schematic representation of the PBPDA targeting strategy is depicted in Fig. 4. The Final Analysis (FNL) of the NCEP Global Forecast System (GFS) at 0000 UTC 11 June was perturbed to generate an initial ensemble. The whole ensemble was then integrated for 48 h. The member that best approximated the observed MCV flow and RAIN24 was used as the “truth” to generate synthetic observations by adding in typical observation error. Then, in a given area around the initial MCV of the truth, synthetic observations were grouped into small data units. By individually assimilating each data unit into the initial field of the other ensemble members using three-dimensional variational data assimilation (3DVar; Barker et al. 2004) followed by a 48-h simulation, the impact of assimilating each data unit on the forecast error of certain forecast metrics relative to the control (no data assimilation) experiment could be determined. 3DVar was used because of its high efficiency, ease of implementation, and generally good performance (Hsiao et al. 2012; Gao et al. 2013). The distribution of the data-impact index could then clearly show the possible target area. To make the analysis of the results easier, we divided the range of positive data-impact index values into four levels and set all negative values to zero. The target area of each member was then defined by the relatively localized and compact area with high data-impact index values.
The model used in this study was the Advanced Research core of the Weather Research and Forecasting Model (WRF), version 3.2 (Skamarock et al. 2008). The vertical coordinate follows the terrain using hydrostatic pressure, and the model uses an Arakawa C grid. Prognostic variables are column mass of dry air (μd), velocity (u, υ, and w), potential temperature (θ), geopotential (ϕ), and mixing ratios for water vapor (qυ), cloud water (qc), rain (qr), ice (qi), snow (qs), and graupel (qg). The model was run with two-way grid nesting. The coarse and fine-grid domains, respectively, have horizontal grid sizes of 40.5 km and 13.5 km, and grid points of 150 × 120 and 181 × 136 (Fig. 5). There are 27 layers in the vertical direction, with a top level at 100 hPa. Both domains use the WRF double-moment 6-class microphysics scheme (Hong et al. 2004), the Rapid Radiative Transfer Model with GCM applications (RRTMG) longwave radiation scheme (Mlawer et al. 1997), the Yonsei University planetary boundary layer scheme (Hong et al. 2006), and the Grell–Devenyi ensemble cumulus parameterization scheme (Grell and Devenyi 2002). The initial and boundary conditions were provided by the FNL/NCEP dataset, which has a time interval of 6 h and a horizontal resolution of 1° × 1°.
b. OSSE configuration
A 48-h ensemble forecast was first performed with an ensemble size of 40 from 0000 UTC 11 June to 0000 UTC 13 June 2008. The initial ensemble was generated with WRF-3DVar (Barker et al. 2004) by adding perturbations randomly sampled from the default background error covariance of “cv3” with the FNL analysis error as the standard deviation. The background error covariance of cv3 was generated through the National Meteorological Center (NMC) method of averaged forecast differences (Parrish and Derber 1992; Barker et al. 2004). The random perturbations were generated initially for control variables (viz., streamfunction, unbalanced velocity potential, unbalanced surface pressure, unbalanced temperature, and “pseudo” relative humidity) with a normal distribution (zero mean and unit standard deviation). The perturbations of the control variables were then transformed into those of the model prognostic variables via empirical orthogonal function (EOF) transformation, a recursive filter, and physical transformation. The perturbed variables included the horizontal wind components, potential temperature, pressure perturbation, and mixing ratio for water vapor. Other prognostic variables, such as vertical velocity (w) and mixing ratios for cloud water (qc), rainwater (qr), snow (qs), and graupel (qg), were not perturbed. The domain-averaged standard deviation of these perturbations was 2.16 m s−1 for u, 2.42 m s−1 for υ, 0.99 K for T, 151 Pa for pressure perturbation (p′), and 0.77 g kg−1 for qυ. This method of generating the initial ensemble has also been used in many other studies (e.g., Houtekamer et al. 2005; Barker 2005; Zhang et al. 2006; Meng and Zhang 2007, 2008a,b, 2011; Wu et al. 2013).
The targeted observations were synthetic wind profilers. Wind profilers measure wind speed and wind direction at various elevations above the ground. Compared with rawinsonde, wind profilers have the advantage of higher temporal and vertical resolution with a fully automatic operation. They are not affected by air traffic control since they do not need an air balloon. Wind profilers are becoming widely used to fill the spatial and temporal gap of rawinsonde observations.
The synthetic wind profilers were generated by adding observation error to the truth (Figs. 6c1–3), an ensemble member whose simulation was the most comparable to FNL analyses in terms of the location and intensity of the MCV (Fig. 6a1), and to observations in terms of its associated composite radar reflectivity at its mature stage (1800 UTC 12 June, provided by the CMA) (Fig. 6a2) and RAIN24 (Fig. 6a3). We assumed that the synthetic profiler winds had independent Gaussian errors with zero mean and a standard deviation as per the default in the WRF-3DVar preprocessing package (Table 1), which has been used previously by Meng and Zhang (2008a) for examining the relative performance of assimilating different observation platforms. To emulate the characteristics of the real-world wind-profiler observations, the limitations of wind profilers, such as impact from ground clutter, and inaccuracy both in rain and within 500 m above the ground (H. Zhang 2011, personal communication) were also taken into account. The observations under the surface and within 500 m above the ground were removed. Furthermore, the wind speed needed to be larger than 5.97 m s−1 so that the error of wind direction was within 10° (Chen and Wang 2000).
The area within which synthetic observations were generated was determined based on the general size of the target area of TCs, because no literature exists on targeting observations for MCVs. The target area of a TC is generally within a radius of 500–1000 km from the TC center (e.g., Majumdar et al. 2006; Wu et al. 2007b, 2009; Yamaguchi et al. 2009; Mu et al. 2009; Harnisch and Weissmann 2010; Qin and Mu 2011; Aberson et al. 2011; Kim et al. 2011). Since MCVs are generally equivalent to small and weak TCs, we selected a square of sides 1100 km in domain 1 with the center of the initial MCV slightly to the southeast of the square’s center (marked in Fig. 3c) in order to cover more upstream area. Each data unit was a small square that consisted of nine neighboring grid points, and thus had sides of 121.5 km (the small squares in the dashed box in the middle of Fig. 4; an enlarged example is shown in Fig. 7). There were a total of 81 units in the selected square area. Through unit-by-unit data assimilation, 81 experiments were performed for each member, consisting of initial one-unit data assimilation and a subsequent 48-h forecast. The impact of individual data units was then assessed for the chosen forecast metrics, described as follows.
c. Forecast metrics
Two forecast metrics were explored in this work, with different degrees of nonlinearity. The first was RAIN24, which is strongly nonlinear, and the other was synoptic-scale wind and temperature in terms of TE, which is weakly nonlinear.
To determine the target area of RAIN24, where assimilating synthetic profiler winds provided greater improvement than other areas, we designed an index based on the threat score (TS) of forecasted RAIN24 that exceeded 50 mm in domain 2:
where TSDA and TSNoDA denote the TS of RAIN24 with and without data assimilation, respectively. A larger R index denotes a greater contribution from the assimilated observation. The impact of data assimilation on the less-nonlinear TE was determined by an index based on the mean DTE of domain 2:
in which DTE is the difference total energy (Zhang et al. 2003) that reflects the overall error of U, V, and T:
where U′, V′, and T′ are the difference between a given member and the truth, (Cp = 1004.9 J kg−1 K−1, Tr = 287 K), and i, j, and k are the indices of the x, y, and z coordinates, respectively.
d. Determination of the target area
To examine the target area quantitatively, the positive data-impact index was categorized into four levels (e.g., the R index distribution for member 27, as shown in Fig. 7). The area covered by the data units with the highest level of data-impact index was defined as a target area if it was limited or localized enough to satisfy the following two criteria: 1) the number of data units (denoted by Nu) were no larger than 30% of the total 81 data units; namely, Nu ≤ 24; and 2) Np + Ns ≤ 10, in which Np denotes the number of area with connected pieces of these data units (“connected” means either sharing a same side or a same point), and Ns denotes the least number of sides of the data unit square that connects different pieces. A larger Np + Ns denotes a less-localized target area. These criteria were trial-and-error based through eyeball examination. For example, for the highest level of the R index of member 27 [L4 (R > 3), as shown in the color bar in Fig. 7]: Nu = 15 ≤ 24, Np = 3, Ns = 4, and Np + Ns = 7 to ≤10. Consequently, the target area for RAIN24 for this member was established as the area marked in orange in Fig. 7.
4. Control experiments
a. Deterministic forecast
Initiated directly from the FNL data at 0000 UTC 11 June, the WRF produced a decent simulation of the MCV, its radar reflectivity, and associated RAIN24 (Figs. 6a,b). The location of the simulated circulation center at 1800 UTC 12 June (Fig. 6b1) lay about 50 km to the southwest of the analysis (Fig. 6a1) with a slightly stronger intensity in terms of maximum vertical vorticity at 700 hPa. The observed radar reflectivity at 1800 UTC 12 June (Fig. 6a2) had a maximum value in northeastern Guangxi, which was well captured by the model (Fig. 6b2).The simulated RAIN24 (Fig. 6b3) approximated the observed heavy rainfall over northern Guangxi Province, reaching as much as 250 mm (Fig. 6a3). Consequently, WRF with the current configuration was able to successfully simulate the evolution of both the MCV and its rainfall.
b. The ensemble forecast
The performance of the ensemble forecast in terms of RAIN24 > 50 mm was examined via the TS against the observation in domain 2. The results showed that six members (15%) successfully captured the rainfall with a TS > 0.3. A total of 12 members (30%) had a TS of 0.2–0.3. The remaining members produced poor rainfall forecasts and accounted for 55% of the members. Thus, the performance of the ensemble was generally reasonable.
The evolutions of the DTE of the 39 members are plotted in Fig. 8a. Since our aim was to explore the features of the target areas among different ensemble members that were almost equally likely, we trimmed the ensemble by dropping six outliers, including M16, which had the largest initial DTE, and M13, M17, M36, M37, and M40, which had the smallest initial DTE. The remaining less-scattered 33 members composed the working ensemble whose results are examined in the rest of the paper (Fig. 8b).
5. Quality of the target area
a. The strongly nonlinear metric: RAIN24
1) Deterministic approach
The distribution of R index of all 33 of the members is demonstrated in Fig. 9. Target areas were detected in about 45% of the ensemble (denoted by black numbers). The effectiveness of the target area was examined through comparing the error of the forecast metric by assimilating all the data units in the detected target area into the initial field, to that by assimilating the same amount of data units but in a different area. We used M27 as an example to calculate the TS of forecasted RAIN24 by assimilating all the 15 data units in the target area (0.11 in Fig. 10). Then, the experiment was repeated 5 times by assimilating the same number of data units but with a random distribution in the data square, as shown in Fig. 10. The median of the TS of the five realizations of randomly distributed data was used to denote the impact of assimilating random observations (0.08 in Fig. 10). First, the TS with targeted data assimilation was significantly higher than that without data assimilation (NoDA). Second, the TS with target data assimilation was significantly higher than that with random data assimilation. In other words, assimilating the targeted observations substantially improved the forecast of RAIN24 compared to assimilating the same amount of data but with a random distribution. Consequently, the target area was effective. Third, the TS with random data assimilation was apparently higher than that of NoDA. This result indicates that a certain amount of forecast error could still be decreased even when the estimated target area was substantially inaccurate, although the reduced error was not as large as that obtained with an accurate target area.
Significant differences were observed in the size, structure and location of the target areas of RAIN24 among the different members (Fig. 9), indicating a large variability of the target area metrics. The size of the target area ranged from L1 to L4. The number of data units in the target area varied from 0 to 24. The location of the target area was scattered all over the domain, such as to the south (M27), northwest (M4), around (M34), north (M2) or southwest (M11) of the vortex center, with different degrees of compactness varying from tightly connected (e.g., M4) to widely separated (e.g., M33).
Substantially different target areas were even found in members very close to each other in terms of both forecast fields and their DTE. For example, M3 and M27 were very similar to each other in terms of forecasted RAIN24, the patterns of the MCV, and their vertical vorticities and radar reflectivities at the mature stage (Figs. 6d,e). The absolute error of initial T, vertical vorticity and horizontal wind speed at 700 hPa, the magnitude and evolution of volume-averaged DTE (Figs. 11a,b), and their horizontal distribution of vertically averaged DTE at the initial and end times were also similar (Fig. 8b). However, there were no data units of L4 in M3, and only one data unit reached L2 to the northwest of the MCV center (Fig. 11a1); while in M27 a target area with L4 was mainly located to the south of the MCV center (Fig. 11b1).
The M6 and M30 were even closer to each other relative to M3 and M27 in terms of volume-averaged DTE, with almost the same magnitude at the initial time (Fig. 8b) and a more similar horizontal pattern of vertically averaged DTE as well as absolute error of initial T, vertical vorticity, and horizontal wind speed (Figs. 11c,d). Their simulated RAIN24, patterns of the MCV, and their vertical vorticities and radar reflectivities at the mature stage were also very similar (Figs. 6f,g). However, significant differences were found in their target area features (Figs. 11c1, d1). M30 displayed a feature similar to M3, with a target area located to the northwest of the MCV center, while the data units of its highest L2 in M6 were too scattered to meet the criteria of target area (Fig. 11c1; Np = 4, Ns = 13, and Np + Ns = 17 to >10).
The above results demonstrate that small differences in the initial field may make a large difference in terms of the data impact and consequently in the existence and the location, structure and size of the target area. The fact that different members produced apparently different locations of the target area suggests that the target area for a strongly nonlinear forecast metric cannot be determined through the deterministic method.
2) Probabilistic approach
With the huge variability of the target area among different ensemble members in mind, we subsequently examined the possibility of determining the target area from a probabilistic point of view. With all the negative R index values in the 33 members set to zero, the ensemble mean R index was calculated (Fig. 12a). The data units with high R-index values were located mainly in a southwest–northeast-oriented zone with three pieces of data units of L4. The L4 data units were so scattered that they barely met the criteria of the target area (Np = 3, Ns = 7, and Np + Ns = 10). Four of the five data units of the target area were located to the southwest of the MCV center, which suggested that adding extra data into the southwest of the MCV center may statistically improve the RAIN24 forecast. The ensemble-based target area was expected to be closer to the true target area and thus should be more statistically accurate than the target area determined by individual members. Meanwhile, a large variability of the target area among different members and a subsequent scattering of data units in the ensemble-based target area would suggest that the target area for highly nonlinear rainfall was probably imprecise and thus possibly not very reliable.
The effectiveness of the ensemble-based target area was verified using a method similar to that described in section 5a(1). The five observation units in the ensemble-based target area shown in Fig. 12a were assimilated into the initial fields of all 33 members. The mean of the TS of the forecasted RAIN24 of the 33 members was used to represent the impact of assimilating the data in the target area. Then, the experiment was repeated 5 times by assimilating the same number of data units but with a random distribution, as shown in Fig. 10. The median of the TS of the five realizations of randomly distributed data was used to denote the impact of assimilating random observations. The results showed that the ensemble-based target area was effective as indicated by the significantly larger TS with targeted data assimilation than that of random data assimilation and NoDA, which was the mean of the TS of the 33 members without data assimilation. An interesting result was that the TS with random data assimilation was not significantly different from that of NoDA, which was apparently different from the result obtained using the deterministic approach. This suggests that targeted observation for rainfall requires that the target area is strictly accurate and adequately sampled to obtain a statistically positive effect.
Overall, the above results indicate that, for a forecast metric of rainfall that is strongly nonlinear, the ensemble-based target area is effective but has a large uncertainty. Considering the small possibility of obtaining an exactly accurate target area beforehand in practice and the similar performance of random data assimilation to NoDA, targeted observation for strongly nonlinear forecast metrics such as rainfall may easily fail.
b. The weakly nonlinear metric: Total energy
The distribution of F index for TE of all 33 members is demonstrated in Fig. 13. About 64% of the ensemble had target areas that satisfied our criteria, which is a larger value than that for rainfall. Although more members had target areas for TE relative to RAIN24, there was still apparently large variability in terms of the location, structure, and size of target areas among the 33 members, such as the opposite direction of location relative to the MCV center of M4 and M19, and the markedly different magnitudes of M29 and M30.
The difference in the target area of the ensemble pairs that had similar MCV features and initial, end values and evolution of DTE (M3 and M27, M6 and M30) became less apparent for the forecast metric of TE (Figs. 11a2–d2). The M6 had a target area for TE with a very similar location as that of M30, though the magnitude was apparently different. The location of the target area of M3 and M27 was still markedly different, but much closer to each other than those for RAIN24. An interesting result was that the target areas of M3 and M27 for TE were almost in the opposite direction relative to the MCV center compared to their counterparts for RAIN24. This phenomenon became less significant for the M6 and M30 pair, whose DTEs were slightly smaller. This result indicates that extra data assimilation that most efficiently decreases the total error in TE does not guarantee the most efficient decrease of error in the associated rainfall.
Together, these results show that, even though the variability of the target areas was lower for TE, which has a weaker nonlinearity than RAIN24, using a deterministic approach to determine their target areas still could not guarantee a statistically positive impact. Initial uncertainties may easily dislocate the target area and thus compromise the improvement in forecast skill. This could be why some targeted observation experiments do not obtain apparent improvement in their long-term verification. The large variability in the target area for synoptic-scale metric was noticed by Ancell and Mass (2008) using adjoint targeting method. Their results are further confirmed by our work using a nonlinear approach.
Using the probabilistic approach, the ensemble-based target area for TE was also examined through calculating the mean F index of TE over the 33 ensemble members (Fig. 12b). The high F index values were mainly located in a zone that was generally northwest–southeast oriented, which was apparently different from the high-index zone for RAIN24. A perfectly compact target area was detected to the north of the MCV center, with five connected L4 data units. The huge difference between the location of the ensemble-based target area for RAIN24 and TE further confirmed that the extra data assimilation that most efficiently decreases the total error in the flow and temperature pattern does not guarantee the most efficient decrease of error in the associated rainfall.
The effectiveness of the ensemble-based target area for total energy was assessed in the same way as for rainfall (Fig. 10). The mean DTE was significantly smaller than that of NoDA but not significantly different from that of random data assimilation, indicating the ineffectiveness of the ensemble-based target area for TE. This result suggests that, for a targeted observation with TE as the forecast metric, the forecast error is likely to be reduced by a similar magnitude no matter whether the target area is accurate or not. This could be what was happening in the previously mentioned field targeted observation campaigns, in which a 10% improvement of forecast skill was obtained using linear-based targeting strategies for forecast metrics of synoptic-scale, weakly nonlinear TE or typhoon track. The impact of these field campaigns was usually verified by comparing the forecast error with and without the assimilation of targeted observations. What was really assessed was the importance of the additional data, but not the importance of all the efforts made to identify a particular target area, as opposed to any other area.
c. The possible role of forecast metric nonlinearity
To examine the possible contribution of error growth during the forecast versus initial data assimilation to the distribution of data-impact index values and their associated target areas, the I index, which reflects the data impact on initial TE, was defined similarly as the F index,
The distribution of I index values is shown in Fig. 14. The percentage of ensemble members with target area (76%) was slightly larger than that for the F index (64%), and both were apparently larger than that of the R index (45%) (Table 2).
The target area showed much less variability than for both the R and F indices, which was quantitatively shown by the standard deviation of their Np + Ns, with values of 6.3, 3.4, and 2.1 for the R, F, and I indices, respectively (Table 2). The mean Np + Ns values for the R, F, and I indices were 5.4, 3.2, and 1.9, respectively (Table 2), indicating that the target area became more compact and thus showed less variability from R to F and I. The target area for the initial TE appeared mostly near or to the southeast of the MCV center. For the two pairs of ensemble members, which had similar initial, end values and evolution of DTE, their target areas showed even closer features (Figs. 11a3–d3). The target area of M3 and M27 were both to the east of the MCV center, with the target area of M27 covering more data units. M6 and M30, which had even more similar initial DTE, showed almost exactly the same target area, which was located to the south of the MCV center.
The target areas for TE at the initial time were generally different from the corresponding target areas for both rainfall and forecasted TE. This feature could be clearly seen in the location of their ensemble-based target areas (Fig. 12). The mean I index showed a perfectly compact target area to the southeast of the MCV center. It was substantially different from the target area based on the F index, located to the north of the MCV center, and the target area based on the R index, which was scattered with a dominant location to the southwest of the MCV center. The difference in the target area for initial TE from those for forecasted rainfall and TE suggested that nonlinear error growth might have played an important role in the distribution of the target area for both strongly nonlinear rainfall and weakly nonlinear TE. The additional data assimilation that was able to reduce the initial DTE the most did not guarantee the largest error reduction in the forecasted DTE, let alone rainfall. For weakly nonlinear forecast metrics, the variability of target areas was smaller and thus resulted in a more compact target area. The weakly nonlinear error growth seemed to change the location of the localized target area only, while the strongly nonlinear error growth seemed to change both the location and compactness of the target area.
6. Summary and discussion
In the present reported study, we examined the quality of the target area for two forecast metrics with different nonlinearity (rainfall versus TE) associated with an MCV in southwest China during 11–13 June 2008. The examination was performed from both a deterministic and statistical perspective using the PBPDA method within the framework of an OSSE. The quality of the target area was examined in terms of its effectiveness and variability.
Data assimilation is a common way to assess the performance of targeted observation. Through examining the direct impact of assimilating synthetic observations, piece by piece, at the initial time by 3DVar on the reduction of the forecast error of certain metrics, the target area was defined as the limited area with the highest level of data-impact index values. This targeting method not only accommodates fully nonlinear error evolution, thus avoiding the limitation of current strategies with linear assumptions, but is also consistent with the data assimilation method.
Results showed that the target area determined via the deterministic approach was not quite reliable for both strongly nonlinear forecasted rainfall and weakly nonlinear TE. Large variability was observed in terms of the location, structure, and size of target areas among different equally likely ensemble members with a reasonable initial ensemble spread. Those members that had very small differences in their flow and rainfall simulations showed substantially different target areas. The high sensitivity of the target area to the uncertainties of the initial field, which were comparable to the analysis error, for both the strongly nonlinear forecasted rainfall and the weakly nonlinear TE, indicates that target areas can be easily misplaced in deterministic scenarios. Larger variability was seen in the target area for rainfall among different members than for TE. Less ensemble members produced target areas for rainfall than for TE. The target area for TE was also more compact than that for rainfall. These results indicated that the target area for rainfall was less reliable than that for TE.
Through averaging the data-impact index values of ensemble members, the ensemble-based target area was obtained for both forecasted rainfall and TE. The data units in the ensemble-based target area for rainfall were much more scattered, almost failing to meet the criteria of the target area, as compared to that for TE, suggesting less reliability. Analyses on the effectiveness of the ensemble-based target area suggest that adding more data in a particular estimated target area, or in a random distribution near the weather system concerned, can both decrease the forecast error of TE by a similar magnitude, thus targeted observing may not be necessary. On the other hand, the forecast error of rainfall tends to be markedly reduced only when the data are added to an accurate target area. Since the target area for rainfall has less precision because of its larger variability relative to that of TE, it is thus likely to be difficult to estimate accurately in practice, and so targeted observation for rainfall may easily fail.
The large variability of the target area for forecasted rainfall was mainly due to its highly nonlinear error-growth feature. By comparing the target areas for forecasted rainfall and TE with that for initial TE, it was found that target areas for forecasted rainfall and TE were substantially different from that for initial TE. This result indicates that assimilating extra data that may improve the initial flow and temperature the most, does not guarantee the greatest improvement in their forecast skill, let alone their associated rainfall. The weak nonlinearity of TE seemed to only change the location of the compact target area for the initial TE, while the strong nonlinearity of rainfall seemed to change both the location and compactness of the target area.
Finally, it is important to note that this study was based on one case using the 3DVar data assimilation method for synthetic wind-profiler observations. The results obtained within an OSSE framework may not be fully applicable to real-world situations. Using real-world satellite data to examine the quality of target areas is worthy of examination in the future. Further work covering more cases, different forecast metrics, data assimilation methods, synthetic observation platforms, and sizes of data units needs to be done to obtain more general conclusions. The direct piece-by-piece data assimilation targeted observation strategy used in this study is an ideal method to examine targeted observation in a theoretical context, but it is almost impossible to apply it operationally since it requires future observations or “truth” to work. Besides assessing the quality of the target area, this PBPDA targeting strategy could also be used to verify the relative performance of various targeting observation strategies.
We thank Fuqing Zhang from Pennsylvania State University and Mu Mu from the Institute of Oceanology, Chinese Academy of Sciences, for their insightful suggestions. We also greatly benefited from comments given by Yunji Zhang and Huizhen Yu from Peking University. This research was sponsored by Ministry of Science and Technology of China Grants 2013CB430104 and GYHY201306004, and Natural Science Foundation of China Grants 41075031 and 41130960.