Over mountainous terrain, ground weather radars face limitations in monitoring surface precipitation as they are affected by radar beam blockages along with the range-dependent biases due to beam broadening and increase in altitude with range. These issues are compounded by precipitation structures that are relatively shallow and experience growth at low levels due to orographic enhancement. To improve surface precipitation estimation, researchers at the University of Oklahoma have demonstrated the benefits of integrating the Tropical Rainfall Measuring Mission (TRMM) Precipitation Radar (PR) products into the ground-based NEXRAD rainfall estimation system using a vertical profile of reflectivity (VPR) identification and enhancement (VPR-IE) approach. However, the temporal resolution of TRMM limits the application of VPR-IE method operationally. To implement the VPR-IE concept into the National Mosaic and Multi-Sensor QPE (NMQ) system in real time, climatological VPRs from 11 years of TRMM PR observations have been characterized for different stratiform/convective rain types, seasons, and surface rain intensities. Then, these representative profiles are used to adjust ground radar–based precipitation estimates in the NMQ system based on different precipitation structures. This study conducts a comprehensive evaluation of the newly developed climatological VPR-IE (CVPR-IE) method on winter events (January, February, and December) in 2011. The statistical analysis reveals that the CVPR-IE method provides a clear improvement over the original radar QPE in the NMQ system for the study region. Compared to physically based VPRs from real-time PR measurements, climatological VPRs have limitations in representing precipitation structure for individual events. A hybrid correction scheme incorporating both climatological and real-time VPR information is desired for better skill in the future.
Precipitation can induce potentially hazardous events such as floods and landslides, especially over complex terrain. Therefore, accurate measurement of precipitation is of vital importance for both hydrological applications and societal benefits. The current Weather Surveillance Radar-1988 Doppler (WSR-88D) Next Generation Weather Radar (NEXRAD) network has proven its value by providing high-resolution continental United States (CONUS)-wide precipitation measurements to the nation. So far, ground weather radar is the most powerful tool to provide high-quality 3D observation at high spatial and temporal resolution. With the upgrade to polarimetric capability finished in 2013, ground weather radar has become a more robust tool for accurate precipitation estimation. Based on the NEXRAD, researchers at the National Oceanic and Atmospheric Administration(NOAA)/National Severe Storms Laboratory (NSSL) and the University of Oklahoma (OU) developed the National Mosaic and Multi-Sensor Quantitative Precipitation Estimation (QPE) (NMQ) system [upgraded to the Multi-Radar Multi-Sensor (MRMS) system since 2013] which is capable of generating high-quality, real-time QPEs over the CONUS (Zhang et al. 2011). However, because of the lack of adequate ground-radar (GR) coverage from intervening terrain blockages (Maddox et al. 2002), reliable ground-based precipitation measurements are difficult to obtain in the Intermountain West region. Corrections should be made for more accurate precipitation estimation.
In complex terrain, ground-based volume-scanning weather radars often rely on scans at higher-elevation angles to avoid beam blockage issues. Surface precipitation estimates are more frequently based on radar measurements within the melting layer or above it in the ice. Also, the broadening radar beam with range may also be too wide to accurately resolve the vertical structure of precipitation. As a result, complete and fine-resolution vertical profiles of reflectivity (VPRs) are essential to improve the ground-radar QPE by mitigating brightband contamination and beam overshooting.
A variety of studies have investigated different approaches to obtain representative VPRs. There are generally two categories. The first relies on ground-radar data or other surface observations to derive VPRs (Kitchen et al. 1994; Andrieu and Creutin 1995; Fabry and Zawadzki 1995; Vignal et al. 1999; Germann and Joss 2002; Tabary 2007; Borga et al. 2000; Kirstetter et al. 2010; Zhang and Qi 2010; Kirstetter et al. 2013). However, in mountainous regions, complete VPRs may be difficult to obtain. The second approach investigates the vertical structure of precipitation using spaceborne radar (Gabella et al. 2006; Wen et al. 2013; Cao et al. 2013a,b), that is, the Precipitation Radar (PR) on board the National Aeronautics and Space Administration (NASA)’s Tropical Rainfall Measuring Mission (TRMM) satellite. Building on the works proposed by Kirstetter et al. (2013), Wen et al. (2013) proposed a concept of QPE enhancement, namely, the VPR identification and enhancement (VPR-IE), which derives a representative, parameterized VPR using PR observations when a local PR pass is available.
The VPR-IE method has been evaluated for several stratiform precipitation events in Arizona. The statistical analysis showed that VPR-IE effectively enhanced ground radar–based QPE, but this improvement was limited to times in which there were PR overpasses. Cao et al. (2013a) summarized the statistical seasonal, spatial, intensity-related, and type-related characteristics of the vertical structure of precipitation in the region of Southern California, Arizona, and western New Mexico through the use of 11+ years of TRMM PR observations. These climatological VPRs can now be integrated into a real-time multisensor scheme.
This study comprehensively evaluates the performance of the climatological VPR-IE (CVPR-IE) system. The remainder of this paper is organized as follows. The description of the CVPR-IE system is provided in section 2. Section 3 presents comprehensive evaluation results of all events during the winter months of 2011. Discussion is provided in section 4. A summary and discussion of future directions follows in the last section.
Our study area is the Intermountain West region of the United States (Fig. 1), where ground weather radar QPE is challenging because of insufficient NEXRAD coverage and high spatial variability of precipitation due to orographic enhancements. In winter, the relatively shallow precipitating clouds make accurate QPE at the surface level even more difficult. VPR correction improves the surface precipitation estimation by considering the vertical structure of hydrometeors and thus linking surface precipitation to the radar measurements aloft. Wen et al. (2013) used a physically based VPR model (Kirstetter et al. 2013) to identify and utilize PR-measured VPRs. The physically based VPR-IE method depends on the availability of PR measurements, which is limited to twice daily from the TRMM satellite orbits.
Cao et al. (2013a) derived climatological VPRs from long-term PR measurements for different seasons, rain intensities, and convective/stratiform rain types. Since the scattering of hydrometeors depends on frequency, the Ku-band climatological VPRs derived from PR measurements have different features compared to S-band VPRs. A conversion from Ku band to S band needs to be applied when the TRMM-based VPR-IE is implemented for ground-based radar measurements. A radar dual-frequency ratio was derived from Ku band using a set of empirical relations for different hydrometeors (snow, ice/hail, rain, and melting particles) and applied to link Ku-band reflectivity to S-band reflectivity (Cao et al. 2013b). The S-band climatological VPRs for the cool season are shown in Fig. 2. The climatological VPR is represented by the ratio of VPR to the reflectivity value at a reference height, which is set to 1 km below the freezing level. This reference height corresponds to where all hydrometeors are in liquid phase. The slope of VPR in the liquid rain region has dramatic influence on surface QPE correction. However, the insufficient sampling size at low level would introduce huge uncertainties. To avoid the correction error caused by inaccurate VPR at low level, we set the part of VPR under the reference level as vertically straight.
The procedure for real-time, climatology-based VPR correction is shown in Fig. 3. First, stratiform precipitation is identified based on the real-time NMQ precipitation type product, since the CVPR-IE is developed for stratiform precipitation, which is the most frequent rain type in the study region during winter months. Second, the surface rain intensity from NMQ is used to select the appropriate climatological VPR. Third, the selected climatological VPR is then combined with the real-time freezing-level height from NMQ to account for local storm structures and the underlying terrain effects. Fourth, representative VPRs are convolved with ground radar–sampling properties (e.g., beam broadening with range) to compute the apparent VPRs (AVPRs) at different radar ranges. Finally, the correction is applied to the reflectivity Z field, which is then converted into rainfall rate R using Z–R relations: Z = 200R1.6 for stratiform rain and Z = 300R1.4 for convective rain. The rainfall rates are then accumulated to hourly rainfall amounts and compared to rain gauge observations. The focus of this paper is on the assessment of the approach. More details of CVPR-IE are discussed in Cao et al. (2014) and Wen et al. (2014).
a. Verification statistics
We select four statistical indices for evaluating CVPR-IE using rain gauges comprising the Hydrometeorological Automated Data System (HADS) network as the reference (Fig. 1). The relative bias (RB; in percent) is used to assess the systematic bias of radar estimations. Spearman’s rank correlation coefficient (CC) is used to assess the agreement between the radar estimates and gauge observations. The mean absolute error (MAE) measures the average magnitude of the error while the root-mean-square error (RMSE) quantifies the average error magnitude, giving more weight to larger errors:
Here, R(i) and G(i) represent the ith matching pair of rainfall amounts estimated with radar reflectivity and observed by rain gauges, respectively, and N represents the total number of data pairs for radar-based and rain gauge data matching. In (2), RankR(i) and RankG(i) represent the assigned rank value in the ascending order of the radar and gauge observation, respectively. Statistics are computed in Table 1 for hourly rainfall estimates after filtering out all points that have a frozen precipitation type according to the NMQ algorithm. Data pairs with nonzero values from both gauge and radar sources are considered as the correction is focused on quantitative measurement rather than detection.
The statistics show improvement with the CVPR-IE method according to all statistical indices except RMSE (Table 1). To evaluate the significance of the improvement and minimize the impact of the sample representativeness, a bootstrap method is implemented by recomputing statistics on the basis of different samples. Efron (1979) introduced the bootstrap method with the idea that the sample values generated by resampling from the original sample repeatedly are the best guide to the true distribution. Based on these bootstrap samples, estimates of the statistical values (bias, CC, etc.) can be derived. Figure 4 shows the probability distribution of statistical parameters derived from 1000 groups of bootstrap samples. Note that a summary statistic fluctuates from sample to sample. In general, all statistical values have improvements with statistical significance after the CVPR-IE correction except RMSE. Relative bias has the largest improvement, with the mode of the distribution of RB shifting from −46% to −40%. The mode of the CC distribution has shifted to higher values following correction and the MAE distribution has shifted to lower values, which means rainfall estimates are more consistent with rain gauge measurements after correction. Figure 4 shows a slight trend of RMSE shifting toward lower values, but not significantly. Further analysis of improvements due to CVPR-IE concentrates on RB, CC, and MAE.
The dependence of the distributions of the statistical values on sample size is shown in Fig. 5. Figure 5 shows the median of the distribution and the interquartile range for the statistics computed on the uncorrected radar data and then the CVPR-IE method. The breadth of the distributions of the statistics in Fig. 5 show a narrowing with larger sample sizes as expected. All statistical indices except RMSE show improvements for all sample sizes. There is a consistent ~6% improvement in RB performance independent of sample size due to the correction method. The negative RB results have shown underestimation both before and after application of the correction methodology. This underestimation may be due to an inappropriate Z–R relationship. Radar precipitation estimates have been deterministically computed using a biased relation between reflectivity and precipitation rate. Previous studies decomposed the Z–R relation uncertainties into several terms (Kirstetter et al. 2010, 2015): 1) the unconditional bias between the radar estimates and the reference; 2) the bias conditioned on factors like precipitation rate, accumulation period, distance from the radar, season, etc.; and 3) the random error remaining after accounting for the conditional (systematic) biases. Kirstetter et al. (2015) provides a new set of Z–R relationships within a probabilistic precipitation-rate framework. The derivation process is using radar measurements where blockage is deemed minimal using a radar quality index. Kirstetter et al. (2015) provides a new set of Z–R relationships within a probabilistic precipitation-rate framework. After applying the newly proposed Z–R relationship, the RB of uncorrected rainfall estimates becomes −11.09% instead of the prior −46.43% (Table 1) using the default Z–R relation for stratiform precipitation in NMQ. After CVPR-IE correction, the RB is improved to −0.10%. The improper former Z–R relationship masked the improvement of the CVPR-IE scheme. It is also notable that the consistent improvements are based on an hourly scale. These improvements may amplify further if assessed at a daily scale. More details regarding the uncertainties associated with radar calibration, Z–R relation, etc., may mask CVPR-IE’s performance and are discussed in section 4.
The statistics shown in Figs. 4 and 5 are aggregated over a large sample size and cannot highlight the improvement in skill for each data pair following correction. To evaluate if the radar estimates are in better agreement with gauge measurements in terms of occurrence of improvement after correction, we apply a difference method similar to Bellon et al. (2007):
where i indicates the ith gauge–radar pair; represents the raw, radar-based estimate; and is for the corrected QPE following the CVPR-IE application. If Ii > 0, then the radar measurement after correction agrees better with gauge measurements and the reverse is true for Ii < 0. The occurrences of positive and negative values are counted and the results are plotted as a function of gauge-based rainfall accumulation in Fig. 6. The solid line indicates that the improvement after correction is consistently better than the radar-only product when the surface hourly rainfall is less than 5 mm. It must be noted that the average stratiform rain rates with no orographic enhancement are typically less than 4 mm h−1 (Schumacher and Houze 2003). Stratiform precipitation is consistently associated with increased occurrences of improved results. The VPR correction scheme performs well in stratiform precipitation. In convective regions, where hourly rain rates are commonly greater than 4 mm h−1, the frequency of deteriorated QPE is balanced by the frequency of improved QPE after correction. Stratiform and convective echoes have different VPR characteristics. VPRs in convective precipitation have less vertical variability without a brightband feature in relation to stratiform echoes. Thus, the current CVPR-IE focuses on stratiform precipitation and shows improvements over a majority of data pairs.
b. Verification with radar beam height
Systematic errors in ground-based radar rainfall estimation, related to the VPR features combined with the geometric effects of the radar beam, create the often-noted radar beam height dependence (Bellon et al. 2005). Figure 7 shows statistics before and after correction as a function of radar sampling height. The uncorrected QPE at surface level shows high CC values and low MAE values indicating the radar QPE is highly consistent with rain gauge measurements. The CC and MAE worsen as radar beams approach and then intercept the melting layer. Both statistics either improve or stay constant for 2000 m above the melting layer (not shown in the figure because of low sampling size) before deteriorating again above this height. Reflectivity in the melting layer is not well correlated with surface rainfall rates. Regarding the RB, both uncorrected and CVPR-IE corrected rainfall slightly underestimate gauge accumulations instead of overestimating them in the melting layer (about 1.75 km below freezing level). The CVPR-IE correction technique detects the melting-layer region and automatically subtracts an offset of reflectivity between the bright band and the liquid rain region as observed in the climatological VPRs (Fig. 2). The unusual underestimation in the melting-layer region is further underestimated after the VPR correction is applied. We note that there is a negative RB (−40%) for the uncorrected rainfall at the surface level, that is, where we expect the best accuracy in rainfall estimates. The method is designed to adjust radar estimates so that they represent rainfall at the reference-level height. If the rainfall estimates are biased there, then the bias will propagate for corrections applied at greater sampling heights. The combination of biased Z–R relations and misidentification of freezing-level height could explain the decreased QPE accuracy. We discuss in section 4b that such biases may be due to an inaccurate Z–R relationship, which tends to mask the improvement in the CVPR-IE method. The overshooting of the radar beam causes the CC to drop significantly for radar beam heights greater than 2000 m above the melting layer (not shown in the figure because of low sampling size). The MAE decreases in this region, but only because the quantitative radar and gauge rainfall amounts are becoming quite light in this region. The significant decrease of the CC means that the data may not be correctable at these heights because of a very poor correlation with surface rainfall. This places an upper limit to which the CVPR-IE can be effective in shallow, stratiform rain. Overall, the CVPR-IE using the climatological PR information mitigates the underestimation above the freezing level by improving the RB by 30%. The CC and MAE are slightly improved in the ice region.
c. Verification with RQI
The NMQ system provides a Radar Quality Index (RQI) product to account for radar beam sampling characteristics (i.e., partial blockage, beam height relative to melting layer, and sampling volume; Zhang et al. 2011). The RQI field ranges from zero to unity, indicating the relative quality of the radar QPE from low to high. Chen et al. (2013) evaluated daily NMQ rainfall accumulations and found that the bias was correlated with RQI values. Figure 8 shows the CVPR-IE correction skill as a function of the anticipated quality of the rainfall estimates. All statistics show a trend of improving values with increasing RQI. The trend of CC is a little more complicated with a linear increase up to an RQI of 0.4 and then no improvements thereafter. For uncorrected radar QPE, the RB is lower than −80% and MAE is higher than 1.6 mm when RQI is less than 0.2. The low RQI score mainly corresponds to poor-quality QPE caused by sampling well above the melting layer in the ice region. The improved performance of the CVPR-IE is more evident when RQI is low and becomes less significant with increasing RQI, that is, when radar QPE is less affected by beam sampling problems. When RQI equals unity, the radar beam samples rain close to the surface. We again note that the RB is negative even when RQI equals unity. This systematic underestimation is not associated with the variability of VPR and may come from other sources of uncertainty such as Z–R relationship and radar miscalibration.
d. Verification with precipitation type
CVPR-IE is applied to stratiform echoes on 5-min radar data, which are then accumulated to hourly scale to match the temporal resolution of rain gauges. After the accumulation process, other precipitation types, such as convective, hail, or undefined precipitation type, may become prevalent at a given grid point and thus obscure the degree of correction that was made to the stratiform rain echoes. To address this, Fig. 9 shows the statistics as a function of the number of occurrences of stratiform echoes that were detected within the hour for each grid point. As the proportion of stratiform rain type increases, RB of both uncorrected and corrected QPE improves and the difference between them increases, indicating that the CVPR-IE functions most effectively with widespread, stratiform rain. The CC stays relatively constant, with increasing stratiform occurrences within the hour. After applying the CVPR-IE correction, CC is slightly improved relative to the uncorrected rainfall estimates. The increasing MAE values with increasing stratiform rain proportion are due to increasing rainfall accumulations. If stratiform rain did not occur, it is likely that precipitation was simply absent at that 5-min time step, thus resulting in lighter hourly accumulations. This inference is supported by the trend of increasing rainfall accumulations in Fig. 9d. There is no difference in MAE following correction using the CVPR-IE method.
Another precipitation type category defined in the NMQ system is called bright band (BB), which indicates areas where the radar beam is within ~800 m of the environmental freezing-level height. As the proportion of the BB type increases (shown in Fig. 10), the RB generally increases, CC does not exhibit a clear trend, and MAE increases in correspondence with increasing precipitation amounts. The corrected QPE shows improvements in all three statistics, especially when the BB occurrence is greater than 4.
a. Limitation of climatological VPR-IE
As demonstrated in section 3, CVPR-IE mitigates range-dependent errors to some extent. The improvement is mainly seen in RB while CC and MAE only have marginal improvements. Also, the improvement in RB is only around 6%, which is more limited than the improvements (~20%) made by the physically based VPR-IE (Wen et al. 2013). To examine the reasons behind these differences, CVPR-IE was applied to the cases investigated in Wen et al. (2013). The absolute error (AE; |Radar − Gauge|), MAE, RMSE, and CC are calculated for radar QPE after VPR-IE processing. To quantify the relative differences between the two different VPR-IE correction methods, the statistics are computed as a percent difference relative to the radar-only QPE as shown in Table 2. Improvements occur if there is a reduction in AE, MAE, RMSE, and an increase in CC. The analysis is conducted for specific hours when PR overpasses are available since the physically based VPR-IE method is limited by the temporal resolution of PR. In general, the physically based VPR-IE method yields more improvements for all five events. The CVPR-IE also improves on the radar-only QPE, but not drastically. The 8 December 2009 event is demonstrated in further detail as an example to show the differences between the physical and climatological VPR correction methods. This is a widespread stratiform event with heavy precipitation and a 0°C-level height of 2500 m. For most of the rainy areas shown in Fig. 11a, the radar beam has overshot the melting layer, which led to underestimation of rainfall on the surface. The physical VPR-IE induces the greatest increase of precipitation rate at ground (Fig. 11c). Rainfall estimates using the CVPR-IE correction are shown in Fig. 11d. Compared to Fig. 11a, the CVPR-IE method increases rainfall estimates around the Flagstaff, Arizona, radar (KFSX), but not sufficiently. Also, the underestimation within the area around 34.2°N, 112.5°W still remains after CVPR-IE by comparing to the gauge measurements on the ground. The reflectivity measured in this area is 25 dBZ. The corresponding climatological VPR (also shown in Fig. 2) for this grid point is plotted in Fig. 12a, along with the physically based VPR coming from actual PR measurements for the specific event. The climatological VPRs are consistent with the physical VPR but have larger uncertainty. The physical VPR has a relatively faster decreasing rate in an ice region than does the climatological VPR of 25 dBZ and thus provides a larger correction on surface-level reflectivity. For the rain areas 100 km east of the Las Vegas, Nevada, radar (KESX), where the beam height is around 1000 m relative to the freezing level (Fig. 12b), the correction of reflectivity using the physical VPR is more than 10 dB, while the climatological one is only ~5 dB. The climatological VPR’s deviation from the physically based VPR and its consequent effect on the correction skill exposes the limitation of a climatological VPR correction and emphasizes the importance of ingesting real-time information from TRMM PR or Global Precipitation Measurement (GPM) Dual-Frequency Precipitation Radar (DPR). Besides the uncertainties from the ice region, the slope of the VPR in the liquid rain region also biases the surface QPE correction. The climatological VPRs relying purely on PR measurements cannot identify collision–coalescence or evaporation processes below bright band. To avoid the correction error caused by inaccurate VPR at low levels, we set the part of the VPR under the reference level as a constant (vertical line). However, the disagreement between the vertical assumption and actual VPRs is partly responsible for the correction bias. Grams et al. (2014) used environmental variables from the Rapid Refresh (RAP) model to identify the enhanced rainfall rates. Blending real-time VPR information from spaceborne radar with climatological VPRs and environmental variables from numerical weather prediction models can improve the VPR variability issue and make the VPR-IE method more robust. To reduce the VPR uncertainty is a major aspect to address in future works.
b. Z–R relation uncertainty
All results up to now have shown underestimation both before and after application of the correction methodology. The RB improved by only 6% from −46% to −40% when considering gauge–radar pairs at all ranges from radar. Moreover, negative biases remained before and after correction for those bins measured well below the melting layer where the method assumes the radar-only estimates are trustworthy. The CVPR-IE method is essentially correcting data measured aloft to the radar-only rainfall estimates measured below the melting layer. If the rainfall estimates are biased in this region that is assumed to be trustworthy, then the bias will propagate to other bins. This underestimation may be due to an inappropriate Z–R relationship. Kirstetter et al. (2015) provide a new set of Z–R relationships within a paradigm of probabilistic precipitation-rate estimates. After applying the newly proposed Z–R relationship, the RB of uncorrected rainfall estimates becomes −11.09% instead of the prior −46.43% (Table 1) using the default Z–R relation for stratiform precipitation in NMQ. After CVPR-IE correction, the RB is improved to −0.10%. The improper former Z–R relationship masked the improvement of the CVPR-IE scheme, which could mitigate range-dependent errors in reflectivity successfully.
5. Summary and conclusions
This study provides a quantitative assessment of a climatological VPR-IE technique for all winter events in 2011 over a study region in the Intermountain West of the United States, where the climatological VPRs are derived from 11+ years of TRMM PR observations. The main results are summarized as follows:
The statistical analysis shows that the CVPR-IE method provides small but significant improvements over the original radar QPE in the current NMQ system.
The statistical significance of the correction skill is further examined using a bootstrapping method based on different sample sizes. The results show that the CVPR-IE method improves radar surface rainfall measurement systematically.
The CVPR-IE mitigates radar underestimation for samples obtained in the ice region but the correction was not enough to remove all negative bias.
The statistics show improvements in radar QPE following application of the CVPR-IE were most effective for bins measured above the melting layer, bins with low radar quality index values, and for gauge–radar pairs that were dominated by stratiform precipitation type.
Compared to a physically based VPR from real-time PR measurements, climatological VPRs have limitations in representing precipitation structures for each individual event. The physically based VPRs, on the other hand, are updated on a twice daily basis corresponding to a satellite overpass. A hybrid VPR correction scheme incorporating both climatological and real-time VPR information is desired to optimize skill in a real-time system.
The current VPR-IE schemes (both physically based and climatological) are restricted by the TRMM satellite coverage between 36°N and 36°S. Compared to the TRMM era, the main advantage of GPM is its extended coverage, which potentially expands the VPR-IE method to higher-latitude regions. Also, the refined DPR and GMI measurements, by better capturing the microphysical processes, could provide more insightful information of storm vertical structure, which potentially improves the representative VPRs and constrains the uncertainty. However, GPM DPR alone updates on a daily or less-than-daily basis, which might pose challenges for the effectiveness of the hybrid VPRs beyond the PR/DPR coverage. We plan to investigate this topic and will certainly communicate any quantitative findings to our community in the scope of our next report.
This research was funded by NASA Precipitation Measuring Mission Program, NASA GPM Ground Validation Program, and NASA Short-Term Prediction Research and Transition Program. Partial support for this research is from Southern-Central Climate Science Center. We also would like to thank the support from NSSL and University of Oklahoma personnel who maintain and operate the MRMS system. Constructive reviews provided by three anonymous reviewers improved the readability of this manuscript. Their time is appreciated.