1. Introduction
Considering that warm season rainfall is among the most poorly forecasted of meteorological parameters (e.g., Doswell et al. 1996; Fritsch and Carbone 2004), numerous efforts have been undertaken to try to improve the forecasts. Stensrud and Fritsch (1994) and Stensrud et al. (1999b) showed that proper initialization of mesoscale features such as cold pools would likely be needed to improve convective system rainfall forecasts; however, Gallus and Segal (2001) found that several techniques to improve the mesoscale initialization, including a technique to ensure depiction of cold pools, did not consistently improve rainfall skill scores significantly. Wang and Seaman (1997) and Gallus (1999), among others, have also shown that the choice of convective scheme strongly influences the simulated rainfall patterns. The convective scheme also affects the response of a model to changes in grid spacing (Gallus 1999) or soil moisture (Gallus and Segal 2000). With such extreme sensitivity to this one parameterization alone, and objective measures showing that no one scheme is better consistently than any other (e.g., Gallus and Segal 2001), the path to improved deterministic forecasts of warm season rainfall appears to be difficult.
Because of the problems in improving deterministic rainfall forecasts, ensemble forecasting techniques have been increasingly used in recent years. At first, ensembles were designed based on perturbed initial conditions, and the ensemble mean values were found to estimate the verifying state (usually large-scale circulations) better than the forecast from a single ensemble member (Molteni et al. 1996; Hamill and Colucci 1997). Similar results using multimodel analyses for initial conditions were found for 2-m temperature and 10-m wind forecasts by Grimit and Mass (2002). Ensembles also are advantageous because they supply probabilistic forecast information that may be of more value to users than a single deterministic forecast (Murphy 1993), and the ensemble dispersion gives an estimate of forecast uncertainty (Tracton and Kalnay 1993).
One of the first studies to investigate ensemble prediction of rainfall was Du et al. (1997), which found in an investigation of errors in initial conditions on cold season synoptic-scale quantitative precipitation forecasts (QPF) that greater improvement over climatology was present in the probabilistic forecast than in a single run using two times higher horizontal grid resolution. However, results from other studies using data from the experimental National Centers for Environmental Prediction (NCEP) Short-Range Ensemble Forecast (SREF) program indicate that these ensembles, which were built using only initial perturbations, generally have insufficient dispersion (Hamill and Colucci 1998; Stensrud et al. 1999a). It should be noted that a goal of increasing ensemble spread is not always an advantage but overall is probably helpful for warm season rainfall forecasts, which are usually characterized by low skill.
Insufficient ensemble dispersion may be a consequence of the original assumption that errors primarily result from uncertainties in the initial conditions. It is likely that the insufficient dispersion problem is more severe in a short-range forecast because initialization perturbations require time to grow and may not be capable of providing consistent dispersion in the short range (Stensrud et al. 2000). In the warm season when forcing and flow are weaker, the growth of the perturbations may be even slower. Due to the fact that errors result from any bias present in a model, an ensemble utilizing variations in both dynamics–numerics and model physics should result in higher spread. Alhamed et al. (2002) showed that model diversity in an ensemble system yields forecasts with greater spread containing more solutions that are possible. Stensrud et al. (2000) discussed the significance of both variations in model physics as well as initial conditions in ensemble forecasting. Based on studies like these, NCEP changed the SREF system in 2004 (Du et al. 2004) to introduce physics uncertainty (through the use of varied convective parameterizations) in addition to initial condition uncertainty.
In the case of a mixed-physics ensemble approach to MCS rainfall forecasting, knowledge of the nature of the impact of different physical schemes on rainfall would be exceptionally useful. As discussed earlier, numerous studies have shown the large impact the convective scheme has on rainfall forecasts. The choice of planetary boundary layer (PBL) scheme can substantially affect temperature and moisture profiles in the lower troposphere, which could interact with other schemes such as the convective parameterization to influence simulation of precipitation (e.g., Bright and Mullen 2002; Wisse and Vila-Guerau de Arellano 2004). However, the impact of different PBL schemes and microphysical schemes on warm season rainfall fields and the interactions of all three of these physical process schemes have received little attention. Our study will use the WRF model to explore these issues. The model selection is of particular merit because the emerging WRF community model will be used increasingly for ensemble forecasting in the near future (Bernardet et al. 2004). The main objective of the present study is to investigate the general impact that various physical schemes as well as their interactions have on warm season MCS rainfall forecasts. For this purpose, high-resolution (12-km grid spacing, 34 vertical levels) simulations from the WRF model of eight International H2O Project (IHOP; Weckwerth et al. 2004) events were examined. For each event, a matrix of 18 WRF model configurations was created by varying the convective parameterization scheme, the PBL scheme, and the microphysical schemes. The various methodologies used in the present study are discussed in section 2, results in section 3, with concluding discussion and summary in the final section.
2. Methodology
A matrix of 18 WRF variants created using different combinations of physical schemes was run for eight IHOP convective cases. The IHOP domain covered a roughly 1500 km × 1500 km region centered over the south-central United States. The cases were purposely selected to represent a range of different synoptic settings in which significant rainfall, primarily from MCSs, was observed and/or forecasted in the IHOP domain over the central United States. For the majority of cases the MCS systems dominated the rainfall field and were captured in the interior of the domain. For each case, three different treatments of convection were used: the Kain–Fritsch (KF) scheme (Kain and Fritsch 1993), the Betts–Miller–Janjić (BMJ) scheme (Betts 1986; Betts and Miller 1986; Janjic 1994), and the use of no convective scheme. For elaborations on differences between KF and BMJ see Jankov and Gallus (2004). For each of these three choices, three different microphysical schemes were used: Lin et al. (1983), NCEP-5 class (Hong et al. 1998), and Ferrier et al. (2002). Within these nine possible configurations, two different PBL schemes were used: the Medium-Range Forecast model (MRF; Troen and Mahrt 1986) and the Eta Model (Janjić 1994). It is important to note that our exploration of impacts and interactions between all possible combinations of physical schemes is slightly affected (only 4 out of 17 possible interactions were neglected) by our choice of the “control run.” To explore all interactions using one control run would involve synergism among three different processes, greatly complicating interpretation. In the present study, the control run, chosen to match the real-time model configuration adopted by the National Oceanic and Atmospheric Administration’s (NOAA) Forecast System Laboratory (FSL) during the IHOP experiment, used the KF convective scheme, MRF PBL scheme, and NCEP class-5 microphysical scheme. The abbreviations for runs using different combinations of the physical schemes are found in Table 1. For the rainfall validation, observed 6-h accumulated precipitation fields from the NCEP stage IV (Baldwin and Mitchell 1997) analysis were used.
All runs were initialized with a diabatic Local Analysis and Prediction System (LAPS) “hot” start initialization (Jian et al. 2003). This technique is based on a three-dimensional analysis of cloud attributes (i.e., coverage and type), which proceeds with a method of estimating mixing ratios, precipitable water, and cloud vertical motions. By using a variational adjustment procedure (involving dynamic balancing and a mass conservation constraint), horizontal wind fields and the mass field are adjusted to produce divergence consistent with the cloud updraft properties (depth, magnitude, and shape of the updraft profiles).
This approach was developed for grid spacings that resolve saturated updrafts and compensating subsidence, but it is still used quasi-operationally for much coarser resolutions (Δx > 10 km).
The notation presented in Table 1 will be used to indicate different model configurations with physical schemes that are changed from the control one (KF-MRF-MPN) presented in boldface throughout the manuscript
3. Results
a. Sensitivity of rainfall forecast skill to physical scheme changes
ETSs for all eight cases for all model versions, during the first six forecast hours valid for four different thresholds (0.01, 0.1, 0.5, and 1.0 in.; the thresholds are stated in inches as commonly used, 1 in. = 25.4 mm) are presented in Table 2. Relatively “good” (“bad”) forecasts [ETS one or more standard deviation above (below) the median for each 6-hourly time period] are indicated. One out of eight cases exhibited relatively good forecast skill for lower thresholds, while a different case had relatively good forecast skill for heavier thresholds. The same analysis but for the 12–18-h forecast period indicated generally lower scores than at earlier times but once again with higher scores for lighter amounts than heavier amounts (Table 3). It should also be noted that a good or bad forecast in the 0–6-h forecast period did not necessarily mean a good or bad forecast at later times. Bias analyses (not shown) indicated that for light amounts, both convective schemes had a substantial high bias (roughly 2.0) during the first 12 h of the forecast, while at later times biases slightly decreased (∼1.6). The worst overestimate occurred during the 6–12-h period. No specific trends in bias were noted for heavier thresholds.
ETS and bias averaged over all eight cases for all 18 configurations indicated that no one configuration was obviously best at all times and thresholds (Tables 4 and 5). However, it should be pointed out that during the 0–6-h forecast period, for lighter thresholds the highest ETSs were clustered among NC runs, possibly due to the positive impact of the hot start initialization. For the heavier thresholds, these same model configurations tended to have the lowest ETSs (Table 4). Based on subjective analyses, these low ETS values were sometimes related to a displacement error, while at other times it is possible that the NC runs were still undergoing “spinup” of strong enough vertical motions to produce heavier rainfall. With regard to bias, once again NC runs appeared to have an advantage as compared to runs that included convective parameterizations. Runs using convective schemes were usually characterized by large biases, especially in the case of BMJ runs for light rainfall. This reflects the BMJ tendency to notably often overpredict areas of light precipitation (Jankov and Gallus 2004). Later in time, during the 12–18-h forecast period (Table 5), the highest ETS values accompanied by bias values near 1.0 were clustered among NC-MRF runs. This is interesting since spinup problems are typically no longer present by this time in a forecast and the hot start might not be expected to be helpful at this time.
Over the four time periods, and for six different rainfall thresholds, the highest ETSs by a particular physics scheme occurred 7 times for MPN, 11 times for MPL, 5 times for MPF, 10 times for MRF, 13 times for ETA, 12 times for KF, 8 for NC, and 4 times for BMJ. It should be noted that differences in ETSs were usually small. Hereafter, discussion will be limited to only two rainfall thresholds: 0.01 and 0.5 in.
b. Sensitivity of rainfall forecast spatial patterns to physical scheme changes
To objectively test the sensitivity of the rainfall forecast pattern to physics changes, CR was calculated using Eq. (4) (neglecting outliers). Based on the CR definition, it is natural to expect CR to decrease as the number of evaluated runs increases. Because the present study investigated three different convective treatments and three different microphysical schemes, but only two different PBL schemes, CR was calculated as an average value of all possible couplets when two of three model physical schemes were held fixed and the third varied (e.g., the PBL scheme and the convective scheme held constant while microphysics varied between two different schemes). Additionally, it should be noted that CR primarily provides information about the spatial variability among the evaluated runs. To determine the variability in terms of rainfall amounts, CR was analyzed for two thresholds (0.01 and 0.5 in.). Figure 1a shows values of CR for changes in the microphysical, PBL, and convective schemes at both thresholds. It can be seen that the sensitivity to the choice of convective treatment dominated during the whole 24-h forecast period. For light rainfall, sensitivity to convective treatment was the highest (lowest CR) among all physics options during the first 6 h of the forecast, becoming at later times more similar to (though still higher than) the sensitivities of the other two physical process schemes. Sensitivity to PBL scheme choice increased with time, while no pronounced trend was present with respect to the choice of microphysical scheme. For heavier rainfall, the CR for the set of different convective schemes was highest in the first 6 h and much lower at later times. At all times, sensitivity to changes in the convective scheme exceeded that of the two other physical schemes. The sensitivity to the PBL scheme was generally comparable to, or a little larger than, that of the microphysical scheme, with changes in both causing more spread (lower CR) for heavier amounts, especially at later times. However, for rainfall amounts in excess of 0.5 in., sensitivity increased rapidly with time for all physics (microphysics, PBL, and convection), a trend not generally observed for the lighter rainfall amounts.
The lowest values of r 2 (largest differences in forecasts) for both thresholds during the whole forecast period also occurred when the convective treatment was changed (Fig. 1b). The r 2 values after hour 6 when the PBL schemes were varied were lower than when microphysics was varied, and the differences increased with time. The largest differences between the impact of changes in convective treatment and changes in other schemes occurred during the two earliest forecast periods. These results and the results from previous studies related to the impacts of resolution and the choice of convective scheme on MCS rainfall (Wang and Seaman 1997; Gallus 1999) imply that in order to achieve a large spread of solutions in a 6- or 12-h forecast with models having horizontal grid spacing of 10 km or more, it is important to vary the convective treatment.
A subjective analysis of rainfall fields for all cases and all model configurations was performed as well. The subjective analysis agreed well with the objective analysis features discussed above, suggesting the greatest variability in the forecasts came from changes in the choice of convective scheme. However, noticeable impacts from changes in the microphysical or PBL schemes were occasionally observed in some events. Figure 2 illustrates an example of the simulated rainfall fields in the domain of integration for the 19 June 2002 case initialized at 1200 UTC for the 6–12-h forecast period and for four different model configurations: KF-MRF-MPN (control run; Fig. 2a), KF-MRF-MPL (Fig. 2b), NC-MRF-MPN (Fig. 2c), and BMJ-MRF-MPN (Fig. 2d). Specific features of Fig. 2 are discussed later in the text. Because rainfall extrema near the edges of the model domain (e.g., Figs. 2a and 2b) may reflect the influence of lateral boundaries, grid points near the boundaries were excluded in the computation of the parameters discussed in this study.
c. Sensitivity of system average rain rate and domain total rain volume to physical scheme changes
Factor separation methodology [analysis of the three terms on the rhs of Eq. (6)] was used as an additional evaluation of sensitivity to changes in the physical schemes. These terms, expressed as a fraction of the control run rainfall amount shown in Table 6, are presented in Tables 7 and 8. Two different rainfall measures were evaluated for this analysis. First, the rhs terms of Eq. (6) were computed using averages over all eight cases for each 6-h forecast period for 18 different model configurations (physical schemes were varied) at the number of points where rainfall exceeded specified thresholds. Essentially, this expresses the system average rain rate (hereafter rain rate) or intensity, where the system is defined to be those points having rainfall above a specified threshold. In addition to system average rain rates, the same terms in Eq. (6) were computed over the entire domain, yielding a domain total rain volume (hereafter rain volume). The use of both measures better characterizes the QPF, since two runs could have the same total rain volume with one achieving it through light rainfall over a large area and the other through heavy rainfall in a small area.
As part of the investigation of changes in rain rate and rain volume due to variations in physical schemes, statistical significance testing was performed. To perform rigorous hypothesis testing, Hamill’s (1999) resampling methodology was used. This procedure was strictly followed and repeated 1000 times for both a separate treatment of each 6-hourly forecast period and for all 6-h periods combined. Combining all forecast periods together helped to increase the small sample size to better evaluate statistical significance. However, using this technique to enlarge sample size was only valid when statistical stationarity was present and was not appropriate for cases in which variables were characterized by strong temporal variability. The synergistic-term-computed values often exhibited such variability and for these parameters, each 6-h period had to be examined separately. With only a few exceptions (to be noted later) the synergistic interactions were not statistically significant. For some parameters where the impacts of changes in schemes or synergistic interactions were large but no statistical significance was found, the small sample size is likely a problem, and future studies should examine the interactions with a larger independent dataset (Nicholls 2001). In these situations, the lack of statistical significance does not necessarily imply that these physical schemes and their interactions have no impact on precipitation simulations. Due to the already extensive size of the present experiment (18 model configurations for eight different cases resulting in 144 model runs), it was not possible to substantially expand the dataset. The discussion to follow will emphasize statistically significant results, although nonsignificant trends occasionally will be noted when they are supported by the results of other studies addressing differences in behavior between physical schemes.
To facilitate a comparison of different model configuration results with the control run and observations, rain rate and rain volume for both are included in Table 6. For the 0.01-in. threshold the control run has a roughly 30% larger areal coverage than observed for the first six forecast hours. During the next 6-h period the control run areal coverage is similar to the observed, while at later times it is smaller, by as much as ∼40% in the 18–24-h period. Control rain rates are 10%–20% smaller than the observed for all 6-h forecast periods. For the 0.5-in. threshold, the control areal coverage is much smaller than the observed at all times, while the rain rate is generally larger except for the 12–18-h forecast period. For both thresholds the control rain volume is always smaller than the observed, particularly for the 0.5-in. threshold, where the forecast is an order of magnitude less than that observed during the 18–24-h period.
1) Change from MRF to ETA combined with changes in microphysical schemes
Factor separation evaluation of changes from MRF to ETA and from MPN to both MPL and MPF are presented in Table 7. The switch from MRF to ETA (run f1) for the 0.01-in. threshold always increased the areal coverage. This result is consistent with a subjective analysis performed within the present study, which indicated that the ETA PBL scheme tends to generate boundary layers that are more moist than MRF, a result agreeing with Bright and Mullen’s (2002) findings. On the other hand, this change did not significantly impact rain rate and rain volume. For the 0.5-in. threshold, the change in the PBL scheme had an even more limited impact.
Changes in microphysics (runs f2 and f3 in Table 7) at all times produced an increase in the areal coverage for both the 0.01- and the 0.5-in. thresholds, especially when MPN (run f0) was replaced with MPL (run f2). This increase in areal coverage for the 0.01-in. threshold was accompanied by an increase in rain rate. For the 0.5-in. threshold, increases in rain rate were usually small and significant only in the case of MPL. Both of the above changes in microphysics, in runs using KF and MRF, resulted in the largest positive impact (compared to all other tested physical schemes changes) on rain volume at all times. Increases were often twice as large for the 0.5-in. threshold compared to the 0.01-in. threshold and exceeded 100% for the 0.5-in. threshold for both f2 and f3 in the last 6-h period. These results (supported by subjective analyses) imply that both MPL and MPF produce larger areas of heavier rainfall amounts as compared to runs using MPN. In addition, runs that use MPL often produced limited areas of excessive rainfall amounts (e.g., Fig. 2b). These results are strictly valid only when KF is used, but information found in upcoming table extends these results to simulations using other convective treatments.
The expressions f̂12/f0 and f̂13/f0 in Table 7 indicate values of the synergistic term normalized by the control run value. For rain rate, synergistic terms were statistically insignificant, implying that the impact on rain rate of the microphysics used is not affected by the PBL scheme used.
Regarding rain volume, the synergistic terms (f̂12/f0 and f̂13/f0) for the 0.01-in. threshold were statistically insignificant with an exception for MPL microphysics during the last 6-h forecast period. For the 0.5-in. threshold, these schemes’ interactions were large and negative after the 0–6-h forecast, especially for MPL in the last 6 h. Thus, it appears the use of ETA limits the impacts of changes in the microphysical scheme. A subjective analysis of the total and convective part of the rainfall indicated that greater moisture in the boundary layer causes more frequent triggering of the convective scheme, leading to more of the rainfall produced by deep convection at the expense of the grid-resolved component, possibly explaining the negative values of the synergistic terms.
2) Change from MRF to ETA combined with changes in convective treatment
Factor separation evaluation of the impact from changes of KF to NC (run f4) and form KF to BMJ (run f5) is presented in Table 8. The largest positive impact on rain rate, compared to impacts produced by changing all other physical schemes, for both the 0.01- and 0.5-in. thresholds, was due to a switch from KF to NC. Although areal coverage decreased, changes were not statistically significant. Figure 2c is an example of a case in which during the early forecast periods areal coverage in the NC runs was considerably smaller but with heavier intensities as compared to runs that used KF and BMJ (Figs. 2a and 2d). It should be noted that in the present study NC often had a higher ETS than runs with a convective scheme, especially at earlier times. This result differs from that of Gallus and Segal (2001) who found in the simulation of warm season cases with a 10-km version of the Eta Model that the run using no convective scheme performed significantly worse than runs using the BMJ or KF schemes. This implies that initialization using the LAPS diabatic analysis (as done here but not in Gallus and Segal 2001) likely helped the NC runs to perform better than they would have otherwise. Rain volume was not significantly impacted by a change from KF to NC.
Previous studies by Gallus and Segal (2001) and Jankov and Gallus (2004) have indicated that Eta Model runs using the BMJ scheme usually produce much wider areas of lighter rainfall amounts compared to runs using KF. In the present study, when KF was replaced by BMJ ( f5 run), the subjective analysis identified the same trend (e.g., Figs. 2a and 2d). For the light threshold at all times a considerable increase in areal coverage occurred (Table 8) when KF was replaced with BMJ. In addition, rain rate and volume typically also decreased but these changes were not statistically significant. For the 0.5-in. threshold the change from KF to BMJ did not impact areal coverage or rain rate significantly, but rain volume did decrease markedly. Synergistic terms (f̂15) or both rain rate and volume were statistically insignificant, implying the PBL scheme does not strongly influence the sensitivity to the convective scheme in our sample of eight cases. This finding was also true for synergistic terms relating to changes from KF to NC and from MRF to ETA (f̂14).
3) Change from KF to NC or BMJ combined with changes in microphysical schemes
Rain volume synergistic terms related to a switch to NC and a change in microphysics to MPL or MPF (f̂24 and f̂34) in Table 8) likewise were not statistically significant. Because results with the KF scheme (Table 7) showed a large impact on rain volume when the microphysical scheme was varied, one might expect even larger impacts when no convective scheme was used since all of the rainfall is produced by the microphysical scheme. However, the variability and statistical insignificance of these synergistic terms indicates that a complex interaction occurs between KF and the microphysics such that the use of no convective scheme does not necessarily result in more sensitivity to the choice of microphysics.
The rain-rate-related synergistic terms associated with a switch to BMJ and a change in microphysics to MPL and MPF were almost always negative (not shown), agreeing with the well-known characteristic of BMJ to generate large areas of light rainfall while substantially drying the atmosphere so that grid-resolved precipitation is often small. Rain-volume-related synergistic terms were generally large and negative especially for the heavier threshold at later times implying that the BMJ and KF schemes exert very different impacts on grid-resolved precipitation processes. Because BMJ generally reduced the microphysical scheme contribution to precipitation, the large positive impact of switching microphysical schemes that existed when KF or NC was used was markedly reduced although still present [e.g., the 180% increase in rain volume that occurred in the 18–24-h period in the KF runs where MPN was switched to MPL (Table 7) decreased to a 49% increase (not shown)].
4. Summary and concluding remarks
The main goal of the present study was to note and quantify general trends in the impact of various physical schemes and their interactions on warm season MCS rainfall forecasts. Knowledge of how different physical schemes or their combinations influence rainfall forecasts may be of major importance in designing and interpreting mixed-physics ensembles. To pursue this goal, a matrix of 18 WRF model configurations, with 12-km grid spacing, was created using different physical scheme combinations for eight IHOP MCS cases. For each case, three different treatments of convection were used (KF, BMJ, and the use of no convective scheme), with three different microphysical schemes (MPN, MPL, and MPF) and two different PBL schemes (MRF and ETA). All runs were initialized with a diabatic Local Analysis and Prediction System (LAPS) hot start initialization (Jian et al. 2003). Also, it should be noted that for the majority of cases the MCS systems dominated the rainfall field and were captured in the interior of the domain.
An analysis of ETS and bias indicated that no single model configuration was clearly better than the rest. The best configuration varied both with time and rainfall threshold. Objective testing of sensitivity to physical scheme changes was performed by evaluating correspondence ratio and squared correlation coefficient values. Both objective measures were computed for sets of two model runs in which two of three model physical schemes were held fixed and the third varied (e.g., the PBL and the convective schemes held fixed while the microphysical scheme varied). Both the correspondence ratio and the correlation coefficient indicated that the highest sensitivity is to the choice of convective treatment, with less sensitivity to the PBL scheme, and the least to microphysics. In addition, the correspondence ratio for light rainfall indicated that sensitivity was highest during the first 6 h, while it was highest at later times for heavier rainfall.
Additional testing of sensitivity of rain rate and rain volume to physics changes was performed using the factor separation method (Stein and Alpert 1993). This method was used to quantify the impacts of the variation of two different physical schemes as compared to a “control run” (KF-MRF-MPN; chosen to match the real-time model configuration used by NOAA’s FSL during the IHOP experiment) and their interaction (synergy) on the simulated rainfall. Statistical significance of the obtained results was tested by following a resampling method suggested by Hamill (1999). A change from KF to NC significantly increased system rain rate. A change from KF to BMJ significantly increased areal coverage of lighter rainfall while lowering system rain rates (though not significantly) compared to KF runs. In general, changes in convective treatment were found to have the largest impact on rain rate when KF was replaced with NC no matter what microphysical and PBL schemes were used. Regarding rain volume, the microphysical scheme choice exerted the largest impact in NC runs and least impact in BMJ runs, as would be expected by the amount of grid-resolved precipitation likely to occur in each.
The impact of interactions (synergy) of different physical schemes, though occasionally of comparable magnitude to that occurring from a change in one scheme alone, was found to vary greatly and typically not to be statistically significant (in our limited sample of eight cases). One exception was for the interaction of ETA with MPL or MPF, which did significantly reduce the rain volume increase that had been noted for the heavier threshold when the microphysics were switched from MPN. These results suggest that most of the significant trends noted for a switch in one physical process scheme (e.g., increase in rain rate when KF is switched to NC) remain consistent even when other physical process schemes are changed. A switch from MPN to either MPL or MPF increased rain volume markedly no matter what convective and PBL schemes were used. A switch from KF to BMJ decreased rain volume, especially for heavier amounts, regardless of what microphysics and PBL schemes were used.
In conclusion, the results imply that if an ensemble designed for MCS rainfall prediction lacks sufficient spread, model runs with different convective schemes should be included as an efficient way to increase spread substantially. On the other hand, if rain volume is a desired quantity (e.g., hydrological purposes), model runs with MPL and MPF microphysical schemes may require different bias correction or weighting in an ensemble compared to runs using MPN.
Future work will focus on more detailed case analyses in order to relate the explicit interaction of physics schemes to the larger-scale environment. These detailed case analyses along with the more general findings from the present study will be used to design and later interpret results from a mixed-physics ensemble.
Acknowledgments
The authors thank three anonymous reviewers for their comments, which helped to improve the manuscript. In addition Linda Wharton at NOAA’s Forecast Systems Laboratory and Eric Aligo and Daryl Herzmann at Iowa State University assisted with the computational work. This research was funded by NSF Grant 0226059 and by a NOAA grant from the U.S. Weather Research Program administered through the Forecast Systems Laboratory. Support was also provided by the Iowa Agriculture and Home Economics Experiment Station Project 3803.
REFERENCES
Alhamed, A., Lakshmivarahan S. , and Stensrud D. J. , 2002: Cluster analysis of multimodel ensemble data from SAMEX. Mon. Wea. Rev., 130 , 226–256.
Baldwin, M. E., and Mitchell K. E. , 1997: The NCEP hourly multi-sensor U.S. precipitation analysis for operations and GCIP research. Preprints, 13th Conf. on Hydrology, Long Beach, CA, Amer. Meteor. Soc., 54–55.
Bernardet, L. R., Nance L. , Chuang H-Y. , Loughe A. , and Koch S. E. , 2004: Verification statistics for the NCEP WRF pre-implementation test. Part 1: Deterministic verification of ensemble members. Preprints, Fifth Joint WRF/14th MM5 User’s Workshop, Boulder, CO, NCAR/Mesoscale and Microscale Meteorology Division, 229–232.
Betts, A. K., 1986: A new convective adjustment scheme. Part I: Observational and theoretical basis. Quart. J. Roy. Meteor. Soc., 112 , 677–692.
Betts, A. K., and Miller M. J. , 1986: A new convective adjustment scheme. Part II: Single column tests using GATE wave, BOMEX, ATEX and arctic air-mass data sets. Quart. J. Roy. Meteor. Soc., 112 , 693–709.
Bright, D. R., and Mullen S. L. , 2002: The sensitivity of the numerical simulation of the southwest monsoon boundary layer to the choice of PBL turbulence scheme in MM5. Wea. Forecasting, 17 , 99–114.
Doswell C. A. III, , Brooks H. E. , and Maddox R. A. , 1996: Flash flood forecasting: An ingredients-based methodology. Wea. Forecasting, 11 , 560–581.
Du, J., Mullen S. L. , and Sanders F. , 1997: Short-range ensemble forecasting of quantitative rainfall. Mon. Wea. Rev., 125 , 2427–2459.
Du, J., and Coauthors, 2004: The NOAA/NWS/NCEP short-range ensemble forecast (SREF) system: Evaluation of an initial condition vs. multi-model physics ensemble approach. Preprints, 16th Conf. on Numerical Weather Prediction, Seattle, WA, Amer. Meteor. Soc., CD-ROM, 21.3.
Ferrier, B. S., Jin Y. , Lin Y. , Black T. , Rogers E. , and DiMego G. , 2002: Implementation of a new grid-scale cloud and rainfall scheme in the NCEP Eta Model. Preprints, 15th Conf. on Numerical Weather Prediction, San Antonio, TX, Amer. Meteor. Soc., 280–283.
Fritsch, J. M., and Carbone R. E. , 2004: Improving quantitative precipitation forecasts in the warm season. Bull. Amer. Meteor. Soc., 85 , 955–965.
Gallus W. A. Jr., , 1999: Eta simulations of three extreme rainfall events: Impact of resolution and choice of convective scheme. Wea. Forecasting, 14 , 405–426.
Gallus W. A. Jr., , and Segal M. , 2000: Sensitivity of forecast rainfall in a Texas convective system to soil moisture and convective scheme. Wea. Forecasting, 15 , 509–526.
Gallus W. A. Jr., , and Segal M. , 2001: Impact of improved initialization of mesoscale features on convective system rainfall in 10-km Eta simulations. Wea. Forecasting, 16 , 680–696.
Grimit, E. P., and Mass C. F. , 2002: Initial results of a mesoscale short-range ensemble forecasting system over the Pacific Northwest. Wea. Forecasting, 17 , 192–205.
Hamill, T. M., 1999: Hypothesis test for evaluating numerical precipitation forecasts. Wea. Forecasting, 14 , 155–167.
Hamill, T. M., and Colucci S. J. , 1997: Verification of Eta–RSM short-range ensemble forecasts. Mon. Wea. Rev., 125 , 1312–1327.
Hamill, T. M., and Colucci S. J. , 1998: Evaluation of Eta–RSM ensemble probabilistic rainfall forecasts. Mon. Wea. Rev., 126 , 711–724.
Hong, S-Y., Juang H-M. H. , and Zhao Q. , 1998: Implementation of prognostic cloud scheme for a regional spectral model. Mon. Wea. Rev., 126 , 2621–2639.
Janjić, Z. I., 1994: The step-mountain Eta coordinate model: Further developments of the convection closure schemes. Mon. Wea. Rev., 122 , 927–945.
Jankov, I., and Gallus W. A. Jr., 2004: Contrast between good and bad forecasts of warm season MCS rainfall. J. Hydrol., 288 , 122–152.
Jian, G-J., Shieh S-L. , and McGinley J. A. , 2003: Precipitation simulation associated with Typhoon Sinlaku (2002) in Taiwan area using the LAPS diabatic initialization for MM5. Terr. Atmos. Oceanic Sci., 14 , 261–288.
Kain, J. S., and Fritsch J. M. , 1993: The role of the convective “trigger function” in numerical prediction of mesoscale convective systems. Meteor. Atmos. Phys., 49 , 93–106.
Lin, Y-L., Farley R. D. , and Orville H. D. , 1983: Bulk scheme of the snow field in a cloud model. J. Climate Appl. Meteor., 22 , 1065–1092.
Molteni, F., Buizza R. , Palmer T. N. , and Petroliagis T. , 1996: The ECMWF ensemble prediction system: Methodology and validation. Quart. J. Roy. Meteor. Soc., 122 , 73–119.
Murphy, J. M., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting, 8 , 281–293.
Nicholls, N., 2001: The insignificance of significance testing. Bull. Amer. Meteor. Soc., 82 , 981–986.
Schaefer, J. T., 1990: The critical success index as an indicator of warning skill. Wea. Forecasting, 5 , 570–575.
Stein, U., and Alpert P. , 1993: Factor separation in numerical simulations. J. Atmos. Sci., 50 , 2107–2115.
Stensrud, D. J., and Fritsch J. M. , 1994: Mesoscale convective systems in weakly forced large-scale environments. Part III: Numerical simulations and implications for operational forecasting. Mon. Wea. Rev., 122 , 2084–2104.
Stensrud, D. J., and Wandishin M. S. , 2000: The correspondence ratio in forecast evaluation. Wea. Forecasting, 15 , 593–602.
Stensrud, D. J., Brooks H. E. , Du J. , Tracton M. S. , and Rogers E. , 1999a: Using ensembles for short-range forecasting. Mon. Wea. Rev., 127 , 433–446.
Stensrud, D. J., Manikin G. S. , Rogers E. , and Mitchell K. E. , 1999b: Importance of cold pools to NCEP mesoscale Eta Model forecasts. Wea. Forecasting, 14 , 650–670.
Stensrud, D. J., Bao J-W. , and Warner T. T. , 2000: Using initial condition and model physics perturbations in short-range ensemble simulations of mesoscale convective systems. Mon. Wea. Rev., 128 , 2077–2107.
Tracton, M. S., and Kalnay E. , 1993: Operational ensemble prediction at the National Meteorological Center: Practical aspects. Wea. Forecasting, 8 , 379–398.
Troen, I., and Mahrt L. , 1986: A simple model of the atmospheric boundary layer: Sensitivity to surface evaporation. Bound.-Layer Meteor., 47 , 129–148.
Wang, W., and Seaman N. L. , 1997: A comparison study of convective schemes in a mesoscale model. Mon. Wea. Rev., 125 , 252–278.
Weckwerth, T. M., and Coauthors, 2004: An overview of the International H2O Project (IHOP_2002) and some preliminary highlights. Bull. Amer. Meteor. Soc., 85 , 253–277.
Wisse, J. S. P., and Vila-Guerau de Arellano J. , 2004: Analysis of the role of the planetary boundary layer schemes during a severe convective storm. Ann. Geophys., 22 , 1861–1874.
Values of (a) CR and (b) r 2 for changes in microphysical (mp), PBL (pbl), and convective schemes (cs). Results are presented for the two thresholds indicated along the abscissa (0.01 and 0.5 in.) and for the four 6-hourly periods ending at the times indicated in the legend.
Citation: Weather and Forecasting 20, 6; 10.1175/WAF888.1
Accumulated rainfall in the simulated domain for the 6–12-h forecast period for the 19 Jun 2002 run initialized at 1200 UTC for different model runs: (a) KF-MRF-MPN (control run), (b) KF-MRF-MPL, (c) NC-MRF-MPN, and (d) BMJ-MRF-MPN. Contours are shown for 1, 10, 50, and 100 mm.
Citation: Weather and Forecasting 20, 6; 10.1175/WAF888.1
Notation used for different physical schemes in the present study.
ETS values for four rainfall thresholds for eight IHOP cases for the 0–6-h forecast period, with relatively “good” forecasts in boldface and relatively “bad” forecasts in italic (see section 3a for definition of good and bad).
ETS and bias (in parentheses) values averaged over the eight IHOP cases for different physical scheme combinations for the 0–6-h forecast period for four different rainfall thresholds. The notation presented in Table 1 is used to indicate different model configurations with physical schemes that are changed from the control run (KF-MRF-MPN) presented in boldface. Boldface ETS values indicate the best single value for each threshold.
Observed and control run areal coverage, rain rate, and rain volume. Areal coverages for observations and the control run are expressed as numbers of grid points where the rainfall amount exceeded a specified thresholds.
Time series of percentage changes in system rain rate and domain rain volume (averaged for all eight cases) due to physics changes ( f1 represents rainfall in the run where the PBL scheme is changed from MRF to ETA, f2 represents rainfall in the run where the microphysics is changed from MPN to MPL, and f3 represents rainfall in the run where the microphysics is changed from MPN to MPF) averaged over points where rainfall exceeded specified thresholds (0.01 and 0.5 in.). Here, f0 represents rainfall in the control run (KF-MRF-MPN). Values presented in italic bold, bold, and italic face indicate results that are statistically significant at the 95%, 90%, and 80% confidence levels, respectively, when the test sample consists of all 6-hourly periods combined. Here, f̂12 and f̂23 represent the corresponding synergistic terms, while A1, A2, and A3 stand for the areal coverage for runs with the physical scheme changed. All values are expressed as a percentage relative to the control run rain rate, rain volume, and areal coverage, which are presented in Table 6.
As in Table 7 except for f4 and f5, where f4 stands for rainfall in the run where no convective scheme (NC) is used and f5 stands for rainfall in the run where the BMJ scheme is used.
