1. Introduction
There continues to be a need for reliable probabilistic forecasts of hazardous convective weather phenomena (e.g., National Research Council 2010), owing to the limited skill of deterministic forecasts of convection (e.g., Zhang et al. 2003). Fortunately, modern computing systems are now making it possible to run ensembles of forecasts at grid spacings where convection is explicitly represented and at least marginally resolved. [Hereafter, we refer to model configurations with horizontal grid spacing from 1–5 km as “convection allowing,” following Clark et al. (2009).] Past studies have established that even relatively small ensembles at these resolutions tend to produce more skillful forecasts of heavy precipitation than larger, coarse-resolution ensembles (e.g., Clark et al. 2009). However, how to best design, implement, and analyze such storm-scale ensembles remains a subject of active research. For global medium-range ensemble prediction systems, the diversity in forecast outcomes primarily comes from perturbations to the initial conditions (ICs), as these uncertainties outweigh uncertainties in model physics on longer time scales (e.g., Gilmour et al. 2001). But at the temporal and spatial scales of convective weather, uncertainties in the representation of boundary layer and cloud-microphysical processes may be just as important as those associated with the ICs (e.g., Stensrud et al. 2000; Eckel and Mass 2005). Furthermore, there are myriad ways in which the diversity in the ICs and/or physical parameterizations can be implemented in a storm-scale ensemble. For example, the National Centers for Environmental Prediction (NCEP) runs a regional ensemble, the short-range ensemble forecast (SREF; Du et al. 1997) system, which uses multiple model cores and model physics, but does not have sufficient resolution to explicitly predict convection. This research aims to examine the strengths and weaknesses of some of the methods for incorporating initial-condition uncertainty and diverse physical parameterizations into convection-allowing ensemble forecasts.
Several research and operational forecasting centers around the world have been testing convection-allowing ensemble forecast systems in recent years (e.g., Xue et al. 2007, 2011; Hohenegger et al. 2008; Schwartz et al. 2010; Vié et al. 2011; Clark et al. 2012; Leoncini et al. 2013). One such ensemble system that has received substantial attention is the storm-scale ensemble forecast (SSEF) system run by the Center for the Analysis and Prediction of Storms (CAPS) at the University of Oklahoma since 2007 (e.g., Xue et al. 2007, 2011; Clark et al. 2012). This system has demonstrated promising results and has led to the development of numerous new analysis and postprocessing methods for ensembles (e.g., Clark et al. 2013). However, considering that the SSEF takes its initial and lateral boundary condition perturbations from selected members of an ensemble (the NCEP SREF) that uses different model physics and runs at coarser resolution, it is possible that improvements could be made by employing other methods for initializing the ensemble. In particular, the methods of ensemble-based data assimilation [as reviewed by Meng and Zhang (2011)] provide some attractive benefits for initializing a storm-scale ensemble, in that the assimilation of observations, the generation of the ensemble ICs, and the forecasts can all be run with a single integrated system. Furthermore, such an approach assimilates observations and develops initial perturbations in a flow-dependent manner; in other words, the way that assimilated observations influence their surrounding environment is a function of the atmospheric flow pattern at that given time.
Recent works by Romine et al. (2013) and Schwartz and Liu (2014) presented the results of using a continuously cycled ensemble-based assimilation system to initialize deterministic convection-allowing forecasts. Romine et al. (2013) found that the choice of physical parameterizations in the assimilation system introduced unique biases into the analyses over time. Furthermore, they found that convection-allowing forecasts initialized with this system were generally more skillful than similar forecasts initialized with the NCEP Global Forecast System (GFS) through forecast hours 12–18, but that the GFS-initialized forecasts performed better at longer lead times. The work presented in this study extends the work of Romine et al. (2013) by using a similar assimilation system but initializing convection-allowing ensemble forecasts, rather than single deterministic forecasts. Furthermore, their finding of consistent biases in their analyses resulting from specific parameterization schemes motivates the exploration of using diversity of parameterizations and a partial (rather than continuous) cycling strategy in the ensemble assimilation-forecast system as a potential way to mitigate these biases.
The period of study will be 8–11 June 2010, when numerous mesoscale convective systems (MCSs) occurred in the central and southern United States that produced heavy to extreme precipitation amounts (Fig. 1a; Higgins et al. 2011; Schumacher et al. 2013). There were two primary focusing mechanisms for organized deep convection during this three-day period: a baroclinic zone that was oriented approximately from west to east across the Great Plains, and a mesoscale convective vortex (MCV) that moved northeastward from Texas into Arkansas. Some of the mechanisms responsible for the heavy rainfall associated with the MCV were addressed by Schumacher et al. (2013), including the importance of convective latent heating and low-level evaporative cooling in the maintenance of the MCV and the associated initiation and organization of convection. Schumacher et al. (2013) also presented some of the characteristics of the SSEF forecasts for this time period, finding that there was a wide range of forecast outcomes within the ensemble for the MCV-related rainfall. In this study, we will evaluate the SSEF forecasts in the context of four other ensemble configurations, with a particular focus on how diversity in ICs and physical parameterizations influence the forecasts in terms of both quantitative metrics and the representation of physical processes. Although we will focus on only a three-day period rather than an entire warm season, the variety of atmospheric processes and heavy rainfall mechanisms that occurred during this time period should provide some insights into the strengths and weaknesses of these configurations.
(a) NCEP stage-IV gridded precipitation analysis (mm; Lin and Mitchell 2005) for the 72-h period ending 11 Jun 2010. (b)–(d) Radar reflectivity mosaic for 0900 UTC (b) 9 Jun, (c) 10 Jun, and (d) 11 Jun 2010. MCSs discussed in the text are annotated with N1–N3 and S1–S3. Images were obtained from the Iowa Environmental Mesonet. An animation of radar images from 8–11 Jun is available in the online supplemental material.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
(a) NCEP stage-IV gridded precipitation analysis (mm; Lin and Mitchell 2005) for the 72-h period ending 11 Jun 2010. (b)–(d) Radar reflectivity mosaic for 0900 UTC (b) 9 Jun, (c) 10 Jun, and (d) 11 Jun 2010. MCSs discussed in the text are annotated with N1–N3 and S1–S3. Images were obtained from the Iowa Environmental Mesonet. An animation of radar images from 8–11 Jun is available in the online supplemental material.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
(a) NCEP stage-IV gridded precipitation analysis (mm; Lin and Mitchell 2005) for the 72-h period ending 11 Jun 2010. (b)–(d) Radar reflectivity mosaic for 0900 UTC (b) 9 Jun, (c) 10 Jun, and (d) 11 Jun 2010. MCSs discussed in the text are annotated with N1–N3 and S1–S3. Images were obtained from the Iowa Environmental Mesonet. An animation of radar images from 8–11 Jun is available in the online supplemental material.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Section 2 provides a brief overview of the 8–11 June 2010 event, and section 3 discusses the configuration of the assimilation and forecast systems. The results of the ensemble-based assimilation itself are presented in section 4, and the evaluation of the ensemble forecasts is given in section 5. Section 6 provides discussion of the implications of these results and their potential applications, and section 7 concludes the paper.
2. Overview of the 8–11 June 2010 period
The large-scale 500-hPa flow over the United States during the period of interest was characterized by an upper-level trough over the northwestern part of the country, and a ridge over the central and southern United States. (Fig. 2a). There was also a weak trough over Texas, which is a reflection of the long-lived MCV that moved across this region (e.g., Schumacher et al. 2013). Associated with the midtropospheric height gradient across the central United States was a low-level baroclinic zone that extended eastward from the Rocky Mountains to the Ohio Valley (Fig. 2b). There was a broad region of southerly 850-hPa winds across the Great Plains with average speeds exceeding 7.5 m s−1, and embedded regions of stronger low-level winds near the Oklahoma and Texas Panhandles and in south Texas near the Gulf of Mexico (Fig. 2b). These areas of strong low-level winds are a regular feature of the warm-season climatology in the Great Plains (e.g., Bonner 1968), and the average wind speeds during this three-day period were slightly stronger than the long-term average (not shown).
(a) NARR (Mesinger et al. 2006) 500-hPa geopotential height (contoured every 30 m), geopotential height anomaly relative to the 1979–2001 climatology (shaded in m), and wind barbs (half barb = 5 kt, full barb = 10 kt, pennant = 50 kt; 1 kt = 0.5144 m s−1) averaged over 8–11 Jun 2010. (b) As in (a), but for 850-hPa temperature (shaded in °C), wind speed (contoured every 2.5 m s−1), and wind barbs.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
(a) NARR (Mesinger et al. 2006) 500-hPa geopotential height (contoured every 30 m), geopotential height anomaly relative to the 1979–2001 climatology (shaded in m), and wind barbs (half barb = 5 kt, full barb = 10 kt, pennant = 50 kt; 1 kt = 0.5144 m s−1) averaged over 8–11 Jun 2010. (b) As in (a), but for 850-hPa temperature (shaded in °C), wind speed (contoured every 2.5 m s−1), and wind barbs.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
(a) NARR (Mesinger et al. 2006) 500-hPa geopotential height (contoured every 30 m), geopotential height anomaly relative to the 1979–2001 climatology (shaded in m), and wind barbs (half barb = 5 kt, full barb = 10 kt, pennant = 50 kt; 1 kt = 0.5144 m s−1) averaged over 8–11 Jun 2010. (b) As in (a), but for 850-hPa temperature (shaded in °C), wind speed (contoured every 2.5 m s−1), and wind barbs.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Intersections of a low-level jet and a baroclinic zone are common breeding grounds for MCSs (e.g., Augustine and Caracena 1994; Tuttle and Davis 2006), and indeed there were MCSs that developed across the central United States during the evenings of 8, 9, and 10 June (Figs. 1b–d). Furthermore, MCSs developed each night in association with the MCV over Texas and Arkansas. On 8–9 June, an MCS developed along the baroclinic zone in eastern Kansas and western Missouri (MCS N1 in Fig. 1b), which moved slowly and produced heavy rainfall. Concurrently, MCS S1 developed and remained nearly stationary over central Texas (Fig. 1b). Both of these systems weakened during the day on 9 June, and two more MCSs developed on the evening of 9 June and persisted through the morning of 10 June, here indicated as MCS N2 in Nebraska and S2 in Texas (Fig. 1c). These MCSs mainly weakened during the day on 10 June, with yet another round of convective development on the evening of 10 June and the morning of 11 June (MCSs N3 and S3 in Fig. 1d). An animation of radar images from this full three-day period is available in the online supplemental material. These six MCSs, along with additional shorter-lived convective systems, produced broad swaths of more than 50 mm of rain with some local areas receiving more than 300 mm (Fig. 1a). Flash flooding was reported in many of the areas affected by the rainfall from these MCSs, including the deadly flash flood that occurred at the Albert Pike Recreational Area in Arkansas [(National Oceanic and Atmospheric Administration) NOAA 2013; Schumacher et al. 2013].
3. Methods
a. Data assimilation system and configuration
To assimilate observations and develop ICs for ensemble forecasts, we use revision 5780 of the Data Assimilation Research Testbed (DART; Anderson et al. 2009) software package in conjunction with version 3.4.1 of the Advanced Research version of the Weather Research and Forecasting (WRF) Model (ARW; Skamarock et al. 2008). A 36-member ensemble is run at 36-km horizontal grid spacing and 51 vertical levels (with stretched grid and a model top at 50 hPa) during the assimilation process on a grid covering much of North America (Fig. 3). Multiple configurations of this ensemble are tested in this study, which will be discussed in detail in section 3b. DART is configured to use an ensemble adjustment Kalman filter (Anderson 2001). To maintain spread in the ensemble during the assimilation cycle, spatially and temporally varying inflation is applied to the prior perturbations about the mean and a sampling error correction is applied (Anderson 2009). To eliminate spurious relationships far from the observation location due to sampling error, horizontal and vertical localization using a Gaspari and Cohn (1999) function is employed, with influence going to zero approximately 580 km away from the observation in the horizontal and 900 hPa away in the vertical. If there are more than 1600 observations within this localization region, the localization length scale is reduced linearly using the method of Torn (2010). The DART options used here are generally similar to those used by Romine et al. (2013), with the primary exceptions being that we use a smaller ensemble (36 vs 50 members) and a somewhat smaller localization radius on a coarser horizontal grid (36- vs 15-km grid spacing) but finer vertical resolution (51 vs 35 vertical levels).
Locations of the WRF-DART assimilation domain (the outer rectangle), the 4-km forecast domain (inner rectangle), the full verification domain (diagonal hatching), and the verification subdomain (cross hatching). (The verification subdomain is included in the calculations over the full verification domain.)
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Locations of the WRF-DART assimilation domain (the outer rectangle), the 4-km forecast domain (inner rectangle), the full verification domain (diagonal hatching), and the verification subdomain (cross hatching). (The verification subdomain is included in the calculations over the full verification domain.)
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Locations of the WRF-DART assimilation domain (the outer rectangle), the 4-km forecast domain (inner rectangle), the full verification domain (diagonal hatching), and the verification subdomain (cross hatching). (The verification subdomain is included in the calculations over the full verification domain.)
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
b. Data assimilation experiments
Four different ensemble and assimilation strategies were tested and then used to initialize convection-allowing ensemble forecasts, and these configurations are summarized schematically in Fig. 4. The configurations with the “single” prefix use the same set of physics parameterizations for all members, whereas the “mixphys” configurations use a variety of physics parameterizations within the ensemble, which are given in Table 1. The parameterizations used in the single ensembles were chosen because they were the parameterizations used by the SSEF members with the best forecasts of the 8–11 June 2010 MCSs in the south-central United States in Schumacher et al. (2013). The specific choices of parameterizations in the mixed-physics ensemble are somewhat arbitrary, but are designed to provide considerable diversity among the schemes available in ARW and as one way to account for model error in the assimilation and forecast process (e.g., Fujita et al. 2007; Wheatley and Stensrud 2010; Meng and Zhang 2011). Romine et al. (2013) established the consistent biases that WRF-DART configurations with single physics parameterizations can introduce in data assimilation (along with the effects those biases can have on convection-allowing forecasts), and the mixphys experiments are designed to identify differences between single-physics and mixed-physics assimilation and forecast systems. The single-physics configuration chosen here will, of course, introduce its own biases, though how those biases compare to other potential choices of parameterizations in a single-physics assimilation and forecast system has not been extensively tested here.
Schematic diagram showing the four WRF-DART ensemble assimilation and forecast configurations. The “single_cc” and “mixphys_cc” experiments are initialized at 0000 UTC 6 Jun 2010 and continuously cycled through 0000 UTC 10 Jun 2010, with 4-km ensemble forecasts initialized at 0000 UTC 8, 9, and 10 Jun. The “single_24hr” and “mixphys_24hr” experiments are initialized 24 h prior to the initialization of the 4-km ensemble forecasts. The colored lines indicate ensemble members with different physics parameterizations.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Schematic diagram showing the four WRF-DART ensemble assimilation and forecast configurations. The “single_cc” and “mixphys_cc” experiments are initialized at 0000 UTC 6 Jun 2010 and continuously cycled through 0000 UTC 10 Jun 2010, with 4-km ensemble forecasts initialized at 0000 UTC 8, 9, and 10 Jun. The “single_24hr” and “mixphys_24hr” experiments are initialized 24 h prior to the initialization of the 4-km ensemble forecasts. The colored lines indicate ensemble members with different physics parameterizations.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Schematic diagram showing the four WRF-DART ensemble assimilation and forecast configurations. The “single_cc” and “mixphys_cc” experiments are initialized at 0000 UTC 6 Jun 2010 and continuously cycled through 0000 UTC 10 Jun 2010, with 4-km ensemble forecasts initialized at 0000 UTC 8, 9, and 10 Jun. The “single_24hr” and “mixphys_24hr” experiments are initialized 24 h prior to the initialization of the 4-km ensemble forecasts. The colored lines indicate ensemble members with different physics parameterizations.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Details of the model configurations for the WRF-DART experiments. The first row shows the configuration for all members of the single ensembles, and the remaining rows show the configurations for the members of the mixphys ensembles. All members use 36-km horizontal grid spacing and 51 vertical levels that are stretched so that the resolution is highest at low levels. Microphysics schemes include Thompson et al. (2008), Morrison et al. (2005), WRF single-moment 6-class microphysics scheme (WSM6; Hong and Lim 2006), and Goddard (Tao et al. 1989) using hail and graupel. Cumulus parameterizations include Kain–Fritsch (KF; Kain 2004); Tiedtke (Tiedtke 1989; Zhang et al. 2011); Betts–Miller–Janjić (BMJ; Janjić 1994, 2000); and Grell-3 (G3). Land surface models include the Noah (Chen and Dudhia 2001) and Rapid Update Cycle (RUC; Smirnova et al. 2000). Planetary boundary layer (PBL) parameterizations include Mellor–Yamada–Janjić (MYJ; Mellor and Yamada 1982; Janjić 2002) and Yonsei University (YSU; Noh et al. 2003). Technical descriptions of the parameterizations are given by Skamarock et al. (2008). Configurations are identical for the 4-km ensemble forecasts initialized from the WRF-DART assimilation, except that no cumulus parameterization is used and only the first 24 members are used. For all of the configurations, the WRF state variables used are u, υ, w, T, QVAPOR, QCLOUD, QRAIN, QICE, QGRAUP, QSNOW, with MU and PH used in the vertical coordinate transforms, and T2, Q2, U10, V10, and PSFC used in the forward operators for the surface observations. As discussed in the text and appendix, number concentrations of microphysical variables are not defined as state variables.
The use of a mixed-physics configuration that includes a mix of single-moment and double-moment microphysical parameterizations means that the number concentration variables cannot be used as state variables for the ensemble filter. This is perhaps not a desirable scenario, but experimentation showed that the influence of the initial number concentrations on the precipitation forecasts were less than those that would be seen with a run initialized from a “cold start,” in which no hydrometeors are present in the ICs. The results of these experiments, along with other discussion of this issue, are presented in appendix A. For consistency with the mixed-physics results, the number concentrations are not used as state variables in the single-physics configuration either.
The configurations with the “cc” suffix are continuously cycled starting at 0000 UTC 6 June 2010, whereas the “24hr” configurations are initialized every 24 h starting at 0000 UTC 7 June. As in Torn and Hakim (2008), all of the ensemble assimilation configurations are initialized with the 36-h forecast of the NCEP GFS valid at that time, which are perturbed with random draws from the NCEP background error covariance using the WRF data assimilation system (Barker et al. 2012) to produce an initial ensemble. In the cc configurations, this process only takes place once, at 0000 UTC 6 June, whereas in the “24hr” experiments this process is repeated every 24 h. These experiments are designed to test two possible ensemble configurations that can initialize a convection-allowing forecast ensemble, and to examine differences between a continuously cycled system and one that draws the initial ensemble from an independent source more frequently.
All of the WRF-DART configurations also use perturbed 36-h GFS forecasts as lateral boundary conditions (LBCs) using the fixed covariance perturbation method described by Torn et al. (2006). The perturbations are applied every 6 h through the assimilation cycle with the 36-h GFS forecast valid at that time, and the same perturbed LBCs are used for all of the configurations.
c. Assimilation of observations
The WRF-DART system primarily assimilates observations from the NCEP Global Data Assimilation System (GDAS), which are summarized in Table 2. This includes radiosondes (wind components, temperature, specific humidity, and altimeter setting are assimilated), surface observations from ships and buoys (wind components, temperature, specific humidity, and altimeter setting are assimilated), Aircraft Meteorological Data Relay (AMDAR; wind components and temperature are assimilated), and atmospheric motion vectors (AMVs; Velden et al. 2005; wind components are assimilated). Surface observations over land and NOAA wind profiler observations, which are processed by the GDAS but not assimilated by that system, are also assimilated here, as are global positioning system (GPS) radio occultation observations (Kursinski et al. 1997) from the Constellation Observing System for Meteorology, Ionosphere and Climate (COSMIC) program. Specific humidity was used as the variable for assimilated moisture observations, consistent with Schwartz and Liu (2014). The observation errors assigned for land surface humidity are relatively large, but are consistent with those assigned by the GDAS for humidity from radiosonde and marine-surface observations. Because most stations observe relative humidity directly before converting to specific humidity or dewpoint, the assigned observation error is a function of relative humidity. Based on a suggestion from an anonymous reviewer, we tested the sensitivity of the results to using dewpoint rather than specific humidity as the assimilated moisture variable, and to using a smaller observation error for surface moisture over land. The results of these tests are presented in appendix B.
Observations assimilated in the WRF-DART system and their assumed errors. “Surface” refers to surface observing stations both over land and ocean. “NCEP stats” refers to the observation error statistics assigned by the NCEP global data assimilation system as obtained from their PREPBUFR format observation files (Keyser 2014).
The assimilated observations were also subject to additional processing in the same manner as described by Romine et al. (2013), including “superobbing” of AMDAR and AMV observations, exclusion of surface observations in complex terrain, and inflation of observation errors for observations near the model lateral boundaries. The horizontal intervals for superobbing were 30 km for AMDAR observations and 60 km for AMVs. Observations more than four standard deviations away from the prior ensemble mean value were rejected. A representative spatial distribution of assimilated observations is shown in Fig. 5 for 0000 UTC 10 June 2010. Neither radar data nor satellite radiances were assimilated in this study; the radar observations were not assimilated because of the focus on forecasts beyond 6 h, where the impact of radar assimilation is small (e.g., Stratman et al. 2013), and satellite radiances were not assimilated because these capabilities are still in their infancy in the DART system (e.g., Jones and Stensrud 2012).
Map showing the locations and types of observations assimilated at 0000 UTC 10 Jun 2010. Here, “ACARS” indicates the AMDAR observations and “SAT” indicates the AMVs. The number below each observation type in the legend indicates the number of individual observations that were assimilated, including those at multiple vertical levels.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Map showing the locations and types of observations assimilated at 0000 UTC 10 Jun 2010. Here, “ACARS” indicates the AMDAR observations and “SAT” indicates the AMVs. The number below each observation type in the legend indicates the number of individual observations that were assimilated, including those at multiple vertical levels.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Map showing the locations and types of observations assimilated at 0000 UTC 10 Jun 2010. Here, “ACARS” indicates the AMDAR observations and “SAT” indicates the AMVs. The number below each observation type in the legend indicates the number of individual observations that were assimilated, including those at multiple vertical levels.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
d. Convection-allowing forecast ensemble
To evaluate their performance as ICs for convection-allowing ensemble forecasts, the analyses from each of the four WRF-DART configurations discussed in section 3c were used to initialize 24-member, 36-h ensemble forecasts. These forecasts were initialized at 0000 UTC 8, 9, and 10 June 2010, using 4-km horizontal grid spacing over a domain covering much of the south-central United States shown in Fig. 3. This domain was designed as a balance between the computational cost of running multiple convection-allowing ensembles and the desire to encompass the features of interest (i.e., both the southern and northern MCSs) in the interior of the domain.
The four convection-allowing ensembles were initialized on a single domain by downscaling the WRF-DART analyses to a 4-km grid using the WRF “ndown” program. They used identical model configurations to their parent assimilation system with the exception of the smaller grid spacing and domain, and the explicit (rather than parameterized) representation of cumulus convection for the 4-km forecasts. In other words, member 1 of the mixphys_cc and mixphys_24hr forecast ensembles used the same physical parameterizations (except for cumulus) as were used in member 1 of the WRF-DART system, and so on. However, only the first 24 members of the ensemble were used to initialize the 4-km forecasts. For the mixed-physics ensembles, initializing the ensemble with the first 24 members (rather than 24 members meeting some other criteria) may mean that the chosen members are not a truly representative sample of the larger ensemble, but the potential effects of this on the forecasts have not been tested.
LBCs for the 4-km forecasts were provided by the GFS forecast initialized 6 h prior to the WRF forecast (i.e., 1800 UTC the previous day). As with the LBCs used for the WRF-DART assimilation system, the LBCs for the 4-km forecasts were perturbed using the fixed covariance perturbation method. All of the ensemble configurations used the same LBC perturbations. A sensitivity test using unperturbed LBCs was also conducted and will be discussed briefly in section 5c. As a point of comparison, these ensemble forecasts will also be evaluated against the SSEF, which was run and evaluated in real time during June 2010.
e. Evaluation of analyses and forecasts
Several methods will be used to evaluate both the analyses generated by the ensemble-based data assimilation system and the output of the 4-km ensemble forecasts. The WRF-DART analyses will be evaluated using standard statistical methods, including the root-mean-square (RMS) error, bias, and ensemble spread of state variables against the observations over the entire domain, as well as by focusing on the analysis of the MCV in the southern United States. The evaluation of the 4-km ensemble forecasts will primarily focus on precipitation, but relationships between the precipitation forecasts and other atmospheric fields will also be presented where appropriate. Two primary methods for evaluating the probabilistic precipitation forecasts are the area under the receiver operating characteristic (ROC) curve and rank histograms (e.g., Mason and Graham 2002; Wilks 2011). For all precipitation forecast evaluation, the NCEP stage-IV precipitation analysis (ST4; Lin and Mitchell 2005) was used as the “truth,” and the WRF Model output was regridded to the stage-IV grid using a neighbor-budget method.
For the area under the ROC curve, a simple “neighborhood” probability of exceeding a given precipitation threshold was first calculated from the ensemble forecasts by using Eq. (5) of Schwartz et al. (2010) with a neighborhood encompassing the six grid points on each side of a given grid point, for a 13 × 13 gridpoint (52 × 52 km2) neighborhood. The WRF-DART forecasts used all 24 members in the calculation of the probabilities, whereas the SSEF had 15 core members that were used (Xue et al. 2011). The ROC curves, and the areas under those curves, were then calculated based on these neighborhood probabilities and unsmoothed stage-IV analyses using the method of Mason and Graham (2002) and subject to a bootstrap procedure with 500 resamplings to calculate confidence intervals. To examine the spread and bias characteristics of the 4-km ensemble precipitation forecasts, rank histograms of the precipitation accumulations at different lead times are also presented. Individual ensemble members were also evaluated using the fractions skill score (FSS; Roberts and Lean 2008), also using a 52 × 52 km2 neighborhood. These neighborhood methods have been shown to balance the objective evaluation of forecasts at very small scales with subjective evaluations of forecast quality (e.g., Ebert 2008; Schwartz et al. 2010). The scores were calculated over two regions: a large region covering nearly the full forecast domain, and a subdomain covering Texas, Arkansas, Oklahoma, and Louisiana (Fig. 3) to isolate the heavy precipitation associated with the MCV. The scores were calculated using the Meteorological Evaluation Tools (MET) software package, version 4.0.
f. MCV tracking
A vortex-tracking method was used to calculate the path of the MCV in the ensemble forecasts and to diagnose the environmental conditions near the MCV. The model-predicted vortices were tracked using the same method as Schumacher et al. (2013), whereby the vortex location is defined by center of a box with the greatest circulation in the 700–500-hPa layer. For the 4-km forecasts, a 183 km × 183 km (61 × 61 model grid point) box was used, whereas for the model ICs at 36-km grid spacing, a 324 × 324 km2 (9 × 9 grid point) box was used. The vortex location in the North American Regional Reanalysis (NARR) was used as “observations” here because it is an independent analysis and because it is available at 3-h intervals. For the NARR, which is on a 32-km grid, a 288 km × 288 km, or 9 × 9 grid point, box was used for calculating the circulation. The sizes of the boxes used for the NARR and 4-km forecasts are the same as in Schumacher et al. (2013), and the size of the box for the 36-km ICs was chosen to be similar to that used for the NARR.
4. Results: Analyses
In this section, similarities and differences between the analyses produced by the four WRF-DART configurations are presented, beginning with the characteristics of the analyses for state variables over the full domain, followed by more detailed aspects of the analyses of the MCV over the south-central United States. Because we are using both continuously cycled configurations and those only cycled for 24 h, the comparisons between analyses focus primarily on a single 24-h period (e.g., five assimilation cycles) from 0000 UTC 9 June to 0000 UTC 10 June, but the results for earlier time periods are generally similar, and a few results showing the evolution of the assimilation diagnostics with time will also be shown in this section.
Consistent with the results of Romine et al. (2013) and Schwartz and Liu (2014), there is a tendency for the “first guess” (i.e., 6-h model forecasts) in all of the WRF-DART systems to have a warm bias with respect to radiosondes through much of the troposphere, but a cool bias with respect to AMDAR temperature observations, especially above 500 hPa (Figs. 6a,b). However, Ballish and Krishna Kumar (2008) point out that the AMDAR temperature observations may themselves have a warm bias. With respect to radiosonde temperature observations, the configurations with one day of cycling have a greater warm bias than those that are continuously cycled. This is also true with respect to AMDAR temperatures above 700 hPa. The continuously cycled configurations have lower values of both RMSE and total spread (i.e., the sum of the observation error standard deviation and the ensemble spread) in their 6-h forecasts than those cycled for only 24 h, an issue that will be revisited later in this section.
Vertical profiles of RMSE (solid lines), total spread (including both ensemble spread and observation error, dashed lines), and bias (dotted lines) for the WRF-DART configurations and observations of temperature for the five assimilation cycles between 0000 UTC 9 Jun and 0000 UTC 10 Jun 2010. Each of the four WRF-DART configurations is shown as indicated by the legend. (a) First guess (i.e., the 6-h forecast) compared with radiosonde temperatures. (b) First guess compared with AMDAR temperatures. (c) Analysis compared with radiosonde temperatures. (d) Analysis compared with AMDAR temperatures. The number of observations available and the number actually assimilated at each level are shown to the right for each type of observation.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Vertical profiles of RMSE (solid lines), total spread (including both ensemble spread and observation error, dashed lines), and bias (dotted lines) for the WRF-DART configurations and observations of temperature for the five assimilation cycles between 0000 UTC 9 Jun and 0000 UTC 10 Jun 2010. Each of the four WRF-DART configurations is shown as indicated by the legend. (a) First guess (i.e., the 6-h forecast) compared with radiosonde temperatures. (b) First guess compared with AMDAR temperatures. (c) Analysis compared with radiosonde temperatures. (d) Analysis compared with AMDAR temperatures. The number of observations available and the number actually assimilated at each level are shown to the right for each type of observation.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Vertical profiles of RMSE (solid lines), total spread (including both ensemble spread and observation error, dashed lines), and bias (dotted lines) for the WRF-DART configurations and observations of temperature for the five assimilation cycles between 0000 UTC 9 Jun and 0000 UTC 10 Jun 2010. Each of the four WRF-DART configurations is shown as indicated by the legend. (a) First guess (i.e., the 6-h forecast) compared with radiosonde temperatures. (b) First guess compared with AMDAR temperatures. (c) Analysis compared with radiosonde temperatures. (d) Analysis compared with AMDAR temperatures. The number of observations available and the number actually assimilated at each level are shown to the right for each type of observation.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
After assimilating observations, all of the WRF-DART analyses accordingly show better fit to those observations. With respect to radiosonde temperature, the analyses still show a slight warm bias, especially below 800 hPa and above 400 hPa (Fig. 6c). The analyses all fit the AMDAR temperatures well in the midtroposphere, with a slight cool bias aloft, again similar to the results shown by Romine et al. (2013). At the radiosonde sites, the total spread is reduced after observations are assimilated, and the spread in temperature is slightly higher than the RMSE through most of the troposphere (Figs. 6c,d).
The first-guess zonal winds exhibit a slow bias, particularly with respect to radiosondes above approximately 700 hPa (Figs. 7a,b). This is also generally consistent with the results of Romine et al. (2013) and Schwartz and Liu (2014). The magnitude of this bias is greater in the systems cycled for only 24 h. As with the radiosonde temperatures, there is less spread and lower RMSE in the continuously cycled systems. After assimilating the observations, only a very minimal slow bias remains (Figs. 7c,d). There is considerably less bias in the first guesses for the meridional (υ) wind component (not shown). Both the first guesses and the analyses of specific humidity show a slight dry bias at low levels with a slight moist bias aloft (Figs. 8a,b). The mixed-physics configurations are consistently drier than the single-physics configurations. The total spread (dashed lines in Figs. 8a,b) is considerably larger than the RMSE for low-level specific humidity. This is an undesirable artifact of the assumed observation error being comparatively large at high values of specific humidity, which are more common at low levels.
As in Fig. 6, but for the zonal (u) component of the wind.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
As in Fig. 6, but for the zonal (u) component of the wind.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
As in Fig. 6, but for the zonal (u) component of the wind.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
As in Fig. 6, but for radiosonde specific humidity.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
As in Fig. 6, but for radiosonde specific humidity.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
As in Fig. 6, but for radiosonde specific humidity.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
The 24-h cycling period does appear sufficient to achieve a magnitude of prior spread that is similar to that in the continuously cycled configuration, at least for the midlevel wind and temperature fields shown in Fig. 9. Other vertical levels (not shown) demonstrate similar behavior. This suggests that it is possible to “spin up” the ensemble assimilation system over a fairly short period of time, but Fig. 9 also shows that it would not be desirable to use a spinup time that was any shorter than 24 h for this type of assimilation system, as the initial “random” spread is comparatively large and the prior spread in the “24hr” runs does not approach that of the “cc” runs until 24 h of cycling is complete.
Comparison of time series of diagnostics for the assimilation of (a),(b) 500-hPa u and (c),(d) 700-hPa temperatures from radiosondes. Shown are RMSE in the first guess (solid red), spread in the first guess (dashed blue), and RMSE in the analysis (dashed red). The thick lines represent the continuously cycled configurations and the thin lines represent the 24-h configurations with cycles starting at 0000 UTC 7, 8, and 9 Jun. (a),(c) The single-physics configurations and (b),(d) the mixed-physics configurations.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Comparison of time series of diagnostics for the assimilation of (a),(b) 500-hPa u and (c),(d) 700-hPa temperatures from radiosondes. Shown are RMSE in the first guess (solid red), spread in the first guess (dashed blue), and RMSE in the analysis (dashed red). The thick lines represent the continuously cycled configurations and the thin lines represent the 24-h configurations with cycles starting at 0000 UTC 7, 8, and 9 Jun. (a),(c) The single-physics configurations and (b),(d) the mixed-physics configurations.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Comparison of time series of diagnostics for the assimilation of (a),(b) 500-hPa u and (c),(d) 700-hPa temperatures from radiosondes. Shown are RMSE in the first guess (solid red), spread in the first guess (dashed blue), and RMSE in the analysis (dashed red). The thick lines represent the continuously cycled configurations and the thin lines represent the 24-h configurations with cycles starting at 0000 UTC 7, 8, and 9 Jun. (a),(c) The single-physics configurations and (b),(d) the mixed-physics configurations.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Diagnostics examining the assimilation of surface observations primarily echo the results presented by Romine et al. (2013), with the continuously cycled analyses being nearly unbiased for surface temperature, but exhibiting a moist bias that is greatest during the daytime hours (Fig. 10). As with the radiosonde moisture observations, the total spread in the surface specific humidity is larger than the RMSE because of the large observation error assigned at high values of specific humidity; this represents a difference from Romine et al. (2013) and Ha and Snyder (2014), where the total spread in surface moisture was closer in magnitude to the RMSE. Prominent differences between the mixed-physics and single-physics configurations are that the mixed-physics systems consistently had much larger ensemble spread in these fields, and they also had larger analysis increments in the surface specific humidity field at the 1800 UTC analysis time. However, the reasons that the increments were larger at this specific time are not clear.
Time series of diagnostics for assimilation of surface observations (i.e., “sawtooth” diagrams) for (a) temperature in single_cc, (b) temperature in mixphys_cc, (c) specific humidity in single_cc, and (d) specific humidity in mixphys_cc. Shown are the RMSE (red), ensemble spread (blue solid), total spread (blue dashed), and bias (green dotted, all in K for temperature and g kg−1 for specific humidity), with two points plotted at each analysis time (the first guess and analysis) over the course of all assimilation cycles. The open black circles indicate the number of assimilated surface observations at each time using the ordinate on the right-hand side of each panel.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Time series of diagnostics for assimilation of surface observations (i.e., “sawtooth” diagrams) for (a) temperature in single_cc, (b) temperature in mixphys_cc, (c) specific humidity in single_cc, and (d) specific humidity in mixphys_cc. Shown are the RMSE (red), ensemble spread (blue solid), total spread (blue dashed), and bias (green dotted, all in K for temperature and g kg−1 for specific humidity), with two points plotted at each analysis time (the first guess and analysis) over the course of all assimilation cycles. The open black circles indicate the number of assimilated surface observations at each time using the ordinate on the right-hand side of each panel.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Time series of diagnostics for assimilation of surface observations (i.e., “sawtooth” diagrams) for (a) temperature in single_cc, (b) temperature in mixphys_cc, (c) specific humidity in single_cc, and (d) specific humidity in mixphys_cc. Shown are the RMSE (red), ensemble spread (blue solid), total spread (blue dashed), and bias (green dotted, all in K for temperature and g kg−1 for specific humidity), with two points plotted at each analysis time (the first guess and analysis) over the course of all assimilation cycles. The open black circles indicate the number of assimilated surface observations at each time using the ordinate on the right-hand side of each panel.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
When visualized spatially, there is considerably more spread in the midtropospheric mass fields in the mixed-physics configurations than in the single-physics analyses. For example, the ensemble-mean 500-hPa height analysis at 0000 UTC 10 June 2010 is very similar across most of the domain in all four configurations, but the spread in the interior of the domain is greater in the mixed-physics configurations (Figs. 11b,d) than in the single-physics configurations (Figs. 11a,c). (The large spread near the domain edges is a result of the perturbations applied at the lateral boundaries, discussed in section 3b.) On the other hand, the analyses from the continuously cycled and 24-h cycling configurations were generally similar, consistent with Fig. 9. All of the assimilation configurations showed a closed 500-hPa height contour over northeast Texas in the ensemble mean, indicative of the MCV, but again the mixed-physics configurations had more spread than the single-physics systems around this feature. Considering the influence of convective and microphysical processes on the development and maintenance of MCVs (e.g., Davis and Trier 2002) and the well-known uncertainties in the representation of these processes by numerical models, a closer look at the analyses of this MCV by the different assimilation configurations is warranted.
Ensemble-mean 500-hPa geopotential height (thick contours every 60 m) and standard deviation (spread; color shading in m) in the analyses at 0000 UTC 10 Jun 2010 from (a) single_cc, (b) mixphys_cc, (c) single_24hr, and (d) mixphys_24hr.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Ensemble-mean 500-hPa geopotential height (thick contours every 60 m) and standard deviation (spread; color shading in m) in the analyses at 0000 UTC 10 Jun 2010 from (a) single_cc, (b) mixphys_cc, (c) single_24hr, and (d) mixphys_24hr.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Ensemble-mean 500-hPa geopotential height (thick contours every 60 m) and standard deviation (spread; color shading in m) in the analyses at 0000 UTC 10 Jun 2010 from (a) single_cc, (b) mixphys_cc, (c) single_24hr, and (d) mixphys_24hr.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Unsurprisingly, the configurations that include multiple microphysics parameterizations in the assimilation system provide a greater diversity of vertical profiles of microphysical properties than the configurations using only a single microphysics parameterization. Particularly in the prior (6-h forecast), the shape of the vertical profiles of hydrometeor mixing ratios are similar across the configurations using only the Morrison microphysics scheme (Figs. 12a,c), with the differences between members primarily appearing in the values of the mixing ratios of the different species. On the other hand, the mixed-physics configurations (Figs. 12b,d) show diversity in both the shapes of the profiles and the magnitudes of the values. In particular, the height of the freezing level (which can be visualized as the transition region between the liquid and ice hydrometeor species) ranges from approximately 600 to 400 hPa across the ensemble members in the mixed-physics configurations, whereas there is a much narrower range around 600 hPa in the single-physics forecasts and analyses. The variability in the shapes and structures of the hydrometeor profiles continues to be greater in the mixed-physics configurations than the single-physics configurations after assimilation, although the overall spread is reduced in all four as expected (Figs. 12e–h).
Vertical profiles of the mixing ratios (g kg−1) of hydrometeor species around the southern Great Plains MCV at 0000 UTC 10 Jun 2010 from the (a)–(d) priors and (e)–(h) analyses. These profiles constitute averages over the 9 × 9 grid point (324 × 324 km2) area with the greatest circulation in the 700–500-hPa layer in the southern Great Plains, with one line for each ensemble member. Examples of these averaging boxes are shown in Fig. 14. Mixing ratios of rainwater are shown in green, cloud liquid water in black, cloud ice in pink, graupel in blue, and snow in cyan. The WRF-DART configurations shown are (a),(e) single_cc; (b),(f) mixphys_cc; (c),(g) single_24hr; and (d),(h) mixphys_24hr.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Vertical profiles of the mixing ratios (g kg−1) of hydrometeor species around the southern Great Plains MCV at 0000 UTC 10 Jun 2010 from the (a)–(d) priors and (e)–(h) analyses. These profiles constitute averages over the 9 × 9 grid point (324 × 324 km2) area with the greatest circulation in the 700–500-hPa layer in the southern Great Plains, with one line for each ensemble member. Examples of these averaging boxes are shown in Fig. 14. Mixing ratios of rainwater are shown in green, cloud liquid water in black, cloud ice in pink, graupel in blue, and snow in cyan. The WRF-DART configurations shown are (a),(e) single_cc; (b),(f) mixphys_cc; (c),(g) single_24hr; and (d),(h) mixphys_24hr.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Vertical profiles of the mixing ratios (g kg−1) of hydrometeor species around the southern Great Plains MCV at 0000 UTC 10 Jun 2010 from the (a)–(d) priors and (e)–(h) analyses. These profiles constitute averages over the 9 × 9 grid point (324 × 324 km2) area with the greatest circulation in the 700–500-hPa layer in the southern Great Plains, with one line for each ensemble member. Examples of these averaging boxes are shown in Fig. 14. Mixing ratios of rainwater are shown in green, cloud liquid water in black, cloud ice in pink, graupel in blue, and snow in cyan. The WRF-DART configurations shown are (a),(e) single_cc; (b),(f) mixphys_cc; (c),(g) single_24hr; and (d),(h) mixphys_24hr.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Because latent heating and cooling processes are key to the vertical distribution of potential vorticity (PV) in MCVs (e.g., Haynes and McIntyre 1987; Davis and Trier 2002), the representation of these processes in the assimilation system are also important in how the MCV is analyzed. Consistent with the greater spread in the microphysical profiles (especially in the first-guess fields), there is also greater spread in the area-averaged vertical structure of PV (Fig. 13), and also in the spatial extent of the higher spread in the midlevel PV (Fig. 14) in the mixed-physics configurations, although the mean profiles and structure of PV are similar among all of the configurations. In particular, the mixed-physics runs demonstrate more spread in upper-level PV than the single-physics ensembles, but there are relatively small differences between the “cc” and “24hr” configurations. This likely reflects the importance of diversity in the microphysical parameterizations in producing diversity in the PV field. Although it is not clear from this brief diagnosis which analysis of the microphysical and PV profiles is closest to the truth, the forecasts initialized from these analyses will provide some evidence as to the strengths and limitations of the different assimilation methods.
Vertical profiles of potential vorticity [PVU (potential vorticity unit); 1 PVU = 10−6 K kg−1 m2 s−1] around the southern Great Plains MCV at 0000 UTC 10 Jun 2010 in (a) single_cc, (b) mixphys_cc, (c) single_24hr, and (d) mixphys_24hr. The profiles were calculated in the same manner as in Fig. 12. The dashed lines represent the prior ensemble mean, and the solid lines represent the analysis ensemble mean. The gray shading shows ±1 standard deviation around those means.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Vertical profiles of potential vorticity [PVU (potential vorticity unit); 1 PVU = 10−6 K kg−1 m2 s−1] around the southern Great Plains MCV at 0000 UTC 10 Jun 2010 in (a) single_cc, (b) mixphys_cc, (c) single_24hr, and (d) mixphys_24hr. The profiles were calculated in the same manner as in Fig. 12. The dashed lines represent the prior ensemble mean, and the solid lines represent the analysis ensemble mean. The gray shading shows ±1 standard deviation around those means.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Vertical profiles of potential vorticity [PVU (potential vorticity unit); 1 PVU = 10−6 K kg−1 m2 s−1] around the southern Great Plains MCV at 0000 UTC 10 Jun 2010 in (a) single_cc, (b) mixphys_cc, (c) single_24hr, and (d) mixphys_24hr. The profiles were calculated in the same manner as in Fig. 12. The dashed lines represent the prior ensemble mean, and the solid lines represent the analysis ensemble mean. The gray shading shows ±1 standard deviation around those means.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
As in Fig. 11, but for the ensemble mean 500-hPa PV (contoured every 1 PVU starting at 1) and spread (PVU; color shading) in the posterior analyses at 0000 UTC 10 Jun 2010. The thick dashed boxes show examples of the averaging regions used in Figs. 12 and 13.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
As in Fig. 11, but for the ensemble mean 500-hPa PV (contoured every 1 PVU starting at 1) and spread (PVU; color shading) in the posterior analyses at 0000 UTC 10 Jun 2010. The thick dashed boxes show examples of the averaging regions used in Figs. 12 and 13.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
As in Fig. 11, but for the ensemble mean 500-hPa PV (contoured every 1 PVU starting at 1) and spread (PVU; color shading) in the posterior analyses at 0000 UTC 10 Jun 2010. The thick dashed boxes show examples of the averaging regions used in Figs. 12 and 13.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
5. Results: Ensemble forecasts
a. Precipitation
We now turn attention to the convection-allowing ensemble forecasts that were initialized from each of these WRF-DART assimilation configurations. This section will begin with an overview of the results obtained from evaluating the different ensemble configurations, and then delve into some detailed aspects of the forecasts. As discussed in section 3e, a neighborhood method was used to derive probabilistic forecasts for which the area under the ROC curve was calculated over two domains: one covering most of the forecast domain, and a subdomain of that larger domain focusing on the southern Great Plains (Fig. 3). The ROC area was calculated at 6-h intervals over each of these domains at the thresholds of 12.5, 25, 50, and 100 mm (6 h)−1, and these results are presented in aggregate in Fig. 15 and separated by forecast lead time in Fig. 16).
Area under the ROC curve for the ensemble forecasts initialized at 0000 UTC 8, 9, and 10 Jun 2010 for the four WRF-DART configurations and the SSEF, aggregated over the 6-h periods beginning with the 6–12-h forecasts and ending with the 24–30-h forecasts. The ROC area was calculated for the thresholds of 12.5, 25, 50, and 100 mm (6 h)−1 for (a) the full verification domain and (b) the subdomain shown in Fig. 3. Confidence intervals are the 5th and 95th percentiles calculated using bootstrap resampling. The number of grid points exceeding the threshold in the stage-IV precipitation analysis is shown at the top of each panel. A ROC area of 1 indicates a perfect forecast; an area of 0.5 corresponds to a random (no skill) forecast.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Area under the ROC curve for the ensemble forecasts initialized at 0000 UTC 8, 9, and 10 Jun 2010 for the four WRF-DART configurations and the SSEF, aggregated over the 6-h periods beginning with the 6–12-h forecasts and ending with the 24–30-h forecasts. The ROC area was calculated for the thresholds of 12.5, 25, 50, and 100 mm (6 h)−1 for (a) the full verification domain and (b) the subdomain shown in Fig. 3. Confidence intervals are the 5th and 95th percentiles calculated using bootstrap resampling. The number of grid points exceeding the threshold in the stage-IV precipitation analysis is shown at the top of each panel. A ROC area of 1 indicates a perfect forecast; an area of 0.5 corresponds to a random (no skill) forecast.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Area under the ROC curve for the ensemble forecasts initialized at 0000 UTC 8, 9, and 10 Jun 2010 for the four WRF-DART configurations and the SSEF, aggregated over the 6-h periods beginning with the 6–12-h forecasts and ending with the 24–30-h forecasts. The ROC area was calculated for the thresholds of 12.5, 25, 50, and 100 mm (6 h)−1 for (a) the full verification domain and (b) the subdomain shown in Fig. 3. Confidence intervals are the 5th and 95th percentiles calculated using bootstrap resampling. The number of grid points exceeding the threshold in the stage-IV precipitation analysis is shown at the top of each panel. A ROC area of 1 indicates a perfect forecast; an area of 0.5 corresponds to a random (no skill) forecast.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
As in Fig. 15, but the ROC areas have been calculated for individual 6-h periods, and for (a) 25 mm (6 h)−1 over the full verification domain, (b) 25 mm (6 h)−1 over the subdomain shown in Fig. 3, (c) 50 mm (6 h)−1 over the full domain, and (d) 50 mm (6 h)−1 over the subdomain.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
As in Fig. 15, but the ROC areas have been calculated for individual 6-h periods, and for (a) 25 mm (6 h)−1 over the full verification domain, (b) 25 mm (6 h)−1 over the subdomain shown in Fig. 3, (c) 50 mm (6 h)−1 over the full domain, and (d) 50 mm (6 h)−1 over the subdomain.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
As in Fig. 15, but the ROC areas have been calculated for individual 6-h periods, and for (a) 25 mm (6 h)−1 over the full verification domain, (b) 25 mm (6 h)−1 over the subdomain shown in Fig. 3, (c) 50 mm (6 h)−1 over the full domain, and (d) 50 mm (6 h)−1 over the subdomain.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
On the whole, all of the ensemble configurations were found to produce skillful forecasts in the 6–36-h timeframe, with aggregate ROC areas above 0.8 for all of the ensembles (Fig. 15), and ROC areas well above 0.7 for most of the ensembles at most times (Fig. 16). The aggregate results (Fig. 15) show that the mixed-physics forecast ensembles had larger ROC areas than the single-physics ensembles at all thresholds over the full domain and at all but the largest threshold for the subdomain. The ROC area for the SSEF generally fell in between the mixed-physics and single-physics WRF-DART ensembles. There was no consistent signal suggesting that the “cc” configurations had larger ROC areas than the “24hr” configurations or vice versa. At the lower precipitation thresholds, the ROC areas for all ensembles were smaller over the subdomain than those for the full domain, whereas areas were larger over the subdomain than the full domain at the higher thresholds for all but the mixphys_24hr and SSEF ensembles. This suggests that the regions of very heavy rainfall near the MCV (which was contained within the verification subdomain) were better predicted than those elsewhere, even though the opposite was true for forecasts of lighter precipitation.
When breaking the results down by lead times, no ensemble configuration was found to be consistently superior or inferior to the others at all lead times and thresholds (Fig. 16). However, consistent with the aggregate results, the ensemble forecasts using mixed physics were typically at or near the top for most lead times in both verification domains, and the two single-physics configurations were consistently at or near the bottom. The SSEF system also performed well, although it does carry the limitation of only being run out to 30 h (instead of 36 for the WRF-DART forecasts) and having an ensemble membership of only 15 for these calculations (cf. 24 members in the WRF-DART ensembles.) However, Clark et al. (2011) showed that increases in precipitation forecast skill were minimal when increasing the SSEF ensemble size beyond approximately 10 members.
For both the full verification domain and the subdomain at the 6–12-h lead time, the WRF-DART ensembles (with the exception of single_24hr on the full domain) outperformed the SSEF (Fig. 16), with these differences being statistically significant on the subdomain (Figs. 16b,d). For 12–18-h forecasts, all of the WRF-DART ensembles outperformed the SSEF when considering the full evaluation domain, but the SSEF showed comparable or better performance over the Arkansas–Texas verification subdomain. In contrast to the short lead times, the SSEF was the best, or nearly the best, forecast system in terms of ROC area for 24–30-h precipitation forecasts. At the longest lead times considered here (30–36 h), the single-physics configurations show an increase in ROC area relative to the mixed-physics ensembles. Possible reasons for this will be examined in the following subsections.
Because the ROC area is insensitive to biases in probabilistic forecasts, an examination of the spread and bias in the different ensemble configurations is also needed. At short lead times, the single-physics configurations have rank histograms that are somewhat “U shaped,” which indicates that the observed precipitation values often fall outside the ensemble envelope (Figs. 17a,f). In other words, these ensembles have insufficient spread at short lead times. The mixed-physics ensembles, on the other hand, have much flatter rank histograms (Figs. 17k,p). The better dispersion characteristics of the mixed-physics ensembles are likely tied to the greater spread in the mixed-physics analyses of the MCV discussed previously and shown in Figs. 12–14. At all lead times, the SSEF has rank histograms that are right skewed, indicating that this system has a consistent wet bias (Figs. 17u–x). This result is in line with the results of Clark et al. (2009) and Schumacher et al. (2013, their Fig. 6) for full seasons of SSEF forecasts. In comparison, the WRF-DART configurations have relatively flat rank histograms at longer lead times, again suggesting that they have more representative spread than the SSEF (Fig. 17). The reasons for the “flattening” of the rank histograms with increasing lead time for the single-physics forecasts are not immediately clear.
Rank histograms for 6-hourly ensemble precipitation forecasts initialized at 0000 UTC 8, 9, and 10 Jun 2010 for the four WRF-DART configurations and the SSEF over the full verification domain. (a)–(e) single_cc, (f)–(j) single_24hr, (k)–(o) mixphys_cc, (p)–(t) mixphys_24hr, and (u)–(x) SSEF. Forecast lead time increases from left to right. Because the SSEF has fewer members, there are fewer ranks in the diagram.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Rank histograms for 6-hourly ensemble precipitation forecasts initialized at 0000 UTC 8, 9, and 10 Jun 2010 for the four WRF-DART configurations and the SSEF over the full verification domain. (a)–(e) single_cc, (f)–(j) single_24hr, (k)–(o) mixphys_cc, (p)–(t) mixphys_24hr, and (u)–(x) SSEF. Forecast lead time increases from left to right. Because the SSEF has fewer members, there are fewer ranks in the diagram.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Rank histograms for 6-hourly ensemble precipitation forecasts initialized at 0000 UTC 8, 9, and 10 Jun 2010 for the four WRF-DART configurations and the SSEF over the full verification domain. (a)–(e) single_cc, (f)–(j) single_24hr, (k)–(o) mixphys_cc, (p)–(t) mixphys_24hr, and (u)–(x) SSEF. Forecast lead time increases from left to right. Because the SSEF has fewer members, there are fewer ranks in the diagram.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Over the three-day period of heavy rainfall in Texas and Arkansas, the predictive skill of each of the ensemble systems generally increased with each run, similar to the findings of Davis et al. (2002), owing to the heavy precipitation becoming more widespread, and the MCV becoming better defined and more robust with time. When considering the full domain, however, the skill was more consistent from run to run. One illustration of this is in the 24–30-h forecasts shown in Fig. 18, which show low probabilities of 25 mm of rain in south-central Texas (MCS S1) in each of the ensemble systems for the forecast initialized on 8 June (Figs. 18a–f), generally higher probabilities in the region where the heavy rain fell for the forecast initialized on 9 June (MCS S2; Figs. 18g–l), and a focused region of low to moderate probabilities near the Texas–Arkansas border in the run initialized on 10 June (MCS S3; Figs. 18m–r). Even for the 9 and 10 June initializations, the exceedence probabilities were less than 30% in the region of the MCV in all ensembles. On the other hand, the locations with high potential for heavy rainfall in the central and northern plains were captured in the runs (with probabilities greater than 30%) from both 8 June and 10 June (MCSs N1 and N3), presumably because the baroclinic zone in that area was already well established and was analyzed well by the assimilation systems. Although not shown here, the heavy precipitation probabilities tend to be focused over smaller areas with higher probabilities at earlier lead times, and then disperse over larger spatial areas as the solutions of the individual ensemble members diverge.
Neighborhood probabilities (color shading) of 25 mm of precipitation in 6 h in the 24–30-h forecasts (a)–(f) initialized at 0000 UTC 8 Jun 2010, valid at 0600 UTC 9 Jun 2010; (g)–(l) initialized at 0000 UTC 9 Jun 2010, valid at 0600 UTC 10 Jun 2010; and (m)–(r) initialized at 0000 UTC 10 Jun 2010, valid at 0600 UTC 11 Jun 2010. Ensemble forecasts shown are from (a),(g),(m) single_cc; (b),(h),(n) single_24hr; (c),(i),(o) mixphys_cc; (d),(j),(p) mixphys_24hr; and (e),(k),(q) SSEF. (f),(l),(r) The stage-IV precipitation analysis is valid at the same times but only values exceeding 25 mm (6 h)−1 are shown. The MCSs are also labeled as in Fig. 1. The 25 mm (6 h)−1 contour from the stage-IV analysis is also shown on each forecast panel.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Neighborhood probabilities (color shading) of 25 mm of precipitation in 6 h in the 24–30-h forecasts (a)–(f) initialized at 0000 UTC 8 Jun 2010, valid at 0600 UTC 9 Jun 2010; (g)–(l) initialized at 0000 UTC 9 Jun 2010, valid at 0600 UTC 10 Jun 2010; and (m)–(r) initialized at 0000 UTC 10 Jun 2010, valid at 0600 UTC 11 Jun 2010. Ensemble forecasts shown are from (a),(g),(m) single_cc; (b),(h),(n) single_24hr; (c),(i),(o) mixphys_cc; (d),(j),(p) mixphys_24hr; and (e),(k),(q) SSEF. (f),(l),(r) The stage-IV precipitation analysis is valid at the same times but only values exceeding 25 mm (6 h)−1 are shown. The MCSs are also labeled as in Fig. 1. The 25 mm (6 h)−1 contour from the stage-IV analysis is also shown on each forecast panel.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Neighborhood probabilities (color shading) of 25 mm of precipitation in 6 h in the 24–30-h forecasts (a)–(f) initialized at 0000 UTC 8 Jun 2010, valid at 0600 UTC 9 Jun 2010; (g)–(l) initialized at 0000 UTC 9 Jun 2010, valid at 0600 UTC 10 Jun 2010; and (m)–(r) initialized at 0000 UTC 10 Jun 2010, valid at 0600 UTC 11 Jun 2010. Ensemble forecasts shown are from (a),(g),(m) single_cc; (b),(h),(n) single_24hr; (c),(i),(o) mixphys_cc; (d),(j),(p) mixphys_24hr; and (e),(k),(q) SSEF. (f),(l),(r) The stage-IV precipitation analysis is valid at the same times but only values exceeding 25 mm (6 h)−1 are shown. The MCSs are also labeled as in Fig. 1. The 25 mm (6 h)−1 contour from the stage-IV analysis is also shown on each forecast panel.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
b. MCV track and structure
Schumacher et al. (2013) showed the close relationship between the MCV and the heavy rainfall over the course of this event, and therefore a discussion of skill and uncertainty for the precipitation forecast necessarily raises the question of the skill and uncertainty for the forecast of the MCV. Visualizing and evaluating ensemble forecasts of tropical cyclones has become commonplace, but similar investigations for continental vortices such as MCVs are limited in the literature [Hawblitzel et al. (2007) and Clark et al. (2010) being two notable exceptions]. This study provides an attempt to evaluate ensemble MCV forecasts in a similar way.
Identifying the MCV during 8 June 2010 was difficult, as that was the day it became more organized and moved inland, and efforts to adequately track the vortex in the model forecasts initialized on 8 June were unsuccessful. In the forecast initialized at 0000 UTC 9 June, there was a tendency for the model-predicted MCVs to move slightly north and west of the NARR-analyzed track; in other words, a left-of-track error (Fig. 19a). (The NARR is used as observations here because it is an independent analysis and because it is available at 3-h intervals.) There were no clear biases or differences between the different model configurations, aside from the fact that the single_cc and mixphys_cc analyses included several members with the initial position of the MCV much farther south than the other analyses or the NARR (Fig. 19a). This left-of-track error in the WRF-DART-initialized ensembles is consistent with the precipitation forecasts shown in Figs. 18g–j, in which the highest precipitation probabilities were also displaced slightly to the north and west of where the heavy rainfall was observed in north Texas. Unfortunately the full three-dimensional output of the SSEF was unavailable for this case to analyze the MCV track in that ensemble.
Forecast track of the MCV in each ensemble member through 30 h for the forecasts initialized at (a) 0000 UTC 9 Jun and (b) 0000 UTC 10 Jun 2010. Members of the different ensemble configurations are colored as shown in the legend; the NARR-analyzed track is shown in thick black. Circles (triangles) are plotted at the beginning and ending points of the forecast (analyzed) tracks. The vortices were tracked using the method described in section 3f.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Forecast track of the MCV in each ensemble member through 30 h for the forecasts initialized at (a) 0000 UTC 9 Jun and (b) 0000 UTC 10 Jun 2010. Members of the different ensemble configurations are colored as shown in the legend; the NARR-analyzed track is shown in thick black. Circles (triangles) are plotted at the beginning and ending points of the forecast (analyzed) tracks. The vortices were tracked using the method described in section 3f.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Forecast track of the MCV in each ensemble member through 30 h for the forecasts initialized at (a) 0000 UTC 9 Jun and (b) 0000 UTC 10 Jun 2010. Members of the different ensemble configurations are colored as shown in the legend; the NARR-analyzed track is shown in thick black. Circles (triangles) are plotted at the beginning and ending points of the forecast (analyzed) tracks. The vortices were tracked using the method described in section 3f.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
For the ensemble forecasts initialized at 0000 UTC 10 June, the ensemble envelope nearly symmetrically surrounded the NARR-analyzed track. The generally accurate track forecasts for the MCV on 10–11 June also contributed to the placement of higher precipitation probabilities near the Texas–Arkansas border in all of the ensembles. Almost all of the SSEF members predicted the MCV to move too slowly (Fig. 19b), which is consistent with the highest precipitation probabilities in the SSEF being located farther southwest than the WRF-DART ensembles (Figs. 18m–r). The connection between the MCV location and the precipitation forecast is further illustrated in Figs. 20a,b, which shows that the ensemble members with the best precipitation forecasts on 11 June (MCS S3 in Fig. 1d) were those that predicted the MCV to be located in southwestern Arkansas and northeast Texas at 0300 UTC (Fig. 20a) and in central Arkansas at 0900 UTC (Fig. 20b). However, at 0300 UTC, the predicted MCV locations in these members were slightly south of the NARR-analyzed location. Furthermore, although the members with poor forecasts of the MCV track generally also had poor precipitation forecasts, there were many members that had poor precipitation forecasts yet an MCV track prediction very similar to other members that had good precipitation forecasts. This suggests that factors beyond simply the track of the MCV were important to the accurate prediction of heavy precipitation. For the SSEF, Schumacher et al. (2013) address some of these possible factors, and the factors that were important in these WRF-DART ensemble forecasts will be addressed in future research. Finally, although relatively few members of the single_cc ensemble had skillful forecasts of the MCV-related precipitation in the 24–30-h forecast (associated with the low ROC area for this ensemble in Figs. 16b,d), multiple members of this ensemble then intensified the MCV and predicted heavy rainfall near the observed location in the 30–36-h forecast (Fig. 20b), improving the forecast skill of that ensemble at this lead time (Figs. 16b,d).
Position of the MCV in each ensemble member at (a) 0300 UTC 11 Jun (27-h forecast) and (b) 0900 UTC 11 Jun (33-h forecast). Each circle is colored based on the ensemble configuration, and the size of the circle is proportional to the FSS for 50 mm or more of precipitation in the 6-h period spanning those times in the verification subdomain. In other words, the larger the circle, the more skillful the precipitation forecast. The NARR-analyzed location is indicated by the “N,” and the stage-IV analyzed 50-mm precipitation contour is shown in gray for each 6-h period.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Position of the MCV in each ensemble member at (a) 0300 UTC 11 Jun (27-h forecast) and (b) 0900 UTC 11 Jun (33-h forecast). Each circle is colored based on the ensemble configuration, and the size of the circle is proportional to the FSS for 50 mm or more of precipitation in the 6-h period spanning those times in the verification subdomain. In other words, the larger the circle, the more skillful the precipitation forecast. The NARR-analyzed location is indicated by the “N,” and the stage-IV analyzed 50-mm precipitation contour is shown in gray for each 6-h period.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Position of the MCV in each ensemble member at (a) 0300 UTC 11 Jun (27-h forecast) and (b) 0900 UTC 11 Jun (33-h forecast). Each circle is colored based on the ensemble configuration, and the size of the circle is proportional to the FSS for 50 mm or more of precipitation in the 6-h period spanning those times in the verification subdomain. In other words, the larger the circle, the more skillful the precipitation forecast. The NARR-analyzed location is indicated by the “N,” and the stage-IV analyzed 50-mm precipitation contour is shown in gray for each 6-h period.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Although space considerations do not make it feasible to show the evolution of the MCV and the precipitation forecasts across the full ensembles, Fig. 21 illustrates some of the differences between the forecasts using a selected single member of all four ensemble configurations (which had identical physics schemes and LBCs) for the heavy rain event on 11 June 2010. In this member of each ensemble, a closed circulation and vorticity maximum was located over Arkansas at 0600 UTC 11 June, though the intensity and specific location of the vorticity maximum varied among these forecasts. In mixphys_24hr, there is a concentrated region of vorticity in southern Arkansas (Fig. 21d), and an associated region of long-lived heavy precipitation, which was located just south of the observed MCS in the 24–30 and 30–36-h forecasts (Figs. 21h,l). In this member in the other three ensembles, the forecast MCV was weaker than that in mixphys_24hr (Figs. 21a–c) and there was too little precipitation in the 24–30-h forecast (Figs. 21e–g). Although there was an increase in precipitation in these members as the MCV moved northeast in hours 30–36 (Figs. 21i–k), there continued to be an underprediction of rainfall surrounding the MCV.
(a)–(d) Absolute vorticity (×105 s−1, color shading), geopotential height (contoured every 20 m), and wind barbs (half barb = 5 kt, full barb = 10 kt, pennant = 50 kt) at 600 hPa in the 30-h forecast valid at 0600 UTC 11 Jun 2010 from ensemble member 3 of the (a) single_cc, (b) single_24hr, (c) mixphys_cc, and (d) mixphys_24hr ensemble forecasts. (e)–(h) Accumulated precipitation in the 6 h ending at 0600 UTC 11 Jun 2010 from the 30-h forecast of ensemble member 3 of the (e) single_cc, (f) single_24hr, (g) mixphys_cc, and (h) mixphys_24hr ensemble forecasts. (i)–(l) As in (e)–(h), but for the 36-h forecast for the 6-h period ending at 1200 UTC 11 Jun. The ICs for these forecasts come from the corresponding WRF-DART analyses, but the physical parameterizations and lateral boundary conditions for each of these forecasts were identical. (m) Stage-IV precipitation analysis for the 6 h ending at 0600 UTC 11 Jun. (n) As in (m), but for 1200 UTC 11 Jun.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
(a)–(d) Absolute vorticity (×105 s−1, color shading), geopotential height (contoured every 20 m), and wind barbs (half barb = 5 kt, full barb = 10 kt, pennant = 50 kt) at 600 hPa in the 30-h forecast valid at 0600 UTC 11 Jun 2010 from ensemble member 3 of the (a) single_cc, (b) single_24hr, (c) mixphys_cc, and (d) mixphys_24hr ensemble forecasts. (e)–(h) Accumulated precipitation in the 6 h ending at 0600 UTC 11 Jun 2010 from the 30-h forecast of ensemble member 3 of the (e) single_cc, (f) single_24hr, (g) mixphys_cc, and (h) mixphys_24hr ensemble forecasts. (i)–(l) As in (e)–(h), but for the 36-h forecast for the 6-h period ending at 1200 UTC 11 Jun. The ICs for these forecasts come from the corresponding WRF-DART analyses, but the physical parameterizations and lateral boundary conditions for each of these forecasts were identical. (m) Stage-IV precipitation analysis for the 6 h ending at 0600 UTC 11 Jun. (n) As in (m), but for 1200 UTC 11 Jun.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
(a)–(d) Absolute vorticity (×105 s−1, color shading), geopotential height (contoured every 20 m), and wind barbs (half barb = 5 kt, full barb = 10 kt, pennant = 50 kt) at 600 hPa in the 30-h forecast valid at 0600 UTC 11 Jun 2010 from ensemble member 3 of the (a) single_cc, (b) single_24hr, (c) mixphys_cc, and (d) mixphys_24hr ensemble forecasts. (e)–(h) Accumulated precipitation in the 6 h ending at 0600 UTC 11 Jun 2010 from the 30-h forecast of ensemble member 3 of the (e) single_cc, (f) single_24hr, (g) mixphys_cc, and (h) mixphys_24hr ensemble forecasts. (i)–(l) As in (e)–(h), but for the 36-h forecast for the 6-h period ending at 1200 UTC 11 Jun. The ICs for these forecasts come from the corresponding WRF-DART analyses, but the physical parameterizations and lateral boundary conditions for each of these forecasts were identical. (m) Stage-IV precipitation analysis for the 6 h ending at 0600 UTC 11 Jun. (n) As in (m), but for 1200 UTC 11 Jun.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
c. Sensitivity to lateral boundary condition perturbations
Some past studies of convection-allowing ensemble forecasts have found that the errors and uncertainties owing to the LBCs can dominate over those of the ICs in relatively short periods of time [on the order of 12 h; e.g., Hohenegger et al. 2008; Vié et al. 2011]. With this in mind, 12 members of the mixphys_24hr ensemble forecast were run for 36 h using unperturbed GFS forecasts for the LBCs, and were then compared to the forecasts discussed previously that used the perturbed LBCs. It was found that, although there were nontrivial differences later in the runs using the perturbed versus unperturbed LBCs, these differences were much smaller than the differences between members owing to the ICs (not shown). In particular, the members that produced accurate forecasts of heavy rainfall at 24–36-h lead times with perturbed LBCs were the same members that did so with the unperturbed LBCs, and similarly those with poor predictions were the same in both experiments. This suggests that although LBC perturbations had an influence on the forecast, they did not overwhelm the IC uncertainties even through 36 h. The discrepancy between this research and the above-cited studies is at least in part attributable to the sizes of the convection-allowing domains in those studies; both used domains covering only a small section of Europe (less than 10° latitude–longitude on a side), whereas this study used a much larger domain that covered approximately the central two-thirds of the continental United States (more than 20° latitude–longitude). In other words, the influence of LBC perturbations is likely very sensitive to the size of the domain chosen, as well as the length of the model integration.
6. Discussion
This study provides an intercomparison of multiple methods of configuring and running a convection-allowing ensemble forecast system. As such systems continue to come into greater use owing to increased computing capabilities and the need for probabilistic information about convective weather hazards, it will be important to evaluate them for both their forecast skill and their representation of uncertainty. This will allow for the systems to be designed such that they optimally use the substantial computing needs required to run them.
This study found that overall, ensemble configurations with diversity in both their ICs and their physical parameterizations provided better probabilistic forecasts of heavy rainfall than those with changes to the ICs but only a single set of parameterization for all members. This result is similar to the findings of Stensrud et al. (2000) and Fujita et al. (2007), among others. These results provide further evidence in support of the idea that uncertainty in the representation of physical processes by models must be included in an ensemble prediction system that explicitly predicts convection. In particular, circulation features such as MCVs that are intimately tied to convective processes may not be analyzed properly by ensemble-based assimilation systems, nor predicted adequately by high-resolution ensembles, if this model uncertainty is not represented in the assimilation and forecast system. However, diversity in physical parameterizations can also complicate the interpretation of the forecasts, and can complicate quantitative methods such as ensemble-based synoptic and mesoscale analysis (e.g., Hakim and Torn 2008), as those methods assume that all ensemble members are equally likely, which may not always be the case for a mixed-physics ensemble. There are also other methods for including model uncertainty without using a diversity of parameterizations, including stochastic perturbations within those parameterizations (e.g., Bouttier et al. 2012), using ensemble data assimilation for parameter estimation (e.g., Nielsen-Gammon et al. 2010), or stochastic kinetic energy perturbations (e.g., Berner et al. 2011). Although these methods have not been thoroughly tested for data assimilation and forecasting on convection-allowing spatial and temporal scales, they may provide means for incorporating model uncertainty in the future.
The results presented here also provide some evidence supporting the design and implementation of the SSEF, which has been run in real-time during several recent spring seasons. Although its design is somewhat ad hoc, it performed comparably to, or better than, the ensemble systems using ensemble data assimilation to initialize their forecasts, especially at longer lead times. For practical purposes this is an important finding, considering that the SSEF is, in a relative sense, computationally inexpensive because it exploits analyses and ensemble perturbations from other readily available sources rather than requiring an ensemble data assimilation system of its own. Of course, this study only presents the results of three cases with somewhat similar large-scale meteorological conditions, so the results should not be generalized too broadly. The SSEF also exhibits a consistent wet bias, suggesting that there may be changes to the configuration of some members, or statistical postprocessing methods, which could reduce this bias and improve forecasts further. Finally, the results of this study suggest that a relatively short cycling period (e.g., 24 h) may lead to comparable forecasts than a continuously cycled system, but again the applicability of this result is limited by the small sample size in this study. The results are consistent with Wheatley et al. (2012), who found that running an ensemble data assimilation system for even a very short time could potentially provide forecast improvements. More research with larger sample sizes is required to better address these issues.
7. Summary and conclusions
In this study, four different ensemble-based data assimilation and forecast systems were run and their performance was evaluated during a period of six heavy-rain-producing MCSs over three days in June 2010. The four systems varied in the length of time over which they were cycled, and in whether they had a single set of physical parameterizations or a diversity of parameterizations. All four assimilation configurations produced results that were generally similar to those reported in previous studies using the WRF-DART system, although there were important differences in the vicinity of the convective systems. In particular, the shapes of the vertical profiles of microphysical variables in the vicinity of an MCV in the southern Plains were more uniform in the single-physics systems but had greater diversity in the mixed-physics systems, which then influenced the structure of the vortex in the analyses.
Convection-allowing ensemble forecasts were then initialized from these analyses, and were also compared to a real-time ensemble from CAPS, known as the SSEF. Using the ROC area as an evaluation method, the systems with mixed physics provided more skillful probabilistic forecasts of the heavy precipitation, though this result is somewhat dependent on the precipitation threshold and lead time being considered. The continuously cycled and partially cycled systems performed similarly to one another for precipitation forecasts. The WRF-DART-initialized forecasts generally outperformed the SSEF at shorter lead times, whereas the SSEF provided the most skillful forecasts, or nearly so, at longer lead times, although it also had a consistent wet bias. The skill of the precipitation forecasts in Arkansas and Texas was partially tied to the accuracy of the forecast track of the MCV, but other factors also influenced the skill of these forecasts. Ongoing work is aimed at further analyzing the processes responsible for the differences between the ensemble systems, and also the differences among the members within each of the ensembles, with an aim of both improved probabilistic prediction of, and increased understanding of the processes responsible for, extreme warm-season precipitation.
Acknowledgments
The authors thank Glen Romine, David Novak, and Wallace Hogsett for helpful discussions and suggestions regarding this work. The authors also thank Ming Xue and Fanyou Kong of CAPS for making the SSEF output available, and for their efforts in developing and running the SSEF. Three anonymous reviewers provided numerous constructive suggestions that substantially improved the quality of this manuscript. Computing resources were provided by the National Center for Atmospheric Research, which is sponsored by the National Science Foundation. NARR analyses were obtained from the NOAA/Physical Sciences Division, and stage-IV analyses were provided by NCAR. R. Schumacher was supported by National Science Foundation Grant AGS-1157425.
APPENDIX A
Influence of Microphysical Fields in Initial Conditions on Convection-Allowing Forecasts
As discussed in the text, the mixed-physics ensembles used in this research include both single- and double-moment microphysical parameterizations. As such, it is not possible to use hydrometeor number concentrations as a state variable during the assimilation in these ensembles, and for an equitable comparison, they are not used as state variables in the single-physics assimilation systems either. In this appendix, we present the results of tests to evaluate the importance of this potential limitation on the resulting forecasts.
As a simple test of the influence of using number concentration as a state variable, we ran the single_24hr assimilation for five cycles from 0000 UTC 9 June to 0000 UTC 10 June, identically to how they were run in the experiments reported in the main part of the manuscript, except with number concentrations of rain, graupel, cloud ice, and snow as additional state variables. Then, we initialized forecasts from a selected single member (member 4) using these initial conditions, along with two experiments: one with all number concentrations set to zero, and one with all hydrometeor mixing ratios and number concentrations set to zero. The first experiment was designed to be the most drastic perturbation one might apply, where the hydrometeor mixing ratios are nonzero but the number concentrations are zero. (In the actual configuration used in the manuscript, the perturbation is much less drastic as the number concentrations remain at the value in the prior.) The second experiment was designed to determine how much influence the initial microphysical profiles have on the forecasts, considering that they are being interpolated from a coarse to a fine grid, and no direct microphysical observations (e.g., radar reflectivity) are being assimilated here. This is akin to a cold start that is often used to initialize WRF forecasts using initial conditions from other modeling systems. We focus on the first 12 h of the forecasts, when the initial microphysical profiles are likely to have the most direct impact on the forecast.
The 0–6-h precipitation forecasts are nearly indistinguishable between these four runs (not shown). By the 6–12-h forecasts, there are nontrivial differences between the runs, especially in the vicinity of the MCV in northeast Texas, but these differences are quite small in the context of the variability within the ensemble as a whole (Fig. A1). Comparing Figs. A1a,b demonstrates the differences arising from using microphysics number concentrations as a state variable during the WRF-DART assimilation cycles. (Thus, there are also differences in all of the other model state variables in the initial conditions of these two forecasts.) The primary differences are small spatial shifts in the location of the heaviest rain in the MCSs in Nebraska and Texas, and also changes in the maximum rainfall amounts from these MCSs. The MCV-related rainfall in the manuscript configuration (Fig. A1a) is slightly less displaced from the observed rainfall location (Fig. A1e). In the forecast with the number concentrations set to zero in the initial condition, the MCSs are in nearly identical locations to those with realistic number concentrations (cf. Figs. A1b,c). The maximum rainfall in northeast Texas is somewhat higher in the run with the initially zero number concentrations, whereas the precipitation from the MCS in Nebraska is very similar. The differences in precipitation amounts are somewhat larger in the cold start run (Fig. A1d), although the locations of the MCSs are again nearly the same.
(a)–(d) Accumulated precipitation in the 6 h ending at 1200 UTC 10 Jun 2010 from the 12-h forecast of ensemble member 4 initialized from (a) the single_cc configuration used in the manuscript and (b) a single_cc configuration with hydrometeor mixing ratio as a state variable during the assimilation cycles. (c) As in (b), but the number concentrations are set to zero in the forecast initial condition. (d) As in (b), but for mixing ratios and number concentrations set to zero in the forecast initial condition. (e) Stage-IV precipitation analysis for the 6 h ending at 1200 UTC 10 Jun.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
(a)–(d) Accumulated precipitation in the 6 h ending at 1200 UTC 10 Jun 2010 from the 12-h forecast of ensemble member 4 initialized from (a) the single_cc configuration used in the manuscript and (b) a single_cc configuration with hydrometeor mixing ratio as a state variable during the assimilation cycles. (c) As in (b), but the number concentrations are set to zero in the forecast initial condition. (d) As in (b), but for mixing ratios and number concentrations set to zero in the forecast initial condition. (e) Stage-IV precipitation analysis for the 6 h ending at 1200 UTC 10 Jun.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
(a)–(d) Accumulated precipitation in the 6 h ending at 1200 UTC 10 Jun 2010 from the 12-h forecast of ensemble member 4 initialized from (a) the single_cc configuration used in the manuscript and (b) a single_cc configuration with hydrometeor mixing ratio as a state variable during the assimilation cycles. (c) As in (b), but the number concentrations are set to zero in the forecast initial condition. (d) As in (b), but for mixing ratios and number concentrations set to zero in the forecast initial condition. (e) Stage-IV precipitation analysis for the 6 h ending at 1200 UTC 10 Jun.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
These results, along with manual inspection of the initial conditions in other ensemble members (not shown) suggests that the effects of not using number concentration as a state variable in the ensemble data assimilation are relatively small in relation to other effects for the configurations used in this study. Thus, configurations with both single- and double-moment microphysics parameterizations are used in the mixed-physics ensembles here. With this being said, for applications where microphysical variables are a more integral part of the data assimilation system (such as when radar reflectivity is assimilated), these conclusions may not hold, and additional testing is suggested for such configurations. There are also some additional differences between these experiments at longer lead times, but whether these differences arise directly from microphysical sensitivities or through feedbacks with other processes in the model is not clear and would warrant additional investigation that is beyond the scope of this study.
APPENDIX B
Sensitivity of Results to Assimilated Moisture Variable
Based on the suggestion of a reviewer, we conducted a limited test to evaluate the sensitivity of the forecasts to the method of assimilating moisture in the DART system. Similar to the tests described in appendix A, we ran the single_24hr assimilation for five cycles from 0000 UTC 9 June to 0000 UTC 10 June, except with dewpoint instead of specific humidity as the assimilated moisture variable. The magnitude of the observation error was kept the same, but converted to dewpoint. Then, a 30-h convection-allowing ensemble forecast (i.e., from 0000 UTC 10 June to 0600 UTC 11 June) was initialized with these analyses. Comparing the ROC area for this ensemble forecast to the one reported in the manuscript shows that the forecasts using the analyses with dewpoint as the assimilated variable were comparable at early lead times, and worse at longer lead times, than those using specific humidity (cf. black and red lines in Fig. B1).
Area under the ROC curve for ensemble forecasts initialized at 0000 UTC 10 Jun 2010, for accumulations of (a) 25 mm (6 h)−1 and (b) 50 mm (6 h)−1 over the full verification domain. The black line is for the single_24hr configuration reported in the main body of the manuscript; the red line is for an experiment where dewpoint, rather than specific humidity, was used as the assimilated moisture variable; and the green line is for an experiment where dewpoint was assimilated but the assumed observation error was halved.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Area under the ROC curve for ensemble forecasts initialized at 0000 UTC 10 Jun 2010, for accumulations of (a) 25 mm (6 h)−1 and (b) 50 mm (6 h)−1 over the full verification domain. The black line is for the single_24hr configuration reported in the main body of the manuscript; the red line is for an experiment where dewpoint, rather than specific humidity, was used as the assimilated moisture variable; and the green line is for an experiment where dewpoint was assimilated but the assumed observation error was halved.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
Area under the ROC curve for ensemble forecasts initialized at 0000 UTC 10 Jun 2010, for accumulations of (a) 25 mm (6 h)−1 and (b) 50 mm (6 h)−1 over the full verification domain. The black line is for the single_24hr configuration reported in the main body of the manuscript; the red line is for an experiment where dewpoint, rather than specific humidity, was used as the assimilated moisture variable; and the green line is for an experiment where dewpoint was assimilated but the assumed observation error was halved.
Citation: Monthly Weather Review 142, 11; 10.1175/MWR-D-13-00357.1
We then conducted the same experiment, except that the assigned observation error for land surface dewpoint was reduced by half in the DART assimilation process. The ensemble forecast initialized from these analyses did show higher ROC area in the 6–12-h forecast than the analyses using specific humidity, but lower ROC area by the 24–30-h forecast (cf. black and green lines in Fig. B1). These tests provide evidence that assimilating specific humidity as the moisture variable is not degrading the resulting ensemble forecasts, at least for this initialization time. Uncovering the reasons for these differences is beyond the scope of this study, and despite the mixed results of this limited testing, recent work has offered some reasons why it may be more desirable to assimilate dewpoint rather than specific humidity (e.g., Fujita et al. 2007; Ha and Snyder 2014). Continued research on the implications of assimilating different moisture variables for the use in convection-allowing forecasts is encouraged.
REFERENCES
Anderson, J. L., 2001: An ensemble adjustment Kalman filter for data assimilation. Mon. Wea. Rev., 129, 2884–2903, doi:10.1175/1520-0493(2001)129<2884:AEAKFF>2.0.CO;2.
Anderson, J. L., 2009: Spatially and temporally varying adaptive covariance inflation for ensemble filters. Tellus, 61A, 72–83, doi:10.1111/j.1600-0870.2008.00361.x.
Anderson, J. L., T. Hoar, K. Raeder, H. Liu, N. Collins, R. Torn, and A. Avellano, 2009: The Data Assimilation Research Testbed: A community facility. Bull. Amer. Meteor. Soc., 90, 1283–1296, doi:10.1175/2009BAMS2618.1.
Augustine, J. A., and F. Caracena, 1994: Lower-tropospheric precursors to nocturnal MCS development over the central United States. Wea. Forecasting, 9, 116–135, doi:10.1175/1520-0434(1994)009<0116:LTPTNM>2.0.CO;2.
Ballish, B. A., and V. Krishna Kumar, 2008: Systematic differences in aircraft and radiosonde temperatures. Bull. Amer. Meteor. Soc., 89, 1689–1708, doi:10.1175/2008BAMS2332.1.
Barker, D., and Coauthors, 2012: The Weather Research and Forecasting model’s community variational/ensemble data assimilation system: WRFDA. Bull. Amer. Meteor. Soc., 93, 831–843, doi:10.1175/BAMS-D-11-00167.1.
Berner, J., S.-Y. Ha, J. P. Hacker, A. Fournier, and C. Snyder, 2011: Model uncertainty in a mesoscale ensemble prediction system: Stochastic versus multiphysics representations. Mon. Wea. Rev., 139, 1972–1995, doi:10.1175/2010MWR3595.1.
Bonner, W. D., 1968: Climatology of the low level jet. Mon. Wea. Rev., 96, 833–850, doi:10.1175/1520-0493(1968)096<0833:COTLLJ>2.0.CO;2.
Bouttier, F., B. Vié, O. Nuissier, and L. Raynaud, 2012: Impact of stochastic physics in a convection-permitting ensemble. Mon. Wea. Rev., 140, 3706–3721, doi:10.1175/MWR-D-12-00031.1.
Chen, F., and J. Dudhia, 2001: Coupling an advanced land surface-hydrology model with the Penn State–NCAR MM5 modeling system. Part I: Model implementation and sensitivity. Mon. Wea. Rev., 129, 569–585, doi:10.1175/1520-0493(2001)129<0569:CAALSH>2.0.CO;2.
Clark, A. J., W. A. Gallus Jr., M. Xue, and F. Kong, 2009: A comparison of precipitation forecast skill between small convection-allowing and large convection-parameterizing ensembles. Wea. Forecasting, 24, 1121–1140, doi:10.1175/2009WAF2222222.1.
Clark, A. J., W. A. Gallus Jr., M. Xue, and F. Kong, 2010: Convection-allowing and convection-parameterizing ensemble forecasts of a mesoscale convective vortex and associated severe weather environment. Wea. Forecasting, 25, 1052–1081, doi:10.1175/2010WAF2222390.1.
Clark, A. J., and Coauthors, 2011: Probabilistic precipitation forecast skill as a function of ensemble size and spatial scale in a convection-allowing ensemble. Mon. Wea. Rev., 139, 1410–1418, doi:10.1175/2010MWR3624.1.
Clark, A. J., and Coauthors, 2012: An overview of the 2010 Hazardous Weather Testbed Experimental Forecast Program spring experiment. Bull. Amer. Meteor. Soc., 93, 55–74, doi:10.1175/BAMS-D-11-00040.1.
Clark, A. J., J. Gao, P. T. Marsh, T. Smith, J. S. Kain, J. Correia Jr., M. Xue, and F. Kong, 2013: Tornado pathlength forecasts from 2010 to 2011 using ensemble updraft helicity. Wea. Forecasting, 28, 387–407, doi:10.1175/WAF-D-12-00038.1.
Davis, C. A., and S. B. Trier, 2002: Cloud-resolving simulations of mesoscale vortex intensification and its effect on a serial mesoscale convective system. Mon. Wea. Rev., 130, 2839–2858, doi:10.1175/1520-0493(2002)130<2839:CRSOMV>2.0.CO;2.
Davis, C. A., D. A. Ahijevych, and S. B. Trier, 2002: Detection and prediction of warm season midtropospheric vortices by the Rapid Update Cycle. Mon. Wea. Rev., 130, 24–42, doi:10.1175/1520-0493(2002)130<0024:DAPOWS>2.0.CO;2.
Du, J., S. L. Mullen, and F. Sanders, 1997: Short-range ensemble forecasting of quantitative precipitation. Mon. Wea. Rev., 125, 2427–2459, doi:10.1175/1520-0493(1997)125<2427:SREFOQ>2.0.CO;2.
Ebert, E. E., 2008: Fuzzy verification of high-resolution gridded forecasts: A review and proposed framework. Meteor. Appl., 15, 51–64, doi:10.1002/met.25.
Eckel, F. A., and C. F. Mass, 2005: Aspects of effective mesoscale, short-range ensemble forecasting. Wea. Forecasting, 20, 328–350, doi:10.1175/WAF843.1.
Fujita, T., D. J. Stensrud, and D. C. Dowell, 2007: Surface data assimilation using an ensemble Kalman filter approach with initial condition and model physics uncertainties. Mon. Wea. Rev., 135, 1846–1868, doi:10.1175/MWR3391.1.
Gaspari, G., and S. E. Cohn, 1999: Construction of correlation functions in two and three dimensions. Quart. J. Roy. Meteor. Soc., 125, 723–757, doi:10.1002/qj.49712555417.
Gilmour, I., L. A. Smith, and R. Buizza, 2001: Linear regime duration: Is 24 hours a long time in synoptic weather forecasting? J. Atmos. Sci., 58, 3525–3539, doi:10.1175/1520-0469(2001)058<3525:LRDIHA>2.0.CO;2.
Ha, S.-Y., and C. Snyder, 2014: Influence of surface observations in mesoscale data assimilation using an ensemble Kalman filter. Mon. Wea. Rev., 142, 1489–1508, doi:10.1175/MWR-D-13-00108.1.
Hakim, G. J., and R. D. Torn, 2008: Ensemble synoptic analysis. Synoptic-Dynamic Meteorology and Weather Analysis and Forecasting: A Tribute to Fred Sanders, Meteor. Monogr., No. 55, Amer. Meteor. Soc., 147–161.
Hawblitzel, D. P., F. Zhang, Z. Meng, and C. A. Davis, 2007: Probabilistic evaluation of the dynamics and predictability of the mesoscale convective vortex of 10–13 June 2003. Mon. Wea. Rev., 135, 1544–1563, doi:10.1175/MWR3346.1.
Haynes, P. H., and M. E. McIntyre, 1987: On the evolution of vorticity and potential vorticity in the presence of diabatic heating and frictional or other forces. J. Atmos. Sci., 44, 828–841, doi:10.1175/1520-0469(1987)044<0828:OTEOVA>2.0.CO;2.
Higgins, R. W., V. E. Kousky, and P. Xie, 2011: Extreme precipitation events in the south-central United States during May and June 2010: Historical perspective, role of ENSO, and trends. J. Hydrometeor., 12, 1056–1070, doi:10.1175/JHM-D-10-05039.1.
Hohenegger, C., A. Walser, W. Langhans, and C. Schär, 2008: Cloud-resolving ensemble simulations of the August 2005 Alpine flood. Quart. J. Roy. Meteor. Soc., 134, 889–904, doi:10.1002/qj.252.
Hong, S.-Y., and J.-O. J. Lim, 2006: The WRF single-moment 6-class microphysics scheme (WSM6). J. Korean Meteor. Soc., 42, 129–151.
Janjić, Z. I., 1994: The step-mountain eta coordinate model: Further developments of the convection, viscous sublayer, and turbulence closure schemes. Mon. Wea. Rev., 122, 927–945, doi:10.1175/1520-0493(1994)122<0927:TSMECM>2.0.CO;2.
Janjić, Z. I., 2000: Comments on “Development and evaluation of a convection scheme for use in climate models.” J. Atmos. Sci., 57, 3686–3686, doi:10.1175/1520-0469(2000)057<3686:CODAEO>2.0.CO;2.
Janjić, Z. I., 2002: Nonsingular implementation of the Mellor–Yamada level 2.5 scheme in the NCEP Meso model. NCEP Office Note 437, 61 pp.
Jones, T. A., and D. J. Stensrud, 2012: Assimilating AIRS temperature and mixing ratio profiles using an ensemble Kalman filter approach for convective-scale forecasts. Wea. Forecasting, 27, 541–564, doi:10.1175/WAF-D-11-00090.1.
Kain, J. S., 2004: The Kain–Fritsch convective parameterization: An update. J. Appl. Meteor., 43, 170–181, doi:10.1175/1520-0450(2004)043<0170:TKCPAU>2.0.CO;2.
Keyser, D., cited2014: PREPBUFR processing at NCEP. NOAA/NWS/NCEP/EMC. [Available online at http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/document.htm.]
Kuo, Y.-H., T.-K. Wee, S. Sokolovskiy, C. Rocken, W. Schreiner, D. Hunt, and R. A. Anthes, 2004: Inversion and error estimation of GPS radio occultation data. J. Meteor. Soc. Japan, 82, 507–531, doi:10.2151/jmsj.2004.507.
Kursinski, E. R., G. A. Hajj, J. T. Schofield, R. P. Linfield, and K. R. Hardy, 1997: Observing Earth’s atmosphere with radio occultation measurements using the Global Positioning System. J. Geophys. Res., 102, 23 429–23 465, doi:10.1029/97JD01569.
Leoncini, G., R. S. Plant, S. L. Gray, and P. A. Clark, 2013: Ensemble forecasts of a flood-producing storm: Comparison of the influence of model-state perturbations and parameter modifications. Quart. J. Roy. Meteor. Soc., 139, 198–211, doi:10.1002/qj.1951.
Lin, Y., and K. E. Mitchell, 2005: The NCEP stage II/IV hourly precipitation analyses: Development and applications. 19th Conf. on Hydrology, San Diego, CA, Amer. Meteor. Soc., 1.2. [Available online at http://ams.confex.com/ams/pdfpapers/83847.pdf.]
Mason, S. J., and N. E. Graham, 2002: Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation. Quart. J. Roy. Meteor. Soc., 128, 2145–2166, doi:10.1256/003590002320603584.
Mellor, G. L., and T. Yamada, 1982: Development of a turbulence closure model for geophysical fluid problems. Rev. Geophys., 20, 851–875, doi:10.1029/RG020i004p00851.
Meng, Z., and F. Zhang, 2011: Limited-area ensemble-based data assimilation. Mon. Wea. Rev., 139, 2025–2045, doi:10.1175/2011MWR3418.1.
Mesinger, F., and Coauthors, 2006: North American Regional Reanalysis. Bull. Amer. Meteor. Soc., 87, 343–360, doi:10.1175/BAMS-87-3-343.
Morrison, H., J. A. Curry, and V. I. Khvorostyanov, 2005: A new double-moment microphysics parameterization for application in cloud and climate models. Part I: Description. J. Atmos. Sci., 62, 1665–1677, doi:10.1175/JAS3446.1.
National Research Council, 2010: When Weather Matters: Science and Services to Meet Critical Societal Needs. Board on Atmospheric Sciences and Climate, National Academies Press, 181 pp.
Nielsen-Gammon, J. W., X.-M. Hu, F. Zhang, and J. E. Pleim, 2010: Evaluation of planetary boundary layer scheme sensitivities for the purpose of parameter estimation. Mon. Wea. Rev., 138, 3400–3417, doi:10.1175/2010MWR3292.1.
NOAA, cited2013: Storm events database. [Available online at http://www.ncdc.noaa.gov/stormevents/.]
Noh, Y., W. G. Cheon, S. Y. Hong, and S. Raasch, 2003: Improvement of the K-profile model for the planetary boundary layer based on large eddy simulation data. Bound.-Layer Meteor., 107, 401–427, doi:10.1023/A:1022146015946.
Roberts, N. M., and H. W. Lean, 2008: Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events. Mon. Wea. Rev., 136, 78–97, doi:10.1175/2007MWR2123.1.
Romine, G. S., C. S. Schwartz, C. Snyder, J. L. Anderson, and M. L. Weisman, 2013: Model bias in a continuously cycled assimilation system and its influence on convection-permitting forecasts. Mon. Wea. Rev., 141, 1263–1284, doi:10.1175/MWR-D-12-00112.1.
Schumacher, R. S., A. J. Clark, M. Xue, and F. Kong, 2013: Factors influencing the development and maintenance of nocturnal heavy-rain-producing convective systems in a storm-scale ensemble. Mon. Wea. Rev., 141, 2778–2801, doi:10.1175/MWR-D-12-00239.1.
Schwartz, C. S., and Z. Liu, 2014: Convection-permitting forecasts initialized with continuously cycling limited-area 3DVAR, ensemble Kalman filter, and hybrid variationalensemble data assimilation systems. Mon. Wea. Rev., 142, 716–738, doi:10.1175/MWR-D-13-00100.1.
Schwartz, C. S., and Coauthors, 2010: Toward improved convection-allowing ensembles: Model physics sensitivities and optimizing probabilistic guidance with small ensemble membership. Wea. Forecasting, 25, 263–280, doi:10.1175/2009WAF2222267.1.
Skamarock, W. C., and Coauthors, 2008: A description of the Advanced Research WRF version 3. NCAR Tech. Note NCAR/TN-475+STR, 113 pp. [Available online at http://www.mmm.ucar.edu/wrf/users/docs/arw_v3_bw.pdf].
Smirnova, T. G., J. M. Brown, S. G. Benjamin, and D. Kim, 2000: Parameterization of cold-season processes in the MAPS land-surface scheme. J. Geophys. Res., 105, 4077–4086, doi:10.1029/1999JD901047.
Stensrud, D. J., J.-W. Bao, and T. T. Warner, 2000: Using initial condition and model physics perturbations in short-range ensemble simulations of mesoscale convective systems. Mon. Wea. Rev., 128, 2077–2107, doi:10.1175/1520-0493(2000)128<2077:UICAMP>2.0.CO;2.
Stratman, D. R., M. C. Coniglio, S. E. Koch, and M. Xue, 2013: Use of multiple verification methods to evaluate forecasts of convection from hot- and cold-start convection-allowing models. Wea. Forecasting, 28, 119–138, doi:10.1175/WAF-D-12-00022.1.
Tao, W.-K., J. Simpson, and M. McCumber, 1989: An ice-water saturation adjustment. Mon. Wea. Rev., 117, 231–235, doi:10.1175/1520-0493(1989)117<0231:AIWSA>2.0.CO;2.
Thompson, G., P. R. Field, R. M. Rasmussen, and W. D. Hall, 2008: Explicit forecasts of winter precipitation using an improved bulk microphysics scheme. Part II: Implementation of a new snow parameterization. Mon. Wea. Rev., 136, 5095–5115, doi:10.1175/2008MWR2387.1.
Tiedtke, M., 1989: A comprehensive mass flux scheme for cumulus parameterization in large-scale models. Mon. Wea. Rev., 117, 1779–1800, doi:10.1175/1520-0493(1989)117<1779:ACMFSF>2.0.CO;2.
Torn, R. D., 2010: Diagnosis of the downstream ridging associated with extratropical transition using short-term ensemble forecasts. J. Atmos. Sci., 67, 817–833, doi:10.1175/2009JAS3093.1.
Torn, R. D., and G. J. Hakim, 2008: Performance characteristics of a pseudo-operational ensemble Kalman filter. Mon. Wea. Rev., 136, 3947–3963, doi:10.1175/2008MWR2443.1.
Torn, R. D., G. J. Hakim, and C. Snyder, 2006: Boundary conditions for limited-area ensemble Kalman filters. Mon. Wea. Rev., 134, 2490–2502, doi:10.1175/MWR3187.1.
Tuttle, J. D., and C. A. Davis, 2006: Corridors of warm season precipitation in the central United States. Mon. Wea. Rev., 134, 2297–2317, doi:10.1175/MWR3188.1.
Velden, C., and Coauthors, 2005: Recent innovations in deriving tropospheric winds from meteorological satellites. Bull. Amer. Meteor. Soc., 86, 205–223, doi:10.1175/BAMS-86-2-205.
Vié, B., O. Nussier, and V. Ducrocq, 2011: Cloud-resolving ensemble simulations of Mediterranean heavy precipitating events: Uncertainty on initial conditions and lateral boundary conditions. Mon. Wea. Rev., 139, 403–423, doi:10.1175/2010MWR3487.1.
Wheatley, D. M., and D. J. Stensrud, 2010: The impact of assimilating surface pressure observations on severe weather events in a WRF mesoscale ensemble system. Mon. Wea. Rev., 138, 1673–1694, doi:10.1175/2009MWR3042.1.