1. Introduction
Severe thunderstorms (tornadoes, large hail, and damaging straight-line wind) are especially frequent in some parts of the United States. In response to these natural hazards, U.S. governmental agencies have coordinated efforts to issue severe weather forecasts, collect storm reports, and train forecasters since the 1950s (Galway 1989). As observing networks, technology, and our physical understanding have improved, so have forecasts of severe thunderstorms (Schaefer 1986). A particular challenge facing severe weather forecasters is that the ability of numerical weather prediction (NWP) models to resolve severe thunderstorms is still limited. In the 1990s, forecasters began to take advantage of research showing that the likelihood of severe thunderstorm occurrence (especially supercell thunderstorms) could be related, at least to first order, with local meteorological conditions in the form of severe thunderstorm indices (Davies and Johns 1993; Johns et al. 1993). Typical ingredients of severe thunderstorm indices are quantities such as convective available potential energy (CAPE) and deep-layer (often 0–6 km) vertical wind shear. These indices, computed from observations or output from numerical weather prediction models, can provide guidance to forecasters (Rasmussen and Blanchard 1998; Togstad et al. 2011), in what has been termed an “ingredients” forecast methodology (Doswell et al. 1996). The same ingredients approach has been used to investigate the climatological frequency of severe thunderstorms (Brooks et al. 2003) and to interpret climate change projections (Diffenbaugh et al. 2013).
Although weather predictability beyond 3 weeks is limited (Simmons and Hollingsworth 2002; Buizza and Leutbecher 2015), a rationale for seasonal climate forecasting has followed from recognizing that the prediction of monthly averages, or more generally, the statistics of weather, can be feasible on the basis of initial condition predictability and forcing from slowly evolving ocean temperatures (Shukla 1981, 1998). Now, seasonal predictions of quantities such as precipitation, near-surface temperature, and hurricane activity are routinely produced and issued. In many regions, including the United States, El Niño–Southern Oscillation (ENSO) is the source of much of the skill of seasonal predictions (Quan et al. 2006). Seasonal forecasts have been limited for the most part to seasonal averages, but there is increasing recognition that forecasts are capable of providing information about other aspects of the forecast distribution, including extremes (Becker et al. 2013). Most recently, there has been a convergence of interest, demand, and scientific capacity on subseasonal-to-seasonal (S2S) forecasts, which have lead times and forecast targets in the space between medium-range weather and long-range or seasonal forecasts (Vitart et al. 2017). Forecasts of extreme events, either in the sense of rarity or impact, are a topic of considerable interest in S2S prediction (Lee et al. 2018; Vitart and Robertson 2018). Extended-range S2S outlooks for severe thunderstorm activity including tornadoes and hail are one of the newest areas of interest and activity (Tippett et al. 2015; Carbin et al. 2016).
At present, there are two general approaches to making long-range forecasts of severe thunderstorm activity. One approach is based on the statistical relationships between severe thunderstorm activity and observed or dynamically forecast sea surface temperature (SST; e.g., Elsner and Widen 2014; Allen et al. 2015b; Lepore et al. 2017). Other studies, while not explicitly making predictions, have identified predictable climate components such as the Madden–Julian oscillation (MJO) and the tropical Pacific SST, which modulate severe thunderstorm activity and have the potential to be useful predictors (Cook and Schaefer 2008; Lee et al. 2013; Barrett and Gensini 2013; Gensini and Marinaro 2016; Molina et al. 2016; Cook et al. 2017). A second, more recent, approach uses dynamical model forecasts of the large-scale environments that are favorable to severe thunderstorm occurrence, essentially extending the ingredients method from short-range weather forecasting to S2S prediction (Carbin et al. 2016).
To quantify the extent to which S2S forecast models are capable of providing useful predictions of tornado or hail frequency at longer leads, here we examine the performance of ensemble monthly forecasts from the Climate Forecasting System, version 2 (CFSv2; Saha et al. 2014). Since the CFSv2 horizontal resolution is not sufficient to resolve thunderstorms, we use an ingredients approach with some modifications for monthly targets. Indices such as the tornado environment index (TEI; Tippett et al. 2012, 2014) and the hail environment index (HEI; Allen et al. 2015a) computed using reanalysis data have been demonstrated to perform well in representing the monthly climatological frequency of hail and tornado occurrence and their interannual variability. Here, monthly CFSv2 forecasts of these severe thunderstorm indices are evaluated to assess their skill in forecasting regimes that are supportive of severe thunderstorms during the upcoming month. To this end, we assess the skill of monthly CFSv2 forecasts of TEI and HEI, and their ingredients using storm report data and the North American Regional Reanalysis (NARR; Mesinger et al. 2006).
2. Data
Tornado and hail reports are taken from the NOAA/Storm Prediction Center’s (SPC) Severe Weather Database and aggregated to monthly scale over the period 1982–2016 and to a 1° × 1° grid over the contiguous United States (CONUS). The gridded tornado data used here are made up of the number of tornado reports (including all intensities) with starting latitudes and longitudes in each grid box each month. The gridded hail data are the number of hail events in each grid box each month, where the number of hail events is defined to be the number of 3-hourly periods with one or more reports of hail greater than 1 in. in diameter (Allen et al. 2015a). Tornado and hail reports are a useful but imperfect representation of the frequency of tornado and hail occurrence, as they are influenced by factors such as population, changing reporting practices, and evolving technology (Verbout et al. 2006; Allen and Tippett 2015). For simplicity, we will refer to the report-based gridded datasets as tornado and hail events.
Monthly averages of cPrcp, 0–3-km SRH, and MLCAPE for the period 1982–2016 are taken from the NARR (Mesinger et al. 2006), averaged from its native 32-km resolution to the 1° × 1° grid. The constants that appear in (1) were estimated using climatological values of NARR data and confirmed SPC reports over the period 1979–2010 for TEI (Tippett et al. 2012) and over the period 1979–2012 for HEI (Allen et al. 2015a). The values of the intercept terms in the indices have been adjusted to account for the use of matching offsets here.
Forecast monthly averages of cPrcp, 0–3-km SRH, and MLCAPE come from the CFSv2, a fully coupled dynamical ocean–atmosphere prediction system (Saha et al. 2014). CFSv2 data from January 1982 through March 2011 are hindcasts, and the CFSv2 data for the period from April 2011 through December 2016 are real-time forecast products. The hindcasts have initializations on every fifth day at 6-h intervals, and these pentad starts are usually organized as monthly ensembles consisting of six or seven pentads. Here, we use the average of the 16 initializations (four pentad starts) up to the seventh day of the month being forecast. The earliest start date is 18 September for forecasts of October. The exact start dates are different each month, and forecasts with start dates in the month being forecast do not include data prior to the start time in their monthly averages. We apply the same pentad sampling of start times to the real-time period for consistency. The 0–3-km SRH and MLCAPE data are provided on the 1° × 1° grid, and cPrcp (CPRAT) data are interpolated from the T126 Gaussian grid. Submonthly data for the same time period and variables are not presently available. The results are presented for both CONUS-wide and regional analyses based on NOAA’s U.S. Climate Regions (Fig. 1; Karl and Koss 1984).
Map of NOAA U.S. Climate Regions (https://www.ncdc.noaa.gov/monitoring-references/maps/us-climate-regions.php); for brevity, the “Northern Rockies and Central” region has been renamed the “Plains” and the “Ohio Valley” region is referred to as the “Central” region.
Citation: Weather and Forecasting 33, 5; 10.1175/WAF-D-18-0054.1
3. Results
a. CONUS-wide events and indices
We first compare CONUS-wide totals of the indices computed using reanalysis and forecast data to monthly numbers of tornado reports and hail events. The rank correlations of the CONUS-wide totals of the NARR-based TEI and HEI with tornado and hail events are 0.83 and 0.89, respectively (Figs. 2a,d). The NARR-based indices represent fairly well the interannual range of the report data, which is not guaranteed, since the methodology used to develop the indices uses only climatological values. Notably, the largest NARR TEI value of 600, which occurred in April 2011 when a record-breaking 700 tornadoes were reported, is outside the data used to estimate the TEI coefficients. The NARR-based and CFSv2-based indices are well correlated (0.77 and 0.92 for TEI and HEI, respectively), but the range of the CFSv2 indices is substantially less than that of the NARR indices (Figs. 2b,e). However, the amplitude of the forecast indices covers most of the data since extreme numbers of tornadoes and hail events are fairly rare. The amplitude of the CFSv2 TEI matches that of the NARR TEI up to about 200, which corresponds to the 99th percentile of the tornado report data. The maximum amplitude of the CFSv2 HEI is about 500, which corresponds to the 96th percentile of the hail event data. The limited range of the CFSv2 indices means that they do not match the range of monthly tornado numbers and hail events. Nonetheless, the correlations between the CFSv2 indices and report data are only slightly less than those between the CFSv2 indices and NARR indices (Figs. 2c,f). However, much of the variability of the monthly tornado and hail events is due to the annual cycle. In fact, the rank correlations of the climatological annual cycle of tornado and hail events with their monthly values are 0.78 and 0.90, respectively, which raises the question of how the forecast climatology and forecast interannual variability contribute to forecast quality.
Scatterplots of CONUS (top) tornado events and TEI and (bottom) hail events and HEI. The Spearman rank correlations of CONUS tornado counts and TEI in the top row and hail events and HEI in the bottom row are shown. The gray line follows the 1:1 ratio.
Citation: Weather and Forecasting 33, 5; 10.1175/WAF-D-18-0054.1
b. Forecast climatology
1) CONUS and regional annual cycles
The phasing and amplitude of the CFSv2 TEI annual cycle matches that of the tornado events and the NARR TEI fairly well CONUS wide and for NOAA regions east of the Rockies (Fig. 3). Peak values of the CFSv2 TEI are too low in the South, Southeast, Central, and Midwest regions, and these biases result in the CONUS values of the CFSv2 TEI being too low, despite peak values being slightly too high in the Northeast region. Peak values of the CFSV2 TEI in the South and Southeast regions are shifted a month earlier to April. The CFSv2 TEI also does not capture the sustained activity through the end of summer and into early fall in the Southeast and has unrealistically high values in December. The phasing of the CFSv2 TEI in the Northwest is similar to that of the NARR TEI, but neither matches the tornado events. The CFSv2 TEI phasing is similar in the West region to both the NARR TEI and tornado events. Since there are comparatively few observations in the western regions, comparisons with the NARR and CFSv2 indices there are more uncertain.
Annual cycle of monthly tornado events, NARR TEI, and CFSv2 TEI for the CONUS and the nine NOAA regions. The numbers in parentheses and square brackets are the Spearman rank correlations of the annual cycle of tornado events with NARR TEI and with CFSv2 TEI, respectively.
Citation: Weather and Forecasting 33, 5; 10.1175/WAF-D-18-0054.1
The annual cycle of the HEI better matches the monthly totals of hail events (Fig. 4) than does TEI and tornado events for the CONUS as well as the South, Northeast, Central, Upper Midwest, and Plains regions. CFSv2 outperforms NARR in terms of both seasonal peak timing and the overall magnitude in the Central and Plains regions, with correlation greater than 0.99 over the Plains. Despite some underestimation in the overall magnitude of the seasonal peak in the South, Southeast, Northeast, and Central regions, generally these differences are not appreciably worse than those for NARR, suggesting that there is good representation of the annual cycle. Similar to the differences noted for TEI (Fig. 3), HEI CFSv2 estimates over the Southwest, Northwest, and West show limited association with the report data, though these differences may reflect the presence of environments producing hail that are not characteristic to other parts of the country (Allen et al. 2015a).
Annual cycle of monthly hail events, NARR HEI, and CFSv2 HEI for the CONUS and the nine NOAA regions. The numbers in parentheses and square brackets are the Spearman rank correlations of the annual cycle of hail events with NARR HEI and with CFSv2 HEI, respectively.
Citation: Weather and Forecasting 33, 5; 10.1175/WAF-D-18-0054.1
2) Annual average spatial distribution
The spatial distribution of the annual averages of CFSv2 TEI are consistent with the largest frequencies for tornadoes being over the Central Plains and Southeast; however, the distribution also shows considerable differences, particularly in terms of magnitude compared to the distribution of the NARR TEI values (Figs. 5a,b). NARR TEI peak values are located over eastern Oklahoma and extend with decreasing magnitude southeast and are characterized by one or two more tornadoes on average per year compared to the CFSv2 TEI values. Peak values of CFSv2 TEI are located farther southeast, over Arkansas, Louisiana, and Mississippi. CFSv2 tends to be overly distributed between the two observed peak regions and biased eastward compared to the observational peak frequency over Kansas and Colorado (Fig. 5c), particularly over the Southern Plains. CFSv2 also tends to overemphasize the frequency over the northern High Plains into Montana compared to both NARR and the report data.
Average yearly totals of (a) NARR TEI, (b) CFSv2 TEI, and (c) tornado events. Yearly averages of (d) NARR HEI, (e) CFSv2 HEI, and (f) hail events.
Citation: Weather and Forecasting 33, 5; 10.1175/WAF-D-18-0054.1
HEI displays some of the same spatial behavior as TEI, but with a much larger spatial disparity between the peak climatologies of CFSv2 and NARR (Figs. 5d,e). CFSv2 HEI better captures the extension of the observed hail frequency westward from the northern Great Plains into Montana, with a reduced emphasis on high frequencies over the southeastern United States. Another region where hail events are frequent is found proximal to the warm Gulf Stream near the East Coast in both CFSv2 and NARR, contrasting a local nadir over the Appalachians. Nonetheless, over the majority of the southeastern United States, both CFSv2 and NARR underrepresent the frequency of reported hail events, consistent with the potential for differences in parameter weighting between regions and potentially problematic observations, as noted by Allen et al. (2015a).
3) Sources of forecast bias
Differences between the CFSv2 and NARR index climatologies are due to differences in their environmental ingredients, and these differences vary by region and calendar month. The largest annual average values of the difference between CFSv2 TEI and NARR TEI are negative and occur in parts of Texas, Oklahoma, Kansas, and Missouri (Fig. 6a). To assess the sources of these differences, we computed the annual average change in the NARR TEI that results when NARR cPrcp values are replaced with CFSv2 cPrcp values (Fig. 6b) and when NARR 0–3-km SRH values are replaced with CFSv2 0–3-km SRH values (Fig. 6c). Both cPrcp and 0–3-km SRH are responsible for overall reductions in the CFSv2 TEI compared to the NARR TEI, though with some spatial differences. The effect in TEI due to using CFSv2 0–3-km SRH is negative almost everywhere. The smaller native horizontal resolution of NARR generally favors a more detailed rendition of the mesoscale features that lead to the largest 0–3-km SRH values. Using CFSv2 values of cPrcp leads to some increases in TEI along the Texas coast that are offset by the decreases due to CFSv2 0–3-km SRH. Decreases in TEI due to CFSv2 cPrcp extend farther northward and into the Central region than those due to CFSv2 0–3-km SRH. This overall picture holds through most of the year, except that higher values of CFSv2 TEI in winter months are due to its higher values of cPrcp (not shown). Differences between the cPrcp products are expected, given the different convective parameterization schemes used for NARR and CFSv2 and differences in the model resolution.
Spatial distribution of yearly totals differences between (a),(d) the CFSv2 and NARR indices, and between mixed indices and NARR for (b),(c) TEI and (e)–(g) HEI.
Citation: Weather and Forecasting 33, 5; 10.1175/WAF-D-18-0054.1
Differences between CFSv2 and NARR HEI values are similar to those for TEI with the addition of positive values in western Kansas, Nebraska, and South Dakota (Fig. 6d). The annual average change in NARR HEI when NARR cPrcp values are replaced with CFSv2 cPrcp values (Fig. 6e) is smaller than for TEI because of the smaller cPrcp coefficient in HEI. The annual average changes in NARR HEI when NARR 0–3-km SRH is replaced by CFSv2 0–3-km SRH are larger. The high values of CFSv2 HEI in western Kansas, Nebraska, and South Dakota are due to higher CFSv2 MLCAPE values there, especially in June–August (not shown). Positive CFSv2 MLCAPE contributions stretch north through the Great Plains and extend westward toward the Rockies, particularly over New Mexico, Colorado, Wyoming, and Montana, and explain much of the northward displacement of the CFSv2-based HEI. This difference between the CFSv2 and NARR HEI climatologies is not necessarily a deficiency, since a limitation of the NARR HEI climatology was its poor representation of the westward extent of hail events (Allen et al. 2015a), driven by overly dry lower-tropospheric profiles in NARR that reduce MLCAPE in response to biases in the convective precipitation scheme (Gensini et al. 2014). The CFSv2 MLCAPE values are considerably lower than those of NARR over the Southeast, South, Gulf Coast, and Florida during July and August (not shown) but have only a modest impact on the annual values of HEI there because conditions are relatively less favorable for hail during this period.
The regional assessment of the annual cycles of the indices and report data (Figs. 3 and 4) reveals biases in the phasing of peak activity. These differences in the month of peak activity according to the indices and report data also have a clear spatial coherence (Fig. 7). CFSv2 depicts a winter peak, with maxima along the Gulf Coast during December and subsequently over Florida in January and February. This contrasts with a far smoother peak frequency during March over this region in NARR. The differences are smaller as the season reaches the May–June annual maximum frequency, with much of the northern Plains showing an earlier peak in CFSv2 compared to NARR. This earlier phasing in CFSv2 is more consistent with the observations, leading to higher correlations over the Midwest, Plains, and Central regions. Similar lagging characteristics are found for the HEI (Figs. 7d–f), with the CFSv2 annual cycle characterized by March peaks over the southeastern United States, before maintaining a very similar pattern to both the NARR and observations through the remainder of the season.
Climatological peak month at each grid point for tornado events and hail events, and corresponding indices for both NARR and CFSv2.
Citation: Weather and Forecasting 33, 5; 10.1175/WAF-D-18-0054.1
c. Forecast skill
1) Environments
The level of association (skill) of CFSv2 forecasts of TEI and HEI with numbers of tornado and hail events is limited by two factors—the skill with which the severe weather indices themselves can be predicted and the degree to which the indices correspond to the observations. Sampling variability might also play a role in the latter, given the large variability in severe thunderstorm activity. Furthermore, the skill with which the indices can be predicted depends on the skill with which the index constituents—cPrcp, MLCAPE, and 0–3-km SRH—can be predicted. While there have been skill assessments of monthly forecasts of quantities such as SST, precipitation, and 200-hPa height (Kumar et al. 2011), limited work is available with respect to the parameters used here (e.g., Jung and Kirtman 2016). As far as we know, this is the first comprehensive skill assessment of monthly forecasts of quantities that are relevant for severe weather.
Rank correlations over the CONUS between NARR and CFSv2 environments are positive overall (Fig. 8). Of the three environmental factors, CFSv2 forecasts of 0–3-km SRH show the highest positive correlations with NARR values, especially for the months from December through May. The higher skill in predicting 0–3-km SRH is somewhat expected, since tropospheric wind fields and upper-air features are known to be forecast with higher skill than thermodynamic and surface climate quantities such as temperature and precipitation (Kumar et al. 2011).
Spatial rank correlations of the monthly environments; values of 
Citation: Weather and Forecasting 33, 5; 10.1175/WAF-D-18-0054.1
Lower correlations for 0–3-km SRH are found over the summer months, during which the mesoscale factors that contribute to the development of 0–3-km SRH are expected to be not well resolved by CFSv2, with a coarser native horizontal resolution compared to NARR, that lead to a less accurate rendition of the small mesoscale features leading to enhanced 0–3-km SRH. The maps of forecast correlation for MLCAPE and cPrcp show lower values and are less spatially coherent, reflecting smaller spatial scales, especially for cPrcp. Overall, forecast correlations for MLCAPE are larger than those for cPrcp. Forecast correlations for April MLCAPE are notably high and widespread.
2) Indices
Errors in the forecasts tend to be compounded when the environmental parameters are combined in the severe weather indices. Despite this, large portions of the CONUS show significant positive correlations between NARR- and CFSv2-based indices, although some of those correlations (e.g., western United States in winter) may be less relevant for severe weather activity (Fig. 9). Overall, the maps of forecast correlation for TEI and HEI are very similar, but with correlations for TEI being somewhat lower and less spatially coherent, consistent with the behavior of MLCAPE relative to cPrcp seen previously. Relatively high correlations in March–July, especially for HEI, are noteworthy as they occur in areas where severe thunderstorms are frequent. A secondary peak in severe weather activity occurs in the end of October and November as the climatological positions of the upper-level jet stream shift southward. Activity during this time of the year often occurs in Gulf Coast states and can spread through the Southeast and Central regions, and the indices are predicted with relatively good skill during these months in these areas.
Spatial rank correlations of the indices; values of 
Citation: Weather and Forecasting 33, 5; 10.1175/WAF-D-18-0054.1
3) Observations
Finally, we look at the rank correlation of CFSv2-predicted values of TEI and HEI with the corresponding NARR-based indices and observed tornado counts and hail events data (Tables 1–6). Correlations can be affected by trends, and the observations, especially the number of hail events, have upward trends that are believed to be related to reporting procedures rather than changes in tornado and hail frequency (Verbout et al. 2006; Allen and Tippett 2015). Detrending, however, can be problematic since the procedure assumes that trends are stable, while the data indicate that these trends have lessened or stopped in recent years. An alternative to detrending is to form year-to-year differences of the data (Tippett 2014). Differencing improved the overall correlations between NARR HEI and the number of hail events (not shown) but has little impact on the correlations between CFSv2 HEI and the number of hail events (not shown).
Rank correlation of NARR TEI with the number of tornado events by NOAA region and month. The values in boldface (78) are statistically significant (p value < 0.05, t test). Missing values are for months with fewer than 10 monthly observations.
Rank correlation of NARR HEI and the number of hail events by NOAA region and month. Statistically significant values (63) are set in boldface, as in Table 1.
Rank correlation of NARR TEI and CFSv2 TEI by NOAA region and month. The values in boldface (94) are statistically significant (p value <0.05, t test).
Rank correlation of NARR HEI and CFSv2 HEI. Statistically significant values (91) are set in boldface, as in Table 1.
Rank correlation of CFSv2 TEI and the number of tornado events by NOAA region and month. Statistically significant values (34) are set in boldface, as in Table 1.
Rank correlation of CFSv2 HEI and the number of hail events. Statistically significant values (30) are set in boldface, as in Table 1.
First, we examine the extent to which the severe weather indices computed using reanalysis data are able to match the year-to-year variability of the observed tornado and hail events. Rank correlations of the NARR indices with the numbers of tornado and hail events are statistically significant in the majority of regions and months, especially those when severe thunderstorm activity is common (Tables 1 and 2). There are slightly fewer regions and months where HEI has a statistically significant correlation with the number of hail events compared to TEI with the number of tornado events. Correlations of CONUS-wide NARR indices with storm report data are statistically significant in most months with the exception of August and September for TEI and August, September, and November for HEI. On a CONUS-wide basis, NARR HEI correlations are weaker in every month than those for TEI, perhaps because of regional biases or reporting issues.
Examining the rank correlation of the CFSv2-based indices with the NARR-based indices (Tables 3 and 4 for TEI and HEI, respectively) shows that indices can be skillfully predicted in most regions and months of the year. Evaluating CFSv2 using storm report data shows rank correlations that are smaller than those with the NARR indices, with fewer significant values. However, there is still some skill, especially for TEI in the South, Southeast, and Central regions, and for HEI in the Upper Midwest and Plain regions; this trend is seen more broadly for both indices over the entire CONUS (Tables 5 and 6).
d. ENSO
A reasonable expectation, given the connections between U.S. severe thunderstorms and ENSO during spring (March–May; Allen et al. 2015b; Lepore et al. 2017) is that the CFSv2 severe thunderstorm forecasts or their skill might depend on the ENSO phase. To address the question of how ENSO modulates the forecasts, we correlated Niño-3.4 with the average of the forecast indices in a region where there is a relatively strong ENSO signal in tornado and hail activity (Allen et al. 2015b). All correlations between index forecasts and Niño-3.4 are negative for March–May targets (first two rows in Table 7, indicating forecasts of higher activity during cool ENSO (La Niña like) conditions, which is in line with previous findings of higher activity expected during cool ENSO (La Niña like) conditions (Lepore et al. 2017). The negative correlations with ENSO are statistically significant in March (−0.37) for TEI and in March–May (−0.47, −0.36, and −0.36, respectively) for HEI.
Rank correlation of seasonal Niño-3.4 values (average of the target month and the two preceding months, i.e., March forecast with January–March Niño-3.4) with CFSv2 TEI and HEI forecast (first two rows) for a region with strong ENSO signal (Allen et al. 2015b), and with NARR/CFSv2 CONUS-wide pattern correlations among environments (MLCAPE, 0–3-km SRH, cPrcp; last three rows). Significant values are set in boldface.
To address the question of whether ENSO modulates forecast skill, we computed the CONUS-wide pattern correlation of the CFSv2 environments with the NARR environments for each forecast and then computed the rank correlation of the environment pattern correlation with Niño-3.4 (last three rows in Table 7). The rank correlation was negative for all environmental variables, indicating a negative relationship between ENSO and skill as measured by pattern correlation. This negative relationship means that the pattern correlation between forecast and reanalysis environments is greater during cool ENSO (La Niña like) conditions. However, the only statistically significant correlations were for MLCAPE and 0–3-km SRH in March (−0.37 and −0.35, respectively). We repeated this analysis for the CONUS-wide pattern correlation of the CFSv2-based and NARR-based indices, but we found no consistent relationship.
Overall, we find that ENSO modulates the amplitude of the forecasts and, to a lesser extent, the skill of the forecast environments during March–May, with higher forecast activity and higher skill during cool ENSO conditions.
4. Summary and discussion
The Climate Forecasting System, version 2, is an operational coupled global ocean–atmosphere model used to produce subseasonal-to-seasonal (S2S) forecasts (Saha et al. 2014), with good skill in predicting ENSO and the MJO, as well as precipitation and near-surface temperature. The horizontal resolution of CFSv2 precludes it from resolving individual storms, but it does resolve large-scale features, including quantities such as MLCAPE and vertical wind shear, which provide information about the favorability of conditions for severe thunderstorms. Here, we have assessed the skill with which CFSv2 predicts environments that favor severe convection, severe thunderstorm indices, and the numbers of tornado and hail events.
Overall, the climatological frequencies of CFSv2-based indices are generally in line with values from NARR, particularly over the central plains and southeastern United States. However, there are some disparities in magnitude, with CFSv2 spreading higher climatological likelihoods farther away from the Great Plains. The source of the biases in the aggregated annual cycle and spatial climatology vary regionally and seasonally (Fig. 6), but overall, forecast monthly averages of 0–3-km SRH and cPrcp are lower than those from the reanalysis. For the majority of the CONUS and the majority of months, differences in the representation of convective precipitation drive TEI biases, and together with MLCAPE, which is generally larger in CFSv2 particularly over the High Plains, impact HEI. The 0–3-km SRH in CFSv2 is generally weaker, especially during the summer months when smaller-scale mesoscale features are more prevalent, contributing locally to the peak of the negative biases.
Whether or not there is skill in predicting storm numbers depends on how well the environment (ingredients) in the index can be predicted and the extent to which a monthly index is able to capture the variability of storm occurrence. CFSv2 monthly forecast skill of the environments is highest for 0–3-km SRH, followed by MLCAPE and cPrcp (Fig. 8). Although errors in predictions of the environments tend to be compounded in the indices, there are statistically significant correlations between the NARR-based and CFSv2-based indices over the CONUS and many of the region east of the Rockies (Tables 1–6), especially for months of peak activity. Forecast skill is reduced when we compare forecasts to storm report data, but still is present in some regions and during some times of the year.
Recent studies have shown that the skill of subseasonal forecasts of near-surface temperature and precipitation drops greatly as the lead time increases and the target period changes from week 2 to week 3 (Li and Robertson 2015; DelSole et al. 2017; Vigaud et al. 2017). Therefore, we expect that much of the skill in monthly forecasts of severe thunderstorm activity results from skill in the first 2 weeks. Since submonthly CFSv2 data for these variables are not available for this period, we partially tested this hypothesis by correlating CFSv2 forecasts of severe thunderstorm indices separately with observations from the first and second halves of the month. The correlations of the forecasts with the observations from the first half of the month are roughly comparable (not shown) with those for the full month, while the correlations between forecasts and observations from the second half of the month are very low, consistent with Carbin et al. (2016). However, because of the wide range of start dates included in the forecast ensemble, the precise dependence on lead time can be confirmed only when submonthly data are available with higher-frequency start times.
In summary, we have assessed the skill of CFSv2 forecasts within the context of severe thunderstorm activity. Indices are forecast with less skill than their constituent environments, but they do have statistically significant skill in some areas and during some times of the year. Forecasts and their skill are modulated by ENSO phase but contain independent information, as well, since there appears to be a strong contribution of the initial conditions to the skill. To the extent that much of the skill appears to be limited to the first half of the month, guidance from subseasonal model forecasts with higher resolution may be of value. Eventually, as computational resources increase, convection-allowing models that currently provide realistic and detailed information at much shorter lead times (Gallo et al. 2016) can be extended to give storm-scale descriptions of severe thunderstorm activity at longer leads.
Acknowledgments
This work was partially supported by a Columbia University Research Initiatives for Science and Engineering (RISE) award, NOAA’s Climate Program Office Modeling, Analysis, Predictions and Projections Program Award NA14OAR4310185, and the Willis Research Network.
REFERENCES
Allen, J. T., and M. K. Tippett, 2015: The characteristics of United States hail reports: 1955–2014. Electron. J. Severe Storms Meteor., 10 (3), http://www.ejssm.org/ojs/index.php/ejssm/article/viewArticle/149.
Allen, J. T., M. K. Tippett, and A. H. Sobel, 2015a: An empirical model relating U.S. monthly hail occurrence to large-scale meteorological environment. J. Adv. Model. Earth Syst., 7, 226–243, https://doi.org/10.1002/2014MS000397.
Allen, J. T., M. K. Tippett, and A. H. Sobel, 2015b: Influence of the El Niño/Southern Oscillation on tornado and hail frequency in the United States. Nat. Geosci., 8, 278–283, https://doi.org/10.1038/ngeo2385.
Barrett, B. S., and V. A. Gensini, 2013: Variability of central United States April–May tornado day likelihood by phase of the Madden–Julian oscillation. Geophys. Res. Lett., 40, 2790–2795, https://doi.org/10.1002/grl.50522.
Becker, E. J., H. Van den Dool, and M. Peña, 2013: Short-term climate extremes: Prediction skill and predictability. J. Climate, 26, 512–531, https://doi.org/10.1175/JCLI-D-12-00177.1.
Brooks, H. E., J. W. Lee, and J. P. Craven, 2003: The spatial distribution of severe thunderstorm and tornado environments from global reanalysis data. Atmos. Res., 67–68, 73–94, https://doi.org/10.1016/S0169-8095(03)00045-0.
Buizza, R., and M. Leutbecher, 2015: The forecast skill horizon. Quart. J. Roy. Meteor. Soc., 141, 3366–3382, https://doi.org/10.1002/qj.2619.
Carbin, G. W., M. K. Tippett, S. P. Lillo, and H. E. Brooks, 2016: Visualizing long-range severe thunderstorm environment guidance from CFSv2. Bull. Amer. Meteor. Soc., 97, 1021–1031, https://doi.org/10.1175/BAMS-D-14-00136.1.
Cook, A. R., and J. T. Schaefer, 2008: The relation of El Niño–Southern Oscillation (ENSO) to winter tornado outbreaks. Mon. Wea. Rev., 136, 3121–3137, https://doi.org/10.1175/2007MWR2171.1.
Cook, A. R., L. M. Leslie, D. B. Parsons, and J. T. Schaefer, 2017: The impact of the El Niño–Southern Oscillation (ENSO) on winter and early spring U.S. tornado outbreaks. J. Appl. Meteor. Climatol., 56, 2455–2478, https://doi.org/10.1175/JAMC-D-16-0249.1.
Davies, J. M., and R. H. Johns, 1993: Some wind and instability parameters associated with strong and violent tornadoes. 1. Wind shear and helicity. The Tornado: Its Structure, Dynamics, Prediction, and Hazards, Geophys. Monogr., No. 79, Amer. Geophys. Union, 573–582.
DelSole, T., L. Trenary, M. K. Tippett, and K. Pegion, 2017: Predictability of week-3–4 average temperature and precipitation over the contiguous United States. J. Climate, 30, 3499–3512, https://doi.org/10.1175/JCLI-D-16-0567.1.
Diffenbaugh, N. S., M. Scherer, and R. J. Trapp, 2013: Robust increases in severe thunderstorm environments in response to greenhouse forcing. Proc. Natl. Acad. Sci. USA, 110, 16 361–16 366, https://doi.org/10.1073/pnas.1307758110.
Doswell, C. A., III, H. E. Brooks, and R. A. Maddox, 1996: Flash flood forecasting: An ingredients-based methodology. Wea. Forecasting, 11, 560–581, https://doi.org/10.1175/1520-0434(1996)011<0560:FFFAIB>2.0.CO;2.
Elsner, J. B., and H. M. Widen, 2014: Predicting spring tornado activity in the central Great Plains by 1 March. Mon. Wea. Rev., 142, 259–267, https://doi.org/10.1175/MWR-D-13-00014.1.
Gallo, B. T., A. J. Clark, and S. R. Dembek, 2016: Forecasting tornadoes using convection-permitting ensembles. Wea. Forecasting, 31, 273–295, https://doi.org/10.1175/WAF-D-15-0134.1.
Galway, J. G., 1989: The evolution of severe thunderstorm criteria within the Weather Service. Wea. Forecasting, 4, 585–592, https://doi.org/10.1175/1520-0434(1989)004<0585:TEOSTC>2.0.CO;2.
Gensini, V. A., and A. Marinaro, 2016: Tornado frequency in the United States related to global relative angular momentum. Mon. Wea. Rev., 144, 801–810, https://doi.org/10.1175/MWR-D-15-0289.1.
Gensini, V. A., T. L. Mote, and H. E. Brooks, 2014: Severe-thunderstorm reanalysis environments and collocated radiosonde observations. J. Appl. Meteor. Climatol., 53, 742–751, https://doi.org/10.1175/JAMC-D-13-0263.1.
Johns, R. H., J. M. Davies, and P. W. Leftwich, 1993: Some wind and instability parameters associated with strong and violent tornadoes. 2. Variations in the combinations of wind and instability parameters. The Tornado: Its Structure, Dynamics, Prediction, and Hazards, Geophys. Monogr., No. 79, Amer. Geophys. Union, 583–590.
Jung, E., and B. P. Kirtman, 2016: Can we predict seasonal changes in high impact weather in the United States? Environ. Res. Lett., 11, 074018, https://doi.org/10.1088/1748-9326/11/7/074018.
Karl, T. R., and W. J. Koss, 1984: Regional and national monthly, seasonal, and annual temperature weighted by area, 1895-1983. Historical Climatology Series 4-3, National Climatic Data Center, Asheville, NC, 38 pp., https://repository.library.noaa.gov/view/noaa/10238.
Kumar, A., M. Chen, and W. Wang, 2011: An analysis of prediction skill of monthly mean climate variability. Climate Dyn., 37, 1119–1131, https://doi.org/10.1007/s00382-010-0901-4.
Lee, C.-Y., S. J. Camargo, F. Vitart, A. H. Sobel, and M. K. Tippett, 2018: Subseasonal tropical cyclone genesis prediction and MJO in the S2S dataset. Wea. Forecasting, 33, 967–988, https://doi.org/10.1175/WAF-D-17-0165.1.
Lee, S.-K., R. Atlas, D. Enfield, C. Wang, and H. Liu, 2013: Is there an optimal ENSO pattern that enhances large-scale atmospheric processes conducive to tornado outbreaks in the United States? J. Climate, 26, 1626–1642, https://doi.org/10.1175/JCLI-D-12-00128.1.
Lepore, C., M. K. Tippett, and J. T. Allen, 2017: ENSO-based probabilistic forecasts of March–May U.S. tornado and hail activity. Geophys. Res. Lett., 44, 9093–9101, https://doi.org/10.1002/2017GL074781.
Li, S., and A. W. Robertson, 2015: Evaluation of submonthly precipitation forecast skill from global ensemble prediction systems. Mon. Wea. Rev., 143, 2871–2889, https://doi.org/10.1175/MWR-D-14-00277.1.
Mesinger, F., and Coauthors, 2006: North American Regional Reanalysis. Bull. Amer. Meteor. Soc., 87, 343–360, https://doi.org/10.1175/BAMS-87-3-343.
Molina, M. J., R. P. Timmer, and J. T. Allen, 2016: Importance of the Gulf of Mexico as a climate driver for U.S. severe thunderstorm activity. Geophys. Res. Lett., 43, 12 295–12 304, https://doi.org/10.1002/2016GL071603.
Quan, X., M. Hoerling, J. Whitaker, G. Bates, and T. Xu, 2006: Diagnosing sources of U.S. seasonal forecast skill. J. Climate, 19, 3279–3293, https://doi.org/10.1175/JCLI3789.1.
Rasmussen, E. N., and D. O. Blanchard, 1998: A baseline climatology of sounding-derived supercell and tornado forecast parameters. Wea. Forecasting, 13, 1148–1164, https://doi.org/10.1175/1520-0434(1998)013<1148:ABCOSD>2.0.CO;2.
Saha, S., and Coauthors, 2014: The NCEP Climate Forecast System version 2. J. Climate, 27, 2185–2208, https://doi.org/10.1175/JCLI-D-12-00823.1.
Schaefer, J. T., 1986: Severe thunderstorm forecasting: A historical perspective. Wea. Forecasting, 1, 164–189, https://doi.org/10.1175/1520-0434(1986)001<0164:STFAHP>2.0.CO;2.
Shukla, J., 1981: Dynamical predictability of monthly means. J. Atmos. Sci., 38, 2547–2572, https://doi.org/10.1175/1520-0469(1981)038<2547:DPOMM>2.0.CO;2.
Shukla, J., 1998: Predictability in the midst of chaos: A scientific basis for climate forecasting. Science, 282, 728–731, https://doi.org/10.1126/science.282.5389.728.
Simmons, A. J., and A. Hollingsworth, 2002: Some aspects of the improvement in skill of numerical weather prediction. Quart. J. Roy. Meteor. Soc., 128, 647–677, https://doi.org/10.1256/003590002321042135.
Tippett, M. K., 2014: Changing volatility of U.S. annual tornado reports. Geophys. Res. Lett., 41, 6956–6961, https://doi.org/10.1002/2014GL061347.
Tippett, M. K., A. H. Sobel, and S. J. Camargo, 2012: Association of U.S. tornado occurrence with monthly environmental parameters. Geophys. Res. Lett., 39, L02801, https://doi.org/10.1029/2011GL050368.
Tippett, M. K., A. H. Sobel, S. J. Camargo, and J. T. Allen, 2014: An empirical relation between U.S. tornado activity and monthly environmental parameters. J. Climate, 27, 2983–2999, https://doi.org/10.1175/JCLI-D-13-00345.1.
Tippett, M. K., J. T. Allen, V. A. Gensini, and H. E. Brooks, 2015: Climate and hazardous convective weather. Curr. Climate Change Rep., 1, 60–73, https://doi.org/10.1007/s40641-015-0006-6.
Togstad, W. E., J. M. Davies, S. J. Corfidi, D. R. Bright, and A. R. Dean, 2011: Conditional probability estimation for significant tornadoes based on rapid update cycle (RUC) profiles. Wea. Forecasting, 26, 729–743, https://doi.org/10.1175/2011WAF2222440.1.
Verbout, S. M., H. E. Brooks, L. M. Leslie, and D. M. Schultz, 2006: Evolution of the U.S. tornado database: 1954–2003. Wea. Forecasting, 21, 86–93, https://doi.org/10.1175/WAF910.1.
Vigaud, N., A. W. Robertson, and M. K. Tippett, 2017: Multimodel ensembling of subseasonal precipitation forecasts over North America. Mon. Wea. Rev., 145, 3913–3928, https://doi.org/10.1175/MWR-D-17-0092.1.
Vitart, F., and A. W. Robertson, 2018: The sub-seasonal to seasonal prediction project (S2S) and the prediction of extreme events. npj Climate Atmos. Sci., 1, 3, https://doi.org/10.1038/s41612-018-0013-0.
Vitart, F., and Coauthors, 2017: The Subseasonal to Seasonal (S2S) Prediction project database. Bull. Amer. Meteor. Soc., 98, 163–173, https://doi.org/10.1175/BAMS-D-16-0017.1.
