Dry conditions in 2013–16 in much of the western United States were responsible for severe drought and led to an exceptional fire season in the Pacific Northwest in 2015. Winter 2015/16 was forecasted to relieve drought in the southern portion of the region as a result of increased precipitation due to a very strong El Niño signal. A student forecasting challenge is summarized in which forecasts of winter hydroclimate across the western United States were made on 1 January 2016 for the winter hydroclimate using several dynamical and statistical forecast methods. They show that the precipitation forecasts had a large spread and none were skillful, while anomalously high observed temperatures were forecasted with a higher skill and precision. The poor forecast performance, particularly for precipitation, is traceable to high uncertainty in the North American Multi-Model Ensemble (NMME) forecast, which appears to be related to the inability of the models to predict an atmospheric blocking pattern over the region. It is found that strong El Niño sensitivities in dynamical models resulted in an overprediction of precipitation in the southern part of the domain. The results suggest the need for a more detailed attribution study of the anomalous meteorological patterns of the 2015/16 El Niño event compared to previous major events.
Much of the western United States has suffered severe droughts in the new millennium. California has endured, beginning in 2013, what is arguably the most severe drought of the ~100-yr instrumental record (Seager et al. 2015). Colorado River reservoirs have declined to record low levels during a prolonged drought that dates to the early 2000s (U.S Bureau of Reclamation 2015). California and the Pacific Northwest suffered record or near-record low snowpacks in the 2014/15 winter (Mote et al. 2016), followed by the warmest summer of the instrumental record, which led to record wildfires (U.S. Forest Service 2016).
In the summer and autumn of 2015, one of the strongest El Niños on record began to evolve and was forecasted by the National Oceanic and Atmospheric Administration (NOAA) to intensify in the 2015/16 winter (Klein 2015). Seasonal forecast models predicted an alleviation of the drought conditions that were generally consistent with the so-called canonical El Niño dipole signature of enhanced winter precipitation in the Southwest accompanied by drier-than-normal conditions in the Pacific Northwest (Jong et al. 2016; Kintisch 2016). It is worth noting that the two previous very strong El Niño years (1982/83 and 1997/98) conformed to the canonical pattern, and there was an expectation that 2015/16 would as well. The prospects of El Niño resulting in an end to the ongoing drought in California and elsewhere in the region opened a discussion on the reliability of seasonal hydroclimate forecasts in the western United States. Since as much as half of the annual precipitation across the western United States falls in the winter months (January–March), accurate seasonal forecasts of winter precipitation, and more importantly snowfall, could help water managers improve the management of the major reservoirs in the region. Therefore, the 2015/16 winter (specifically January–March 2016, which we hereafter refer to as winter 2016) provides an excellent opportunity to assess the accuracy of seasonal forecasts across the western United States during exceptionally strong El Niño conditions.
Against this backdrop, graduate students at Princeton University and the University of California, Los Angeles (UCLA) were asked to produce seasonal forecasts of winter 2016 precipitation P, temperature T, and snow water equivalent (SWE) at a set of stations across the region. Specifically, they were asked to provide their best forecasts for an effective date of 1 January 2016, using methods of their choice, and using only data and/or model output available to them as of the forecast effective date. The Princeton University students were enrolled in a stochastic hydrology course while the UCLA students were enrolled in a physical hydrology course, which influenced the methods used to some extent. For both groups, however, the experiment challenged the students to apply and use seasonal forecast information from both dynamical and statistical models and combinations thereof.
2. Experimental design
a. The challenge
The goal of the student challenge was to forecast accumulated monthly P and average monthly T at a set of 10 stations (separate but paired stations were used for P and T and SWE) in the western United States for the months of January–March (1 April for SWE) using historical and/or climate forecast information that was available by 1 January 2016 (see Fig. S1 and Table S1 in the supplemental material for locations of the stations). All participants were provided with historic observations for each station, and there were no restrictions as to the methods that could be used to produce the forecasts.
In total, 25 forecasts for P, 24 forecasts for T, and nine forecasts for SWE were submitted. We partitioned the techniques and data that were used into eight categories: six empirical data methods [Autoregressive Integrated Moving Average (ARIMA), artificial neural networks (ANNs), climatological-derived forecasts (Clim), regression (Reg), copula, and vine copulas], and two dynamical seasonal forecast systems [NCEP Climate Forecast System, version 2 (CFSv2), and a subset of the North American Multi-Model Ensemble (NMME) suite]. We provide detailed descriptions of each forecast method in the supplemental material.
The observations for the 10 P and T stations (average record length 80 years) used for the forecast challenge were obtained from the University of Washington/UCLA Surface Water Monitor (SWM; Wood and Lettenmaier 2006). The stations were selected to have a relatively uniform spatial distribution across the western United States (Tables S1 and S2 and Fig. S1 in the supplemental material). Ten SWE stations (average record length 67 years) were taken from Natural Resources Conservation Service (NRCS) archives and were paired (i.e., chosen to be relatively close to) the P and T stations. Gridded observations from the SWM (at 1/16° spatial resolution) were used to study the relationship between the Niño-3.4 signal and precipitation and temperature anomalies. We used meteorological data from the NCEP–NCAR reanalyses (Kalnay et al. 1996) to study patterns in sea surface temperature (SST), 500-hPa height anomaly, and wind.
c. Retrospective forecast analysis
The performance of the dynamical NMME forecast (Kirtman et al. 2014) was determined by evaluating the January retrospective forecasts (reforecasts) for January–March accumulated P and average T in comparison with station observations for the period 1982–2010. For each individual forecast year, the root-mean-square error (RMSE) between the ensemble mean NNME forecast anomalies and observed anomalies is given by
where are the forecast P and T anomalies derived from linear interpolation of NMME to the stations (n = 10) and are the observed station anomalies. The RMSE values are used to assess the skill of NMME for individual years and the spread in forecast performance among years. Figure 3 (described in greater detail below) shows histograms of the P and T RMSEs over the 29-yr reforecast period.
a. What happened?
In contrast to the predictions and general expectations (widely reported in the press), winter 2016 started out with a relatively wet January across most of the western United States, and especially in the Pacific Northwest, which is expected to be anomalously dry in El Niño years. This was followed by below-average to normal precipitation in February across much of the region. March was wet across most of the region, except the extreme southern part of the domain, which is expected to be anomalously wet in El Niño years. Temperatures were above normal across most of the coastal portions of the domain in January, which propagated to most of the region later in the winter.
Accumulated over the January–March season, precipitation was abnormally wet over Washington, Oregon, and Northern California and below normal over Southern California and most of the western interior (Fig. 1). This pattern contrasts with the expected El Niño sensitivities for both the historic observations and the dynamical forecast models (Fig. 2). Averaged over the winter, temperatures were anomalously warm over the region, which is opposite to the expectation for the southern part of the region but consistent for the northern part. SWE anomalies were positive over much of the region, although warm temperatures reduced the anomalies somewhat, especially at the lowest elevation stations in Washington and Montana (Fig. 1).
The unexpected evolution of precipitation patterns in winter 2016 resulted in poor forecasts by the dynamical forecast models, leading to some of the highest forecast errors on record. A retrospective forecast analysis (1982–2010; section 2c) showed that the average forecast error for winter 2016 dynamical P forecasts in the NMME across all the stations was the highest on record. This resulted in a 160-mm or 355% increase in the RMSE compared to the long-term average RMSE for all the other January winter precipitation forecasts for the region (Fig. 3). As shown in the supplemental material, winter forecast errors generally show no relationship with El Niño strength, indicating that the anomalies in the winter 2016 forecast were not caused by the strong El Niño signal. The predictive skill of the dynamical forecast models for winter 2016 T anomalies shows that the RMSE was 2.38°C or 287% higher than long-term average, while the 1982/83 and 1998 strong El Niño winters had RMSE values around or below normal for El Niño winters (Fig. 3). This confirms that the failure in the dynamical model forecasts for winter 2016 P and T anomalies is not directly related to El Niño strength and is more likely related to the changes in the atmospheric circulation that occurred as a result of an atmospheric blocking pattern over the region that differed from canonical El Niño atmospheric circulation patterns (Fig. S3 in the supplemental material).
b. The forecasts
Although many different techniques were used by the students to generate the individual forecasts, most P forecasts are close to either 1) the long-term climatological mean or 2) the average observed anomaly in El Niño years (Fig. 4). The P forecasts based on information from either CFSv2 or the NMME show stronger anomalies, while the other forecasts are closer to climatology. Even though most of the forecasts used El Niño information, there is a wide disparity, even as to directionality in the forecast anomalies. This suggests an overall low predictability for forecasting total P over winter 2016.
The observed winter 2016 P anomalies indicate that most of the forecasts were far off target, and for some stations none of the forecasts came close to the observed P (e.g., stations 1, 2, and 5). On average, only 46% of the P forecasts had the correct directionality, where the strongest agreement in directionality was in the central part of the domain (stations 5, 7, and 9). The predictive skill of most of the statistical methods was severely hampered by the poor performance of the dynamical forecast models, as these were used as a covariate in most of the statistical forecast methods. Overall, this resulted in poor performance of most forecasts for the winter 2016 P anomalies.
Most forecasts show positive anomalies for all stations for the average winter 2016 T (Fig. 5). The agreement among the forecasts is the highest for the northern portion of the domain, whereas there is more spread in the south. The NMME forecasts (including CFSv2) produce very large anomalies, while this signal is dampened by the two copula methods. For the southern stations (6–10), the errors for the ARIMA models are relatively low, most likely because they effectively weigh recent years, which have also been warm.
In general, the T forecasts have lower spread and greater agreement as to the direction of anomalies than the P forecasts; 87% of the forecasts had the correct directionality in the anomalies, with decreasing skill from north to south. This suggests that the predictability of T based on historic data and the initial conditions is higher than for P, quite likely as a result of the relatively strong observed T trend over much of the region in recent decades, while the dynamical forecast models profit from strong teleconnections between observed T and SST.
The SWE forecasts mostly were for strong positive anomalies (Fig. 6), with some exceptions among the Pacific Northwest stations. In general, the forecasts tended to overestimate 1 April SWE in the eastern part of the domain (western interior), with the forecasts more accurate in the western part. The variation among the SWE forecasts is higher than for most of the P and T forecasts. The accuracy in the directionality of the anomalies is 67%, which is intermediate between the accuracy for P and T forecasts. Most of the SWE forecast methods relied on their P and T forecasts to make SWE predictions; therefore, errors and spread in either of those forecasts are propagated to the SWE forecasts leading to amplified spread.
4. El Niño sensitivity in the western United States
As noted in section 1, many studies have shown that there is a strong teleconnection between ENSO and western U.S. precipitation (e.g., Jong et al. 2016), temperatures (Seager and Hoerling 2014), and streamflow (Hamlet and Lettenmaier 1999; Ward et al. 2010; Wanders and Wada 2015). Most studies show a dipole effect, with El Niño warm and dry in the north and cool and wet in the south, and roughly the reverse for La Niña. There also is a perception (to our knowledge, not well supported by analysis) that the magnitude of the precipitation and temperature anomalies depends on the strength of the ENSO event. We investigated the sensitivity of anomalies in precipitation and temperature to El Niño signal, as defined by the regression slope between the Niño-3.4 signal seasonal precipitation and temperature anomalies over the western United States (see Fig. S3 in the supplemental material). The historical observations for the 29-yr period from 1982 to 2010, which include the strong 1982/83 and 1997/98 El Niño winters, were used to detect areas with a high El Niño sensitivity.
Although El Niño teleconnection is widely acknowledged in the literature, the sensitivity of large-scale climate patterns (and especially precipitation) to the ENSO signal is not as strong as generally believed (Fig. 2). Apart from a small warm anomaly in the Pacific Northwest and a wet anomaly in the western United States and the Southwest, teleconnections between ENSO and observed anomalies are absent throughout most of the region. When the sensitivities are upscaled using gridded observations at the 1° spatial resolution of the NMME models, we find a discrepancy between the two patterns. The dynamical forecast models show a much higher sensitivity to ENSO compared to observations (Fig. 2). A decomposition of the sensitivity signal between El Niño and La Niña years indicates that the strongest signal in the precipitation anomalies comes from El Niño years, while La Niña years are more important for the temperature anomalies.
Various studies show that a large part of the predictive skill of the current-generation seasonal forecast models of precipitation and temperature anomalies comes from ENSO teleconnections (e.g., Jia et al. 2015). During 2016, an anomalous region of high SSTs was observed offshore of western Canada. Earlier strong El Niño episodes in 1982/83 and 1997 showed lower SST temperatures offshore of western Canada (Jacox et al. 2016; Stanley 2016). In addition to this region, a smaller negative geopotential height anomaly appeared over the western states compared to earlier years (Fig. S2 in the supplemental material), resulting in relatively unobstructed airflow from the Pacific to the Pacific Northwest. This may have been a potential source of the abnormally high temperatures and anomalously high precipitation totals in the Northwest compared to other regions.
An important question for the scientific community is whether this El Niño winter 2016 was in itself an anomaly, or if it will become the new canonical pattern in a warmer western U.S. climate (Fig. S2 in the supplemental material). The observed anomalies in SST and 500-hPa height show that the 2016 El Niño winter was significantly different from the patterns that were observed during the 1982/83 and 1998 winters. As recently observed by Swain et al. (2016), the observed 500-hPa positive anomaly during winter 2016 could be a result of an increasing trend in 500-hPa height during winter. If this feature were to become more common in El Niño years, then it is of vital importance to fully understand the impact of this pattern on the large-scale circulation patterns and resulting precipitation and temperature anomalies.
We also found that the length of the historic record significantly impacts the forecasting results, particularly in terms of the inferred ENSO sensitivities for P and T (section 4). The P and T anomalies in ENSO years in particular are quite sensitive to the length of the historic record, given the strong trends in P and T for California (see Fig. S4 in the supplemental material). In the context of a continuously changing climate, this suggests that the choice of reference period has an impact on seasonal forecasts that use teleconnections as predictors. If the historical teleconnections in models are based on different periods, they can exhibit different teleconnection patterns for new events like winter 2016. A key question is what length of historic record is long enough from a statistical standpoint, but not so long as to include past events that are no longer representative of the current climate.
None of the P forecasts, regardless of method, came close to observed winter P at any of the stations. Many of the forecasts, statistical including dynamical, used information from the NMME, and the poor forecast performance particularly for P is traceable to a major NMME forecast “bust,” which appears to be related to the inability of the models to predict an atmospheric blocking pattern over the region that differed from canonical El Niño patterns.
The T forecasts showed some skill, with most forecasts indicating positive T anomalies as did the observations at all stations. However, many of the T forecasts showed apparent skill primarily because recent years have been anomalously warm across most of the region. This trend is detected by the T forecasts, resulting in an apparent skill as a result of accurate forecasts of the positive T trend in the region.
Forecasts based on direct transfer (with bias correction) of global forecast anomalies to the stations tended to amplify the magnitude of El Niño and La Niña anomalies relative to the sensitivities inferred directly from long-term P and T observations. This resulted in very large errors in P forecasts in both the Pacific Northwest (for which forecasts based on global models were for large negative anomalies) and for P forecasts in the southern part of the domain (large positive anomalies). In both cases, the observed anomalies were large and of opposite sign to those in the forecasts.
N.W. was supported by the Netherlands Organisation for Scientific Research (NOW) Rubicon Fellowship 825.15.003 (Forecasting to Reduce Socio-Economic Effects of Droughts). E.F.W. was supported by the NOAA Climate Program Office under grant NA15OAR4310075. D.P.L. was supported by the NOAA Climate Program Office under grant NA14OAR4310293. E.F.W. and D.P.L. designed the forecast experiment and initiated the experiment. X.G.H. and K.J.W. collected and aggregated the Princeton University and UCLA student forecasts. M.X. collected all the observations and made them available. A.B., X.G.H., H.H., A.K., Z.T.M., B.R.P., L.Q.P., N.V., K.J.W., M.X., S.Z., and N.W. produced the forecasts. N.W., E.F.W., and D.P.L. contributed to the design of the figures and all contributed to the writing of the manuscript. The station observations for this study can be obtained at http://www.wcc.nrcs.usda.gov/. The gridded SWM observations are available at http://hydro.washington.edu/forecast/monitor_cali/index.shtml. The NMME forecasts are available at https://iridl.ldeo.columbia.edu/SOURCES/.Models/.NMME/#info. The student forecast can be obtained by contacting the authors responsible for the forecast of interest (see the supplemental material).
Supplemental information related to this paper is available at the Journals Online website: http://dx.doi.org/10.1175/JHM-D-16-0230.s1.
Publisher’s Note: This article was revised on 1 February 2017 to include the open access designation that was missing when originally published.