SubX is a research to operations project in service of developing better operational subseasonal forecasts.
Early warning of heat waves, extreme cold, flooding rains, flash drought, or other weather hazards as far as 4 weeks into the future could allow for risk reduction and disaster preparedness, potentially preserving life and resources. Less extreme, but no less important, reliable probabilistic forecasts about the potential for warmer, colder, wetter, or drier conditions at a few weeks lead are valuable for routine planning and resource management. Many sectors would benefit from these predictions, including emergency management, public health, energy, water management, agriculture, and marine fisheries [see White et al. (2017) for a review of potential applications]. However, a well-known “gap” exists in our current prediction systems at this subseasonal time scale of 2 weeks to 1 month. This gap falls between the prediction of weather, where atmospheric initial conditions contribute to skillful forecasts, and seasonal prediction, which is guided by slowly evolving surface boundary conditions such as sea surface temperatures and soil moisture (National Research Council 2010; Brunet et al. 2010; National Academies of Sciences, Engineering and Medicine 2017; Mariotti et al. 2018; Black et al. 2017; DelSole et al. 2017).
The potential for successful prediction at the subseasonal time scale has been established for some regions and seasons (e.g., Pegion and Sardeshmukh 2011; DelSole et al. 2017; Li et al. 2015), but it is not clear whether the full potential predictability has been realized. Additionally, many questions remain regarding our fundamental understanding of the physical processes giving rise to predictability, as well as how best to design, build, postprocess, and verify a subseasonal prediction system. Amidst these questions, the U.S. National Oceanic and Atmospheric Administration (NOAA) was mandated to begin issuing week 3–4 outlooks for temperature and precipitation. NOAA has for many years released official outlooks for 1-week, 2-week, 1-month, and 3-month averages; week 3–4 prediction is a new area with many unique research and development concerns.
The Subseasonal Experiment (SubX), a research-to-operations project, was launched to fulfill both the immediate need for real-time subseasonal prediction guidance and to allow for the exploration of relevant research questions, in order to develop more skillful and useful subseasonal predictions in the future. SubX takes a multimodel ensemble approach and includes global climate prediction models from both operational and research centers. As a research database designed around operational standards, SubX improves our ability to directly answer research questions in the service of developing better operational forecasts.
Combining models together into multimodel ensembles has been a successful technique to improve forecast quality for weather and seasonal predictions (e.g., Hagedorn et al. 2005; Weigel et al. 2008; Kirtman et al. 2014; Krishnamurti et al. 2000, 1999). The skill improvement comes from two sources: first, the collection of a larger ensemble of model predictions than that available from any individual forecast system, which allows for a better estimation of forecast uncertainty, probability distribution, and signal-to-noise ratio; equally advantageous is so-called “complementary skill,” or the additive skill from the different models. Also, as new versions of constituent models are introduced to the ensemble, a multimodel system can evolve faster than the typical improvement cycle for a single model. Examples of current multimodel systems include the North American Multimodel Ensemble (NMME; Kirtman et al. 2014) and European Seasonal to Interannual Prediction (EUROSIP; Mishra et al. 2019), both seasonal forecast systems, and the North American Ensemble Forecast System (NAEFS; Candille 2009; Candille et al. 2010), which produces forecasts out to 14 days.
THE SUBX DATABASE.
SubX provides a publicly available database of 17 years of historical reforecasts (1999–2015), plus more than 18 months of real-time forecasts from seven U.S. and Canadian modeling groups. All forecasts include daily values for at least 32 days beyond the initialization date. See Table 1 for model descriptions and the appendix A for protocol details.
Summary of models participating in SubX. In the components column, A = atmosphere, O = ocean, I = sea ice, and L = land. Numbers in the ensemble members column apply to reforecasts and real-time forecasts unless indicated by brackets, which indicate a different number of ensemble members used in real-time forecasts than those used in the reforecasts. Initial day of week refers to the day of the week the real-time forecasts fall on for each model. Community column indicates SEAS for seasonal prediction community and NWP for numerical weather prediction community. The R/O column indicates O for operational models and R for research models.
SubX has two unique aspects that distinguish it from other subseasonal forecast databases, such as the World Weather Research Programme (WWRP)/World Climate Research Programme (WCRP) Subseasonal to Seasonal (S2S) Prediction Project (Robertson et al. 2015; Vitart et al. 2017). The first of these is the inclusion of research models alongside operational models from NOAA and Environment and Climate Change Canada, facilitating feedback between research and operations on model development. A second distinction is the almost immediate availability of forecasts, allowing for use in real-time applications, including the NOAA Climate Prediction Center’s week 3–4 outlooks. This aspect of SubX has provided forecasters with additional forecast guidance, and allows for a research experiment to assess and guide best practices and priorities for real-time predictions.
HOW SKILLFUL ARE SUBSEASONAL PREDICTIONS WITH THE SUBX MODELS?
In addition to physical scientific questions, the design of a subseasonal multimodel ensemble mean (MME) presents practical complications beyond those of a weather or seasonal system. For example, a common challenge for subseasonal reforecast databases is that different models are initialized on different days, making it difficult to produce a traditional multimodel ensemble, typically made by averaging all forecasts from the same start date (Vitart et al. 2017). The implications of this practical consideration are explored in the SubX project, wherein forecasts from different start dates over the course of 1 week are combined and verified for the same verification period. This methodology, called a lagged average ensemble, has been used in weather and seasonal forecasting with single models (e.g., Hoffman and Kalnay 1983; Kalnay and Dalcher 1987; Trenary et al. 2018; DelSole et al. 2017).
Here, we evaluate the skill of the week 3 averages (average of days 15–21 of the forecast period) over all seasons from the individual SubX models ensemble means, as well as the MME, for anomalous temperature and precipitation over land. Skill is assessed using the anomaly correlation coefficient (ACC; Wilks 2006). The ACC provides information about how well the variability of the forecasted anomalies matches the observed variability, and is calculated as the temporal correlation of temporal anomalies at each grid point (Becker et al. 2014), shown as maps in Figs. 1 and 2. Details of the observational datasets used for verification are provided in the sidebar “Verification datasets” and details of the methodology used for making climatology and anomalies are provided in the appendix B.
ACC of 2-m temperature for week 3 (average of forecast days 15–21). Numbers in parentheses indicate the average ACC value over all land points in the domain. ACC values greater than 0.12 are statistically different from zero at the 5% level using a t test based on 219 degrees of freedom (17 years × 52 weeks = 884 forecasts/4-week decorrelation estimate). For reference, an ACC of 0.4 (0.2) means that the model can explain 16% (4%) of the observed variance. The calculation is performed over reforecasts with initial conditions for all months from the years 1999–2014.
Citation: Bulletin of the American Meteorological Society 100, 10; 10.1175/BAMS-D-18-0270.1
ACC of precipitation for week 3 (average of forecast days 15–21). Numbers in parentheses indicate the average ACC value over all land points in the domain. ACC values greater than 0.12 are statistically different from zero at the 5% level using a t test based on 219 degrees of freedom (17 years × 52 weeks = 884 forecasts/4-week decorrelation estimate). For reference, an ACC of 0.4 (0.2) means that the model can explain 16% (4%) of the observed variance. The calculation is performed over reforecasts with initial conditions for all months from the years 1999–2014. South America is shown as the only region with statistically significant skill.
Citation: Bulletin of the American Meteorological Society 100, 10; 10.1175/BAMS-D-18-0270.1
VERIFICATION DATASETS
Calculation of skill requires a verifying observational dataset. Where applicable, the datasets used correspond to those used by CPC for verification of their forecasts. For 2-m temperature over land, the CPC daily temperature dataset with horizontal resolution of 0.5° × 0.5° is used.SB3 These data are provided as a maximum (Tmax) and minimum (Tmin) daily temperature, thus the average daily temperature is calculated as the average of Tmax and Tmin (Fan and van den Dool 2008). For precipitation over land, the CPC Global Daily Precipitation dataset (0.5° × 0.5°) is used (Xie et al. 2007; Chen et al. 2008). Verification datasets are regridded to the coarser SubX model resolution of 1° × 1° prior to performing model evaluation. The years 1999–2014 are used for evaluation of the 2-m temperature and precipitation skill.
We also evaluate the skill of indices representing two subseasonal phenomena that are known sources of S2S predictability–the MJO and the NAO. The MJO skill is evaluated using the RMM index without interannual variability removed (Wheeler and Hendon 2004). The observed index is calculated using the NCEP–NCAR reanalysis (Kalnay et al. 1996) and NOAA Interpolated OLR (Liebmann and Smith 1996). The NAO is defined as the projection of the December–February geopotential height at 500 hPa (Z500) onto the leading North Atlantic EOF spatial pattern of Z500 (0°–90°N, 93°W–47°E). The observed NAO index is calculated using 500-hPa geopotential height from the NCEP–NCAR reanalysis (Kalnay et al. 1996). The years 1999–2014 are used for the evaluation of MJO and NAO skill. Both indices are calculated daily and then averaged to weekly values for skill calculations.
The original data can be found at
The skill of the individual models and MME are also compared to a forecast based on the persistence of the initial conditions, where the anomaly at the initial forecast time is predicted to continue throughout the forecast. Week 3 is beyond weather time scales, and predictability due to atmospheric initial conditions is largely absent (Lorenz 1965, 1969). However, predictability due to slower varying components of the climate system, such as the global warming trend or the El Niño–Southern Oscillation, present in the initial anomaly will change little over a 3-week forecast. Therefore, skill due to these mechanisms would be present in a persistence forecast. Comparison of forecast skill with the skill of a persistence forecast provides insights into whether forecast skill can be attributed to any of these slowly varying components.
Over all months, positive ACC for temperature forecasts is present over much of the land for most models and the MME, with substantial regional variations (Fig. 1). The ACC of the individual models and the MME are higher than the skill of a persistence forecast, indicating that there is skill from sources other than the trend and/or ENSO (Fig. 1). While skill here is shown for the 15–21-day average forecasts for the individual models, the MME is produced from lagged averaged forecasts, and contains older model initializations (see the appendix C for details). However, the MME shows skill improvement over the individual models. For precipitation, anomaly correlation maps for week 3 indicate that the only region of statistically significant skill when calculated over all months is in Brazil (Fig. 2). This region of precipitation skill is consistent across the individual models and has higher skill than a persistence forecast; again, the MME has higher ACC than individual models, despite the inclusion of older model initializations.
While the multimodel ensemble mean methodology improves skill over the individual models, on average, skill at subseasonal time scales is low. However, there is evidence that skill varies over time. For example, there is seasonal dependence of skill for North America, with winter being more skillful than summer (e.g., DelSole et al. 2017). Skill also varies from year-to-year. This is evident in the SubX MME skill of spatial pattern correlations of North America temperature and precipitation anomalies for January initial conditions, which exhibits substantial variation with time (Fig. 3). At times, the ACC exceeds 0.5, a common threshold for “useful” skill (Murphy and Epstein 1989; Barnston and den Dool 1994; Jones et al. 2000) while at other times, the ACC is zero or even negative. This indicates there may be potential for higher skill forecasts at certain times, called “forecasts of opportunity.” While a thorough diagnosis of these higher skill periods is outside the current scope of this paper, in the next section we examine some potential sources of subseasonal prediction skill.
ACC between observed and SubX MME spatial anomalies for week 3 forecasts of (a) 2-m temperature and (b) precipitation over North America (15°–75°N, 190°–305°E) for the 71 MME January reforecasts over the 1999–2014 reforecast period. Blue dashed and dotted lines indicate ACC of 0.0, 0.12, and 0.5.
Citation: Bulletin of the American Meteorological Society 100, 10; 10.1175/BAMS-D-18-0270.1
SUBSEASONAL SOURCES OF PREDICTABILITY.
Subseasonal predictability is likely influenced by a number of modes of climate variability that vary on time scales of weeks, including the Madden–Julian oscillation (MJO; Madden and Julian 1971, 1972) or the North Atlantic Oscillation (NAO; Hurrell et al. 2010). Several studies have suggested these modes may be predictable on subseasonal time scales, and present potential sources of predictability, allowing for the identification of “forecasts of opportunity” (National Research Council 2010; National Academies of Sciences, Engineering and Medicine 2017). That is, due to known impacts from the subseasonal modes, model forecasts may be more skillful when these modes are active, allowing for more confidence in their output. Correctly simulating and predicting these processes and their impacts are the key to successful subseasonal prediction.
The Madden–Julian oscillation.
The Madden–Julian oscillation, a dominant mode of tropical variability on subseasonal time scales, is a system of large-scale convective anomalies and associated circulation anomalies that propagates eastward from the tropical Indian Ocean and affects global weather [e.g., Cassou 2008; Lin et al. 2009; Guan et al. 2012; Mundhenk et al. 2018; Zhang 2013; see Stan et al. (2017) for a review of MJO teleconnections].
Therefore, accurate simulation and prediction of the MJO and its propagation is crucial to extend global subseasonal forecast skill. Observed convective anomalies associated with the MJO, as indicated by outgoing longwave radiation (OLR) anomalies, propagate eastward from the Indian Ocean (60°E) to the date line (Fig. 4, top). Most of the SubX models can reproduce the observed propagation of the OLR anomalies in week 3 forecasts, although some appear to have difficulty propagating them across the Maritime Continent, approximately 120°E—a well-known challenge for global climate models (Kim et al. 2018).
Week 3 (average of days 15–21) composite OLR (W m–2) averaged 5°S–5°N as a function of longitude (x axis) and phase (y axis) for MJO events identified based on RMM index amplitude ≥1.
Citation: Bulletin of the American Meteorological Society 100, 10; 10.1175/BAMS-D-18-0270.1
A common measurement of the MJO uses two Real-time Multivariate MJO (RMM) indices that combine OLR with winds at 200 and 850 hPa and measure the strength and phase of the MJO (Wheeler and Hendon 2004). A model’s ability to predict the combination of both RMM indices in terms of the bivariate correlation of the two indices provides insight into its overall capability to simulate and predict the MJO (Rashid et al. 2011). Most of the individual SubX models have ACC for these indices >0.5 out to week 4 (Fig. 5). This range of prediction skill is similar to the MJO skill of the WWRP/WCRP S2S models, with the exception of the skill of the European Centre for Medium-Range Weather Forecasts (ECMWF) model, which far exceeds that of any other S2S or SubX model (Vitart 2017). The SubX MME has similar skill to the best individual models for weeks 1–3 and higher skill at week 4 (Fig. 5). The MME is consistent with the ECMWF model from the S2S database, which has ACC for RMM indices of 0.6 out to 28 days (i.e., the end of the 4-week period; Vitart 2017).
RMM index skill in terms of bivariate anomaly correlation for November–March initialized reforecasts. NCEP-CFSv2 OLR data were not provided to the SubX database and are not included here.
Citation: Bulletin of the American Meteorological Society 100, 10; 10.1175/BAMS-D-18-0270.1
It is of interest that the two most skillful SubX models at weeks 3 and 4 have very different configurations. The GMAO-GEOS model is a fully coupled atmosphere–ocean–land–sea ice model that has contributed to the monthly and seasonal NMME; this model contributes 4 ensemble members in SubX (see sidebar “SubX models”). In contrast, the base model of the EMC-GEFS is a numerical weather prediction atmosphere–land model forced with prescribed sea surface temperatures (SSTs) and contributes 11 ensemble members to the SubX reforecasts. The comparable MJO prediction skill from these two models illustrates an open question of S2S ensemble prediction, as the varying contributions of model configuration, ensemble size, and the role of a fully interactive ocean model remain to be clarified.
SUBX MODELS
Seven modeling groups participate in SubX:
National Centers for Environmental Prediction (NCEP) Climate Forecast System, version 2 (NCEP-CFSv2);
NCEP Environmental Modeling Center, Global Ensemble Forecast System (EMC-GEFS);
Environment and Climate Change Canada Global Ensemble Prediction System, Global Environmental Multi-scale Model (ECCC-GEM);
National Aeronautics and Space Administration, Global Modeling and Assimilation Office, Goddard Earth Observing System (GMAO-GEOS);
Navy Earth System Prediction Capability (NAVY-ESPC);SB1
National Center for Atmospheric Research Community Climate System Model, version 4, run at the University of Miami Rosenstiel School for Marine and Atmospheric Science (RSMAS-CCSM4);
National Oceanic and Atmospheric Administration, Earth System Research Laboratory, Flow-Following Icosahedral Model (ESRL-FIM).
(For additional details, see Table 1.)
All groups have provided reforecasts for the 1999–2015 period with the exception of ECCC-GEM (1999–2014)SB2 and most have provided additional reforecasts to fill the gap between the end of the SubX reforecast period and beginning of the real-time forecasts in July 2017. Five of the groups use fully coupled atmosphere–ocean–land–sea ice models (NCEP-CFSv2, GMAO-GEOS, NAVY-ESPC, RSMAS-CCSM4, ESRL-FIM), while two groups use models with atmosphere and land components forced with prescribed SSTs (EMC-GEFS, ECCC-GEM). In the EMC-GEFS forecast system, SSTs are specified by relaxing the SST analysis to a combination of climatological SST and bias-corrected SST from operational NCEP-CFSv2 forecasts. The longer the lead time, the more weight given to the bias-corrected NCEP-CFSv2 forecast SST. In the ECCC-GEM forecast system, the SST anomaly averaged from the previous 30 days is persisted in the forecast. The sea ice cover is adjusted in order to be consistent with the SST change [see Gagnon et al. (2013) for details]. Most groups provide four ensemble members for the reforecasts (NCEP-CFSv2, ECCC-GEM, GMAO-GEOS, NAVY-ESPC, ESRL-FIM) with some groups creating ensembles by combining different start times and others using their own ensemble generation systems to produce initial conditions. Some groups provide additional ensemble members in real-time (e.g., RSMAS-CCSM4, EMC-GEFS).
The NAVY-ESPC model is referred to as NRL-NESM in the SubX database and the change of name to NAVY-ESPC in the database is currently in progress. NRL-NESM and NAVY-ESPC refer to the same model.
ECCC-GEM runs its reforecasts on the fly as part of their operational practice and will fill in 2015 at a later date.
The North Atlantic Oscillation.
The NAO, indicated by an oscillation in surface pressure and geopotential height between the Iceland low and the Azores high, is a key source of extratropical subseasonal variability (Hurrell et al. 2010). The NAO has been linked to periods of extreme winter weather on subseasonal time scales in eastern North America and Europe (e.g., Hurrell et al. 2010). Until recently, there was little evidence that the NAO could be skillfully predicted beyond weather time scales (e.g., Johansson 2007; Kim et al. 2012); however, recent studies have found that the Met Office seasonal prediction system can produce skillful monthly predictions of the NAO up to 1 year into the future using large ensembles (>20 members) and long reforecasts (∼40 years; Scaife et al. 2014; Dunstone et al. 2016).
Given both this newly discovered predictability of the NAO and its potential impacts on extreme weather at S2S time scales, we evaluate the skill of NAO prediction by the SubX models using a daily index representing the NAO (see “Verification datasets” sidebar for details of the index calculation). All individual models, as well as the MME, exhibit ACC > 0.5 when forecasting this NAO index through week 2 (average of days 8–14), using initialization dates from the Northern Hemisphere winter (Fig. 6). While ACC drops for forecasts of week 3 and week 4, one individual model has ACC = 0.5, while all models have significant skill at week 3. Only for forecasts of week 4 does the ACC of the MME clearly exceed any individual model.
NAO index skill in terms of ACC for December–February initialized reforecasts.
Citation: Bulletin of the American Meteorological Society 100, 10; 10.1175/BAMS-D-18-0270.1
REAL-TIME FORECASTS.
The SubX participating modeling centers have produced new forecasts each week since July 2017. These are provided to the NOAA Climate Prediction Center (CPC) as dynamical guidance for their official week 3–4 temperature outlook and experimental week 3–4 precipitation outlook, issued every Friday. The CPC outlooks show regions of increased probability of above-normal or below-normal temperature and precipitation, and regions where the probabilities of above or below normal are equally likely (i.e., 50/50 chance). Using guidance from the real-time SubX forecasts for 2-m temperature, precipitation, and 500-hPa geopotential heights as well as other tools, CPC forecasters produce the official maps for week 3–4 outlooks. For example, the maps for 6 July 2018 temperature and precipitation show above- and below-normal areas consistent with the corresponding probabilities and anomalies from the SubX multimodel ensemble mean, demonstrating the use of SubX in the CPC official outlooks (Fig. 7).
SubX real-time multimodel ensemble mean anomaly and probability guidance for (a),(b) temperature and (d),(e) precipitation and corresponding CPC official week 3–4 outlook products for (c) temperature and (f) precipitation. Forecasts were made 6 Jul 2018. The temperature in (b) and precipitation in (e) probability maps are for above-normal categories.
Citation: Bulletin of the American Meteorological Society 100, 10; 10.1175/BAMS-D-18-0270.1
We also evaluate the skill of the SubX real-time 2-m temperature forecasts produced from July 2017 to December 2018. Overall the real-time forecasts have similar skill to the reforecasts (Figs. 1 and 8). The real-time forecasts are also substantially more skillful over the continental United States than the reforecasts. Skill is expected to vary from year to year, depending on the presence or absence of major modes of climate variation, land surface conditions, and other factors. The sources of the higher skill over the continental United States during this period remain to be identified, but could come from the trend, ENSO, or other sources.
SubX real-time week 3 (average of forecast days 15–21) forecast skill for 2-m temperature over the period July 2017 to December 2018. Numbers in parentheses indicate the average ACC value over all land points in the domain. Statistical significance is not calculated or shown for the real-time forecasts due to small sample size.
Citation: Bulletin of the American Meteorological Society 100, 10; 10.1175/BAMS-D-18-0270.1
REAL-TIME PREDICTION OF HAZARDOUS AND EXTREME EVENTS.
Disaster preparedness and emergency management is one sector for which prediction of hazardous and extreme weather on S2S time scales is of particular interest (e.g., White et al. 2017). As an example of how SubX real-time forecasts can potentially provide information useful to this sector, Fig. 9 shows precipitation forecasts associated with Hurricane Michael for the SubX real-time forecasts. These forecasts were issued on 20 September 2018, prior to the formation of Michael, and were valid for the 2-week period of 6–19 October. All SubX models indicated the potential for precipitation anomalies in this period in excess of 50 mm over the 2-week period along a line stretching from southwest to northeast across Florida at 3 weeks lead time. Tropical Storm Michael formed on 7 October and made landfall as a hurricane along the Florida panhandle on 10 October. The storm tracked across the panhandle and through the southeastern United States, delivering heavy rainfall. Although the actual track is not accurately predicted at this lead time, the forecast for a potential tropical cyclone and associated enhanced precipitation during this period is useful information, potentially helping emergency managers to plan and aid organizations to stage supplies in anticipation of a disaster. A similar early picture was provided by SubX for Hurricane Harvey. SubX models forecasted anomalously high precipitation over the week spanning 24–31 August in Texas and Louisiana at 3–4-week lead times (not shown). Case studies such as these add to our understanding of the prediction and predictability of extreme events, especially in the context of a database designed for operational considerations.
SubX real-time forecasts for total precipitation anomalies (mm) for the 2-week period of 6–19 Oct issued on 17 Sep 2019. The observed track of Hurricane Michael from 7 to 12 Oct is shown in the bottom-right panel. Hurricane track data are from the initial tropical cyclone position (i.e., TC Vitals) obtained from the National Hurricane Center.
Citation: Bulletin of the American Meteorological Society 100, 10; 10.1175/BAMS-D-18-0270.1
CONCLUDING REMARKS.
SubX provides a comprehensive, publicly available research infrastructure in the service of developing better S2S forecasts. It consists of a database of seven global models that have produced a suite of 17 years of historical reforecasts and also have provided weekly real-time forecasts since July 2017. The inclusion of research and operational models and availability of both real-time and retrospective forecasts in SubX provides a unique contribution to community efforts in subseasonal predictability and prediction.
With the availability of subseasonal reforecast databases such as SubX and WWRP/WCRP S2S, it is now possible for the research community to extensively explore the full range of subseasonal predictability, and to develop methodologies for S2S postprocessing including forecast calibration and multimodel ensemble weighting (e.g., Vigaud et al. 2017a,b). Additionally, the contribution of individual models to an MME can be explored comprehensively. The inclusion of research models in SubX makes it possible for this research to directly feedback to model development. The availability of real-time subseasonal forecasts in SubX also enables the development of real-time forecast demonstration prototypes for applications in various socioeconomic sectors. We hope that the community will use the SubX database to provide input into pressing questions in S2S predictability and prediction, design tools relevant to decision making on the S2S time scale, and test and compare model developments for better S2S predictions.
Some important questions regarding S2S predictions remain unanswerable with the current datasets, including SubX. For example, in a second phase of SubX, with a more strict protocol aligning model initialization dates, it would be easier to combine models into an MME and we could better untangle questions about the contributions of individual models. Another improvement for a second phase would be to produce a longer reforecast period and a larger ensemble to evaluate the number of years and ensemble members needed to robustly quantify S2S skill and identify forecasts of opportunity.
ACKNOWLEDGMENTS
The SubX project is funded and was initiated by NOAA’s Climate Program Office’s Modeling, Analysis, Predictions, and Projections program (MAPP) in partnership with the NASA Modeling, Analysis, and Prediction program (MAP); the Office of Naval Research; and NOAA’s NWS Office of Science and Technology Integration. Relevant NOAA award numbers are NA16OAR4310149, NA16OAR4310151, NA16OAR4310150, NA16OAR4310143, NA16OAR4310141, NA16OAR4310146, NA16OAR4310145, NA16OAR4310148. S. Sun and B. W. Green are supported by funding from NOAA Award NA17OAR4320101. N. Barton and E.J. Metzger were funded by the “Navy ESPC in the North-American Multi Model Ensemble” project sponsored by the Office of Naval Research. Computer time for Navy-ESPC was provided by the Department of Defense High Performance Computing Modernization Program. This is NRL contribution NRL/JA/7320-18-4121. Global Modeling and Assimilation Office, NASA Goddard Space Flight Center, Greenbelt, Maryland, and Universities Space Research Association, Columbia, Maryland. CPC precipitation and temperature, NCEP/NCAR Reanalysis, and NOAA Interpolated OLR data provided by the NOAA/OAR/ESRL PSD, Boulder, Colorado, from their website at www.esrl.noaa.gov/psd/. The Center for Ocean–Land–Atmosphere studies (COLA) provided extensive disk space for performing the model evaluations and also hosts the SubX Website. COLA support for SubX is provided by grants from NSF (1338427) and NASA (NNX14AM19G) and a Cooperative Agreement with NOAA (NA14OAR4310160).
APPENDIX A: SUBX PROTOCOL.
The SubX protocol required that each modeling group adhere to a rigid scope of retrospective and real-time forecasts. The groups agreed to produce 17 years of reforecasts out to a minimum of 32 days for the years 1999–2015. Initialization was required at least weekly, and a minimum of three ensemble members were required, although more were encouraged. Since the land surface (e.g., soil moisture) is an important source of subseasonal predictability (Koster et al. 2010, 2011), all models were required to include a land surface model and initialize both the atmosphere and land. Additionally, coupled ocean–atmosphere models were also required to initialize the ocean. The SubX project has also performed more than 1 year of real-time forecasts. During this demonstration period, forecasts were required to be made available to CPC by 1800 local time every Wednesday. This requirement was relaxed to 0800 local time Thursday partway through the real-time demonstration period. All data were provided on a uniform 1° × 1° longitude–latitude grid as full fields to both CPC for their internal use and the International Research Institute for Climate and Society Data Library (IRIDL) for public dissemination (Kirtman et al. 2017).
APPENDIX B: CLIMATOLOGY AND BIAS CORRECTION.
A forecast is typically initialized with an analysis in which observations have been assimilated, thereby constraining the initial state to represent the observed state as closely as possible. As the forecast time increases, the model state on average moves from the observed climate toward a model-intrinsic climate, which is typically biased. Therefore, it is common practice in S2S predictions to estimate and remove the mean forecast bias using a set of reforecasts (e.g., Zhu et al. 2014). Additionally, the skill of forecasts at S2S time scales is typically evaluated in terms of anomalies or differences from the mean climate, thus requiring a climatology based on reforecasts. Both of these needs are met by determining the model climatology as a function of lead time and initialization date. For seasonal predictions using monthly data, it is typical to calculate the model climatology as a multiyear average for each forecast start month and lead or target time (Tippett et al. 2018). However, calculation of the climatology is not trivial due to differences in initialization day and frequency among models. For example, some forecast models are initialized on the same Julian days every year while others are initialized on a day-of-the-week schedule, meaning that the Julian initialization dates shift from year to year. In the first case, the 17-yr reforecast period yields 17 model runs on some calendar dates and none on the rest. In the second case, only 2–3 model runs are available for each day of the year from which to determine the climatology. An additional challenge for the SubX project was that a climatology was needed to produce bias-corrected forecast anomalies in real-time for CPC prior to the completion of the reforecasts at some centers. The need to compute model climatology adaptively will recur because some models will likely change during the forecast phase due to routine model improvements. Additionally, many operational models used by the CPC only provide reforecasts “on the fly” (e.g., ECMWF and Environment and Climate Change Canada ensembles generate reforecasts for a single day of the year with each real-time forecast initialization).
To compute the climatology, the first step is to calculate ensemble means for individual days of each forecast run. For most groups, ensembles are produced by averaging initialization dates from different hours of the same initialization day; these are averaged to yield ensemble means for the 24-h period spanning each forecast day. In the case of the NAVY-ESPC, which produces ensemble means over runs started on four consecutive days because ocean data assimilation is based on a 24-h data cycle, the ensemble mean consists of a single member for each day. Next, for each day of the year (1–366), a multiyear average of the ensemble means is calculated. Depending on how model runs are scheduled, this may not produce a climatology for each day of the year for some models. Finally, a triangular window is applied to the (fairly noisy as well as sparse in some cases) climatology, meaning that weight decreases linearly with distance from the center point. A smoothing window of 31 days (±15 days) is applied in a periodic fashion such that December smoothing includes January values and vice versa. This approach means that the forecast climatology can be computed from a partial reforecast database whereby only reforecasts with nearby initializations are required. Due to drift from the initial quasi-observed state to the model’s own internal mean state, the climatology for a given calendar day is expected to be different for different lead times. Therefore, the above procedure is performed for each lead time and each model individually. Removal of this climatology from the corresponding full fields produces anomalies and effectively performs a mean bias correction (Becker et al. 2014). Climatologies have been computed for many variables following this procedure and are available from the IRIDL.
Another common methodology is to fit harmonics to the data (Saha et al. 2014; Tippett et al. 2018). Both our smoothing methodology and the fitting of harmonics can be viewed as a special case of local linear regression [Tippett and DelSole 2013; see Hastie et al. (2009) for a review]. Mahlstein et al. (2015) previously proposed using local linear regression to compute climatologies of daily data. Local linear regression estimates a simple function of the predictors using data close to the desired climatology target in such a way as to yield a smooth function of the predictors. Figure A1 demonstrates that with synthetic data and a known climatology, the methodology used in SubX (green line) produces a climatology very close to the one obtained with a harmonic (red) using a similar number of years (16 years) and initial condition sampling (every 7 days) as SubX.
Results of estimating the climatological mean of a synthetic time series. The mean of each calendar day is shown as the gray curve (sample mean), a harmonic fit is shown as the red curve (harmonic), and a local linear regression fit based on p = 1 and quadratic function p = 2 are shown as the green and orange curves (using = 28)
Citation: Bulletin of the American Meteorological Society 100, 10; 10.1175/BAMS-D-18-0270.1
APPENDIX C: MULTIMODEL ENSEMBLE MEAN.
Since the SubX models are initialized on different days, producing an MME becomes a challenging problem (e.g., Vitart et al. 2017). In SubX, we choose to align the verification dates of each model to produce an MME so that skill could be assessed for the same verification period in observations. Additionally, this choice reproduces well the setup for weekly real-time forecasting. Following the same procedure used by CPC for producing real-time forecasts, Saturday is defined as the first day of a given week. All reforecasts for all models that are produced during the prior week (previous Friday–Thursday) are used to produce an MME forecast for weeks 1–4 individually, where week 1 is defined as the first Saturday–Friday interval. Friday initializations are not included in an attempt to mimic real-time forecast procedures. In real time, forecasts provided after 0800 local time Thursday cannot be processed in time to be used by the forecasters because forecasters must review forecast guidance on Thursday and issue the forecast on Friday. This procedure, which also involves forming averages of daily forecasts over the appropriate week, is repeated for weeks 2–4. Weeks 3 and 4 are then averaged together to produce week 3–4 forecasts. Using this procedure, a multimodel ensemble mean, equally weighted by model can be produced by averaging the ensemble means of each of the models for their week 3–4 forecasts. There are some potential drawbacks to this procedure. For example, some models will contribute older forecasts to the MME than others, depending on their initialization date. The extent to which decreased skill with longer lead time is balanced by increased ensemble size and model diversity in such an ensemble remains an open research question to be addressed in future research. Additionally, since the period over which forecasts are obtained is Saturday–Thursday (a 6-day period, used to mimic the 6-day period of real-time forecast initializations) and some of the models initialize once every 7 days, there are times when a model will not be included in the MME, depending on how the reforecast dates fall. For example, this occurs with the ECCC-GEM model in approximately 13% of the weekly forecasts. Finally, in rare cases, it is not possible to produce a week 3–4 forecast for the ECCC-GEM model since part of week 4 is not available due to the reforecast initialization day and 32-day reforecast length.
REFERENCES
Barnston, A. G., and H. V. den Dool, 1994: Long-lead seasonal forecasts—Where do we stand? Bull. Amer. Meteor. Soc., 75, 2097–2114, https://doi.org/10.1175/1520-0477(1994)075<2097:LLSFDW>2.0.CO;2.
Becker, E., H. v. den Dool, and Q. Zhang, 2014: Predictability and forecast skill in NMME. J. Climate, 27, 5891–5906, https://doi.org/10.1175/JCLI-D-13-00597.1.
Black, J., N. C. Johnson, S. Baxter, S. B. Feldstein, D. S. Harnos, and M. L. L’Heureux, 2017: The predictors and forecast skill of Northern Hemisphere teleconnection patterns for lead times of 3–4 weeks. Mon. Wea. Rev., 145, 2855–2877, https://doi.org/10.1175/MWR-D-16-0394.1.
Brunet, G., and Coauthors, 2010: Collaboration of the weather and climate communities to advance subseasonal-to-seasonal prediction. Bull. Amer. Meteor. Soc., 91, 1397–1406, https://doi.org/10.1175/2010BAMS3013.1.
Candille, G., 2009: The multiensemble approach: The NAEFS example. Mon. Wea. Rev., 137, 1655–1665, https://doi.org/10.1175/2008MWR2682.1.
Candille, G., S. Beauregard, and N. Gagnon, 2010: Bias correction and multiensemble in the NAEFS context or how to get a “free calibration” through a multiensemble approach. Mon. Wea. Rev., 138, 4268–4281, https://doi.org/10.1175/2010MWR3349.1.
Cassou, C., 2008: Intraseasonal interaction between the Madden–Julian Oscillation and the North Atlantic Oscillation. Nature, 455, 523–527, https://doi.org/10.1038/nature07286.
Chen, M., W. Shi, P. Xie, V. B. S. Silva, V. E. Kousky, R. W. Higgins, and J. E. Janowiak, 2008: Assessing objective techniques for gauge-based analyses of global daily precipitation. J. Geophys. Res., 113, D04110, https://doi.org/10.1029/2007JD009132.
DelSole, T., L. Trenary, M. K. Tippett, and K. Pegion, 2017: Predictability of week-3–4 average temperature and precipitation over the contiguous United States. J. Climate, 30, 3499–3512, https://doi.org/10.1175/JCLI-D-16-0567.1.
Dunstone, N., D. Smith, A. Scaife, and L. Hermanson, 2016: Skilful predictions of the winter North Atlantic Oscillation one year ahead. Nature, 9, 809–814, https://doi.org/10.1038/ngeo2824.
Fan, Y., and H. van den Dool, 2008: A global monthly land surface air temperature analysis for 1948–present. J. Geophys. Res., 113, D01103, https://doi.org/10.1029/2007JD008470.
Gagnon, N., H. Lin, S. Beauregard, M. Charron, B. Archambault, R. Lahlou, and C. Côté, 2013: Improvements to the Global Ensemble Prediction System (GEPS) from version 3.0.0 to version 3.1.0. Canadian Meteorological Centre Tech. Note, Environment Canada, 38 pp., http://collaboration.cmc.ec.gc.ca/cmc/cmoi/product_guide/docs/lib/technote_geps310_20131204_e.pdf.
Guan, B., D. E. Waliser, N. P. Molotch, E. J. Fetzer, and P. J. Neiman, 2012: Does the Madden–Julian oscillation influence wintertime atmospheric rivers and snowpack in the Sierra Nevada? Mon. Wea. Rev., 140, 325–342, https://doi.org/10.1175/MWR-D-11-00087.1.
Hagedorn, R., F. Doblas-Reyes, and T. N. Palmer, 2005: The rationale behind the success of multi-model ensembles in seasonal forecasting. Tellus, 57A, 219–233, https://doi.org/10.1111/j.1600-0870.2005.00103.x.
Hastie, T., R. Tibshirani, and J. H. Friedman, 2009: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed. Springer, 745 pp
Hoffman, R. N., and E. Kalnay, 1983: Lagged average forecasting, an alternative to Monte Carlo forecasting. Tellus, 35A, 100–118, https://doi.org/10.3402/tellusa.v35i2.11425.
Hogan, T., and Coauthors, 2014: The Navy Global Environmental Model. Oceanography, 27 (3), 116–125, https://doi.org/10.5670/oceanog.2014.73.
Hurrell, J. W., Y. Kushnir, G. Ottersen, and M. Visbeck, 2010: An overview of the North Atlantic Oscillation. The North Atlantic Oscillation: Climatic Significance and Environmental Impact, Geophys. Monogr., Vol. 134, Amer. Geophys. Union, 1–35.
Infanti, J. M., and B. P. Kirtman, 2016: Prediction and predictability of land and atmosphere initialized CCSM4 climate forecasts over North America. J. Geophys. Res. Atmos., 121, 12 690–12 701, https://doi.org/10.1002/2016JD024932.
Johansson Å., 2007: Prediction skill of the NAO and PNA from daily to seasonal time scales. J. Climate, 20, 1957–1975, https://doi.org/10.1175/JCLI4072.1.
Jones, C., D. E. Waliser, J. K. E. Schemm, and W. K. M. Lau, 2000: Prediction skill of the Madden and Julian Oscillation in dynamical extended range forecasts. Climate Dyn ., 16, 273–289, https://doi.org/10.1007/s003820050327.
Kalnay, E., and A. Dalcher, 1987: Forecasting forecast skill. Mon. Wea. Rev., 115, 349–356, https://doi.org/10.1175/1520-0493(1987)115<0349:FFS>2.0.CO;2.
Kalnay, E., and Coauthors, 1996: The NCEP/NCAR 40-Year Reanalysis Project. Bull. Amer. Meteor. Soc., 77, 437–472, https://doi.org/10.1175/1520-0477(1996)077<0437:TNYRP>2.0.CO;2.
Kim, H., P. J. Webster, and J. A. Curry, 2012: Seasonal prediction skill of ECMWF System 4 and NCEP CFSv2 retrospective forecast for the Northern Hemisphere Winter. Climate Dyn ., 39, 2957–2973, https://doi.org/10.1007/s00382-012-1364-6.
Kim, H., F. Vitart, and D. E. Waliser, 2018: Prediction of the Madden–Julian oscillation: A review. J. Climate, 31, 9425–9443, https://doi.org/10.1175/JCLI-D-18-0210.1.
Kirtman, B. P., and Coauthors, 2014: The North American Multimodel Ensemble: Phase-1 seasonal-to-interannual prediction; Phase-2 toward developing intraseasonal prediction. Bull. Amer. Meteor. Soc., 95, 585–601, https://doi.org/10.1175/BAMS-D-12-00050.1.
Kirtman, B. P., and Coauthors, 2017: The Subseasonal Experiment (SubX). IRI Data Library, https://doi.org/10.7916/D8PG249H.
Koster, R. D., M. J. Suarez, A. Ducharne, M. Stieglitz, and P. Kumar, 2000: A catchment-based approach to modeling land surface processes in a general circulation model: 1. Model structure. J. Geophys. Res., 105, 24 809–24 822, https://doi.org/10.1029/2000JD900327.
Koster, R. D., and Coauthors, 2010: Contribution of land surface initialization to subseasonal forecast skill: First results from a multi-model experiment. Geophys. Res. Lett., 37, L02402, https://doi.org/10.1029/2009GL041677.
Koster, R. D., and Coauthors, 2011: The second phase of the Global Land–Atmosphere Coupling Experiment: Soil moisture contributions to subseasonal forecast skill. J. Hydrometeor., 12, 805–822, https://doi.org/10.1175/2011JHM1365.1.
Krishnamurti, T. N., C. M. Kishtawal, T. E. LaRow, D. R. Bachiochi, Z. Zhang, C. E. Williford, S. Gadgil, and S. Surendran, 1999: Improved weather and seasonal climate forecasts from multimodel superensemble. Science, 285, 1548–1550, https://doi.org/10.1126/science.285.5433.1548.
Krishnamurti, T. N., C. M. Kishtawal, Z. Zhang, T. LaRow, D. Bachiochi, E. Williford, S. Gadgil, and S. Surendran, 2000: Multimodel ensemble forecasts for weather and seasonal climate. J. Climate, 13, 4196–4216, https://doi.org/10.1175/1520-0442(2000)013<4196:MEFFWA>2.0.CO;2.
Li, S., A. W. Roberston, S. Li, and A. W. Robertson, 2015: Evaluation of submonthly precipitation forecast skill from global ensemble prediction systems. Mon. Wea. Rev., 143, 2871–2889, https://doi.org/10.1175/MWR-D-14-00277.1.
Liebmann, B., and C. Smith, 1996: Description of a complete (interpolated) outgoing longwave radiation dataset. Bull. Amer. Meteor. Soc., 77, 1275–1277, https://doi.org/10.1175/1520-0477-77.6.1274.
Lin, H., G. Brunet, and J. Derome, 2009: An observed connection between the North Atlantic oscillation and the Madden–Julian oscillation. J. Climate, 22, 364–380, https://doi.org/10.1175/2008JCLI2515.1.
Lin, H., N. Gagnon, S. Beauregard, R. Muncaster, M. Markovic, B. Denis, and M. Charron, 2016: GEPS-based monthly prediction at the Canadian Meteorological Centre. Mon. Wea. Rev., 144, 4867–4883, https://doi.org/10.1175/MWR-D-16-0138.1.
Lorenz, E. N., 1965: A study of the predictability of a 28-variable atmospheric model. Tellus, 17, 321–333, https://doi.org/10.3402/tellusa.v17i3.9076.
Lorenz, E. N., 1969: The predictability of a flow which possesses many scales of motion. Tellus, 21, 289–307, https://doi.org/10.3402/tellusa.v21i3.10086.
Madden, R. A., and P. R. Julian, 1971: Detection of a 40–50 day oscillation in the zonal wind in the tropical Pacific. J. Atmos. Sci., 28, 702–708, https://doi.org/10.1175/1520-0469(1971)028<0702:DOADOI>2.0.CO;2.
Madden, R. A., and P. R. Julian, 1972: Description of global-scale circulation cells in the tropics with a 40–50 day period. J. Atmos. Sci., 29, 1109–1123, https://doi.org/10.1175/1520-0469(1972)029<1109:DOGSCC>2.0.CO;2.
Mahlstein, I., C. Spirig, and M. Liniger, 2015: Estimating daily climatologies for climate indices derived from climate model data and observations. J. Geophys. Res. Atmos., 120, 2808–2828, https://doi.org/10.1002/2014JD022327.
Mariotti, A., P. M. Ruti, and M. Rixen, 2018: Progress in subseasonal to seasonal prediction through a joint weather and climate community effort. npj Climate Atmos. Sci., 1, 4, https://doi.org/10.1038/s41612-018-0014-z.
Metzger, E. J., and Coauthors, 2014: US Navy operational global ocean and arctic ice prediction systems. Oceanography, 27, 32–43, https://doi.org/10.5670/oceanog.2014.66.
Mishra, N., C. Prodhomme, and V. Guemas, 2019: Multi-model skill assessment of seasonal temperature and precipitation forecasts over Europe. Climate Dyn ., 52, 4207–4225, https://doi.org/10.1007/s00382-018-4404-z.
Molod, A., L. Takacs, M. J. Suarez, J. Bacmeister, I.-S. Song, and A. Eichmann, 2012: The GEOS-5 atmospheric general circulation model: Mean climate and development from MERRA to Fortuna. Tech. Memo. NASA/TM-2012-104606, Vol. 28, 115 pp., https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20120011790.pdf.
Mundhenk, B. D., E. A. Barnes, E. D. Maloney, and C. F. Baggett, 2018: Skillful empirical subseasonal prediction of landfalling atmospheric river activity using the Madden-Julian oscillation and quasi-biennial oscillation. npj Climate Atmos. Sci., 1, 20177, https://doi.org/10.1038/s41612-017-0008-2.
Murphy, A. H., and E. Epstein, 1989: Skill scores and correlation coefficients in model verification. Mon. Wea. Rev., 117, 572–582, https://doi.org/10.1175/1520-0493(1989)117<0572:SSACCI>2.0.CO;2.
National Academies of Sciences, Engineering and Medicine, 2017: Next Generation Earth System Prediction: Strategies for Subseasonal to Seasonal Forecasts. National Academies Press, 350 pp., https://doi.org/10.17226/21873.
National Research Council, 2010: Assessment of Intraseasonal to Interannual Climate Prediction and Predictability. National Academies Press, 192 pp., https://doi.org/10.17226/12878.
Pegion, K., and P. D. Sardeshmukh, 2011: Prospects for improving subseasonal predictions. Mon. Wea. Rev., 139, 3648–3666, https://doi.org/10.1175/MWR-D-11-00004.1.
Rashid, H. A., H. H. Hendon, M. C. Wheeler, and O. Alves, 2010: Prediction of the Madden–Julian oscillation with the POAMA dynamical prediction system. Climate Dyn ., 36, 649–661, https://doi.org/10.1007/s00382-010-0754-x.
Reichle, R., and Q. Liu, 2014: Observation-corrected precipitation estimates in GEOS-5. Tech. Memo. NASA/TM-2014-104606, Vol. 35, 18 pp., https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20150000725.pdf.
Rienecker, M. M., and Coauthors, 2008: The GEOS-5 Data Assimilation System—Documentation of versions 5.0.1, 5.1.0, and 5.2.0. Tech. Memo. NASA/TM-2008-104606, Vol. 27, 97 pp., https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20120011955.pdf.
Robertson, A. W., A. Kumar, M. Peña, and F. Vitart, 2015: Improving and promoting subseasonal to seasonal prediction. Bull. Amer. Meteor. Soc., 96, ES49–ES53, https://doi.org/10.1175/BAMS-D-14-00139.1.
Saha, S., and Coauthors, 2014: The NCEP Climate Forecast System version 2. J. Climate, 27, 2185–2208, https://doi.org/10.1175/JCLI-D-12-00823.1.
Scaife, A. A., A. Arribas, and E. Blockley, 2014: Skillful long-range prediction of European and North American winters. Geophys. Res. Lett., 41, 2514–2519, https://doi.org/10.1002/2014GL059637.
Stan, C., D. M. Straus, J. S. Frederiksen, H. Lin, E. D. Maloney, and C. Schumacher, 2017: Review of tropical-extratropical teleconnections on intraseasonal time scales. Rev. Geophys., 55, 902–937, https://doi.org/10.1002/2016RG000538.
Sun, S., R. Bleck, S. G. Benjamin, B. W. Green, and G. A. Grell, 2018a: Subseasonal forecasting with an icosahedral, vertically quasi-Lagrangian coupled model. Part I: Model overview and evaluation of systematic errors. Mon. Wea. Rev., 146, 1601–1617, https://doi.org/10.1175/MWR-D-18-0006.1.
Sun, S., B. W. Green, R. Bleck, and S. G. Benjamin, 2018b: Subseasonal forecasting with an icosahedral, vertically quasi-Lagrangian coupled model. Part II: Probabilistic and deterministic forecast skill. Mon. Wea. Rev., 146, 1619–1639, https://doi.org/10.1175/MWR-D-18-0007.1.
Tippett, M. K., and T. DelSole, 2013: Constructed analogs and linear regression. Mon. Wea. Rev., 141, 2519–2525, https://doi.org/10.1175/MWR-D-12-00223.1.
Tippett, M. K., L. Trenary, T. DelSole, K. Pegion, and M. L. L’Heureux, 2018: Sources of bias in the monthly CFSv2 forecast climatology. J. Appl. Meteor. Climatol., 57, 1111–1122, https://doi.org/10.1175/JAMC-D-17-0299.1.
Trenary, L., T. DelSole, M. Tippett, and K. Pegion, 2018: Monthly ENSO forecast skill and lagged ensemble size. J. Adv. Model. Earth Syst., 10, 1074–1086, https://doi.org/10.1002/2017MS001204.
Vigaud, N., A. W. Robertson, and M. K. Tippett, 2017a: Multimodel ensembling of subseasonal precipitation forecasts over North America. Mon. Wea. Rev., 145, 3913–3928, https://doi.org/10.1175/MWR-D-17-0092.1.
Vigaud, N., A. W. Robertson, M. K. Tippett, and N. Acharya, 2017b: Subseasonal predictability of boreal summer monsoon rainfall from ensemble forecasts. Front. Environ. Sci., 5, 2197–19, https://doi.org/10.3389/fenvs.2017.00067.
Vitart, F., 2017: Madden-Julian oscillation prediction and teleconnections in the S2S database. Quart. J. Roy. Meteor. Soc., 143, 2210–2220, https://doi.org/10.1002/qj.3079.
Vitart, F., and Coauthors, 2017: The Subseasonal to Seasonal (S2S) Prediction project database. Bull. Amer. Meteor. Soc., 98, 163–173, https://doi.org/10.1175/BAMS-D-16-0017.1.
Weigel, A. P., M. A. Liniger, and C. Appenzeller, 2008: Can multi-model combination really enhance the prediction skill of probabilistic ensemble forecasts? Quart. J. Roy. Meteor. Soc., 134, 241–260, https://doi.org/10.1002/qj.210.
Wheeler, M. C., and H. Hendon, 2004: An all-season real-time multivariate MJO index: Development of an index for monitoring and prediction. Mon. Wea. Rev., 132, 1917–1932, https://doi.org/10.1175/1520-0493(2004)132<1917:AARMMI>2.0.CO;2.
White, C. J., and Coauthors, 2017: Potential applications of subseasonal-to-seasonal (s2s) predictions. Meteor. Appl., 24, 315–325, https://doi.org/10.1002/met.1654.
Wilks, D. S., 2006: Statistical Methods in the Atmospheric Sciences. 2nd ed. International Geophysics Series, Vol. 100, Academic Press, 648 pp.
Xie, P., M. Chen, S. Yang, A. Yatagai, T. Hayasaka, Y. Fukushima, and C. Liu, 2007: A gauge-based analysis of daily precipitation over East Asia. J. Hydrometeor., 8, 607–626, https://doi.org/10.1175/JHM583.1.
Zhang, C., 2013: Madden–Julian oscillation: Bridging weather and climate. Bull. Amer. Meteor. Soc., 94, 1849–1870, https://doi.org/10.1175/BAMS-D-12-00026.1.
Zhou, X., Y. Zhu, D. Hou, and D. Kleist, 2016: A comparison of perturbations from an ensemble transform and an ensemble Kalman filter for the NCEP Global Ensemble Forecast System. Wea. Forecasting, 31, 2057–2074, https://doi.org/10.1175/WAF-D-16-0109.1.
Zhou, X., Y. Zhu, D. Hou, Y. Luo, J. Peng, and R. Wobus, 2017: Performance of the new NCEP Global Ensemble Forecast System in a parallel experiment. Wea. Forecasting, 32, 1989–2004, https://doi.org/10.1175/WAF-D-17-0023.1.
Zhu, H., M. C. Wheeler, and A. Sobel, 2014: Seamless precipitation prediction skill in the tropics and extratropics from a global model. Mon. Wea. Rev., 142, 1556–1569, https://doi.org/10.1175/MWR-D-13-00222.1.
Zhu, Y., and Coauthors, 2018: Towards the improvement of sub-seasonal prediction in the NCEP Global Ensemble Forecast System (GEFS). J. Geophys. Res. Atmos., 123, 6732–6745, https://doi.org/10.1029/2018JD028506.
