1. Introduction
In early May 2013, widespread heavy rains and subsequent flooding in Saudi Arabia resulted in 25 deaths (http://reliefweb.int/report/saudi-arabia/25-saudis-killed-heavy-rain-floods). The overall climate of Saudi Arabia is arid–semiarid, and rainfall in most regions is scant and irregular. However, episodes of heavy rainfall during the wet season (November–April) are not uncommon, and when combined with the topography and hydrology of the region, intense rainfall can result in substantial flood risk. In fact, floods constitute 8 of the top 10 natural disasters in Saudi Arabia when ranked by the total number of affected people during the period 1900–2013 [the Emergency Events Database (EM-DAT), which is hosted by the Office of U.S. Foreign Disaster Assistance/Centre for Research on the Epidemiology of Disasters (OFDA/CRED), online at http://www.emdat.be; Université catholique de Louvain, Brussels, Belgium]. The substantial impact of such rainfall events on society raises the following questions regarding their predictability. To what extent was the May 2013 event predictable? More generally, what is the skill of rainfall forecasts for the region as a function of lead time and season? Can reliable probabilistic predictions of high-impact rainfall for the region be constructed?
Extended-range (beyond 5 days) forecasts are considered challenging because they stretch toward the limit of our deterministic predictability, as provided by initial conditions (Palmer 1993), but provide limited opportunity for boundary forcing such as sea surface temperature to play a major role (Shukla 1981, 1998). The Madden–Julian oscillation (MJO) is a mode of intraseasonal variability whose predictability and climate impacts provide a scientific basis for extended-range prediction (Waliser et al. 2003). Although a tropical phenomenon, the MJO influences precipitation around the globe (Jones et al. 2004). In particular, as reviewed in Barlow (2012), the MJO and other tropical forcings play an important role in the precipitation of the Arabian Peninsula (AP) region, and OLR-based estimates indicate that upward of 80% of the November–April precipitation across the region might be related to the MJO. The strength and location of the MJO and its associated tropical convection can be characterized using the phases (1–8) of the Real-Time Multivariate MJO (RMM) indices (Wheeler and Hendon 2004). The impact of the MJO on the AP region varies by phase. Using November–April daily station data for Riyadh, Saudi Arabia, Barlow (2012) found a statistically significant increase in the frequency of wet conditions during phase 1 and a statistically significant increase in dry conditions during phases 3–6, which correspond to positive values of RMM, series 1 (RMM1), and enhanced equatorial convection over the Indian Ocean (IO) and the Maritime Continent from about 70°E to the date line.
The leading pattern of November–April monthly precipitation anomalies for the IO precipitation variability on intraseasonal time scales includes the equatorial region from ~80° to 110°E (Hoell et al. 2012). Observational analysis shows that enhanced convection in this region on intraseasonal time scales is associated with suppressed convection over much of the AP region and an upper-level anticyclone centered across South Asia. Analysis of the thermodynamic balance shows cold temperature advection in the AP region to be mostly balanced by downward motion consistent with dry conditions. This pattern of IO precipitation variability has hemispheric impacts, and Hoell et al. (2013) used barotropic Rossby wave ray tracing to show that IO convection can force stationary Rossby waves that propagate through the Northern Hemisphere and reach the Middle East from the west in 15 days. In addition, the MJO also influences low-level circulation and local moisture transport in the AP region (Nazemosadat and Ghaedamini 2010). Nazemosadat and Ghaedamini (2010) characterized the MJO using an index based on principle components of filtered low-level zonal winds (Maloney and Hartmann 1998), which corresponds to IO convection in the equatorial region from ~80° to 140°E, slightly to the east of the region identified by Hoell et al. (2012). Suppressed IO convection in this region is associated with positive anomalies of precipitable water, consistent with the transport of moisture to the region by enhanced low-level southerly winds.
This negative association between AP rainfall and IO convection tends to be weaker when analyzed on interannual time scales, in part because of the additional effect of El Niño––Southern Oscillation (ENSO). Athar (2015) examined correlations of seasonal (3-month averages) Saudi station rainfall data with the Indian Ocean dipole (IOD) and ENSO over the period 1978–2010 and found generally positive correlations, consistent with the observed southward shift of the upper-level subtropical jet stream and enhanced moisture transport to the region during positive IOD and ENSO events (Chakraborty et al. 2006). The positive phase of the IOD is associated with negative SST anomalies in the eastern tropical IO and positive SST anomalies in the west.
The potentially competing roles of IO and ENSO SST on the climate of the AP region was highlighted in the analysis of Hoell et al. (2014) of climate responses to different La Niña “flavors” (Johnson 2013). When La Niña conditions are accompanied by negative IO SST anomalies, there is little robust precipitation response in the AP region in the observations or model experiments, reflecting the opposing IO and ENSO factors. However, when La Niña is accompanied by small or positive IO SST anomalies, the AP region shows a strong decrease in rainfall in the observations and model experiments. Interestingly, these two La Niña types correspond roughly to the pre- (cool IO SST anomalies) and post-1980 (warm or no IO SST anomalies) periods. Kang et al. (2015) correlated the November–April average of Arabian Peninsula rainfall with concurrent values of the Niño-3.4 index and found the correlation to be insignificant (0.1) over the 60-yr period of 1950–2010, but positive and significant (0.51) over the recent 30-yr period of 1980–2010. This changing relation between AP rainfall and the Niño-3.4 index can be attributed to changes in the extent to which IO SST anomalies accompany those of the Niño-3.4 index. Modeling experiments show that in isolation AP precipitation is negatively correlated with IO SST anomalies and positively correlated with ENSO (Kang et al. 2015).
In the synoptic context, de Vries et al. (2013) examined the dynamical characteristics associated with the so-called active Red Sea trough (ARST) and extreme rainfall events in the Middle East, including flooding in Jeddah, Saudi Arabia. The Red Sea trough (RST), a surface trough extending from East Africa through the Red Sea, is an increasingly common climatological feature of the region present in autumn, and to a lesser extent in winter and spring, that is related to the local topography and thermal forcing (Krichak et al. 1997a,b; Alpert et al. 2004). The RST is usually associated with dry warm weather in the region, but when the RST is accompanied by an upper-tropospheric trough, instability may result and lead to severe precipitation. Such ARST events are characterized by tropical–extratropical interactions, and de Vries et al. (2013) found that most extreme rainfall events were associated with an amplification of a stationary wave that interacts with the low-level circulation. This feature of the ARST is highly reminiscent of the tropically forced stationary Rossby waves described by Hoell et al. (2013), and suggests that the midlatitude Rossby wave amplification and forcing described by de Vries et al. (2013) may potentially have a tropical origin. Other aspects of ARST events are moisture transport, primarily from the Arabian and Red Seas, and pronounced midtropospheric ascent, both features present in the intraseasonal time-scale analysis as well.
The predictability of the MJO combined with its substantial precipitation impacts in the region provide a conceptual basis for skillful extended-range forecasts, and Barlow et al. (2005) suggested that there was potential for 3-week forecasts. Realization of this potential for skillful extended-range forecasts in the AP region by dynamical prediction models requires that the model both predict the MJO skillfully and represent the regional MJO response (teleconnection) with adequate fidelity. The complexity of the observed synoptic tropical–extratropical interactions discussed above means that despite the apparent importance of tropical forcing and the predictability it provides, forecast skill for the region is not assured. Assessment of MJO forecast skill requires a substantial archive of forecasts, and the Climate Forecast System (CFS), version 2, provides such a reforecast archive (Saha et al. 2014), which has been used to document the considerable improvement of MJO forecasts in CFS, version 2, compared to those of the previous version of CFS; MJO forecast skill in CFS, version 2, now extends out to 2–3 weeks (Zhang and van den Dool 2012; Wang et al. 2014). Here, we use the CFS, version 2, reforecast archive to assess regional precipitation forecast skill.
The paper is organized as follows. In section 2 we describe the observational and forecast data, as well as the verification metrics. The observed and forecast climatologies of the region are described in section 3. Section 4 describes forecasts of the 2013 flooding event mentioned in the introduction. A 12-yr reforecast dataset is analyzed in section 5. Finally, a summary and conclusions are given in section 6.
2. Data and methods
We consider forecasts and observation-based estimates of rainfall over the Arabian Peninsula region. We restrict most of our analysis, in particular forecast skill verification, to the areal average of rainfall over Saudi Arabia, and we refer to this quantity as the SA rainfall index. The SA rainfall index, while not capturing rainfall variability at smaller scales, does provide a concise summary of rainfall variability affecting large regions of the country. Intense localized rainfall events are not well resolved by the index because of the areal averaging. For instance, the rainfall amounts associated with the 25 November 2009 flooding event in Jeddah were locally extreme (Almazroui 2011b), but the corresponding values of the SA rainfall index, while above normal, were not so extreme. We use the SA rainfall index as a tool to identify widespread rainfall situations of interest whose spatial features can then be further investigated.
a. Observations
The observation-based rainfall estimate used here is the 3B42 product from the TRMM Multisatellite Precipitation Analysis, version 7 (hereafter referred to as simply TRMM; Huffman et al. 2007, 2010). This product combines precipitation estimates from various satellite systems (both infrared and radar) as well as surface gauge analysis on a 0.25° × 0.25° spatial grid with 3-hourly resolution. Here, we use the daily resolution of the research version of the TRMM product for the period from 1 January 1999 to 14 February 2011, a total of 4428 days. TRMM, version 6, was previously compared to rainfall data from 30 Saudi stations over the period 1998–2009 (Almazroui 2011a) and was found to match well with daily station data on a countrywide basis. The TRMM data matched less well with individual station data on a daily basis. Monthly averaging improved the relation between station and TRMM data. Our focus here is on widespread rainfall events, and we expect that the spatial averaging of the SA rainfall index will serve to reduce the impact of small-scale discrepancies between the TRMM data and actual rainfall amounts.
b. Forecasts
We analyze Climate Forecast System (CFS), version 2, reforecasts for the period 1999–2010 and real-time forecasts around the April–May 2013 flooding event (Saha et al. 2014). The CFS atmospheric model has a spectral resolution of T126 (1° × 1°) in the horizontal and 64 sigma-pressure hybrid layers in the vertical. CFS reforecasts are initialized four times daily at 6-hourly intervals using the Climate Forecast System Reanalysis (CFSR; Saha et al. 2010), and one ensemble member per initialization time is integrated for at least 45 days, with output available at 6-hourly intervals; we use only the reforecast output out to 45 days. There are 17 532 reforecasts over the 12-yr period. The CFS reforecast dataset is available from the NOAA/National Climate Date Center (NCDC); forecasts are missing from 25 days (100 starts). We group the four reforecast starts per day (0000–1800 UTC) together to form a four-member ensemble. This reforecast ensemble is “lagged” in the sense that each reforecast starts at a different time, and ensemble reforecasts for a given target contain forecasts of varying (by up to 18 h) lead times (Hoffman and Kalnay 1983). Real-time forecasts for the period around April–May 2013 are likewise initialized four times daily but with four ensemble members per initialization time as compared to one in the reforecast. For the real-time forecasts of the 2013 event, we average the four ensemble members per start time and retain the 6-hourly start and forecast resolution.
Deterministic reforecasts are constructed from the lagged ensemble for two SA rainfall index quantities: the rainfall amount in a given forecast window and the number of days in a given forecast window that the SA rainfall index exceeds 0.25 mm day−1. This threshold corresponds to about the 75th percentile of the SA rainfall index during the wet season, and its relatively low value reflects the rarity of widespread rainfall. The forecast rainfall amount is computed by taking the ensemble average, and the rainfall exceedance frequency is computed by counting the number of days that an ensemble member has an SA rainfall index greater than 0.1 mm day−1, which is roughly the 75th percentile of the forecast values during the wet season; the precise value of the percentile depends somewhat on lead time as will be discussed in section 3. The results are not particularly sensitive to either the choice of observations or forecast thresholds. We consider forecast windows of differing lengths and leads. For instance, we consider forecasts of the 5-day average of the SA rainfall index and forecasts of the number of days in that 5-day window for which the SA rainfall index exceeds the 0.25 mm day−1 threshold.
Probabilistic forecasts of heavy rainfall occurrence are also considered. Heavy rainfall is defined as occurring when the SA rainfall index exceeds 1.0 mm day−1 on any day during the forecast target window. This threshold corresponds to about the 90th percentile of daily rainfall during the wet season. Two methods are used to construct probability forecasts. In the first method, the forecast probability is simply the fraction of the four ensemble members exceeding 0.50 mm day−1 (the lower threshold value is used to account for model bias), and in the second, the forecast probability comes from a logistic regression using the fourth root of the total ensemble mean rainfall during the forecast target window as a predictor. The two logistic regression coefficients vary with season (wet or dry) and lead, and there are roughly 2100 samples for each season and lead.
Our convention with regard to lead time and forecast target window is illustrated by the following examples. The shortest lead (first lead) forecast of 2 January 1999 daily rainfall is computed from the lagged ensemble with forecasts initialized at 0000, 0600, 1200, and 1800 UTC 1 January 1999, and we define such a forecast to have a lead time of 6 h, the time from the last initialization to the beginning of the forecast target window. Likewise, the shortest lead (first lead) forecast of the 2–6 January 1999 average is computed from the same 1 January 1999 forecasts and is also defined to have a lead time of 6 h.
c. Verification measures
Forecasts are verified separately during two 6-month seasons: wet and dry (May–October). We use a rank (Spearman) correlation for the verification of the deterministic forecasts. Rank correlation, unlike the usual Pearson correlation, is invariant to monotonic transformations and not affected by outliers. We verify probabilistic forecasts using the Brier skill score (Murphy 1973). The Brier skill score is the squared difference between forecast probabilities and observed occurrence, and measures both forecast reliability and resolution. The Brier skill score compares the Brier score of a forecast to that of a reference forecast, here the climatological probability of the events during each of the 6-month seasons. The Brier skill score is oriented so that values greater than zero indicate skill greater than that of a forecast whose prediction probability is equal to the climatological probability. Reliability diagrams and relative operating characteristic curves (ROCs) provide additional information about the quality of the probabilistic forecasts.
3. Climatology
The climate of Saudi Arabia is mostly arid and semiarid, though regions in the southwest do have a steppe climate and receive considerable rainfall year-round (Abdullah and Al-Mazroui 1998). Overall, the annual cycle of the country can be divided into a 6-month wet and a 6-month dry season (Almazroui 2006, 2011a). Figures 1a and 1b show TRMM estimates of the November–April and May–October average rainfall, respectively, over the 12-yr period 1999–2010. There is some precipitation throughout the country on average during the wet season, with the largest values found over a broad region in the northeast of the country, and along a narrow zone in the southwest following the central (Asir) and the southern (Sarat al Yemen) parts of the Sarawat mountain range as they parallel the Red Sea coast, about 100 km inland, to the southernmost tip of Yemen (Abdullah and Al-Mazroui 1998). Figure 2 shows the topography of the region, highlighted by the rapid rise in elevation from the Red Sea coast eastward toward the interior of the country. Annual rainfall amounts in the southwestern region vary strongly with elevation, and there is more rainfall on the western side of the mountain ranges. Annual amounts on the order of 400 mm are measured at some stations (Abdullah and Al-Mazroui 1998). Much of the winter rainfall is the result of tropical–extratropical interactions. Moisture at low levels is supplied via the Sudan low–Red Sea trough and interacts with upper-level Mediterranean cyclones and Rossby waves associated with the subtropical jet (Abdullah and Al-Mazroui 1998; de Vries et al. 2013).
TRMM rainfall during the (a) wet and (b) dry seasons during 1999–2010. (c),(d) As in (a),(b), but for CFS first-lead rainfall.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
TRMM rainfall during the (a) wet and (b) dry seasons during 1999–2010. (c),(d) As in (a),(b), but for CFS first-lead rainfall.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
TRMM rainfall during the (a) wet and (b) dry seasons during 1999–2010. (c),(d) As in (a),(b), but for CFS first-lead rainfall.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
The topography of the region.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
The topography of the region.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
The topography of the region.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
During the dry season, mid- and upper-level subsidence over the region serves to suppress rainfall, and rainfall is mostly isolated to the region in the southwest where the combination of steep elevation, southwesterly winds, and strong orographic effects produces frequent rainfall on the western slopes. Figure 1b indicates that the dry season rainfall in this region actually exceeds that of the wet season. Little rainfall occurs in the remainder of the country during the dry season. Two exceptionally dry zones are the Rub al Khali, known in English as the Empty Quarter, in the eastern and the northwestern corner of the country including the Tabuk region.
The first-lead CFS forecast climatology for the region is represented by the average of 6-h forecasts made four times daily over the 12-yr reforecast period (1999–2010). Figures 1c and 1d show the first-lead CFS precipitation climatology for November–April and May–October, respectively. The first-lead CFS climatology captures the contrast between the wet and dry seasons well, and the patterns compare relatively well with those of the TRMM estimates. The first-lead CFS climatology underestimates rainfall in the north, especially during the wet season, and does not resolve orographic rainfall in the southwest. Some aspects of this southwestern orographic rainfall are present in the first-lead CFS forecasts during the wet season, but do not appear at all in the dry season. The properties of the CFS first-lead climatologies are constrained by the CFSR data used to initialize the CFS forecasts, and next we present lead time–dependent features of the CFS forecast climatology.
The annual cycle of the SA rainfall index computed from 10 annual harmonics of the TRMM estimates is shown in Fig. 3. As estimated during this particular period, the SA rainfall index has its maximum values in April, January, and the beginning of December. The annual cycle of the CFS SA rainfall index is also shown for forecast leads varying from 1 to 15 days. Overall, the CFS forecast SA rainfall index shows the same seasonality, but with generally lower values. However, the annual cycle of the CFS SA rainfall index forecasts shows considerable dependence on lead time. In April, the CFS SA rainfall index becomes wetter as lead time increases. In contrast, the CFS SA rainfall index becomes drier in January as lead time increases. During the July–August period, the shortest lead CFS SA rainfall index forecasts are too dry while the longer-lead examples are too wet. The CFS forecast of the SA rainfall index during the November–December period shows relatively little systematic dependence on lead time.
Annual cycle (1999–2010) computed from 10 harmonics of the TRMM estimate (black) of the SA rainfall index and the CFS forecast with leads of 1–5 (brown), 6–10 (blue), and 11–15 days (green).
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Annual cycle (1999–2010) computed from 10 harmonics of the TRMM estimate (black) of the SA rainfall index and the CFS forecast with leads of 1–5 (brown), 6–10 (blue), and 11–15 days (green).
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Annual cycle (1999–2010) computed from 10 harmonics of the TRMM estimate (black) of the SA rainfall index and the CFS forecast with leads of 1–5 (brown), 6–10 (blue), and 11–15 days (green).
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
The spatial expression of the dependence of the CFS forecast climatology on lead time is shown in Fig. 4 for January and April. The January climatology of the initial CFS lead matches the TRMM January climatology well (Fig. 4b). However, for a 15-day lead, the CFS January climatology is too dry, especially in the north and west of the country (Fig. 4c). In April, the initial lead CFS climatology is too dry, especially along the southwestern coastal region and in the center of the country (Fig. 4e). At a 15-day lead, the CFS climatology becomes wetter in the central portion of the country (Fig. 4f), in contrast to the drying seen with the increasing lead in January. Figure 5 shows variations in July–August forecast climatology with lead. The model climatology for the first lead is too dry, especially in the southwest of the AP. However, the climatology based on 15-day forecasts is much wetter, perhaps overly so, in this region.
January rainfall from (a) TRMM, (b) CFS lead 1, and (c) CFS lead 15. (d)–(f) As in (a)–(c), but for April rainfall.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
January rainfall from (a) TRMM, (b) CFS lead 1, and (c) CFS lead 15. (d)–(f) As in (a)–(c), but for April rainfall.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
January rainfall from (a) TRMM, (b) CFS lead 1, and (c) CFS lead 15. (d)–(f) As in (a)–(c), but for April rainfall.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Total July–August rainfall from (a) TRMM, (b) CFS lead 1, and (c) CFS lead 15.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Total July–August rainfall from (a) TRMM, (b) CFS lead 1, and (c) CFS lead 15.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Total July–August rainfall from (a) TRMM, (b) CFS lead 1, and (c) CFS lead 15.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
There are substantial differences between the observed and forecast daily climatological distributions of the SA rainfall index. Figure 6 shows the cumulative distribution functions of the observed and forecast daily SA rainfall index for the wet and dry seasons separately. One striking difference between the observed and forecast daily SA rainfall index climatological distributions is the frequency of zero values. The observed TRMM SA rainfall index has no zero values while for the frequency of CFS forecasts with zero SA rainfall index the value is around 35% during the wet season and between 40% and 70% during the dry season depending on lead. This discrepancy is primarily due to the inability of the CFS to resolve the frequency of sparse orographic rainfall in the southwest. On days when the CFS SA rainfall index is zero, the TRMM data often indicate isolated rainfall located along the western side on the steep orography of the southwest. The forecast climatological statistics depend on lead time in both seasons, with longer-lead forecasts tending toward more frequent precipitation. The lead time dependence of the forecast statistics is more pronounced in the dry season forecasts. In addition to having more zero values, the forecast SA rainfall index percentiles are smaller than the corresponding observed SA percentiles, indicating model bias and that forecast values of the SA rainfall index should not be directly compared with observed values.
Cumulative distribution functions of the observed and forecast SA rainfall index for the (a) wet and (b) dry seasons. CFS forecast with leads of 1–5 (brown), 6–10 (blue), and 11–15 days (green) are shown.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Cumulative distribution functions of the observed and forecast SA rainfall index for the (a) wet and (b) dry seasons. CFS forecast with leads of 1–5 (brown), 6–10 (blue), and 11–15 days (green) are shown.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Cumulative distribution functions of the observed and forecast SA rainfall index for the (a) wet and (b) dry seasons. CFS forecast with leads of 1–5 (brown), 6–10 (blue), and 11–15 days (green) are shown.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
4. Forecasts of the April–May 2013 flooding event
A slow-moving frontal system around the end of April and beginning of May 2013 caused heavy rain, hail, and damaging wind, and resulted in widespread flooding over the Arabian Peninsula. The extreme rainfall resulted from the interaction of an upper-level trough, the low-level Red Sea trough, and cross-equatorial moisture transport (A. J. de Vries et al. 2014, personal communication). TRMM estimates shown in Fig. 7a for the period from 27 April to 1 May 2013 indicate heavy rainfall (>50 mm) over wide areas of Saudi Arabia, Yemen, and Iran, with rainfall amounts exceeding 70 mm in many regions. A CFS forecast of the total rainfall amount for the 5-day period from 27 April to 1 May 2013 (initialized at 0000 UTC 23 April 2013 and given as the mean of four ensemble members with lead times of 4–9 days) is shown in Fig. 7b and indicates rainfall amounts and a spatial pattern similar to that seen in the TRMM data.
(a) Total TRMM-estimated rainfall from 27 Apr to 1 May 2013. (b) Total CFS forecast rainfall from 27 Apr to 1 May 2013 for the forecast started at 0000 UTC 23 Apr 2013.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
(a) Total TRMM-estimated rainfall from 27 Apr to 1 May 2013. (b) Total CFS forecast rainfall from 27 Apr to 1 May 2013 for the forecast started at 0000 UTC 23 Apr 2013.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
(a) Total TRMM-estimated rainfall from 27 Apr to 1 May 2013. (b) Total CFS forecast rainfall from 27 Apr to 1 May 2013 for the forecast started at 0000 UTC 23 Apr 2013.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
It is encouraging that a particular CFS forecast run skillfully indicated extreme rainfall with a lead time of 4 days. However, conclusions about the predictability, even for a single event, cannot be based on a single forecast. We compute 6-hourly forecast values of the SA rainfall index (Fig. 8a) for different forecast times and leads and compare them with the observed values (Fig. 8b). Forecast values of the SA rainfall index are arranged by target day (x axis) and lead time (y axis). This arrangement of forecast values was used in Tippett et al. (2012) and Barnston et al. (2012) for ENSO forecasts and permits the visualization of a large number of forecast runs in a way that emphasizes forecast timing and consistency from one forecast to another. The earliest forecast start date shown is 27 March 2013. Vertical features in the figure are an indication of forecast consistency, that is, consistency between forecasts with different start times but the same forecast target (verification) time. Large values of the observed daily SA rainfall index are seen beginning around 25 April and continue through 7 May 2013 with one value reaching 5 mm day−1. Forecasts with leads up to 10 days indicated heavy rainfall (many 6-h periods with rain rates exceeding 4 mm day−1), and forecasts with leads up to 20 days indicated a distinct transition from dry to wet conditions. The largest forecast precipitation rate in any 6-h period exceeded 10 mm day−1. The forecasts also give an indication of the length of the rainfall event and the transition back to dry conditions. There is a notable indication of unrealistically extended wet conditions in the longer-lead (greater than 10 day) forecasts.
(a) The 6-hourly CFS SA rainfall index as a function of target (x axis) and lead (y axis) time and (b) the daily TRMM SA rainfall index (mm day−1) for target times from 27 Mar to 26 May 2013. In (a), forecast amounts in the intervals 0–0.125, 0.125–1, 1–4, and >4 mm day−1 are shown in light brown, green, purple, and red, respectively.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
(a) The 6-hourly CFS SA rainfall index as a function of target (x axis) and lead (y axis) time and (b) the daily TRMM SA rainfall index (mm day−1) for target times from 27 Mar to 26 May 2013. In (a), forecast amounts in the intervals 0–0.125, 0.125–1, 1–4, and >4 mm day−1 are shown in light brown, green, purple, and red, respectively.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
(a) The 6-hourly CFS SA rainfall index as a function of target (x axis) and lead (y axis) time and (b) the daily TRMM SA rainfall index (mm day−1) for target times from 27 Mar to 26 May 2013. In (a), forecast amounts in the intervals 0–0.125, 0.125–1, 1–4, and >4 mm day−1 are shown in light brown, green, purple, and red, respectively.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
A predictable feature of the climate system that may be relevant to this rainfall event is the MJO, as discussed in the introduction. ENSO conditions at this time were neutral. Barlow et al. (2005) related tropical heating in the Indian Ocean to the MJO to precipitation in the region and found that negative values of RMM1 favored precipitation in the region. Figure 9 indicates that the RMM1 was negative during the April–May flooding event, consistent with wet conditions in the region. From about 20 April 2013 until the end of the month, the MJO was in phase 8, but mostly with weak amplitude. By the end of the month, the MJO magnitude, defined as 
Values of the RMM indices from 20 Apr to 15 May 2013. April (May) dates are shown in black (gray). Corresponding phases 1–8 are indicated.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Values of the RMM indices from 20 Apr to 15 May 2013. April (May) dates are shown in black (gray). Corresponding phases 1–8 are indicated.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Values of the RMM indices from 20 Apr to 15 May 2013. April (May) dates are shown in black (gray). Corresponding phases 1–8 are indicated.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
5. Reforecast skill
We now examine CFS forecasts of the SA rainfall index over the 12-yr period (1999–2010). Figure 10 shows CFS forecasts for 3 yr (2007–09) of this period, arranged as in Fig. 8 as a function of lead time and target date along with the corresponding TRMM SA rainfall estimates. There is a clear visual qualitative correspondence between forecasts and observed values of the SA rainfall index over the period shown. There are several cases when the forecasts consistently indicate wet conditions 10 or more days in advance of the occurrence of wet conditions. Intense rainfall events are captured to some extent by the forecasts. Transitions between wet and dry periods are also well represented in the forecasts. There are a number of cases, especially in August, when longer-lead (greater than 10 days) forecasts indicate substantial rainfall, while little or scant rainfall is forecast at shorter leads, a feature consistent with the excess rainfall in the longer-lead forecast climatology seen earlier in Fig. 3.
CFS forecasts (colors) of the SA rainfall index as a function of target (x axis) and lead (y axis) time for the period 2007–09 as well as TRMM estimates (line plots). Forecast amounts greater than 0, 0.02, and 0.2 mm are shown in light brown, green, and purple, respectively.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
CFS forecasts (colors) of the SA rainfall index as a function of target (x axis) and lead (y axis) time for the period 2007–09 as well as TRMM estimates (line plots). Forecast amounts greater than 0, 0.02, and 0.2 mm are shown in light brown, green, and purple, respectively.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
CFS forecasts (colors) of the SA rainfall index as a function of target (x axis) and lead (y axis) time for the period 2007–09 as well as TRMM estimates (line plots). Forecast amounts greater than 0, 0.02, and 0.2 mm are shown in light brown, green, and purple, respectively.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Quantitative measures of the skill (rank correlation) of the deterministic forecasts of rainfall amount and number of days exceeding 0.25 mm day−1 are shown in Fig. 11 as a function of lead time for forecast target windows ranging in width from 1 to 30 days. The maximum lead time shown depends on the forecast window, with forecasts for 30-day windows having a maximum lead of about 15 days. The correlations for amount and for number of days exceeding the threshold are qualitatively similar in the wet season. At the shortest leads, increasing the forecast window width initially increases skill, but eventually as the forecast window expands to include increasingly distant target time, skill decreases. For both quantities, forecasts of 5-day windows have the highest skill initially (closely followed by forecasts of 10-day windows), and forecasts of 30-day windows have correlations of nearly 0.6 at the shortest lead. Skill at the longer leads increases as the forecast window increases and decreases as lead increases. However, skill for forecast windows greater than 10 days decreases more slowly during the wet season than that of short windows especially for longer leads. This distinction is less evident during the dry season, suggesting different predictability mechanisms. The skill during the wet season is greater than during the dry season for both quantities, with the wet season advantage greater in the case of the forecasts of exceedance days.
Rank correlation as a function of lead time (days) for forecasts of the (a),(b) SA rainfall index and (b),(d) number of days for which the SA rainfall index exceeds the threshold 0.25 mm day−1 for the (left) wet and (right) dry seasons.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Rank correlation as a function of lead time (days) for forecasts of the (a),(b) SA rainfall index and (b),(d) number of days for which the SA rainfall index exceeds the threshold 0.25 mm day−1 for the (left) wet and (right) dry seasons.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Rank correlation as a function of lead time (days) for forecasts of the (a),(b) SA rainfall index and (b),(d) number of days for which the SA rainfall index exceeds the threshold 0.25 mm day−1 for the (left) wet and (right) dry seasons.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Figure 12 shows the Brier skill score for forecasts of the probability of heavy rainfall occurrence based on lagged ensemble and logistic regression. This quantity is challenging to forecast, and even in the wet season when skill is higher, there is only modest, if any, skill for leads greater than 10 days. In the dry season, skill drops sharply with lead, and there is little skill for windows greater than 5 days, even with the logistic regression. Logistic regression increases forecast skill, especially for leads of less than 10 days during the wet season. The Brier skill score of the first lead 15-day forecast increases from 0.2 to more than 0.3 with logistic regression. The benefit of the logistic regression is limited to the first couple of leads and the shorter windows (5 days or less) during the dry season.
Brier skill scores for forecasts of the probability of exceeding a threshold. Probability computed from a (a),(b) lagged ensemble and (c),(d) logistic regression for the (left) wet and (right) dry seasons.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Brier skill scores for forecasts of the probability of exceeding a threshold. Probability computed from a (a),(b) lagged ensemble and (c),(d) logistic regression for the (left) wet and (right) dry seasons.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Brier skill scores for forecasts of the probability of exceeding a threshold. Probability computed from a (a),(b) lagged ensemble and (c),(d) logistic regression for the (left) wet and (right) dry seasons.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
We can understand how logistic regression improves probabilistic forecast quality by considering reliability. Figure 13 shows reliability and ROCs for the shortest lead lagged-ensemble and logistic regression probability forecasts of heavy rainfall during a 15-day window. The lagged ensemble probability forecasts show substantial overconfidence in the sense that heavy rainfall occurs more (less) often in the wet season than low (high) forecast probability values would indicate. For instance, when the forecast probability is 0% (80%), heavy rainfall occurs about 20% (65%) of the time. This problem of overconfidence is more severe in the dry season. Some of the overconfidence is likely due to the quantization of probabilities by the four-member ensemble, evident in the forecast issuance frequencies. Despite the overconfidence, the ROCs indicate that the lagged ensemble probabilities are able to discriminate fairly well between occurrence and nonoccurrence, with the discrimination (as measured by the area under ROC) being greater during the wet season. The logistic regression probabilities are considerably more reliable. Dry season logistic regression probabilities of heavy rainfall do not exceed 20%, while in the wet season they nearly cover the range of possible values. Logistic regression has a modest impact on the wet season ROC, which suggests that the primary result of the logistic regression is to temper the forecast probabilities in a manner that mostly preserves their ordering. Logistic regression has the potential to correct for lead-dependent and other systematic biases in the forecast model output. Greater improvement from the logistic regression is seen in the dry season ROC.
Reliability diagrams of (a) lagged ensemble and (b) logistic regression first-lead probability forecasts of heavy rainfall over the subsequent 15-day window; inset histograms indicate the forecast issuance frequency. Corresponding ROCs for (c) lagged ensemble and (d) logistic regression probability forecasts.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Reliability diagrams of (a) lagged ensemble and (b) logistic regression first-lead probability forecasts of heavy rainfall over the subsequent 15-day window; inset histograms indicate the forecast issuance frequency. Corresponding ROCs for (c) lagged ensemble and (d) logistic regression probability forecasts.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Reliability diagrams of (a) lagged ensemble and (b) logistic regression first-lead probability forecasts of heavy rainfall over the subsequent 15-day window; inset histograms indicate the forecast issuance frequency. Corresponding ROCs for (c) lagged ensemble and (d) logistic regression probability forecasts.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
To connect forecast skill with the MJO phase, Fig. 14 shows first-lead logistic regression forecast probabilities for heavy rainfall during the subsequent 5-day window conditional on the MJO phase. The MJO phase used is the value from 3 days before the start of the forecast, and only cases with RMM magnitude (
Average first-lead logistic regression forecast probabilities of heavy rainfall during the subsequent 5 days stratified by the MJO phase 3 days prior to the start of the (a) wet and (b) dry seasons. Heavy and light dashed lines show the unconditional mean and its 95% confidence intervals, respectively, based on the number of forecasts in each phase.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Average first-lead logistic regression forecast probabilities of heavy rainfall during the subsequent 5 days stratified by the MJO phase 3 days prior to the start of the (a) wet and (b) dry seasons. Heavy and light dashed lines show the unconditional mean and its 95% confidence intervals, respectively, based on the number of forecasts in each phase.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
Average first-lead logistic regression forecast probabilities of heavy rainfall during the subsequent 5 days stratified by the MJO phase 3 days prior to the start of the (a) wet and (b) dry seasons. Heavy and light dashed lines show the unconditional mean and its 95% confidence intervals, respectively, based on the number of forecasts in each phase.
Citation: Weather and Forecasting 30, 4; 10.1175/WAF-D-15-0011.1
6. Summary
The climate of Saudi Arabia is arid–semiarid, and rainfall occurrence is infrequent and scattered (Abdullah and Al-Mazroui 1998). However, when rainfall does occur in the region, it is sometimes intense and causes flooding, loss of life, and damage to property. The serious consequences of heavy rainfall in the region make its prediction important. Several studies have related precipitation in the region to remote forcing, especially tropical convection related to ENSO and the MJO (Barlow 2012; Hoell et al. 2012, 2013, 2014; Athar 2015; Kang et al. 2015). These climate signals are predictable, and based on the predictability of the MJO, it has been proposed that rainfall in the region should be predictable up to 2–3 weeks in advance (Barlow et al. 2005). However, to this point there has been no assessment of the extent to which this predictability is realizable either in statistical or dynamical forecasts.
The NOAA Climate Forecast System (CFS), version 2, is a state-of-the-art coupled global ocean–atmosphere model used to produce forecasts out to 9 months (Saha et al. 2014). The CFS has good skill in predicting ENSO and the MJO (Zhang and van den Dool 2012; Barnston and Tippett 2013; Wang et al. 2014) and therefore has the potential to forecast the region teleconnections associated with these signals. Here, we have taken advantage of the extensive CFS reforecast dataset with 6-hourly forecast starts over a 12-yr (1999–2010) period to document the ability of the CFS to forecast areal-averaged Saudi Arabian rainfall (SA rainfall index). While there are deficiencies in the forecast climatology likely related to orography and resolution, as well as lead-dependent biases, CFS represents the climatology of the region both in terms of spatial patterns and the annual cycle reasonably well.
A motivation for this work was the heavy rainfall that occurred between the end of April and the beginning May 2013, which resulted in widespread flooding over the Arabian Peninsula. We find that the CFS was able to predict features of the 2013 event well, although not the local intensity. Forecasts indicated heavy rainfall up to 10 days in advance of the event and a distinct transition from dry to wet conditions up to 20 days in advance. Analysis of the reforecasts shows that the CFS can skillfully predict rainfall amount, the number of days exceeding a threshold, and the probability of heavy rainfall occurrence for varying forecast windows. Heavy rainfall occurrence is the more difficult quantity to forecast, and skill is markedly lower in the dry season (May–October). While the probability forecasts of heavy rainfall occurrence show good discrimination, the probability values indicate overconfident forecasts. We demonstrate that logistic regression with the ensemble mean value as a predictor improves forecast skill and reliability. Forecast probability signals are found to have a clear relation with MJO phase in the wet season (November–April) consistent with previous variability analyses on interseasonal and intraseasonal time scales, providing a physical basis for observed forecast skill.
Acknowledgments
This study was supported in part by NOAA Awards NA12OAR4310091 and NA14OAR4310184 and Office of Naval Research Award N00014-12-1-091. The views expressed herein are those of the authors and do not necessarily reflect the views of NOAA or any of its subagencies.
REFERENCES
Abdullah, M. A., and Al-Mazroui M. A. , 1998: Climatological study of the southwestern region of Saudi Arabia. I. Rainfall analysis. Climate Res., 9, 213–223, doi:10.3354/cr009213.
Almazroui, M., 2006: The relationships between atmospheric circulation patterns and surface climatic elements in Saudi Arabia. Ph.D. thesis, Climatic Research Unit, University of East Anglia, Norwich, United Kingdom, 432 pp.
Almazroui, M., 2011a: Calibration of TRMM rainfall climatology over Saudi Arabia during 1998–2009. Atmos. Res., 99, 400–414, doi:10.1016/j.atmosres.2010.11.006.
Almazroui, M., 2011b: Sensitivity of a regional climate model on the simulation of high intensity rainfall events over the Arabian Peninsula and around Jeddah (Saudi Arabia). Theor. Appl. Climatol., 104, 261–276, doi:10.1007/s00704-010-0387-3.
Alpert, P., Osetinsky I. , Ziv B. , and Shafir H. , 2004: Semi-objective classification for daily synoptic systems: Application to the eastern Mediterranean climate change. Int. J. Climatol., 24, 1001–1011, doi:10.1002/joc.1036.
Athar, H., 2015: Teleconnections and variability in observed rainfall over Saudi Arabia during 1978–2010. Atmos. Sci. Lett., doi:10.1002/asl2.570, in press.
Barlow, M., 2012: Africa and West Asia. Intraseasonal Variability in the Atmosphere–Ocean Climate System, W. K.-M. Lau and D. E. Waliser, Eds., 2nd ed., Springer-Verlag, 477–495.
Barlow, M., Wheeler M. , Lyon B. , and Cullen H. , 2005: Modulation of daily precipitation over southwest Asia by the Madden–Julian oscillation. Mon. Wea. Rev., 133, 3579–3594, doi:10.1175/MWR3026.1.
Barnston, A. G., and Tippett M. K. , 2013: Predictions of Nino3.4 SST in CFSv1 and CFSv2: A diagnostic comparison. Climate Dyn., 41, 1615–1633, doi:10.1007/s00382-013-1845-2.
Barnston, A. G., Tippett M. K. , L’Heureux M. L. , Li S. , and DeWitt D. G. , 2012: Skill of real-time seasonal ENSO model predictions during 2002–11: Is our capability increasing? Bull. Amer. Meteor. Soc., 93, 631–651, doi:10.1175/BAMS-D-11-00111.1.
Chakraborty, A., Behera S. K. , Mujumdar M. , Ohba R. , and Yamagata T. , 2006: Diagnosis of tropospheric moisture over Saudi Arabia and influences of IOD and ENSO. Mon. Wea. Rev., 134, 598–617, doi:10.1175/MWR3085.1.
de Vries, A. J., Tyrlis E. , Edry D. , Krichak S. O. , Steil B. , and Lelieveld J. , 2013: Extreme precipitation events in the Middle East: Dynamics of the active Red Sea trough. J. Geophys. Res. Atmos., 118, 7087–7108, doi:10.1002/jgrd.50569.
Hoell, A., Barlow M. , and Saini R. , 2012: The leading pattern of intraseasonal and interannual Indian Ocean precipitation variability and its relationship with Asian circulation during the boreal cold season. J. Climate, 25, 7509–7526, doi:10.1175/JCLI-D-11-00572.1.
Hoell, A., Barlow M. , and Saini R. , 2013: Intraseasonal and seasonal-to-interannual Indian Ocean convection and hemispheric teleconnections. J. Climate, 26, 8850–8867, doi:10.1175/JCLI-D-12-00306.1.
Hoell, A., Funk C. , and Barlow M. , 2014: La Niña diversity and Northwest Indian Ocean Rim teleconnections. Climate Dyn., 43, 2707–2724, doi:10.1007/s00382-014-2083-y.
Hoffman, R. N., and Kalnay E. , 1983: Lagged average forecasting, an alternative to Monte Carlo forecasting. Tellus, 35A, 100–118, doi:10.1111/j.1600-0870.1983.tb00189.x.
Huffman, G. J., and Coauthors, 2007: The TRMM Multisatellite Precipitation Analysis (TMPA): Quasi-global, multiyear, combined-sensor precipitation estimates at fine scales. J. Hydrometeor., 8, 38–55, doi:10.1175/JHM560.1.
Huffman, G. J., Adler R. , Bolvin D. , and Nelkiny E. , 2010: The TRMM Multi-Satellite Precipitation Analysis (TMPA). Satellite Rainfall Applications for Surface Hydrology, F. Hossain and M. Gebremichael, Eds., Springer Verlag, 3–22.
Johnson, N. C., 2013: How many ENSO flavors can we distinguish? J. Climate,26, 4816–4827, doi:10.1175/JCLI-D-12-00649.1.
Jones, C., Waliser D. E. , Lau K. M. , and Stern W. , 2004: Global occurrences of extreme precipitation and the Madden–Julian oscillation: Observations and predictability. J. Climate, 17, 4575–4589, doi:10.1175/3238.1.
Kang, I.-S., Rashid I. U. , Kucharski F. , Almazroui M. , and Al Khalaf A. A. , 2015: Multidecadal changes in the relationship between ENSO and wet-season precipitation in the Arabian Peninsula. J. Climate, doi:10.1175/JCLI-D-14-00388.1, in press.
Krichak, S. O., Alpert P. , and Krishnamurti T. N. , 1997a: Interaction of topography and tropospheric flow—A possible generator for the Red Sea trough? Meteor. Atmos. Phys., 63, 149–158, doi:10.1007/BF01027381.
Krichak, S. O., Alpert P. , and Krishnamurti T. N. , 1997b: Red Sea trough/cyclone development—Numerical investigation. Meteor. Atmos. Phys., 63, 159–169, doi:10.1007/BF01027382.
Maloney, E. D., and Hartmann D. L. , 1998: Frictional moisture convergence in a composite life cycle of the Madden–Julian oscillation. J. Climate, 11, 2387–2403, doi:10.1175/1520-0442(1998)011<2387:FMCIAC>2.0.CO;2.
Murphy, A. H., 1973: A new vector partition of the probability score. J. Appl. Meteor., 12, 595–600, doi:10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2.
Nazemosadat, M. J., and Ghaedamini H. , 2010: On the relationships between the Madden–Julian oscillation and precipitation variability in southern Iran and the Arabian Peninsula: Atmospheric circulation analysis. J. Climate, 23, 887–904, doi:10.1175/2009JCLI2141.1.
Palmer, T. N., 1993: Extended-range atmospheric prediction and the Lorenz model. Bull. Amer. Meteor. Soc., 74, 49–65, doi:10.1175/1520-0477(1993)074<0049:ERAPAT>2.0.CO;2.
Saha, S., and Coauthors, 2010: The NCEP Climate Forecast System Reanalysis. Bull. Amer. Meteor. Soc., 91, 1015–1057, doi:10.1175/2010BAMS3001.1.
Saha, S., and Coauthors, 2014: The NCEP Climate Forecast System version 2. J. Climate, 27,2185–2208, doi:10.1175/JCLI-D-12-00823.1.
Shukla, J., 1981: Dynamical predictability of monthly means. J. Atmos. Sci., 38, 2547–2572, doi:10.1175/1520-0469(1981)038<2547:DPOMM>2.0.CO;2.
Shukla, J., 1998: Predictability in the midst of chaos: A scientific basis for climate forecasting. Science, 282, 728–731, doi:10.1126/science.282.5389.728.
Tippett, M. K., Barnston A. G. , and Li S. , 2012: Performance of recent multimodel ENSO forecasts. J. Appl. Meteor. Climatol., 51, 637–654, doi:10.1175/JAMC-D-11-093.1.
Waliser, D. E., Lau K. M. , Stern W. , and Jones C. , 2003: Potential predictability of the Madden–Julian oscillation. Bull. Amer. Meteor. Soc., 84, 33–50, doi:10.1175/BAMS-84-1-33.
Wang, W., Hung M.-P. , Weaver S. , Kumar A. , and Fu X. , 2014: MJO prediction in the NCEP Climate Forecast System version 2. Climate Dyn., 42, 2509–2520, doi:10.1007/s00382-013-1806-9.
Wheeler, M. C., and Hendon H. , 2004: An all-season real-time multivariate MJO index: Development of an index for monitoring and prediction. Mon. Wea. Rev., 132, 1917–1932, doi:10.1175/1520-0493(2004)132<1917:AARMMI>2.0.CO;2.
Zhang, Q., and van den Dool H. , 2012: Relative merit of model improvement versus availability of retrospective forecasts: The case of climate forecast system MJO prediction. Wea. Forecasting, 27, 1045–1051, doi:10.1175/WAF-D-11-00133.1.
