1. Introduction
Accurate predictions of weather phenomena by numerical models are limited by the uncertainties in the initial atmospheric, oceanic (sea surface temperature and sea ice), land surface, and soil moisture conditions along with inaccurate representations of true physical processes. Uncertainties in the initial conditions (ICs) are bracketed by creating IC ensembles, using various methods as surveyed in Buizza and Palmer (1995), for example. One simple method is to slightly perturb the original atmospheric or oceanic states, as described in Abhilash et al. (2013, 2014a). Imperfect model uncertainties are more challenging to represent. In a posterior approach, ensemble predictions from different models are pooled to produce a final forecast probability distribution. In any multimodel approach, the independent skills of the participating models are combined in a judicious manner to reinforce the total skill of the multimodel ensemble (MME) mean (or other statistics). One example of such an MME combination approach using historically trained regression (i.e., linear combinations of the inputs) is documented in Krishnamurti et al. (1999) and Weigel et al. (2008). Such a multimodel superensemble forecast provides a better deterministic forecast than any individual model or any of its single-model ensemble (SME) subsets (Weigel et al. 2008). However, such a “best estimate” use of superensembles does not address statistical skill issues like the spread–error relationship (SER).
Forecasts are more useful if uncertainty is quantified. In the extended range (ER) especially, beyond the weather scale (2–3 weeks), a single deterministic rainfall forecast is not sufficient: the user community also should be given probabilistic forecasts that quantify the uncertainty. One chronic problem in numerical ensembles is underdispersion, leading to overconfidence errors in the resulting statistical forecast distributions. Single-model ensemble prediction systems (EPSs) are especially prone to underdispersion, since they do not represent all sources of forecast error (Buizza et al. 2008). Some single-model approaches for producing more-diverse inputs to the ensemble product generator include stochastic postprocessing [e.g., “Bayesian model averaging” method of Raftery et al. (2005)] or stochastic parameterization (Buizza et al. 1999; Palmer 2001). But true MME approaches may better represent physical uncertainties and increase the ensemble spread accordingly. Seeking a robust operational approach for ER monsoon forecasts, this study considers a partial MME approach.
Employing a broad-based MME in real-time forecasts would require reliable, coordinated multi-institutional effort (Sahai et al. 2013; Borah et al. 2015; Prasad et al. 2014). While frameworks like the National Monsoon Mission (http://www.tropmet.res.in/monsoon/) might foster such collaborations in the long run, the current study reports on an MME derived from multiversion, multiresolution, and two-tier extended-range forecasts computed at one center. Results are based on the National Centers for Environmental Prediction’s (NCEP) Climate Forecast System (CFS), version 2. CFSv2 is a coupled model whose atmosphere component, the Global Forecast System (GFS), has been subsequently improved in ways not yet propagated to the coupled system. The Indian adaptation of the NCEP CFS and its operational reliability for extended-range forecasting have already been reported upon (Sahai et al. 2013; Borah et al. 2015). The goal here is to demonstrate and quantify improvements in the MME forecasts as compared with the CFS-based SME approach (Sahai et al. 2013) in the extended range.
For this study we run three different versions of CFS/GFS to generate three sets of SMEs and then pool them to form the MME, an approach to ensemble generation described by Houtekamer et al. (1996). The versions of CFS/GFS are diverse in resolution as well as in their model physics. Since extended-range skill stems from both slowly varying lower boundary effects and from the model’s memory of the initial atmospheric state, our three component SMEs were devised with both predictability sources in mind. Specifically, the MME includes both ocean-initialized coupled CFS integrations with slightly different atmospheric initial conditions, and “two tier” integrations with stand-alone atmospheric GFS forced by bias-corrected CFS-forecast sea surface temperature (SST), a setup hereinafter called GFSbc. The combined forecast system addresses the prime goal of the National Monsoon Mission, as initiated by the government of India (http://www.tropmet.res.in/National%20Monsoon%20Mission-121-Page), which is to improve the monsoon prediction capabilities of the CFS system.
Although these three SMEs show similar prediction skill and their errors saturate at about the same lead time of around 25 days, there are many instances where the three models disagree in predicting particular events, such as the amplitude and phase of monsoon intraseasonal oscillation (MISO) propagation. Even though the SMEs show similar skill on average, there are years when any one of the configurations is the most or least skillful (Abhilash et al. 2013), motivating this so-called CFS-based Grand Ensemble Prediction System (CGEPS), for ER prediction (ERP) of active/break spells of Indian summer monsoon rainfall. Another important operational reason to combine SMEs into an MME (rather than disseminating each variant separately) is to clarify communication to the users, with a single consensus forecast and associated uncertainties and reliability estimates.
This paper’s main objective is to demonstrate the operational efficacy of CGEPS in the extended-range rainfall forecast during the summer monsoon season. Section 2 describes the model and methodology; section 3 presents the results. Section 3a presents the skill during the hindcast period, section 3b presents the MISO prediction skill and spread–error relationship, and section 3c presents the probabilistic prediction skill. Section 4 gives our conclusions.
2. Model and methodology
We have used the latest version1 of NCEP’s CFSv2 (Saha et al. 2014). The atmospheric component (GFS) is coupled to an ocean model, a sea ice model, and a land surface model. For its ocean component, the CFSv2 uses the GFDL Modular Ocean Model, version 4p0d (MOM4; Griffies et al. 2004). The initial conditions have been prepared from a coupled data assimilation system (CDAS) with T574L64 resolution atmospheric assimilation and MOM4-based oceanic assimilation, a real-time extension of the CFSR (Saha et al. 2010).
The stand-alone GFS with slightly different physics options is forced with daily bias-corrected forecast SSTs from CFSv2. Bias correction involves subtracting a climatology of bias as a function of calendar day and lead time, with Optimum Interpolation Sea Surface Temperature observations as the reference. We denote this two-tier forecast as GFSbc, with “bc” indicating bias-corrected boundary conditions. For more model and experimental details and skills of GFSbc, CFST126, and CFST382, see Abhilash et al. (2013, 2014 a,b) and Sahai et al. (2015), respectively.
Based on performance experience, and aiming to maximize the operational skill for our available computer resources, we have chosen to pool these variants: 11 members of CFST126 (~100 km), 11 members of CFST382 (~38 km), and 21 members of GFSbc. A schematic diagram of the CGEPS generation is shown in Fig. 1. The model integrations run to generate the ensemble members are initiated with updated initial conditions every 5 days from 16 May through 28 September during the summer monsoon season. The forecast consensus is given by averaging the 43 ensemble members. The skill (defined as the correlation between the observed and the forecast rainfall anomalies from a 10-yr hindcast climatology) is evaluated during the hindcast period of 2001–12 summer monsoon seasons. Skill is computed by averaging the forecast on a pentad (5-day mean) scale. For example, if the forecast starts from 16 May IC, the pentad 1 (P1) lead-time forecast corresponds to the forecast averaged over the period 17–21 May, the pentad 2 (P2) lead-time forecast corresponds to the forecast averaged over the period 22–26 May, and so on. The pentad averaging is done since the daily prediction of summer monsoon rainfall by dynamical models is not skillful beyond 10 days.
Schematic diagram of CGEPS.
Citation: Journal of Applied Meteorology and Climatology 54, 7; 10.1175/JAMC-D-14-0200.1
To assess the performance of the method, various skill estimates are calculated for several boxed regions over India. These skill metrics and scores include temporal correlation and root-mean-square error (RMSE) evaluated at the pentad scale, MISO-index prediction skill and its spread–error relationship, the relative operating characteristic (ROC) curve, the area under the ROC curve (AUC), Brier skill score (BSS), and resolution and reliability (detailed in section 3 below). Rainfall data for all scoring are taken from the Tropical Rainfall Measuring Mission (TRMM) gauge merged rainfall dataset (Mitra et al. 2009) obtained from the India Meteorological Department.
Probabilistic forecasts from ensemble members at different lead times are validated with respect to predefined categories (quantiles) based on observations. The categories are defined by classifying the observed rainfall into above normal (AN), below normal (BN), and near normal (NN) categories using the tercile classification method. Terciles are the three intervals, that is, the lower, middle, and upper thirds of the climatologically distributed values of a variable like rainfall where each category has an equal climatological probability of 33.33%. To determine the tercile ranges, the observed rainfall data were ranked in descending order. The three categories are defined as AN, NN, or BN and are separated by the values ⅓ and ⅔ of the way down the ranked list. In a similar manner the three categories are defined for the three SMEs based on model hindcast climatological data (2001–12). By counting the proportion of the model’s ensemble members falling into the respective terciles, corresponding probabilities for each forecast category can be generated. These categorical forecasts are very important for the probabilistic prediction of ISM. This tercile classification approach is used over some approximately homogenous regions, selected based on the climatology and variances of the rainfall over the Indian landmass. For the monsoon zone of India (MZI) as discussed in Borah et al. (2013), central India (CEI), northeast India (NEI), northwest India (NWI), and south peninsular India (SPI), the percentage departure of rainfall from the long-term pentad climatology is evaluated in the next section.
3. Results
The climatological seasonal [June–September (JJAS)] mean bias of rainfall forecast over the Indian region at P1, P2, P3, and P4 lead times from the three variants of CFS (i.e., CFST126, GFSbc, and CFST382) along with the MME are shown in Fig. 2 from left to right, respectively. The top panel in Fig. 2 shows the observed seasonal (JJAS) mean climatological rainfall. In general, the dry bias over the Indian region increases with lead. The serious overestimation (wet bias) in rainfall over the Arabian Sea and western equatorial Indian Ocean and the underestimation (dry bias) over the Indian land region are common in most of the climate models (e.g., Sperber et al. 2013; Kim et al. 2012; Bush et al. 2014). The overestimation in rainfall over the Arabian Sea is reduced in GFSbc when forced with bias-corrected forecast SST. However, the dry bias over the Indian landmass remains an unresolved problem in all models. Near-zero values off the west coast mark the boundary between the wet and dry biases in model rainfall, which are large along the west coast of India. Figure 2 also shows that, among all variants of CFS, the CFST382 has the smallest climatological June–September precipitation biases in the P1–P4 lead forecast over the Indian land region, followed by the MME average. The wet bias over the adjoining oceanic regions is considerably reduced in MME for all pentad leads. Since MME is a simple average of all variants, and GFSbc has the maximum number of ensemble members, this climatological bias in MME over the oceanic and central Indian regions can be attributed largely to GFSbc. Sahai et al. (2015) show that the drastic reduction of climatological biases in the P1–P4 lead forecast in CFST382 over the Indian land region is not necessarily reflected in the ERP skill of CFST382. In contrast, in the MME, reduction in the climatological biases over the Indian land region (though lesser than that of CFST382) has been translated to useful ERP skill, as evident from Table 1, which shows the spatial correlations for forecast rainfall at different pentad lead times (P1–P4). The MME is seen to be retaining a large spatial correlation (above 0.85) of up to four-pentad lead time. Further improvements are discussed in the following sections.
(top) Seasonal climatological mean precipitation during the JJAS season (mm day−1) from TRMM observations, and the difference in pentad lead forecast climatology from observations for CFST126, GFSbc, CFST382, and MME.
Citation: Journal of Applied Meteorology and Climatology 54, 7; 10.1175/JAMC-D-14-0200.1
Spatial pattern correlation coefficients between the climatological rainfall forecast at different lead pentads and the observations for the SMEs and the MME. The region for pattern correlation is taken to be 40°S–40°N, 40°–120°E.
a. Hindcast skill
Hindcast skill of the MME from CGEPS is analyzed for pentad-averaged rainfall over five homogeneous regions of India (Fig. 3a). Skill in terms of the correlation coefficient (CC) between the predicted and observed pentad mean rainfall series is computed for MME and for all individual models during the hindcast period of 2001–12 for P1–P4 lead time. For most of the cases, CC from MME shows the highest skill relative to that from all of the individual models (Figs. 3b–f) for all lead pentads and for all homogeneous regions except for CEI and the core monsoon zone of India (MZI; Rajeevan et al. 2010). For those regions, GFSbc skill is almost the same or marginally better than MME for longer leads (i.e., P3 and P4). In summary, the deterministic forecast skill from MME is reasonably improved relative to all individual models and for all pentad leads.
(a) Homogeneous zones of India: MZI, CEI, NEI, NWI, and SPI. (b)–(f) Temporal correlation is shown for the pentad lead prediction of area-averaged rainfall with observations for the hindcast period 2001–12 over the homogeneous zones of India.
Citation: Journal of Applied Meteorology and Climatology 54, 7; 10.1175/JAMC-D-14-0200.1
b. MISO prediction skill and spread–error relationship
We compare the prediction skill of large-scale MISO by computing the MISO indices following Suhas et al. (2012) and Borah et al. (2013), which define the eight-phase northward-propagating MISO rainfall anomalies from the Indian Ocean region to the foothills of the Himalayas. This is an empirical orthogonal function (EOF) analysis similar to that of Wheeler and Hendon (2004) except that the EOF analysis is performed on an extended data matrix. Like those authors’ verification measures proposed to compute the Madden–Julian oscillation (MJO) prediction skill, we have also calculated the bivariate CC and RMSE (Lin et al. 2008; Gottschalck et al. 2010; Rashid et al. 2011) between the predicted and observed MISO indices. The MISO indices (MISO1 and MISO2) are the principal components computed by projecting the MME forecast and observation onto the extended EOFs.
Figure 4a shows that the limit of useful prediction (as measured by a threshold CC of 0.5) for MME is up to 20 days and is comparable to the best participating model (GFSbc). The forecast error saturates at longer leads near 25 days for both MME and GFSbc (Fig. 4b). Up to 15 days, both MME and GFSbc show similar forecast skill; then, the forecast error grows slightly faster for GFSbc relative to the MME. In summary, the CGEPS MME shows better MISO prediction skill than does the best SME.
Bivariate (a) correlation and (b) RMSE and spread of MISO indices from CFST126, CFST382, GFSbc, and MME. Phase diagram of MISO evolution from observations (blue line), models (red line), and forecast spread (green shades, ±1 standard deviation) from a typical break in the initial conditions starting from 24 Aug 2012 for (c) CFST126, (d) CFST382, (e) GFSbc, and (f) MME.
Citation: Journal of Applied Meteorology and Climatology 54, 7; 10.1175/JAMC-D-14-0200.1
Reliability and utility of the EPS are generally assessed by plotting the ensemble spread versus RMSE. Spread is evaluated by plotting the standard deviation of all individual models with respect to their ensemble mean. Both the spread and RMSE of MME are computed by considering all of the 43 ensemble members. Figure 4b shows the RMSE (solid line) and the spread (dotted line) for each SME and the MME as a function of lead time. Ideally for a better EPS, the spread should be close to the RMSE, which should be minimized at any lead time. It is evident from Fig. 4b that the individual SME has lower spread (overconfident) and higher RMSE at long lead times, indicating a poorer SER. However, the MME shows considerable improvement in SER, which is mostly contributed by the increased spread (i.e., reduction in overconfidence penalty) at longer lead time, without compromising the skill.
To better illustrate the impact of this increased spread on the prediction of MISO, an example is given in Figs. 4c–f. The panels show a scatterplot of MISO1 and MISO2 indices, to evaluate the ability to forecast the MISO propagation in a [MISO1, MISO2] phase space from a typical break state starting from 24 August 2012. While the spread of MISO indices (green shaded) does not encompass the verification curve (blue) for the SMEs (Figs. 4c–e), MME spread does include it (phases 1–3 in Fig. 4f). MME also improves SER in a statistical sense, like the MISO amplitude of this particular event.
c. Skill of probabilistic forecast
Since deterministic predictions of daily weather features by dynamical models are not accurate beyond 10 days, probabilistic forecasts from ensembles add extra value in ERP. Probabilistic categorical prediction skill from the EPS is evaluated by computing the ROC score, measured as AUC (Sahai et al. 2008), and BSS (Wilks 2005). The ROC diagram represents the ability of a set of probabilistic forecasts to discriminate a dichotomous event, relating the hit rate and the false alarm rate (Kharin and Zwiers 2003). In a ROC curve, the closer clustering of probability values indicates less spread among the ensemble members. The ROC curves for the below normal category at P3 lead is shown in Fig. 5a for different probability thresholds. The ROC curve for MME (black curve) shows a better probability distribution than do the individual participating SMEs. Figure 5b shows the AUC for all the three categories and for all lead pentads. The AUC for MME is always better than or similar to the best participating SME, for all lead pentads and for all three categories. However, AUC is highest for the BN and AN categories. Thus, with the present MME system, the BN and AN categories are well predicted and clearly discriminated.
(a) ROC curve for the BN category for third pentad lead predictions and (b) AUC for BN, NN, and AN categories for one- to four-pentad lead prediction.
Citation: Journal of Applied Meteorology and Climatology 54, 7; 10.1175/JAMC-D-14-0200.1
Figure 6a shows the BSS of categorical probabilistic forecasts. BSS measures the relative skill of forecasts compared to a standard or climatological or persistence forecast (Wilks 2005). Again the MME is seen to be skillful (BSS is positive) for below and above normal categories. However, for the other SMEs, the BSS is negative for all lead pentads except for the first lead pentad. The BSS for the near-normal category is very low or even negative for all lead pentads and for all SMEs as well as MME, consistent with the ROC analysis discussed in the previous paragraph. The skill of categorical prediction for the near-normal category is lower because the width of the categories for predictands with bell-shaped distributions is narrow near the center: it is easier for the verifying observation to escape from the closed middle category than the open-ended outer categories (Van den Dool and Toth 1991).
(a) BSS for BN, NN, and AN categories of one- to four-pentad lead probabilistic forecast. (b) Resolution (solid line) and reliability (dashed line) of the corresponding categories for one- to four-pentad lead prediction.
Citation: Journal of Applied Meteorology and Climatology 54, 7; 10.1175/JAMC-D-14-0200.1
Is running a cheaper variant (e.g., GFSbc) with a larger number of ensemble members necessarily a good strategy in general? The answer is negative. The BSS plots show why the best-performing SME (e.g., GFSbc), which is also cheaper to run, cannot compete with the MME. The yellow bar plot (MME21) in Fig. 6a is obtained if we generate a (3 × 7) 21-member MME by randomly selecting seven ensemble members from each of the three variants of the participating models in CGEPS and compute the BSS. It can be seen that it is still better than the 21-member GFSbc (green bar) for all three categories. Thus, the improvement in the probability prediction skill is not coming solely from running more instances of GFSbc (i.e., the model with better skill). Furthermore, it is not attributable to the fact that a larger number of ensemble members are run for GFSbc. The observed improvement is the result of combining the three variants of the model. Thus, the MME approach is a better choice for MISO forecasting than running the best SME with a larger number of ensembles, as long as the SME is underdispersive.
The Brier score can be decomposed into its reliability, resolution, and uncertainty. The performance of the EPS is further evaluated by comparing its two main attributes: reliability and resolution. The statistical reliability measures the degree to which the sample of forecasts is statistically indistinguishable from the corresponding observations. On the other hand, statistical resolution reflects the system’s ability to distinguish between different forecast events. For a better EPS, the reliability should be closer to zero and the resolution should be higher at all lead times. From Fig. 6b, it is seen that at shorter lead times the resolution is the main contributor to positive BSSs. For all SMEs reliability is higher than resolution after P1 lead, resulting in a negative BSS for all SMEs. However, in the case of MME, the reliability value is always closer to zero and lower than the resolution, for the below normal and above normal categories, resulting in positive BSS. Thus, the MME performs better than any of its three participating SMEs. However, for the near-normal category, the reliability from MME remains more or less constant and slightly higher than the resolution for all longer leads beyond the first pentad. It is evident from the above analyses that the three-category probabilistic forecasts from MME outperform all SMEs and this better skill flows from greater spread and thus a better SER.
4. Conclusions
The prediction of monsoon weather and the MISO in the extended range is a challenging problem for the operational monsoon meteorologist. It is found that the CGEPS MME provides multiple benefits: by encompassing the errors in both the ICs and forecast model physics, it provides better probability forecasts from the users’ perspective (measured by increased reliability). We conclude that deterministic prediction of the MISO using this CGEPS MME is operationally acceptable, although the evaluation of deterministic skill has not yet been thoroughly compared with the forecasts of other operational prediction systems, such as the ECMWF Variable Ensemble Prediction System (VarEPS) system, which is reported to outperform CFSv2 for MJO forecasts (Kim et al. 2014).
Part of the overconfidence penalty involved in SMEs is overcome in the MME, which improves the spread–error relationship, so that the MME approach adds value to both the deterministic and probabilistic forecasts. The results encourage us to pursue a comprehensive adoption of this broadened NCEP-CFSv2 CGEPS MME framework for the operational extended-range prediction of the Indian summer monsoon. Based on the results above, further diversification efforts such as a perturbed physics-parameter approach would probably further enhance the ensemble spread and hence may improve the CGEPS’s skill in these metrics, so long as the diversity does not involve bringing in models of such poor quality that their performance degrades the system. In this setting, physical experimentation can easily be justified within an operationally oriented computing workflow, an auspicious circumstance for sustained efforts by which the National Monsoon Mission’s goals can drive improvements in tropical physics for these much-used global models.
Acknowledgments
IITM is fully supported by the Ministry of Earth Sciences, government of India, New Delhi. SS thanks the Council for Scientific and Industrial Research (CSIR), New Delhi for a research fellowship. BEM gratefully acknowledges National Monsoon Mission financial support given by the Earth System Science Organization, Ministry of Earth Sciences, government of India (Grant/Project MM/SERP/Univ_Miami_USA/2013/INT-1/002).
REFERENCES
Abhilash, S., A. K. Sahai, N. Borah, R. Chattopadhyay, S. Joseph, S. Sharmila, S. De, and B. N. Goswami, 2013: Does bias correction in the forecasted SST improve the extended range prediction skill of active-break spells of Indian summer monsoon rainfall? Atmos. Sci. Lett., 15, 114–119, doi:10.1002/asl2.477.
Abhilash, S., A. K. Sahai, S. Pattnaik, B. N. Goswami, and A. Kumar, 2014a: Extended range prediction of active-break spells of Indian summer monsoon rainfall using an ensemble prediction system in NCEP Climate Forecast System. Int. J. Climatol., 34, 98–113, doi:10.1002/joc.3668.
Abhilash, S., and Coauthors, 2014b: Prediction and monitoring of monsoon intraseasonal oscillations over Indian monsoon region in an ensemble prediction system using CFSv2. Climate Dyn., 42, 2801–2815, doi:10.1007/s00382-013-2045-9.
Borah, N., S. Abhilash, S. Joseph, R. Chattopadhyay, S. Sharmila, and A. K. Sahai, 2013: Development of extended range prediction system using CFSv2 and its verification. IITM Research Rep. RR-130, 62 pp. [Available online at http://www.tropmet.res.in/~lip/Publication/Scientific-Reports/SR-16.pdf.]
Borah, N., A. K. Sahai, S. Abhilash, R. Chattopadhyay, S. Joseph, S. Sharmila, and A. Kumar, 2015: An assessment of real-time extended range forecast of 2013 Indian summer monsoon. Int. J. Climatol., doi:10.1002/joc.4178, in press.
Buizza, R., and T. N. Palmer, 1995: The singular-vector structure of the atmospheric global circulation. J. Atmos. Sci., 52, 1434–1456, doi:10.1175/1520-0469(1995)052<1434:TSVSOT>2.0.CO;2.
Buizza, R., M. Miller, and T. N. Palmer, 1999: Stochastic representation of model uncertainties in the ECMWF Ensemble Prediction System. Quart. J. Roy. Meteor. Soc., 125, 2887–2908, doi:10.1002/qj.49712556006.
Buizza, R., M. Leutbecher, and L. Isaksen, 2008: Potential use of an ensemble of analyses in the ECMWF Ensemble Prediction System. Quart. J. Roy. Meteor. Soc., 134, 2051–2066, doi:10.1002/qj.346.
Bush, S. J., A. G. Turner, S. J. Woolnough, G. M. Martin, and N. P. Klingaman, 2014: The effect of increased convective entrainment on Asian monsoon biases in the MetUM general circulation model. Quart. J. Roy. Meteor. Soc., 141, 311–326, doi:10.1002/qj.2371.
Gottschalck, J., and Coauthors, 2010: A framework for assessing operational Madden–Julian oscillation forecasts: A CLIVAR MJO working group project. Bull. Amer. Meteor. Soc., 91, 1247–1258, doi:10.1175/2010BAMS2816.1.
Griffies, S. M., M. J. Harrison, R. C. Pacanowski, and A. Rosati, 2004: A technical guide to MOM4. GFDL Ocean Group Tech. Rep. 5, NOAA/Geophysical Fluid Dynamics Laboratory, Princeton, NJ, 342 pp.
Houtekamer, P. L., L. Lefaivre, J. Derome, H. Ritchie, and H. L. Mitchel, 1996: A system simulation approach to ensemble prediction. Mon. Wea. Rev., 124, 1225–1242, doi:10.1175/1520-0493(1996)124<1225:ASSATE>2.0.CO;2.
Kharin, V. V., and F. W. Zwiers, 2003: On the ROC score of probability forecasts. J. Climate, 16, 4145–4150, doi:10.1175/1520-0442(2003)016<4145:OTRSOP>2.0.CO;2.
Kim, H.-M., P. J. Webster, and J. A. Curry, 2012: Seasonal prediction skill of ECMWF system 4 and NCEP CFSv2 retrospective forecast for the Northern Hemisphere winter. Climate Dyn., 39, 2957–2973, doi:10.1007/s00382-012-1364-6.
Kim, H.-M., P. J. Webster, V. E. Toma, and D. Kim, 2014: Predictability and prediction skill of the MJO in two operational forecasting systems. J. Climate, 27, 5364–5378, doi:10.1175/JCLI-D-13-00480.1.
Krishnamurti, T. N., and Coauthors, 1999: Improved weather and seasonal climate forecasts from multimodel superensemble. Science, 285, 1548–1550, doi:10.1126/science.285.5433.1548.
Lin, H., G. Brunet, and J. Derome, 2008: Forecast skill of the Madden–Julian oscillation in two Canadian atmospheric models. Mon. Wea. Rev., 136, 4130–4149, doi:10.1175/2008MWR2459.1.
Mitra, A. K., A. K. Bohra, M. N. Rajeevan, and T. N. Krishnamurti, 2009: Daily Indian precipitation analysis formed from a merge of rain-gauge data with the TRMM TMPA satellite-derived rainfall estimates. J. Meteor. Soc. Japan, 87A, 265–279, doi:10.2151/jmsj.87A.265.
Palmer, T. N., 2001: A nonlinear dynamical perspective on model error: A proposal for non-local stochastic-dynamic parametrization in weather and climate prediction models. Quart. J. Roy. Meteor. Soc., 127, 279–304, doi:10.1002/qj.49712757202.
Prasad, V. S., S. Mohandas, S. K. Dutta, M. Das Gupta, G. R. Iyengar, E. N. Rajagopal, and S. Basu, 2014: Improvements in medium range weather forecasting system of India. J. Earth Syst. Sci., 123, 247–258, doi:10.1007/s12040-014-0404-5.
Raftery, A. E., T. Gneiting, F. Balabdaoui, and M. Polakowski, 2005: Using Bayesian model averaging to calibrate forecast ensembles. Mon. Wea. Rev., 133, 1155–1174, doi:10.1175/MWR2906.1.
Rajeevan, M., S. Gadgil, and J. Bhate, 2010: Active and break spells of the Indian summer monsoon. J. Earth Syst. Sci., 119, 229–247, doi:10.1007/s12040-010-0019-4.
Rashid, H. A., H. H. Hendon, M. C. Wheeler, and O. Alves, 2011: Prediction of the Madden–Julian oscillation with the POAMA dynamical prediction system. Climate Dyn., 36, 649–661, doi:10.1007/s00382-010-0754-x.
Saha, S., and Coauthors, 2010: The NCEP Climate Forecast System Reanalysis. Bull. Amer. Meteor. Soc., 91, 1015–1057, doi:10.1175/2010BAMS3001.1.
Saha, S., and Coauthors, 2014: The NCEP Climate Forecast System version 2. J. Climate, 27, 2185–2208, doi:10.1175/JCLI-D-12-00823.1.
Sahai, A. K., R. Chattopadhyay, and B. N. Goswami, 2008: A SST based large multi-model ensemble forecasting system for Indian summer monsoon rainfall. Geophys. Res. Lett., 35, L19705, doi:10.1029/2008GL035461.
Sahai, A. K., and Coauthors, 2013: Simulation and extended range prediction of monsoon intraseasonal oscillations in NCEP CFS/GFS version 2 framework. Curr. Sci., 104, 1398–1408.
Sahai, A. K., S. Abhilash, R. Chattopadhyay, N. Borah, S. Joseph, S. Sharmila, and M. Rajeevan, 2015: High-resolution operational monsoon forecasts: An objective assessment. Climate Dyn., 44, 3129–3140, doi:10.1007/s00382-014-2210-9.
Sperber, K. R., H. Annamalai, I. S. Kang, A. Kitoh, A. Moise, A. G. Turner, B. Wang, and T. Zhou, 2013: The Asian summer monsoon: An intercomparison of CMIP5 versus CMIP3 simulations of the late 20th century. Climate Dyn., 41, 2711–2744, doi:10.1007/s00382-012-1607-6.
Suhas, E., J. M. Neena, and B. N. Goswami, 2012: An Indian monsoon intraseasonal oscillations (MISO) index for real time monitoring and forecast verification. Climate Dyn., 40, 2605–2616, doi:10.1007/s00382-012-1462-5.
Van den Dool, H. M., and Z. Toth, 1991: Why do forecasts for “near normal” often fail? Wea. Forecasting, 6, 76–85, doi:10.1175/1520-0434(1991)006<0076:WDFFNO>2.0.CO;2.
Weigel, A. P., M. A. Liniger, and C. Appenzeller, 2008: Can multi-model combination really enhance the prediction skill of probabilistic ensemble forecasts? Quart. J. Roy. Meteor. Soc., 134, 241–260, doi:10.1002/qj.210.
Wheeler, M. C., and H. H. Hendon, 2004: An all-season real-time multivariate MJO index: Development of an index for monitoring and prediction. Mon. Wea. Rev., 132, 1917–1932, doi:10.1175/1520-0493(2004)132<1917:AARMMI>2.0.CO;2.
Wilks, D. S., 2005: Statistical Methods in the Atmospheric Sciences. International Geophysical Series, Vol. 91, Academic Press, 627 pp.
As of 2014. Version numbering will soon be instituted in the CFS.
