• Barnston, A. G., , and T. M. Smith, 1996: Specification and prediction of global surface temperature and precipitation from global SST using CCA. J. Climate, 9 , 26602696.

    • Search Google Scholar
    • Export Citation
  • Barnston, A. G., and Coauthors, 1994: Long-lead seasonal forecasts—Where do we stand? Bull. Amer. Meteor. Soc., 75 , 20972114.

  • Bretherton, C. S., , M. Widman, , V. P. Dymnikov, , J. M. Wallace, , and I. Blade, 1999: The effective number of spatial degrees of freedom of a time-varying field. J. Climate, 12 , 19902009.

    • Search Google Scholar
    • Export Citation
  • Chelton, D. B., 1983: Effects of sampling errors in statistical estimation. Deep-Sea Res., 30 , 10831103.

  • Davis, R., 1977: Techniques for statistical analysis and prediction of geophysical fluid systems. Geophys. Astrophys. Fluid Dyn., 8 , 245277.

    • Search Google Scholar
    • Export Citation
  • DelSole, T., 2001: Optimally persistent patterns in time-varying fields. J. Atmos. Sci., 58 , 13411356.

  • DelSole, T., 2007: A Bayesian framework for multimodel regression. J. Climate, 20 , 28102826.

  • DelSole, T., , and J. Shukla, 2002: Linear prediction of Indian monsoon rainfall. J. Climate, 15 , 36453658.

  • Gadgil, S., , M. Rajeevan, , and R. Nanjndiah, 2005: Monsoon prediction–Why yet another failure? Curr. Sci., 88 , 13891400.

  • Gowariker, V., , V. Thapliyal, , R. P. Sarker, , G. S. Mandal, , and D. R. Sikka, 1989: Parametric and power regression models: New approach to long range forecasting of monsoon rainfall in India. Mausam, 40 , 115122.

    • Search Google Scholar
    • Export Citation
  • Gowariker, V., , V. Thapliyal, , S. M. Kulshrestha, , G. S. Mandal, , N. Sen Roy, , and D. R. Sikka, 1991: A power regression model for long range forecast of southwest monsoon rainfall over India. Mausam, 42 , 125130.

    • Search Google Scholar
    • Export Citation
  • Hastie, T., , R. Tibshirani, , and J. Friedman, 2001: The Elements of Statistical Learning. Springer-Verlag, 552 pp.

  • Jagannathan, P., 1960: Seasonal forecasting in India: A review. FMU:1-80, India Meteorological Department, Pune, India, 120 pp.

  • Kharin, V. V., , and F. W. Zwiers, 2002: Climate predictions with multimodel ensembles. J. Climate, 15 , 793799.

  • Klotzbach, P. J., 2007: Revised prediction of seasonal Atlantic basin tropical cyclone activity from 1 August. Wea. Forecasting, 22 , 937949.

    • Search Google Scholar
    • Export Citation
  • Klotzbach, P. J., , and W. M. Gray, 2003: Forecasting September Atlantic basin tropical cyclone activity. Wea. Forecasting, 18 , 11091128.

    • Search Google Scholar
    • Export Citation
  • Klotzbach, P. J., , and W. M. Gray, 2004: Updated 6–11-month prediction of Atlantic basin seasonal hurricane activity. Wea. Forecasting, 19 , 917934.

    • Search Google Scholar
    • Export Citation
  • Lanzante, J. R., 1984: Strategies for assessing skill and significance of screening regression models with emphasis on Monte Carlo techniques. J. Climate Appl. Meteor., 23 , 14541458.

    • Search Google Scholar
    • Export Citation
  • Lawley, N. D., 1956: Tests of significance for the latent roots of covariance and correlation matrices. Biometrika, 43 , 128136.

  • Michaelson, J., 1987: Cross-validation in statistical climate forecast models. J. Climate Appl. Meteor., 26 , 15891600.

  • Montgomery, R. B., 1940: Report on the work of G. T. Walker. Mon. Wea. Rev., 68 , (Suppl. 39). 126.

  • North, G. R., , F. J. Moeng, , T. L. Bell, , and R. F. Cahalan, 1982: The latitude dependence of the variance of zonally averaged quantities. Mon. Wea. Rev., 110 , 319326.

    • Search Google Scholar
    • Export Citation
  • Penland, C., , and T. Magorian, 1993: Prediction of Niño 3 sea surface temperatures using linear inverse modeling. J. Climate, 6 , 10671076.

    • Search Google Scholar
    • Export Citation
  • Rajeevan, M., , D. S. Pai, , R. Anil Kumar, , and B. Lal, 2007: New statistical models for long-range forecasting of southwest monsoon rainfall over India. Climate Dyn., 28 , 813828.

    • Search Google Scholar
    • Export Citation
  • Shapiro, L. J., 1984: Sampling errors in statistical models of tropical cyclone motion: A comparison of predictor screening and EOF techniques. Mon. Wea. Rev., 112 , 13781388.

    • Search Google Scholar
    • Export Citation
  • Shapiro, L. J., , and D. B. Chelton, 1986: Comments on “Strategies for assessing skill and significance of screening regression models with emphasis on Monte Carlo techniques.”. J. Climate Appl. Meteor., 25 , 12951298.

    • Search Google Scholar
    • Export Citation
  • Shukla, J., , and J. L. Kinter III, 2006: Predictability of seasonal climate variations: A pedagogical review. Predictability in Weather and Climate. T. Palmer and R. Hagedorn, Eds., Cambridge University Press, 306–341.

    • Search Google Scholar
    • Export Citation
  • Stefanick, M., 1981: Space and time scales of atmospheric variability. J. Atmos. Sci., 38 , 9881002.

  • Stone, M., 1974: Cross-validatory choice and assessment of statistical predictions. J. Roy. Stat. Soc. Ser. A, 36 , 111147.

  • Thapliyal, V., , and S. M. Kulshrestha, 1992: Recent models for long range forecasting of southwest monsoon rainfall in India. Mausam, 43 , 239248.

    • Search Google Scholar
    • Export Citation
  • Van den Dool, H., 2007: Empirical Methods in Short-Term Climate Prediction. Oxford University Press, 215 pp.

  • Walker, G. T., 1914: Correlation in seasonal variation of weather III: On the criterion for the reality of relationship or periodicities. Mem. India Meteor. Dep., 21 , 1215.

    • Search Google Scholar
    • Export Citation
  • Walker, G. T., 1922: Correlation in seasonal variation of weather VII: The local distribution of monsoon rainfall. Mem. India Meteor. Dep., 23 , 2339.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 81 81 11
PDF Downloads 47 47 8

Artificial Skill due to Predictor Screening

View More View Less
  • 1 George Mason University, Fairfax, Virginia, and Center for Ocean–Land–Atmosphere Studies, Calverton, Maryland
© Get Permissions
Restricted access

Abstract

This paper shows that if predictors are selected preferentially because of their strong correlation with a prediction variable, then standard methods for validating prediction models derived from these predictors will be biased. This bias is demonstrated by screening random numbers and showing that regression models derived from these random numbers have apparent skill, in a cross-validation sense, even though the predictors cannot possibly have the slightest predictive usefulness. This result seemingly implies that random numbers can give useful predictions, since the sample being predicted is separate from the sample used to estimate the regression model. The resolution of this paradox is that, prior to cross validation, all of the data had been used to evaluate correlations for selecting predictors. This situation differs from real-time forecasts in that the future sample is not available for screening. These results clarify the fallacy in assuming that if a model performs well in cross-validation mode, then it will perform well in real-time forecasts. This bias appears to afflict several forecast schemes that have been proposed in the literature, including operational forecasts of Indian monsoon rainfall and number of Atlantic hurricanes. The cross-validated skill of these models probably would not be distinguishable from that of a no-skill model if prior screening were taken into account.

Corresponding author address: Timothy DelSole, 4041 Powder Mill Rd., Suite 302, Calverton, MD 20705-3106. Email: delsole@cola.iges.org

Abstract

This paper shows that if predictors are selected preferentially because of their strong correlation with a prediction variable, then standard methods for validating prediction models derived from these predictors will be biased. This bias is demonstrated by screening random numbers and showing that regression models derived from these random numbers have apparent skill, in a cross-validation sense, even though the predictors cannot possibly have the slightest predictive usefulness. This result seemingly implies that random numbers can give useful predictions, since the sample being predicted is separate from the sample used to estimate the regression model. The resolution of this paradox is that, prior to cross validation, all of the data had been used to evaluate correlations for selecting predictors. This situation differs from real-time forecasts in that the future sample is not available for screening. These results clarify the fallacy in assuming that if a model performs well in cross-validation mode, then it will perform well in real-time forecasts. This bias appears to afflict several forecast schemes that have been proposed in the literature, including operational forecasts of Indian monsoon rainfall and number of Atlantic hurricanes. The cross-validated skill of these models probably would not be distinguishable from that of a no-skill model if prior screening were taken into account.

Corresponding author address: Timothy DelSole, 4041 Powder Mill Rd., Suite 302, Calverton, MD 20705-3106. Email: delsole@cola.iges.org

Save