A Single-Sample Estimate of Shrinkage in Meteorological Forecasting

Paul W. Mielke Jr. Department of Statistics, Colorado State University, Fort Collins, Colorado

Search for other papers by Paul W. Mielke Jr. in
Current site
Google Scholar
PubMed
Close
,
Kenneth J. Berry Department of Sociology, Colorado State University, Fort Collins, Colorado

Search for other papers by Kenneth J. Berry in
Current site
Google Scholar
PubMed
Close
,
Christopher W. Landsea NOAA Climate and Global Change Fellowship, NOAA/AOML/Hurricane Research Division, Miami, Florida

Search for other papers by Christopher W. Landsea in
Current site
Google Scholar
PubMed
Close
, and
William M. Gray Department of Atmospheric Science, Colorado State University, Fort Collins, Colorado

Search for other papers by William M. Gray in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

An estimator of shrinkage based on information contained in a single sample is presented and the results of a simulation study are reported. The effects of sample size, amount, and severity of nonrepresentative data in the population, inclusion of noninformative predictors, and least (sum of) absolute deviations and least (sum of) squared deviations regression models are examined on the estimator. A single-sample estimator of shrinkage based on drop-one cross-validation is shown to be highly accurate under a wide variety of research conditions.

* Current affiliation: NOAA/AOML/Hurricane Research Division, Miami, Florida.

Corresponding author address: Dr. Paul W. Mielke Jr., Department of Statistics, Colorado State University, Fort Collins, CO 80523-1877.

Abstract

An estimator of shrinkage based on information contained in a single sample is presented and the results of a simulation study are reported. The effects of sample size, amount, and severity of nonrepresentative data in the population, inclusion of noninformative predictors, and least (sum of) absolute deviations and least (sum of) squared deviations regression models are examined on the estimator. A single-sample estimator of shrinkage based on drop-one cross-validation is shown to be highly accurate under a wide variety of research conditions.

* Current affiliation: NOAA/AOML/Hurricane Research Division, Miami, Florida.

Corresponding author address: Dr. Paul W. Mielke Jr., Department of Statistics, Colorado State University, Fort Collins, CO 80523-1877.

Save
  • Badescu, V., 1993: Use of Willmott’s index of agreement to the validation of meteorological models. Meteor. Mag.,122, 282–286.

  • Barnston, A. G., and H. M. Van den Dool, 1993: A degeneracy in cross-validated skill in regression-based forecasts. J. Climate,6, 963–977.

    • Crossref
    • Export Citation
  • Browne, M. W., 1970: A critical evaluation of some reduced-rank regression procedures. Research Bulletin 70-21, Educational Testing Service, Princeton, NJ.

    • Crossref
    • Export Citation
  • ——, 1975a: Predictive validity of a linear regression equation. Br. J. Math. Statist. Psychol.,28, 79–87.

    • Crossref
    • Export Citation
  • ——, 1975b: A comparison of single sample and cross-validation methods for estimating the mean squared error of prediction in multiple linear regression. Br. J. Math. Statist. Psychol.,28, 112–120.

    • Crossref
    • Export Citation
  • ——, and R. Cudeck, 1989: Single sample cross-validation indices for covariance structures. Mult. Behav. Res.,24, 445–455.

    • Crossref
    • Export Citation
  • ——, and ——, 1992: Alternative ways of assessing model fit. Sociol. Meth. Res.,21, 230–258.

    • Crossref
    • Export Citation
  • Camstra, A., and A. Boomsma, 1992: Cross-validation in regression and covariance structure analysis. Soc. Meth. Res.,21, 89–115.

    • Crossref
    • Export Citation
  • Copas, J. B., 1983: Regression, prediction, and shrinkage. J. Roy. Statist. Soc.,45B, 311–354.

    • Crossref
    • Export Citation
  • Cotton, W. R., G. Thompson, and P. W. Mielke, 1994: Real-time mesoscale prediction on workstations. Bull. Amer. Meteor. Soc.,75, 349–362.

    • Crossref
    • Export Citation
  • Efron, B., 1983: Estimating the error rate of a prediction rule: Improvement on cross-validation. J. Amer. Statist. Assoc.,78, 316–331.

    • Crossref
    • Export Citation
  • Elsner, J. B., and C. P. Schmertmann, 1993: Improving extended- range seasonal predictions of intense Atlantic hurricane activity. Wea. Forecasting,8, 345–351.

  • ——, and ——, 1994: Assessing forecast skill through cross-validation. Wea. Forecasting,9, 619–624.

    • Crossref
    • Export Citation
  • Geisser, S., 1975: The predictive sample reuse method with applications. J. Amer. Statist. Assoc.,70, 320–328.

    • Crossref
    • Export Citation
  • Glick, N., 1978: Additive estimators for probabilities of correct classification. Pattern Recog.,10, 211–222.

  • Gray, W. M., C. W. Landsea, P. W. Mielke, and K. J. Berry, 1992: Predicting Atlantic seasonal hurricane activity 6–11 months in advance. Wea. Forecasting,7, 440–455.

  • Hess, J. C., and J. B. Elsner, 1994: Extended-range hindcasts of tropical-origin Atlantic hurricane activity. Geophys. Res. Lett.,21, 365–368.

    • Crossref
    • Export Citation
  • Hora, S. C., and J. B. Wilcox, 1982: Estimation of error rates in several-population discriminant analysis. J. Marketing Res.,19, 57–61.

    • Crossref
    • Export Citation
  • Horst, P., 1966: Psychological Measurement and Prediction. Wadsworth, 455 pp.

  • Huberty, C. J., J. M. Wisenbaker, and J. C. Smith, 1987: Assessing predictive accuracy in discriminant analysis. Mult. Behav. Res.,22, 307–329.

    • Crossref
    • Export Citation
  • Kelly, F. P., T. H. Vonder Haar, and P. W. Mielke, 1989: Imagery randomized block analysis (IRBA) applied to the verification of cloud edge detectors. J. Atmos. Oceanic Technol.,6, 671–679.

    • Crossref
    • Export Citation
  • Lachenbruch, P. A., 1967: An almost unbiased method of obtaining confidence intervals for the probability of misclassification in discriminant analysis. Biometrics,23, 639–645.

    • Crossref
    • Export Citation
  • ——, and M. R. Mickey, 1968: Estimation of error rates in discriminant analysis. Technometrics,10, 1–11.

    • Crossref
    • Export Citation
  • Lee, T. J., R. A. Pielke, and P. W. Mielke, 1995: Modeling the clear- sky surface energy budget during FIFE 1987. J. Geophys. Res.,100, 25585–25593.

    • Crossref
    • Export Citation
  • Livezey, R. E., A. G. Barnston, and B. K. Neumeister, 1990: Mixed analog/persistence prediction of seasonal mean temperatures for the USA. Int. J. Climatol.,10, 329–340.

  • MacCallum, R. C., M. Roznowski, C. M. Mar, and J. V. Reith, 1994:Alternative strategies for cross-validation of covariance structure models. Mult. Behav. Res.,29, 1–32.

    • Crossref
    • Export Citation
  • Maltz, M. D., 1994: Deviating from the mean: The declining significance of significance. J. Res. Crime Delinq.,31, 434–463.

    • Crossref
    • Export Citation
  • McCabe, G. J., and D. R. Legates, 1992: General-circulation model simulations of winter and summer sea-level pressures over North America. Int. J. Climatol.,12, 815–827.

    • Crossref
    • Export Citation
  • Michaelsen, J., 1987: Cross-validation in statistical climate forecast models. J. Climate Appl. Meteor.,26, 1589–1600.

    • Crossref
    • Export Citation
  • Mielke, P. W., K. J. Berry, C. W. Landsea, and W. M. Gray, 1996: Artificial skill and validation in meteorological forecasting. Wea. Forecasting,11, 153–169.

    • Crossref
    • Export Citation
  • Mosier, C. I., 1951: Symposium: The need and means of cross-validation, I. Problems and designs of cross-validation. Educ. Psych. Meas.,11, 5–11.

    • Crossref
    • Export Citation
  • Mosteller, F., and J. W. Tukey, 1977: Data Analysis and Regression. Addison-Wesley, 586 pp.

  • Murphy, A. H., and R. L. Winkler, 1984: Probability forecasting in meteorology. J. Amer. Statist. Assoc.,79, 489–500.

    • Crossref
    • Export Citation
  • Nicholls, N., 1985: Predictability of interannual variations of Australian seasonal tropical cyclone activity. Mon. Wea. Rev.,113, 1144–1149.

    • Crossref
    • Export Citation
  • Picard, R. R., and R. D. Cook, 1984: Cross-validation of regression models. J. Amer. Statist. Assoc.,79, 575–583.

    • Crossref
    • Export Citation
  • ——, and K. N. Berk, 1990: Data splitting. Amer. Statist.,44, 140–147.

    • Crossref
    • Export Citation
  • Snee, R. D., 1977: Validation of regression models: Methods and examples. Technometrics,19, 415–428.

    • Crossref
    • Export Citation
  • Stone, M., 1974: Cross-validatory choice and assessment of statistical predictions. J. Roy. Statist. Soc.,36B, 111–147.

    • Crossref
    • Export Citation
  • ——, 1978: Cross-validation: A review. Math. Operationsforsch. Statist., Ser. Statistics,9, 127–139.

    • Crossref
    • Export Citation
  • Subrahmanyam, M., 1972: A property of simple least squares estimates. Sankhya,34B, 355–356.

  • Toussaint, G. T., 1974: Bibliography on estimation of missclassification. IEEE Trans. Inf. Theory,20, 472–479.

    • Crossref
    • Export Citation
  • Tucker, D. F., P. W. Mielke, and E. R. Reiter, 1989: The verification of numerical models with multivariate randomized block permutation procedures. Meteor. Atmos. Phys.,40, 181–188.

    • Crossref
    • Export Citation
  • Watterson, I. G., 1996: Nondimensional measures of climate model performance. Int. J. Climatol.,16, 379–391.

    • Crossref
    • Export Citation
  • Willmott, C. J., 1982: Some comments on the evaluation of model performance. Bull. Amer. Meteor. Soc.,63, 1309–1313.

    • Crossref
    • Export Citation
  • ——, S. G. Ackleson, R. E. Davis, J. J. Feddema, K. M. Klink, D. R. Legates, J. O’Donnell, and C. M. Rowe, 1985: Statistics for the evaluation and comparison of models. J. Geophys. Res.,90, 8995–9005.

    • Crossref
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 226 63 5
PDF Downloads 70 37 3