Significance Tests in Climate Science

Maarten H. P. Ambaum Department of Meteorology, University of Reading, Reading, United Kingdom

Search for other papers by Maarten H. P. Ambaum in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

A large fraction of papers in the climate literature includes erroneous uses of significance tests. A Bayesian analysis is presented to highlight the meaning of significance tests and why typical misuse occurs. The significance statistic is not a quantitative measure of how confident one can be of the “reality” of a given result. It is concluded that a significance test very rarely provides useful quantitative information.

Corresponding author address: Maarten H. P. Ambaum, Department of Meteorology, University of Reading, P.O. Box 243, Reading RG6 6BB, United Kingdom. Email: m.h.p.ambaum@reading.ac.uk

Abstract

A large fraction of papers in the climate literature includes erroneous uses of significance tests. A Bayesian analysis is presented to highlight the meaning of significance tests and why typical misuse occurs. The significance statistic is not a quantitative measure of how confident one can be of the “reality” of a given result. It is concluded that a significance test very rarely provides useful quantitative information.

Corresponding author address: Maarten H. P. Ambaum, Department of Meteorology, University of Reading, P.O. Box 243, Reading RG6 6BB, United Kingdom. Email: m.h.p.ambaum@reading.ac.uk

Save
  • Armstrong, J. S., 2007: Significance tests harm progress in forecasting. Int. J. Forecasting, 23 , 321327.

  • Cohen, J., 1994: The Earth is round (p < 0.05). Amer. Psychol., 49 , 9971003.

  • Cox, R. T., 1961: The Algebra of Probable Inference. Johns Hopkins Press, 144 pp.

  • Hunter, J. E., 1997: Needed: A ban on the significance test. Psychol. Sci., 8 , 37.

  • Jaynes, E. T., 1968: Prior probabilities. IEEE Trans. Syst. Sci. Cybern., 4 , 227241.

  • Jaynes, E. T., 2003: Probability Theory: The Logic of Science. Cambridge University Press, 727 pp.

  • Jolliffe, I. T., 2004: P stands for …. Weather, 59 , 7779.

  • Nicholls, N., 2001: The insignificance of significance testing. Bull. Amer. Meteor. Soc., 82 , 981986.

  • Ryle, G., 1949: The Concept of Mind. Hutchinson’s University Library, 334 pp.

  • Von Storch, H., and F. W. Zwiers, 1999: Statistical Analysis in Climate Research. Cambridge University Press, 484 pp.

  • Wilks, D. S., 1995: Statistical Methods in the Atmospheric Sciences: An Introduction. Academic Press, 467 pp.

  • Ziliak, S. T., and D. N. McCloskey, 2008: The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives. Economics, Cognition and Society Series, University of Michigan Press, 321 pp.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 2577 824 74
PDF Downloads 2091 592 48