• Arcones, M. A., and E. Giné, 1989: The bootstrap of the mean with arbitrary bootstrap sample. Ann. Inst. Henri Poincaré, 25, 457481.

    • Search Google Scholar
    • Export Citation
  • Arcones, M. A., and E. Giné, 1991: Additions and corrections to “The bootstrap of the mean with arbitrary bootstrap sample.” Ann. Inst. Henri Poincaré, 27, 583595.

    • Search Google Scholar
    • Export Citation
  • Athreya, K. B., 1987a: Bootstrap of the mean in the infinite variance case. Proc. First World Congress of the Bernoulli Society, Utrecht, Netherlands, Bernoulli Society, 95–98.

  • Athreya, K. B., 1987b: Bootstrap of the mean in the infinite variance case. Ann. Stat., 15, 724731, https://doi.org/10.1214/aos/1176350371.

  • Azzalini, A., and A. Genz, 2016: mnormt: The Multivariate Normal and t Distributions, version 1.5-5. R package, https://azzalini.stat.unipd.it/SW/Pkg-mnormt.

    • Crossref
    • Export Citation
  • Bickel, P. J., and D. A. Freedman, 1981: Some asymptotic theory for the bootstrap. Ann. Stat., 9, 11961217, https://doi.org/10.1214/aos/1176345637.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bickel, P. J., F. Götze, and W. R. van Zwet, 1997: Resampling fewer than n observations: Gains, losses, and remedies for losses. Stat. Sin., 7, 131.

    • Search Google Scholar
    • Export Citation
  • Brockwell, P. J., and R. A. Davis, 1996: Introduction to Time Series and Forecasting. Springer, 420 pp.

    • Crossref
    • Export Citation
  • Canty, A., and B. Ripley, 2017: boot: Bootstrap R (S-Plus) Functions, version 1.3-20. R package, http://statwww.epfl.ch/davison/BMA/.

  • Carpenter, J., 1999: Test inversion bootstrap confidence intervals. J. Roy. Stat. Soc., 61B, 159172, https://doi.org/10.1111/1467-9868.00169.

  • Carpenter, J., and J. Bithell, 2000: Bootstrap confidence intervals: When, which, what? A practical guide for medical statisticians. Stat. Med., 19, 11411164, https://doi.org/10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Davison, A., and D. Hinkley, 1997: Bootstrap Methods and Their Application. Cambridge University Press, 582 pp.

    • Crossref
    • Export Citation
  • Deheuvels, P., D. M. Mason, and G. R. Shorack, 1993: Some results on the influence of extremes on the bootstrap. Ann. Inst. Henri Poincaré Probab. Stat., 29, 83103.

    • Search Google Scholar
    • Export Citation
  • DelSole, T., and M. K. Tippett, 2014: Comparing forecast skill. Mon. Wea. Rev., 142, 46584678, https://doi.org/10.1175/MWR-D-14-00045.1.

  • Efron, B., 1979: Bootstrap methods: Another look at the jackknife. Ann. Stat., 7, 126, https://doi.org/10.1214/aos/1176344552.

  • Efron, B., and R. Tibshirani, 1993: An Introduction to the Bootstrap. Vol. 57. Chapman and Hall, 436 pp.

    • Crossref
    • Export Citation
  • Feigin, P., and S. I. Resnick, 1997: Linear programming estimators and bootstrapping for heavy-tailed phenomena. Adv. Appl. Probab., 29, 759805, https://doi.org/10.2307/1428085.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Fukuchi, J.-I., 1994: Bootstrapping extremes of random variables. Ph.D. thesis, Dept. of Statistics, Iowa State University, 96 pp., https://doi.org/10.31274/rtd-180813-10322.

    • Crossref
    • Export Citation
  • Garthwaite, P. H., and S. T. Buckland, 1992: Generating Monte Carlo confidence intervals by the Robbins-Monro process. Appl. Stat., 41, 159171, https://doi.org/10.2307/2347625.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gilleland, E., 2010: Confidence intervals for forecast verification. NCAR Tech. Rep. NCAR/TN-479+STR, 71 pp.

  • Gilleland, E., 2013: Testing competing precipitation forecasts accurately and efficiently: The spatial prediction comparison test. Mon. Wea. Rev., 141, 340355, https://doi.org/10.1175/MWR-D-12-00155.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gilleland, E., 2017: distillery: Method Functions for Confidence Intervals and to Distill Information from an Object, version 1.0-4. R package, https://www.ral.ucar.edu/staff/ericg.

  • Gilleland, E., A. S. Hering, T. L. Fowler, and B. G. Brown, 2018: Testing the tests: What are the impacts of incorrect assumptions when applying confidence intervals or hypothesis tests to compare competing forecasts? Mon. Wea. Rev., 146, 16851703, https://doi.org/10.1175/MWR-D-17-0295.1.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Giné, E., and J. Zinn, 1989: Necessary conditions for the bootstrap of the mean. Ann. Stat., 17, 684691, https://doi.org/10.1214/aos/1176347134.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hall, P., 1990: Asymptotic properties of the bootstrap for heavy-tailed distributions. Ann. Probab., 18, 13421360, https://doi.org/10.1214/aop/1176990748.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hamill, T. M., 1999: Hypothesis tests for evaluating numerical precipitation forecasts. Wea. Forecasting, 14, 155167, https://doi.org/10.1175/1520-0434(1999)014<0155:HTFENP>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hamilton, J. D., 1994: Time Series Analysis. Princeton University Press, 799 pp.

    • Crossref
    • Export Citation
  • Hannig, J., 2009: On generalized fiducial inference. Stat. Sin., 19, 491544.

  • Hannig, J., H. K. Iyer, and P. Patterson, 2006: Fiducial generalized confidence intervals. J. Amer. Stat. Assoc., 101, 254269, https://doi.org/10.1198/016214505000000736.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hannig, J., H. K. Iyer, and C. M. Wang, 2007: Fiducial approach to uncertainty assessment accounting for error due to instrument resolution. Meteorologia, 44, 476483, https://doi.org/10.1088/0026-1394/44/6/006.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Hering, A. S., and M. G. Genton, 2011: Comparing spatial predictions. Technometrics, 53, 414425, https://doi.org/10.1198/TECH.2011.10136.

  • Jolliffe, I. T., 2007: Uncertainty and inference for verification measures. Wea. Forecasting, 22, 637650, https://doi.org/10.1175/WAF989.1.

  • Jolliffe, I. T., and D. B. Stephenson, 2003: Forecast Verification: A Practitioner’s Guide in Atmospheric Science. Wiley, 254 pp.

  • Kabaila, P., 1993: Some properties of profile bootstrap confidence intervals. Aust. J. Stat., 35, 205214, https://doi.org/10.1111/j.1467-842X.1993.tb01326.x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kinateder, J. G., 1992: An invariance principle applicable to the bootstrap. Exploring the Limits of Bootstrap, R. Lepage and L. Billard, Eds., Wiley Series in Probability and Mathematical Statistics, Wiley, 157–181.

  • Knight, K., 1989: On the bootstrap of the sample mean in the infinite variance case. Ann. Stat., 17, 11681175, https://doi.org/10.1214/aos/1176347262.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Lahiri, S. N., 2003: Resampling Methods for Dependent Data. Springer, 374 pp.

    • Crossref
    • Export Citation
  • Lee, S., 1999: On a class of m out of n bootstrap confidence intervals. J. Roy. Stat. Soc., 61B, 901911, https://doi.org/10.1111/1467-9868.00209.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • LePage, R., 1992: Bootstrapping signs. Exploring the Limits of Bootstrap, R. Lepage and L. Billard, Eds., Wiley Series in Probability and Mathematical Statistics, Wiley, 215–224.

  • Lidong, E., J., Hannig, and H. Iyer, 2008: Fiducial intervals for variance components in an unbalanced two-component normal mixed linear model. J. Amer. Stat. Assoc., 103, 854865, https://doi.org/10.1198/016214508000000229.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Liu, Y., and et al. , 2009: An operational mesoscale ensemble data assimilation and prediction system: E-RTFDDA—System design and verification. 19th Numerical Weather Prediction Conf./23rd Weather Forecasting Conf., Omaha, NE, Amer. Meteor. Soc., 16A.4, https://ams.confex.com/ams/23WAF19NWP/techprogram/paper_154271.htm.

  • Mood, A. M., F. A. Graybill, and D. C. Boes, 1963: Introduction to the Theory of Statistics. 3rd ed. McGraw Hill, 564 pp.

  • Okkan, U., and U. Kirdemir, 2018: Investigation of the behavior of an agricultural-operated dam reservoir under RCP scenarios of AR5-IPCC. Water Resour. Manage., 32, 28472866, https://doi.org/10.1007/s11269-018-1962-0.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • R Core Team, 2017: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, https://www.R-project.org/.

  • Resnick, S. I., 2007: Heavy-Tail Phenomena: Probabilistic and Statistical Modeling. Springer Series in Operations Research and Financial Engineering, Springer, 404 pp.

  • Ripley, B. D., 2002: Time series in R 1.5.0. R News, No. 2(2), R Foundation, Vienna, Austria, 2–7, https://www.r-project.org/doc/Rnews/Rnews_2002-2.pdf.

  • Robbins, H., and S. Monro, 1951: A stochastic approximation method. Ann. Math. Stat., 22, 400407, https://doi.org/10.1214/aoms/1177729586.

  • Rubin, D. B., 1981: The Bayesian bootstrap. Ann. Stat., 9, 130134, https://doi.org/10.1214/aos/1176345338.

  • Schendel, T., and R. Thongwichian, 2015: Flood frequency analysis: Confidence interval estimation by test inversion bootstrapping. Adv. Water Resour., 83, 19, https://doi.org/10.1016/j.advwatres.2015.05.004.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Schendel, T., and R. Thongwichian, 2017a: Confidence intervals for return levels for the peaks-over-threshold approach. Adv. Water Resour., 99, 5359, https://doi.org/10.1016/j.advwatres.2016.11.011.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Schendel, T., and R. Thongwichian, 2017b: Considering historical flood events in flood frequency analysis: Is it worth the effort? Adv. Water Resour., 105, 144153, https://doi.org/10.1016/j.advwatres.2017.05.002.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Shao, J., and T. Dongsheng, 1995: The Jackknife and the Bootstrap. Springer-Verlag, 123 pp.

    • Crossref
    • Export Citation
  • von Storch, H., and F. W. Zwiers, 2001: Statistical Analysis in Climate Research. 1st ed. Cambridge University Press, 484 pp.

  • Wandler, D., and J. Hannig, 2012: Generalized fiducial confidence intervals for extremes. Extremes, 15, 6787, https://doi.org/10.1007/s10687-011-0127-9.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wasserman, L., 2005: All of Statistics: A Concise Course in Statistical Inference. Springer, 442 pp.

    • Crossref
    • Export Citation
  • Wilks, D. S., 1997: Resampling hypothesis tests for autocorrelated fields. J. Climate, 10, 6582, https://doi.org/10.1175/1520-0442(1997)010<0065:RHTFAF>2.0.CO;2.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2006: Statistical Methods in the Atmospheric Sciences. An Introduction. 2nd ed. Academic Press, 627 pp.

All Time Past Year Past 30 Days
Abstract Views 297 297 11
Full Text Views 61 61 7
PDF Downloads 75 75 8

Bootstrap Methods for Statistical Inference. Part I: Comparative Forecast Verification for Continuous Variables

View More View Less
  • 1 National Center for Atmospheric Research, Boulder, Colorado
© Get Permissions Rent on DeepDyve
Restricted access

Abstract

When making statistical inferences, bootstrap resampling methods are often appealing because of less stringent assumptions about the distribution of the statistic(s) of interest. However, the procedures are not free of assumptions. This paper addresses a specific situation that occurs frequently in atmospheric sciences where the standard bootstrap is not appropriate: comparative forecast verification of continuous variables. In this setting, the question to be answered concerns which of two weather or climate models is better in the sense of some type of average deviation from observations. The series to be compared are generally strongly dependent, which invalidates the most basic bootstrap technique. This paper also introduces new bootstrap code from the R package “distillery” that facilitates easy implementation of appropriate methods for paired-difference-of-means bootstrap procedures for dependent data.

Supplemental information related to this paper is available at the Journals Online website: https://doi.org/10.1175/JTECH-D-20-0069.s1.

© 2020 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Eric Gilleland, ericg@ucar.edu

This article has a companion article which can be found at http://journals.ametsoc.org/doi/abs/10.1175/JTECH-D-20-0070.1.

Abstract

When making statistical inferences, bootstrap resampling methods are often appealing because of less stringent assumptions about the distribution of the statistic(s) of interest. However, the procedures are not free of assumptions. This paper addresses a specific situation that occurs frequently in atmospheric sciences where the standard bootstrap is not appropriate: comparative forecast verification of continuous variables. In this setting, the question to be answered concerns which of two weather or climate models is better in the sense of some type of average deviation from observations. The series to be compared are generally strongly dependent, which invalidates the most basic bootstrap technique. This paper also introduces new bootstrap code from the R package “distillery” that facilitates easy implementation of appropriate methods for paired-difference-of-means bootstrap procedures for dependent data.

Supplemental information related to this paper is available at the Journals Online website: https://doi.org/10.1175/JTECH-D-20-0069.s1.

© 2020 American Meteorological Society. For information regarding reuse of this content and general copyright information, consult the AMS Copyright Policy (www.ametsoc.org/PUBSReuseLicenses).

Corresponding author: Eric Gilleland, ericg@ucar.edu

This article has a companion article which can be found at http://journals.ametsoc.org/doi/abs/10.1175/JTECH-D-20-0070.1.

Supplementary Materials

    • Supplemental Materials (PDF 117.74 KB)
Save