Finite Samples and Uncertainty Estimates for Skill Measures for Seasonal Prediction

Arun Kumar NOAA Climate Prediction Center, Washington, D.C

Search for other papers by Arun Kumar in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

The expected value for various measures of skill for seasonal climate predictions is determined by the signal-to-noise ratio. The expected value, however, is only realized for long verification time series. In practice, the verifications for specific seasons—for example, forecasts for the December–February seasonal mean—seldom exceed a sample size of 30. The estimates of skill measure based on small verification time series, because of sampling errors, can have large departures from their expected value. An analysis of spread in the estimates of skill measures with the length of verification time series and for different signal-to-noise ratios is made. The analysis is based on the Monte Carlo approach and skill measures for deterministic, categorical, and probabilistic forecasts are considered. It is shown that the behavior of spread for various skill measures can be very different and it is not always the largest for the small values of signal-to-noise ratios.

Corresponding author address: Dr. Arun Kumar, NOAA/NWS/NCEP, Climate Prediction Center, 5200 Auth Rd., Rm. 800, Camp Springs, MD 20746. Email: arun.kumar@noaa.gov

Abstract

The expected value for various measures of skill for seasonal climate predictions is determined by the signal-to-noise ratio. The expected value, however, is only realized for long verification time series. In practice, the verifications for specific seasons—for example, forecasts for the December–February seasonal mean—seldom exceed a sample size of 30. The estimates of skill measure based on small verification time series, because of sampling errors, can have large departures from their expected value. An analysis of spread in the estimates of skill measures with the length of verification time series and for different signal-to-noise ratios is made. The analysis is based on the Monte Carlo approach and skill measures for deterministic, categorical, and probabilistic forecasts are considered. It is shown that the behavior of spread for various skill measures can be very different and it is not always the largest for the small values of signal-to-noise ratios.

Corresponding author address: Dr. Arun Kumar, NOAA/NWS/NCEP, Climate Prediction Center, 5200 Auth Rd., Rm. 800, Camp Springs, MD 20746. Email: arun.kumar@noaa.gov

Save
  • Anderson, T. W., 1984: An Introduction to Multivariate Statistical Analysis. 2nd ed. John Wiley and Sons, 675 pp.

  • Barnston, A. G., 1992: Correspondence among the correlation, RMSE, and Heidke forecast verification measures; refinement of the Heidke score. Wea. Forecasting, 7 , 699707.

    • Search Google Scholar
    • Export Citation
  • Kumar, A., 2007: On the interpretation and utility of skill information for seasonal climate predictions. Mon. Wea. Rev., 135 , 19741984.

    • Search Google Scholar
    • Export Citation
  • Kumar, A., and M. P. Hoerling, 1995: Prospects and limitations of seasonal atmospheric GCM predictions. Bull. Amer. Meteor. Soc., 76 , 335345.

    • Search Google Scholar
    • Export Citation
  • Kumar, A., and M. P. Hoerling, 2000: Analysis of a conceptual model of seasonal climate variability and implications for seasonal predictions. Bull. Amer. Meteor. Soc., 81 , 255264.

    • Search Google Scholar
    • Export Citation
  • Kumar, A., A. G. Barnston, P. Peng, M. P. Hoerling, and L. Goddard, 2000: Changes in the spread of the variability of the seasonal mean atmospheric states associated with ENSO. J. Climate, 13 , 31393151.

    • Search Google Scholar
    • Export Citation
  • Kumar, A., A. G. Barnston, and M. P. Hoerling, 2001: Seasonal predictions, probabilistic verifications, and ensemble size. J. Climate, 14 , 16711676.

    • Search Google Scholar
    • Export Citation
  • Ropelewski, C. F., and M. S. Halpert, 1987: Global and regional scale precipitation patterns associated with the El Niño–Southern Oscillation. Mon. Wea. Rev., 115 , 16061626.

    • Search Google Scholar
    • Export Citation
  • Sardeshmukh, P. D., G. P. Compo, and C. Penland, 2000: Changes of probability associated with El Niño. J. Climate, 13 , 42684286.

  • Tang, Y., H. Lin, J. Derome, and M. K. Tippett, 2007: A predictability measure applied to seasonal prediction of the Arctic Oscillation. J. Climate, 20 , 47334750.

    • Search Google Scholar
    • Export Citation
  • Tippett, M. K., R. Kleeman, and Y. Tang, 2004: Measuring the potential predictability of seasonal climate predictions. Geophys. Res. Lett., 31 , L22201. doi:10.1029/2004GL021575.

    • Search Google Scholar
    • Export Citation
  • Trenberth, E. K., G. W. Branstrator, D. Karoly, A. Kumar, N-C. Lau, and C. Ropelewski, 1998: Progress during TOGA in understanding and modeling global teleconnections associated with tropical sea surface temperatures. J. Geophys. Res., 103 , (C7). 1429114324.

    • Search Google Scholar
    • Export Citation
  • van den Dool, H. M., and Z. Toth, 1991: Why do forecasts for “near normal” often fail. Wea. Forecasting, 6 , 7685.

  • Wu, R., and B. P. Kirtman, 2006: Changes in the spread and predictability associated with ENSO in an ensemble coupled GCM. J. Climate, 19 , 43784396.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 561 246 24
PDF Downloads 432 145 16