• Bender, M. A., and Ginis I. , 2000: Real-case simulations of hurricane–ocean interaction using a high-resolution coupled model: Effects on hurricane intensity. Mon. Wea. Rev., 128 , 917946.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bender, M. A., Ginis I. , Tuleya R. , Thomas B. , and Marchok T. , 2007: The operational GFDL coupled hurricane–ocean prediction system and summary of its performance. Mon. Wea. Rev., 135 , 39653989.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bradley, A. A., Hashino T. , and Schwartz S. S. , 2003: Distributions-oriented verification of probability forecasts for small data samples. Wea. Forecasting, 18 , 903917.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Bradley, A. A., Schwartz S. S. , and Hashino T. , 2004: Distributions-oriented verification of ensemble streamflow predictions. J. Hydrometeor., 5 , 532545.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Brooks, H. E., and Doswell C. A. III, 1996: A comparison of measures-oriented and distributions-oriented approaches to forecast verification. Wea. Forecasting, 11 , 288303.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Brooks, H. E., Witt A. , and Eilts M. D. , 1997: Verification of public weather forecasts available via the media. Bull. Amer. Meteor. Soc., 78 , 21672177.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Charba, J. P., Reynolds D. W. , McDonald B. E. , and Carter G. M. , 2003: Comparative verification of recent quantitative precipitation forecasts in the National Weather Service: A simple approach for scoring forecast accuracy. Wea. Forecasting, 18 , 161183.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • de Elía, R., and Laprise R. , 2003: Distributions-oriented verification of limited-area model forecasts in a perfect-model framework. Mon. Wea. Rev., 131 , 24922509.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • DelSole, T., 2005: Predictability and information theory. Part II: Imperfect forecasts. J. Atmos. Sci., 62 , 33683381.

  • DeMaria, M., 2006: Statistical tropical cyclone intensity forecast improvements using GOES and aircraft reconnaissance data. Preprints, 27th Conf. on Hurricanes and Tropical Meteorology, Monterey, CA, Amer. Meteor. Soc., 14A.3. [Available online at http://ams.confex.com/ams/pdfpapers/108035.pdf.].

  • DeMaria, M., and Kaplan J. , 1994: A Statistical Hurricane Intensity Prediction Scheme (SHIPS) for the Atlantic basin. Wea. Forecasting, 9 , 209220.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • DeMaria, M., and Kaplan J. , 1999: An updated Statistical Hurricane Intensity Prediction Scheme (SHIPS) for the Atlantic and eastern North Pacific basins. Wea. Forecasting, 14 , 326337.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • DeMaria, M., Mainelli M. , Shay L. K. , Knaff J. A. , and Kaplan J. , 2005: Further improvements to the Statistical Hurricane Intensity Prediction Scheme (SHIPS). Wea. Forecasting, 20 , 531543.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • DeMaria, M., Knaff J. A. , and Kaplan J. , 2006: On the decay of tropical cyclone winds crossing narrow landmasses. J. Appl. Meteor. Climatol., 45 , 491499.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Déqué, M., 2003: Continuous variables. Forecast Verification: A Practitioner’s Guide in Atmospheric Science, I. T. Jolliffe and D. B. Stephenson, Eds., Wiley, 97–119.

    • Search Google Scholar
    • Export Citation
  • Elsberry, R. L., Lambert T. , and Boothe M. , 2007: Accuracy of Atlantic and eastern North Pacific tropical cyclone intensity forecast guidance. Wea. Forecasting, 22 , 747762.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Emanuel, K., DesAutels C. , Holloway C. , and Korty R. , 2004: Environmental control of tropical cyclone intensity. J. Atmos. Sci., 61 , 843858.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Engel, C., and Ebert E. , 2007: Performance of hourly operational consensus forecasts (OCFs) in the Australian region. Wea. Forecasting, 22 , 13451359.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Falkovich, A., Ginis I. , and Lord S. , 2005: Ocean data assimilation and initialization procedure for the coupled GFDL/URI hurricane prediction system. J. Atmos. Oceanic Technol., 22 , 19181932.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Franklin, J., cited. 2006: National Hurricane Center forecast verification. [Available online at http://www.nhc.noaa.gov/verification].

  • Jolliffe, I. T., and Stephenson D. B. , 2003: Forecast Verification: A Practitioner’s Guide in Atmospheric Science. Wiley, 240 pp.

  • Knaff, J. A., DeMaria M. , Sampson C. R. , and Gross J. M. , 2003: Statistical, 5-day tropical cyclone intensity forecasts derived from climatology and persistence. Wea. Forecasting, 18 , 8092.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Knaff, J. A., Sampson C. R. , and DeMaria M. , 2005: An operational Statistical Typhoon Intensity Prediction Scheme for the western North Pacific. Wea. Forecasting, 20 , 688699.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kumar, T. S. V. V., Krishnamurti T. N. , Fiorino M. , and Nagata M. , 2003: Multimodel superensemble forecasting of tropical cyclones in the Pacific. Mon. Wea. Rev., 131 , 574583.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kurihara, Y., Tuleya R. E. , and Bender M. A. , 1998: The GFDL hurricane prediction system and its performance in the 1995 hurricane season. Mon. Wea. Rev., 126 , 13061322.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Leung, L., and North G. R. , 1990: Information theory and climate prediction. J. Climate, 3 , 514.

  • Maini, P., Kumar A. , Rathore L. S. , and Singh S. V. , 2003: Forecasting maximum and minimum temperatures by statistical interpretation of numerical weather prediction model output. Wea. Forecasting, 18 , 938952.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1973: A new vector partition of the probability score. J. Appl. Meteor., 12 , 595600.

  • Murphy, A. H., 1991: Forecast verification: Its complexity and dimensionality. Mon. Wea. Rev., 119 , 15901601.

  • Murphy, A. H., 1996: General decompositions of MSE-based skill scores: Measures of some basic aspects of forecast quality. Mon. Wea. Rev., 124 , 23532369.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1997: Forecast verification. The Economic Value of Weather and Climate Forecasts, R. W. Katz and A. H. Murphy, Eds., Cambridge University Press, 19–74.

    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., and Ehrendorfer M. , 1987: On the relationship between accuracy and value of forecasts in the cost–loss ratio situation. Wea. Forecasting, 2 , 243251.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., and Winkler R. L. , 1987: A general framework for forecast verification. Mon. Wea. Rev., 115 , 13301338.

  • Murphy, A. H., and Wilks D. S. , 1998: A case study in the use of statistical models in forecast verification: Precipitation probability forecasts. Wea. Forecasting, 13 , 795810.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., Brown B. G. , and Chen Y. , 1989: Diagnostic verification of temperature forecasts. Wea. Forecasting, 4 , 485501.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Myrick, D. T., and Horel J. D. , 2006: Verification of surface temperature forecasts from the National Digital Forecast Database over the western United States. Wea. Forecasting, 21 , 869892.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Nachamkin, J. E., Chen S. , and Schmidt J. , 2005: Evaluation of heavy precipitation forecasts using composite-based methods: A distributions-oriented approach. Mon. Wea. Rev., 133 , 21632177.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Potts, J. M., 2003: Basic concepts. Forecast Verification: A Practitioner’s Guide in Atmospheric Science, I. T. Jolliffe and D. B. Stephenson, Eds., Wiley, 13–36.

    • Search Google Scholar
    • Export Citation
  • Schulz, E. W., Kepert J. D. , and Greenslade D. , 2007: An assessment of marine surface winds from the Australian Bureau of Meteorology numerical weather prediction systems. Wea. Forecasting, 22 , 613636.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2000: Diagnostic verification of the Climate Prediction Center long-lead outlooks, 1995–98. J. Climate, 13 , 23892403.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2006: Statistical Methods in the Atmospheric Sciences. Academic Press, 648 pp.

  • Wilks, D. S., and Godfrey C. M. , 2002: Diagnostic verification of the IRI net assessment forecasts, 1997–2000. J. Climate, 15 , 13691377.

    • Crossref
    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 3 3 2
PDF Downloads 2 2 1

A Case Study of Deterministic Forecast Verification: Tropical Cyclone Intensity

View More View Less
  • 1 Massachusetts Institute of Technology, Cambridge, Massachusetts
Restricted access

Abstract

Deterministic predictions of tropical cyclone (TC) intensity from operational forecast systems traditionally have been verified with a summary accuracy measure (e.g., mean absolute error). Since the forecast system development process is coupled to the verification procedure, it follows that TC intensity forecast systems have been developed with the goal of producing predictions that optimize the chosen summary accuracy measure. Here, the consequences of this development process for the quality of the resultant forecasts are diagnosed through a distributions-oriented (DO) verification of operational TC intensity forecasts. DO verification techniques examine the full relationship between a set of forecasts and the corresponding set of observations (i.e., forecast quality), rather than just the accuracy attribute of that relationship.

The DO verification results reveal similar first-order characteristics in the quality of predictions from four TC intensity forecast systems. These characteristics are shown to be consistent with the theoretical response of a forecast system to the imposed goal of summary accuracy measure optimization: production of forecasts that asymptote with lead time to the central tendency of the observed distribution. While such forecasts perform well with respect to the accuracy, unconditional bias, and type I conditional bias attributes of forecast quality, they perform poorly with respect to type II conditional bias. Thus, it is clear that optimization of forecast accuracy is not equivalent to optimization of forecast quality. Ultimately, developers of deterministic forecast systems must take care to employ a verification procedure that promotes good performance with respect to the most desired attributes of forecast quality.

Corresponding author address: Jonathan R. Moskaitis, MIT, Rm. 54-1721, 77 Massachusetts Ave., Cambridge, MA 02139. Email: jonmosk@mit.edu

Abstract

Deterministic predictions of tropical cyclone (TC) intensity from operational forecast systems traditionally have been verified with a summary accuracy measure (e.g., mean absolute error). Since the forecast system development process is coupled to the verification procedure, it follows that TC intensity forecast systems have been developed with the goal of producing predictions that optimize the chosen summary accuracy measure. Here, the consequences of this development process for the quality of the resultant forecasts are diagnosed through a distributions-oriented (DO) verification of operational TC intensity forecasts. DO verification techniques examine the full relationship between a set of forecasts and the corresponding set of observations (i.e., forecast quality), rather than just the accuracy attribute of that relationship.

The DO verification results reveal similar first-order characteristics in the quality of predictions from four TC intensity forecast systems. These characteristics are shown to be consistent with the theoretical response of a forecast system to the imposed goal of summary accuracy measure optimization: production of forecasts that asymptote with lead time to the central tendency of the observed distribution. While such forecasts perform well with respect to the accuracy, unconditional bias, and type I conditional bias attributes of forecast quality, they perform poorly with respect to type II conditional bias. Thus, it is clear that optimization of forecast accuracy is not equivalent to optimization of forecast quality. Ultimately, developers of deterministic forecast systems must take care to employ a verification procedure that promotes good performance with respect to the most desired attributes of forecast quality.

Corresponding author address: Jonathan R. Moskaitis, MIT, Rm. 54-1721, 77 Massachusetts Ave., Cambridge, MA 02139. Email: jonmosk@mit.edu

Save