Application of Spatial Verification Methods to Idealized and NWP-Gridded Precipitation Forecasts

David Ahijevych National Center for Atmospheric Research, * Boulder, Colorado

Search for other papers by David Ahijevych in
Current site
Google Scholar
PubMed
Close
,
Eric Gilleland National Center for Atmospheric Research, * Boulder, Colorado

Search for other papers by Eric Gilleland in
Current site
Google Scholar
PubMed
Close
,
Barbara G. Brown National Center for Atmospheric Research, * Boulder, Colorado

Search for other papers by Barbara G. Brown in
Current site
Google Scholar
PubMed
Close
, and
Elizabeth E. Ebert Centre for Australian Weather and Climate Research, Melbourne, Victoria, Australia

Search for other papers by Elizabeth E. Ebert in
Current site
Google Scholar
PubMed
Close
Restricted access

Abstract

Several spatial forecast verification methods have been developed that are suited for high-resolution precipitation forecasts. They can account for the spatial coherence of precipitation and give credit to a forecast that does not necessarily match the observation at any particular grid point. The methods were grouped into four broad categories (neighborhood, scale separation, features based, and field deformation) for the Spatial Forecast Verification Methods Intercomparison Project (ICP). Participants were asked to apply their new methods to a set of artificial geometric and perturbed forecasts with prescribed errors, and a set of real forecasts of convective precipitation on a 4-km grid. This paper describes the intercomparison test cases, summarizes results from the geometric cases, and presents subjective scores and traditional scores from the real cases.

All the new methods could detect bias error, and the features-based and field deformation methods were also able to diagnose displacement errors of precipitation features. The best approach for capturing errors in aspect ratio was field deformation. When comparing model forecasts with real cases, the traditional verification scores did not agree with the subjective assessment of the forecasts.

Corresponding author address: David Ahijevych, National Center for Atmospheric Research, P.O. Box 3000, Boulder, CO 80307-3000. Email: ahijevyc@ucar.edu

This article included in the Spatial Forecast Verification Methods Inter-Comparison Project (ICP) special collection.

Abstract

Several spatial forecast verification methods have been developed that are suited for high-resolution precipitation forecasts. They can account for the spatial coherence of precipitation and give credit to a forecast that does not necessarily match the observation at any particular grid point. The methods were grouped into four broad categories (neighborhood, scale separation, features based, and field deformation) for the Spatial Forecast Verification Methods Intercomparison Project (ICP). Participants were asked to apply their new methods to a set of artificial geometric and perturbed forecasts with prescribed errors, and a set of real forecasts of convective precipitation on a 4-km grid. This paper describes the intercomparison test cases, summarizes results from the geometric cases, and presents subjective scores and traditional scores from the real cases.

All the new methods could detect bias error, and the features-based and field deformation methods were also able to diagnose displacement errors of precipitation features. The best approach for capturing errors in aspect ratio was field deformation. When comparing model forecasts with real cases, the traditional verification scores did not agree with the subjective assessment of the forecasts.

Corresponding author address: David Ahijevych, National Center for Atmospheric Research, P.O. Box 3000, Boulder, CO 80307-3000. Email: ahijevyc@ucar.edu

This article included in the Spatial Forecast Verification Methods Inter-Comparison Project (ICP) special collection.

Save
  • Atger, F., 2001: Verification of intense precipitation forecasts from single models and ensemble prediction systems. Nonlinear Processes Geophys., 8 , 401417.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Brill, K. F., and Mesinger F. , 2009: Applying a general analytic method for assessing bias sensitivity to bias-adjusted threat and equitable threat scores. Wea. Forecasting, 24 , 17481754.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Brooks, H. E., Kay M. , and Hart J. A. , 1998: Objective limits on forecasting skill of rare events. Preprints, 19th Conf. on Severe Local Storms, Minneapolis, MN, Amer. Meteor. Soc., 552–555.

    • Search Google Scholar
    • Export Citation
  • Casati, B., 2010: New developments of the intensity-scale technique within the Spatial Verification Methods Inter-Comparison Project. Wea. Forecasting, in press.

    • Search Google Scholar
    • Export Citation
  • Davis, C. A., Brown B. G. , Bullock R. , and Halley-Gotway J. , 2009: The method for object-based diagnostic evaluation (MODE) applied to numerical forecasts from the 2005 NSSL/SPC Spring Program. Wea. Forecasting, 24 , 12521267.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Dey, C. H., cited. 1998: Grid identification (PDS Octet 7): Master list of NCEP storage grids. U.S. Department of Commerce Office Note 388, GRIB ed. 1 (FM92), NOAA/NWS. [Available online at http://www.nco.ncep.noaa.gov/pmb/docs/on388/tableb.html#GRID240].

    • Search Google Scholar
    • Export Citation
  • Ebert, E. E., 2009: Neighborhood verification: A strategy for rewarding close forecasts. Wea. Forecasting, 24 , 14981510.

  • Ebert, E. E., and Gallus W. A. , 2009: Toward better understanding of the contiguous rain area (CRA) method for spatial forecast verification. Wea. Forecasting, 24 , 14011415.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Gilleland, E., Ahijevych D. , Brown B. G. , Casati B. , and Ebert E. E. , 2009a: Intercomparison of spatial forecast verification methods. Wea. Forecasting, 24 , 14161430.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Kain, J. S., and Coauthors, 2008: Some practical considerations regarding horizontal resolution in the first generation of operational convection-allowing NWP. Wea. Forecasting, 23 , 931952.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Keil, C., and Craig G. C. , 2009: A displacement and amplitude score employing an optical flow technique. Wea. Forecasting, 24 , 12971308.

  • Lack, S. A., Limpert G. L. , and Fox N. I. , 2010: An object-oriented multiscale verification scheme. Wea. Forecasting, in press.

  • Lin, Y., and Mitchell K. E. , 2005: The NCEP Stage II/IV hourly precipitation analyses: Development and applications. Preprints, 19th Conf. on Hydrology, San Diego, CA, Amer. Meteor. Soc., 1.2. [Available online at http://ams.confex.com/ams/pdfpapers/83847pdf].

    • Search Google Scholar
    • Export Citation
  • Marzban, C., and Sandgathe S. , 2009: Verification with variograms. Wea. Forecasting, 24 , 11021120.

  • Marzban, C., Sandgathe S. , Lyons H. , and Lederer N. , 2009: Three spatial verification techniques: Cluster analysis, variogram, and optical flow. Wea. Forecasting, 24 , 14571471.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mittermaier, M. P., and Roberts N. , 2010: Intercomparison of spatial forecast verification methods: Identifying skillful spatial scales using the fractions skill score. Wea. Forecasting, in press.

    • Search Google Scholar
    • Export Citation
  • Nachamkin, J. E., 2009: Application of the composite method to the spatial forecast verification methods intercomparison dataset. Wea. Forecasting, 24 , 13901400.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • NCAR, cited. 2009: Model Evalution Tools (MET) users page. [Available online at http://www.dtcenter.org/met/users].

  • Venugopal, V., Basu S. , and Foufoula-Georgiou E. , 2005: A new metric for comparing precipitation patterns with an application to ensemble forecasts. J. Geophys. Res., 110 , D08111. doi:10.1029/2004JD005395.

    • Search Google Scholar
    • Export Citation
  • Weisman, M. L., Davis C. , Wang W. , Manning K. W. , and Klemp J. B. , 2008: Experiences with 0–36-h explicit convective forecasts with the WRF-ARW model. Wea. Forecasting, 23 , 407437.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wernli, H., Paulat M. , Hagen M. , and Frei C. , 2008: SAL—A novel quality measure for the verification of quantitative precipitation forecasts. Mon. Wea. Rev., 136 , 44704487.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wernli, H., Hofmann C. , and Zimmer M. , 2009: Spatial forecast verification methods intercomparison project: Application of the SAL technique. Wea. Forecasting, 24 , 14721484.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Wilks, D. S., 2006: Statistical Methods in the Atmospheric Sciences. 2nd ed. Elsevier, 627 pp.

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 2865 1153 107
PDF Downloads 4582 224 38