Editorial Type:
Article
Article Type:
Research Article

Diagnosing the Sensitivity of Binary Image Measures to Bias, Location, and Event Frequency within a Forecast Verification Framework

Benjamin R. J. Schwedler Department of Earth and Atmospheric Sciences, Purdue University, West Lafayette, Indiana

Search for other papers by Benjamin R. J. Schwedler in
Current site
Google Scholar
PubMed Close
and
Michael E. Baldwin Department of Earth and Atmospheric Sciences, Purdue University, West Lafayette, Indiana

Search for other papers by Michael E. Baldwin in
Current site
Google Scholar
PubMed Close
Print Publication:
01 Dec 2011
DOI:
https://doi.org/10.1175/WAF-D-11-00032.1
Page(s):
1032–1044
Restricted access

Abstract

While the use of binary distance measures has a substantial history in the field of image processing, these techniques have only recently been applied in the area of forecast verification. Designed to quantify the distance between two images, these measures can easily be extended for use with paired forecast and observation fields. The behavior of traditional forecast verification metrics based on the dichotomous contingency table continues to be an area of active study, but the sensitivity of image metrics has not yet been analyzed within the framework of forecast verification. Four binary distance measures are presented and the response of each to changes in event frequency, bias, and displacement error is documented. The Hausdorff distance and its derivatives, the modified and partial Hausdorff distances, are shown only to be sensitive to changes in base rate, bias, and displacement between the forecast and observation. In addition to its sensitivity to these three parameters, the Baddeley image metric is also sensitive to additional aspects of the forecast situation. It is shown that the Baddeley metric is dependent not only on the spatial relationship between a forecast and observation but also the location of the events within the domain. This behavior may have considerable impact on the results obtained when using this measure for forecast verification. For ease of comparison, a hypothetical forecast event is presented to quantitatively analyze the various sensitivities of these distance measures.

Corresponding author address: Benjamin Schwedler, Purdue University, 550 Stadium Mall Dr., West Lafayette, IN 47907. E-mail: bschwedl@purdue.edu

Abstract

While the use of binary distance measures has a substantial history in the field of image processing, these techniques have only recently been applied in the area of forecast verification. Designed to quantify the distance between two images, these measures can easily be extended for use with paired forecast and observation fields. The behavior of traditional forecast verification metrics based on the dichotomous contingency table continues to be an area of active study, but the sensitivity of image metrics has not yet been analyzed within the framework of forecast verification. Four binary distance measures are presented and the response of each to changes in event frequency, bias, and displacement error is documented. The Hausdorff distance and its derivatives, the modified and partial Hausdorff distances, are shown only to be sensitive to changes in base rate, bias, and displacement between the forecast and observation. In addition to its sensitivity to these three parameters, the Baddeley image metric is also sensitive to additional aspects of the forecast situation. It is shown that the Baddeley metric is dependent not only on the spatial relationship between a forecast and observation but also the location of the events within the domain. This behavior may have considerable impact on the results obtained when using this measure for forecast verification. For ease of comparison, a hypothetical forecast event is presented to quantitatively analyze the various sensitivities of these distance measures.

Corresponding author address: Benjamin Schwedler, Purdue University, 550 Stadium Mall Dr., West Lafayette, IN 47907. E-mail: bschwedl@purdue.edu
Save
Cover Weather and Forecasting
Weather and Forecasting

  • Ahijevych, D., Gilleland E. , Brown B. G. , and Ebert E. E. , 2009: Application of spatial verification methods to idealized and NWP-gridded precipitation forecasts. Wea. Forecasting, 24, 1485–1497.

    • Search Google Scholar
    • Export Citation

  • Baddeley, A. J., 1992a: An error metric for binary images. Robust Computer Vision: Quality of Vision Algorithms, W. Förstner and S. Ruwiedel, Eds., Wichmann, 59–78.

    • Search Google Scholar
    • Export Citation

  • Baddeley, A. J., 1992b: Errors in binary images and an Lp version of the Hausdorff metric. Nieuw Arch. Wiskunde, 10, 157–183.

  • Baldwin, M. E., and Kain J. S. , 2006: Sensitivity of several performance measures to displacement error, bias, and event frequency. Wea. Forecasting, 21, 636–648.

    • Search Google Scholar
    • Export Citation

  • Borgefors, G., 1984: Distance transformations in arbitrary dimensions. Comput. Vision Graph. Image Process., 27, 321–345, doi:10.1016/0734-189X(84)90035-5.

    • Search Google Scholar
    • Export Citation

  • Borgefors, G., 1986: Distance transformations in digital images. Comput. Vision Graph. Image Process., 34, 344–371, doi:10.1016/S0734-189X(86)80047-0.

    • Search Google Scholar
    • Export Citation

  • Breu, H., Gil J. , Kirkpatrick D. , and Werman M. , 1995: Linear time euclidean distance transform algorithms. IEEE Trans. Pattern Anal. Mach. Intell., 17, 529–533, doi:10.1109/34.391389.

    • Search Google Scholar
    • Export Citation

  • Brill, K. F., 2009: A general analytic method for assessing sensitivity to bias of performance measures for dichotomous forecasts. Wea. Forecasting, 24, 307–318.

    • Search Google Scholar
    • Export Citation

  • Brill, K. F., and Mesinger F. , 2009: Applying a general analytic method for assessing bias sensitivity to bias-adjusted threat and equitable threat scores. Wea. Forecasting, 24, 1748–1754.

    • Search Google Scholar
    • Export Citation

  • Danielsson, P., 1980: Euclidean distance mapping. Comput. Vision Graph. Image Process., 14, 227–248, doi:10.1016/0146-664X(80)90054-4.

    • Search Google Scholar
    • Export Citation

  • Dubuisson, M.-P., and Jain A. K. , 1994: A modified Hausdorff distance for object matching. Proc. Int. Conf. on Pattern Recognition, Jerusalem, Israel, IEEE, 566–568, doi:10.1109/ICPR.1994.576361.

    • Search Google Scholar
    • Export Citation

  • Gilleland, E., 2011: Spatial forecast verification: Baddeley's delta metric applied to the ICP test cases. Wea. Foreasting, 26, 409–415.

    • Search Google Scholar
    • Export Citation

  • Gilleland, E., Lee T. C. M. , Halley Gotway J. , Bullock R. G. , and Brown B. G. , 2008: Computationally efficient spatial forecast verification using Baddeley's delta image metric. Mon. Wea. Rev., 136, 1747–1757.

    • Search Google Scholar
    • Export Citation

  • Gilleland, E., Ahijevych D. , Brown B. G. , Casati B. , and Ebert E. E. , 2009: Intercomparison of Spatial Forecast Verification Methods. Wea. Forecasting, 24, 1416–1430.

    • Search Google Scholar
    • Export Citation

  • Moeckel, R., and Murray A. B. , 1997: Measuring the distance between time series. Physica D, 102, 187–194, doi:10.1016/S0167-2789(96)00154-6.

    • Search Google Scholar
    • Export Citation

  • Murphy, A. H., 1991: Forecast verification: Its complexity and dimensionality. Mon. Wea. Rev., 119, 1590–1601.

  • Murphy, A. H., and Winkler R. L. , 1987: A general framework for forecast verification. Mon. Wea. Rev., 115, 1330–1338.

  • Rosenfeld, A., and Pfaltz J. , 1966: Sequential operations in digital picture processing. J. Assoc. Comput. Mach., 13, 471–494.

  • Rosenfeld, A., and Pfaltz J. , 1968: Distance functions on digital pictures. Pattern Recognit., 1, 33–61, doi:10.1016/0031-3203(68)90013-7.

    • Search Google Scholar
    • Export Citation

  • Rucklidge, W., 1996: Efficient Visual Recognition Using the Hausdorff Distance. Springer, 178 pp.

  • Venugopal, V., Basu S. , and Foufoula-Georgiou E. , 2005: A new metric for comparing precipitation patterns with an application to ensemble forecasts. J. Geophys. Res., 110, D081111, doi:10.1029/2004JD005395.

    • Search Google Scholar
    • Export Citation
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 93 41 2
PDF Downloads 29 13 1