1. Introduction
When selecting a method to evaluate forecasts, it is important to understand the behavior of the performance measure(s) used to assess forecast quality. Performance measures are often computed for a dichotomous forecast using values in the 2 × 2 contingency table. A single measure computed in this manner cannot completely describe the joint distribution of the forecast and observations, and it is thus necessary to use multiple measures when summarizing the performance of a forecast (e.g., Murphy and Winkler 1987; Murphy 1991).
During the selection process it is crucial that the behavior of the summary measures under consideration be understood in terms of the joint distribution of the forecast and observations. Even if the problem has been simplified to a dichotomous forecast and observation to reduce its complexity and dimensionality, the evaluation measures are still affected by attributes of the forecast and observation such as the bias and base rate. In particular, the behavior of several measures for forecasts with varying bias has received attention recently. Using a conceptual model, Baldwin and Kain (2006) examine the effects of base rate, bias, and displacement error on six verification measures. Brill (2009) develops an analytic method for determining the response of several performance measures to changes in bias and probability of detection and applies this technique to assess numerical forecasts in Brill and Mesinger (2009).
Recently, binary image metrics have been applied to the forecast verification problem. Gilleland et al. (2008) presented a method that uses Baddeley's Δ image metric (Baddeley 1992a,b) in an object-based forecast verification technique. This metric was applied to the cases from the Spatial Verification Methods Intercomparison Project (ICP; Ahijevych et al. 2009; Gilleland et al. 2009; also see information online at http://www.ral.ucar.edu/projects/icp/) by calculating Δ across the entire field (Gilleland 2011). The use of distance measures is motivated by their ability to quantify a distance between the forecast and observed fields in addition to capturing the usual measures of “goodness.” While these metrics have been used to verify forecasts, it is not clear how they are affected by changes in the forecast situation, such as bias, displacement, and base rate. Because the wider forecast verification community may not be familiar with the behavior of these metrics, the thrust of this paper is to analyze and present the response of binary image metrics to changes in the forecast situation.
Section 2 presents the formulation of the binary distance measures analyzed in this work and includes a discussion of the properties a distance measure must satisfy in order to be classified as a metric. Section 3 details the synthetic forecast situation used to study the response of these distance measures to changes in base rate, bias, and displacement. In section 4, three scenarios with varying base rates are presented and the behavior of each metric with changes to bias and displacement is described. Next, a discussion of the causes of these responses appears in section 5, and a brief summary is provided in section 6.
2. Distance measures for binary images
Applying image processing techniques to numerical weather prediction (NWP) forecasts can provide a measure of the discrepancy with observations. Distance measures that are commonly applied to images often need little or no modification to work with these data, since both observations and forecasts are often provided on rectangular grids. Gridded datasets can naturally be represented by an image raster X, with each pixel corresponding to the location and value of each grid box. For a binary distance measure to be applied, a threshold is first applied to X. All pixels exceeding the threshold value become part of the foreground of the image raster (valued at 1) with the remaining pixels composing the background (valued at 0). Foreground pixels from image X are now contained in the set that we will define as G.
In this paper, the distance transform will be applied to hypothetical forecasts and observations that exceed a specified threshold. Sets O and F will refer to the set of foreground pixels in the observation and forecast, respectively. The objective is then to describe the manner in which the distance measures change as the relationship between O and F changes.
In order for a distance measure m to be classified a metric, it must satisfy the following three properties (Baddeley 1992a):
symmetry—m(O, F) = m(F, O);
positivity and separation—m(O, F) ≥ 0 and m(O, F) = 0 if and only if O = F; and
triangle inequality—m(O, F) ≤ m(O, G) + m(G, F), where G is any arbitrary third set.
In an attempt to alleviate the susceptibility to outliers and noise, Dubuisson and Jain (1994) present 24 different distance measures based on the Hausdorff distance. These alternatives are composed of six different directed differences between sets O and F, and four ways to combine these directed distances. The directed distances include the minimum, maximum, and mean Euclidean distances between one set to the other, as well as three different kth percentiles of the distances. The four combinations of the directed distances are the maximum, minimum, arithmetic average, and weighted averages. Of these 24 combinations, the Hausdorff distance is the only measure that always obeys the three properties of a metric.
As presented in this section, all measures have units of pixels but are easily converted to units of physical distance by scaling the pixel distance by the appropriate amount. As discussed at the beginning of this section, not all of the distance measures discussed here are image metrics. Table 1 indicates which of the three properties of a metric are satisfied by each distance measure. Only Hd and Δ satisfy all three properties, consequently MHD and PHD are distance measures but not metrics.
Distance measures in this study, with columns indicating the three properties the measure must satisfy to be a metric. A checkmark is present if the measure satisfies the particular property.
3. Idealized forecast and observations
This paper utilizes a method similar to that employed by Baldwin and Kain (2006) to determine the sensitivity of distance measures to various aspects of a forecast and observation. Like Baldwin and Kain (2006), we examine the sensitivity to bias (B), displacement errors (D), and changes in base rate (P), and we add an investigation of additional sensitivities of the Baddeley metric to location and cutoff distance. For ease of understanding and simplicity of computation, both the forecast and observation will be represented by circular shapes having radii of rf and ro, respectively.
To further simplify the problem, the domain of interest is constructed as a 1001 pixel × 1001 pixel square with a total area defined to be equal to 1. This construction produces a four-connected between-pixel distance of 0.001. While in Baldwin and Kain (2006) the scalar performance measure could be computed analytically by simply knowing B, D, and ro, the computation of the distance transform for these measures requires that binary fields be generated for each combination of these variables. For each set to retain its membership and cardinality, it is necessary that the circles representing both the forecast and observations must be completely inside the square domain. This means that as the base rate for the event increases, the ranges of B and D over which these metrics can be computed is limited. Since the displacement distance is defined from the centroid of the observed circle, its location also limits the range of computation. Section 5 contains additional discussion on the location dependence of these measures.
To illustrate how the distance calculations are impacted by the changing relationship between the idealized forecast and observations, a schematic diagram of three different cases is shown in Fig. 1. In all panels, the observed feature (light gray) is a circle with radius 0.20 centered at (0.25, 0.50), which has a base rate of P ≊ 0.13. For all cases in this illustration, the center of the forecast feature (dark gray) is located a distance of 2ro from the observed center, resulting in a placement at (0.65, 0.50). From top to bottom in Fig. 1, the bias is assigned the values of B = 1, B = ½, and B = 2. Panels on the left in Fig. 1 demonstrate how the forecast radius rf changes with varying magnitudes of B. Each panel on the right in Fig. 1 provides an instance of an arbitrary pixel distance ρ(x, y), the minimum Euclidean distance from this pixel d(x, F), and the Hausdorff distance Hd. Note that for the case of B = 1, both directed distances are equivalent, as emphasized by the two measures of Hd shown in Fig. 1b. Table 2 summarizes the values of the distance measures computed for each of the three cases show in Fig. 1. All measures are valued in terms of the distance in this example (i.e., the distance between each pixel is equal to 0.001).
Schematic examples of observed (light gray) and forecast (dark gray) features for ro = 0.20. Each outlined domain has a total area of 1. The distance between feature centroids is held constant at D = 2ro, with bias values ranging between ½ and 2. An arbitrary distance from the pixel x at the top of the observed circle ρ(x, y), (d) the minimum distance from x to the forecast feature d(x, F), and (f) the Hausdorff distance Hd.
Citation: Weather and Forecasting 26, 6; 10.1175/WAF-D-11-00032.1
Values of distance metrics computed for the schematic examples shown in Fig. 1 [ro = 0.20, (xo, yo) = (0.25, 0.50), and D′ = 2].
The following section computes each distance measure for three values of base rate over a range of normalized displacement (D′ = D/ro) and bias errors. For each base rate, distance measures are computed for the ranges 0 ≤ D′ ≤ 2.5 and 0 < B ≤ 3, the same ranges used by Baldwin and Kain (2006).
4. Results
Results are presented for all distance measures discussed in section 2, beginning with the example case shown in Fig. 1 with a base rate of P ≊ 0.13. In addition to this event with marginal spatial coverage, events with minuscule and moderate coverage will also be examined.
a. Marginal coverage
Figure 2 shows the response of binary distance measures to changes in B and D′ for an event with marginal coverage (P ≊ 0.13, ro = 0.20). This particular scenario contains the three cases shown in Fig. 1. Values presented in Table 2 can be roughly ascertained from this graphic by examining how the measure changes with increasing bias for a constant displacement of D′ = 2. Because the area of the domain is fixed at 1.0, it is not possible to compute the values of the distance measures over the entire range of B and D while at the same time keeping both circles inside the domain. For larger bias values, the forecast begins to fall outside of the domain. This can occur for both large and small displacements between the observation and forecast.
Values of binary distance measures [P ≊ 0.13, observed circle radius ro = 0.20, observed circle center (xo, yo) = (0.25, 0.50)] as a function of bias B and normalized displacement D′. Shown are (a) Hd, (b) PHD, (c) Δ(c = ∞), and (d) MHD. A dashed line indicates the axis of minimum value. Contour interval is 0.05.
Citation: Weather and Forecasting 26, 6; 10.1175/WAF-D-11-00032.1
As would be expected, all distance measures reach their minimum value when the forecast is identical to the observations (B = 1, D′ = 0). For the three Hausdorff-based distances in Fig. 2, the axis of minimum value is realized for an unbiased forecast (B = 1). This pattern of behavior, which at first may not seem intuitive, can be explained by taking a close look at the example in Fig. 1. Because of the maximum in its definition (3), for a given displacement, Hd will be minimized when the forecast and observed circles are identical in size. For a constant displacement distance and a reduction in bias, the distance from the left edge of the forecast circle to the left edge of the observed circle increases. Similarly, when bias is increased, the distance between the right edges of the forecast and the observed circles becomes larger. In each case, the result is an increase in the largest possible distance between two pixels in O and F. For a given base rate, it appears that Hd changes faster for an underforecast (B < 1) than for an overforecast (B > 1) (Fig. 2a). In actuality, this is a consequence of the B–D′ coordinate system used here. The radii of the forecast and observation are directly proportional to each other by a factor of the square root of the bias 
Results for the adjustments to the Hausdorff distance made by Venugopal et al. (2005) and Dubuisson and Jain (1994) are also shown here in Figs. 2b and 2d, respectively. The pattern of behavior for both MHD and PHD is very similar to that of Hd, with the primary difference being the magnitude of the distance measure. Additional differences are a result of using the 75th percentile and mean of the distribution of distance values. As would be expected from their definitions, the modifications result in a reduced value for MHD and PHD when compared to Hd, with the values of PHD being greater than those of MHD.
The Baddeley metric with a cutoff value of c = ∞ is shown in Fig. 2c. For an infinite c, the cutoff transformation is not applied to any pixels in the image. With no application of this transformation, Δ(c = ∞) does not exhibit strictly monotonic behavior with changing D′. In fact, in the lower-left corner, Δ first decreases for increasing D′ and then begins to increase after reaching its minimum value for a constant B. This behavior is apparent along the axis of the minimum value, which initially drifts toward the D′ axis before merging away toward greater bias with increasing D′.
b. Minuscule coverage
Figure 3 shows the values of two distance measures for an observation with minuscule areal coverage (P ≊ 0.03, ro = 0.10). By excluding MHD and PHD because of their similarity to Hd, all distance measures shown from this point on satisfy all three properties required to be classified as a metric. Because a discussion of the effects of choosing different cutoff values for Δ is pursued in the next section, we also only show the Baddeley metric with no cutoff value here.
Values of (a) Hd and (b) Δ(c = ∞) for the situation with minuscule coverage [P ≊ 0.03, observed circle radius ro = 0.10, observed circle center (xo, yo) = (0.15, 0.50)]. Contour interval is 0.05 for all panels.
Citation: Weather and Forecasting 26, 6; 10.1175/WAF-D-11-00032.1
Comparing the Hausdorff distance for minuscule coverage to the marginal coverage event reveals that the magnitude of Hd for ro = 0.10 (Fig. 3a) is exactly half the value for ro = 0.20 (Fig. 2a). Minimum values are again achieved at B = 1 for the same reason that was provided for the event with marginal areal extent. The proportional response for Hd results from the direct proportionality of both the radius of the forecast circle and its displacement to ro.
While the Δ values for ro = 0.10 (Fig. 3b) are approximately half the values calculated for ro = 0.20 (Fig. 2c), the correspondence is not exact. This can be clearly demonstrated by comparing the axis of minimum Δ values between these two figures. For the smaller base rate, Δ achieves its minimum value through a decrease of B as D′ increases (Fig. 3b). When the observed circle is larger, an increase in B for larger values of D′ minimizes Δ (Fig. 2c). This change in the axis of minimum value for Δ is a response to both the change of the location of the observed circle and a change in base rate. In both cases, the left edge of the observed circle is placed a distance of 0.05 from the left edge of the domain. The effects of the observation location relative to the domain are isolated in the next section.
c. Moderate coverage
Figure 4 presents Hd and Δ(c = ∞) for an event with moderate spatial extent (P ≊ 0.28, ro = 0.30). As was the case for the previous two scenarios, the left edge of the observed circle is placed 0.05 from the left edge of the domain. Similar to Fig. 2, portions of each frame are blank where the larger forecast circle would escape the domain.
As in Fig. 3, but for an event with moderate coverage [P ≊ 0.28, observed circle radius ro = 0.30, observed circle center (xo, yo) = (0.35, 0.50)].
Citation: Weather and Forecasting 26, 6; 10.1175/WAF-D-11-00032.1
By comparing the Hausdorff distances for this scenario (Fig. 4a) to the previous figures, we see the continued trend for the change in Hd to be proportional to the change in ro. For values of D′ where the forecast circle is completely inside the domain, Hd is again minimized at B = 1.
Values for Δ do not follow this proportional trend (Fig. 4b). Although the magnitude of the values has again increased, the peak associated with the axis of minimum values has shifted toward even larger bias values as the center of the forecast circle moves to the right.
5. Discussion
Results from the previous section demonstrate the behavior of four binary distance measures for a simplified forecast situation. The Hausdorff distance and those based on it (MHD and PHD) all show a similar behavior that only changes in magnitude with a changing base rate. As alluded to earlier, this relationship between the magnitude of Hd and ro is a consequence of the geometry of the simplified cases in this paper. For a case where the forecast is larger than the observation (B > 1), the Hausdorff distance will be from the right edge of the observed circle to the right edge of the forecast circle. Likewise, when B < 1, Hd is achieved along the path from the left edge of the observation to the left edge of the forecast. If the forecast is unbiased (B = 1), both of these distances are equal (Fig. 1b). Thus, the direct relationship between Hd and ro arises because both the normalized displacement distance D′ and the forecast circle radius rf are proportional to the size of the observation.
As demonstrated in the previous section, the Hausdorff-based distance measures have a consistent shape for all values of the base rate, which is not the case for the Baddeley metric. In addition to its sensitivity to base rate, Δ is also sensitive to changes in the location of the forecast and observation relative to the edge of the domain. To help us determine why Δ has this response, we need to first understand why the other distance measures are not sensitive to this change.
For the Hausdorff distance (as well as its two related distance measures), the distance between sets O and F is a combination of two directed distances. Each of these distances depends only on the distribution of the minimum Euclidean distance from all points in a first set to the points in a second set. Regardless of whether the maximum, kth percentile, or the mean of this distribution is taken, the directed distances are only dependent upon the relative size and location of the two sets; the absolute location of the two circles within the domain does not matter. Thus, Hd, MHD, and PHD are not sensitive to the locations of F and O relative to the domain, but only to their locations relative to each other.
An understanding of the relationship of the Baddeley distance metric to the location of the forecast and observations stems from an understanding of how the distance transform of a set changes with domain-relative location. Because of the nature of the hypothetical forecast problem, the distance transform of either O or F is represented by a series of concentric circular contours surrounding the set. For a constant base rate, if O is located closer to the left edge of the domain, the magnitude of the distance transform values will be greater in the right portion of the domain than in the left portion. Additionally, the values of the distance transform will change more quickly in the y direction on the left side of the domain than on the right side of the domain. This can easily be discerned by picturing a set of concentric contours and shifting them horizontally within a fixed area.
From its definition, Δ is an Lp norm of the absolute difference between these two distance maps. To illustrate different sensitivities of Δ, this difference is calculated for four combinations of base rate, bias, displacement distance, cutoff distance, and location (Fig. 5). Specific values of each parameter for each panel are detailed in Table 3. For a case where B = 2 and the forecast is displaced to the right of the observation, the difference between the distance maps will be greater on the right edge of the domain than on the left edge. As the observation and forecast shift to the right, the number of larger values from the right portion of the domain is reduced and replaced by smaller differences on the left. This change results in a decrease of the Euclidean norm of the distances, which in turn will reduce Δ. This case is illustrated by comparing Figs. 5a and 5b and the associated Δ values. The only difference between these two panels is that the observation and forecast have been shifted 0.15 to the right. For the opposite situation, an event that is underforecast (B < 1), the magnitude of the differences between the two distance maps will be greater on the left edge of the domain than on the right. In this case, a shift to the right would have the opposite effect, with an increase in Δ arising from the addition of larger values from the left portion of the domain.
Absolute difference between distance transform of observations and forecasts (|d(x, O) − d(x, F)| for all x ∈ X) with parameters from Table 3. In each panel, the observation is shown with a solid line, and the forecast with a dashed line. The result of a change in horizontal location is demonstrated by comparing (a) to (b). An increase in base rate is illustrated from (a) to (c), and the effects of including a cutoff distance are demonstrated between (b) and (d).
Citation: Weather and Forecasting 26, 6; 10.1175/WAF-D-11-00032.1
Values of parameters for illustration of sensitivities of Δ shown in Fig. 5. The y coordinate for both the forecast and observed circles is 0.50.
To further illustrate the effects of changing the domain-relative location of the forecast and observation, values of Δ for the minuscule coverage event are calculated for three locations of the observed circle (Fig. 6). Distances from the left edge of the domain to the observed circle (de = xo − ro) are set at 0.05, 0.20, and 0.35. As the forecast and observations move farther from the left edge of the domain, the axis of the minimum Δ values shifts from being realized where underforecasting is occurring (B < 1) to cases dominated by overforecasting. This shift summarizes the changes to the Baddeley metric throughout the B–D′ coordinates. As de becomes larger, Δ generally increases for underforecasts and decreases for an overforecast.
Values of Δ with no cutoff value (c = ∞) for an observed circle radius of ro = 0.10. The center of the circle is shifted three different distances from the edge of the domain: (a) (xo, yo) = (0.45, 0.50), (b) (xo, yo) = (0.30, 0.50), and (c) (xo, yo) = (0.45, 0.50). A contour interval of 0.025 is used for all panels.
Citation: Weather and Forecasting 26, 6; 10.1175/WAF-D-11-00032.1
This relationship for the distance map difference holds for all base rates. When ro is increased, as is pictured in the change between Figs. 5a and 5c, difference values remain greater on the right side of the domain and are increased in magnitude. With the increase in base rate, both the forecast and observation are closer to the edges of the domain. This fact means that although the difference between the distance transforms has increased, the value of the individual distance maps near the edges of the domain has decreased. Accompanied by a greater overlap between the forecast and observation, this change results in neither the difference between the distance transforms nor the magnitude Δ being exactly doubled, even though the radii of the forecast and observations have both been doubled.
Applying a cutoff distance when calculating Δ drastically changes its behavior. This effect is particularly strong for small cutoff values and events with small areal extent. By applying a cutoff value of c = 0.30 for an observed radius of 0.10, a large portion of the distance maps for both the forecast and observation are set at a constant value of 0.30. Pixels where these regions overlap [d(x, O) = d(x, F) = c] have zero contribution to the norm calculation of the Baddeley metric. Additionally, larger difference values have a greater effect on Δ because of the square in the Euclidean norm. This case is illustrated in the change between Figs. 5b and 5d. With the cutoff transformation applied, nonzero differences are replaced by zero values and the value of Δ is decreased. Because smaller cutoff values result in larger portions of a field being set to a constant value, smaller values of c will likewise result in a smaller Δ when applied to the same fields.
The cutoff distance also has an important role related to the domain edge effects previously discussed. Including a cutoff distance can reduce the sensitivity to the location of O and F within the domain. To illustrate this point, consider the situation of a relatively small cutoff distance. Points that are at least this distance away from both the forecast and observations will have no contribution to Δ. If the relative location of F to O remains constant (B, D′ unchanged), as long as both the forecast and observation are a distance greater than c from all edges of the domain, the value of the Baddeley distance metric will also remain constant. If either F or O moves within a distance c from any of the domain edges, the value of Δ will change. In this case Δ will be decreased due to the exclusion of nonzero differences that were previously used to compute the metric value, and the addition of zero difference points near the opposite edge of the domain. The cutoff distance then acts as a buffer to the domain, inside which the location of the forecast and observation has no effect on Δ.
For an event with minuscule coverage located near the center of the domain, the effects of changing cutoff distance are quite apparent (Fig. 7). The increase in the magnitude of Δ with increasing cutoff distance is evident immediately. This is caused by the increased number of points that contribute nonzero values to the Euclidean norm in the calculation of the Baddeley metric. The axis of the minimum value of Δ(c = 0.10) moves toward smaller bias values with increasing displacement, and above D′ ≳ 1.5, the smallest values of Δ occur as B → 0 (Fig. 7a). To understand why this is the case, consider what happens as the forecast circle moves a greater distance from the observation. As the forecast is displaced farther from the observation, it enters the portion of the domain where the distance transform d(x, O) is equal to a constant value of c. Minimizing the difference between the distance transforms of O and F will minimize the value of Δ. The simplest way to do this is to make d(x, O) = d(x, F) = c for as many pixels as possible, since the contribution to Δ at these pixels is zero. By reducing B as D′ increases, the overlap of these regions of constant distance transform increases and thus the value of Δ is decreased.
Values of Δ for an observed circle radius of ro = 0.10 with the observation centered on (xo, yo) = (0.45, 0.40). Increasing values of cutoff distance used in the calculation are (a) c = 0.10, (b) c = 0.30, and (c) c = 0.50. Contour interval is 0.01 for (a) and 0.025 for (b) and (c).
Citation: Weather and Forecasting 26, 6; 10.1175/WAF-D-11-00032.1
By setting the value of the cutoff distance to c = 0.30, the forecast circle is no longer within the buffer from the edge of the domain for large values of bias and displacement. For small values of B and D′, the axis of the minimum value has a location that is similar to the smaller cutoff value of 0.10, but moves toward greater values of bias more slowly with increasing D′ (Fig. 7b). Because of the larger cutoff value, a larger portion of the domain has a nonzero contribution to Δ, and thus the total value is increased.
For the large cutoff distance (c = 0.50), the distance between the observed circle and the edge of the domain is greater than the cutoff distance in most places, with the only regions where the cutoff transformation is applied are located in the corners of the domain. As the displacement of the forecast circle increases, it moves farther from the left edge of the domain, and the cutoff distance is applied to the left-most portion of the domain. As a result, the behavior of both the magnitude of the Baddeley metric and the axis of minimum values is very different than for lower cutoff values (Fig. 7c). Instead of being convex with respect to the origin, as was the case for smaller cutoff values, the axis of minimum value is concave, with smaller values occurring for larger bias as D′ increases. For further increases in the cutoff distance, this axis would become even more concave and the magnitude of the Δ values would increase (as can be seen in Fig. 6c).
If the verification is to be performed on a fixed domain, one possible way to eliminate the sensitivity of the Baddeley metric to the domain-relative location would be to both employ the cutoff transformation and append the constant values (e.g., zeros, for precipitation) to the edge of the domain for a distance equaling the cutoff value. In this way, a synthetic buffer zone of width c is added to the edges of the domain, eliminating the edge effects previously discussed. Implementing this approach would reduce the value of Δ because the inclusion of additional zero values within the padding area would decrease the value of the Lp norm. While reducing the sensitivity to edge effects, this modification would only be practical for values of c that are relatively small compared to the extent of the domain because of the increased computational expense associated with the larger, padded domain.
Each measure examined here is sensitive to many aspects of the forecast situation. It is of particular relevance that when compared to the Hausdorff distance, the Baddeley image metric is influenced by additional characteristics related to the arrangement of the forecast and observation. As such, it might not be appropriate to use a particular distance measure in certain circumstances. Establishing the facets of the forecast problem that are most important will assist in determining the types of sensitivities that are acceptable in a given application.
6. Summary
This paper examines the behavior of four binary distance measures, focusing on two that obey the properties of binary image metrics. To facilitate a better understanding of these metrics, they were applied to a hypothetical forecast situation designed to test the sensitivity of these metrics to base rate, displacement error, and bias.
For three of the measures (Hd, MHD, and PHD), only the magnitude of each measure was influenced by the base rate and changed proportionally to the size of the circle used to represent the observation. The minimum value of these three measures was achieved for B = 1, regardless of the base rate. This was not the case with Baddeley's delta metric. The shape of the Δ contours was found to be sensitive to the base rate of the observation, as well as the location of the forecast and observation in relation to the domain, and the cutoff distance used in its computation. In addition, Δ attained its minimum value for increasingly biased forecasts as the base rate, cutoff distance, and distance to the edge of the domain were independently increased.
Three of these distance measures were designed to circumvent the instability that the Hausdorff distance can have in the presence of outliers or in noisy images. Because of the mixed discrete/continuous nature of precipitation fields in particular, many forecasts and observations are susceptible to outliers when thresholded. While this could result in unpredictable behavior in the distance measures, this matter is beyond the scope of the current study.
While all forecast situations cannot readily be condensed into the simplified two-circle design used in this study, this framework facilitates a basic understanding of how these distance measures may behave in more complex forecast scenarios. Factors such as the shape, orientation, texture, and mixed discrete/continuous nature of precipitation fields may influence and mask the impacts of changes in bias, base rate, and simple displacement. The Baddeley image metric has already been applied to the data used in the ICP, which provides some measure for how these metrics respond to more complex scenarios (Gilleland 2011). If any of these measures are used in verifying forecasts made for actual events, their sensitivity to base rate, bias, and displacement errors needs to be considered when interpreting the results.
Acknowledgments
Portions of this work were supported by NSF Grant ATM-0756624. Benjamin Schwedler was supported through a NSF Graduate Research Fellowship. Computational resources were provided by the Purdue University Rosen Center for Advanced Computing. Comments and suggestions from the two anonymous reviewers helped improve the clarity of this manuscript.
REFERENCES
Ahijevych, D., Gilleland E. , Brown B. G. , and Ebert E. E. , 2009: Application of spatial verification methods to idealized and NWP-gridded precipitation forecasts. Wea. Forecasting, 24, 1485–1497.
Baddeley, A. J., 1992a: An error metric for binary images. Robust Computer Vision: Quality of Vision Algorithms, W. Förstner and S. Ruwiedel, Eds., Wichmann, 59–78.
Baddeley, A. J., 1992b: Errors in binary images and an Lp version of the Hausdorff metric. Nieuw Arch. Wiskunde, 10, 157–183.
Baldwin, M. E., and Kain J. S. , 2006: Sensitivity of several performance measures to displacement error, bias, and event frequency. Wea. Forecasting, 21, 636–648.
Borgefors, G., 1984: Distance transformations in arbitrary dimensions. Comput. Vision Graph. Image Process., 27, 321–345, doi:10.1016/0734-189X(84)90035-5.
Borgefors, G., 1986: Distance transformations in digital images. Comput. Vision Graph. Image Process., 34, 344–371, doi:10.1016/S0734-189X(86)80047-0.
Breu, H., Gil J. , Kirkpatrick D. , and Werman M. , 1995: Linear time euclidean distance transform algorithms. IEEE Trans. Pattern Anal. Mach. Intell., 17, 529–533, doi:10.1109/34.391389.
Brill, K. F., 2009: A general analytic method for assessing sensitivity to bias of performance measures for dichotomous forecasts. Wea. Forecasting, 24, 307–318.
Brill, K. F., and Mesinger F. , 2009: Applying a general analytic method for assessing bias sensitivity to bias-adjusted threat and equitable threat scores. Wea. Forecasting, 24, 1748–1754.
Danielsson, P., 1980: Euclidean distance mapping. Comput. Vision Graph. Image Process., 14, 227–248, doi:10.1016/0146-664X(80)90054-4.
Dubuisson, M.-P., and Jain A. K. , 1994: A modified Hausdorff distance for object matching. Proc. Int. Conf. on Pattern Recognition, Jerusalem, Israel, IEEE, 566–568, doi:10.1109/ICPR.1994.576361.
Gilleland, E., 2011: Spatial forecast verification: Baddeley's delta metric applied to the ICP test cases. Wea. Foreasting, 26, 409–415.
Gilleland, E., Lee T. C. M. , Halley Gotway J. , Bullock R. G. , and Brown B. G. , 2008: Computationally efficient spatial forecast verification using Baddeley's delta image metric. Mon. Wea. Rev., 136, 1747–1757.
Gilleland, E., Ahijevych D. , Brown B. G. , Casati B. , and Ebert E. E. , 2009: Intercomparison of Spatial Forecast Verification Methods. Wea. Forecasting, 24, 1416–1430.
Moeckel, R., and Murray A. B. , 1997: Measuring the distance between time series. Physica D, 102, 187–194, doi:10.1016/S0167-2789(96)00154-6.
Murphy, A. H., 1991: Forecast verification: Its complexity and dimensionality. Mon. Wea. Rev., 119, 1590–1601.
Murphy, A. H., and Winkler R. L. , 1987: A general framework for forecast verification. Mon. Wea. Rev., 115, 1330–1338.
Rosenfeld, A., and Pfaltz J. , 1966: Sequential operations in digital picture processing. J. Assoc. Comput. Mach., 13, 471–494.
Rosenfeld, A., and Pfaltz J. , 1968: Distance functions on digital pictures. Pattern Recognit., 1, 33–61, doi:10.1016/0031-3203(68)90013-7.
Rucklidge, W., 1996: Efficient Visual Recognition Using the Hausdorff Distance. Springer, 178 pp.
Venugopal, V., Basu S. , and Foufoula-Georgiou E. , 2005: A new metric for comparing precipitation patterns with an application to ensemble forecasts. J. Geophys. Res., 110, D081111, doi:10.1029/2004JD005395.
