a. Geometric indices

The geometric indices are those proposed in AghaKouchak et al. (2011), and are aimed at describing the spatial pattern of individual binary fields, rather than a direct comparison between two fields. While they propose applying the indices to the binary field as a whole, they are also useful to apply to individual features; one of which, the area index, has been in use in the MODE method since its inception (Davis et al. 2006a,b, 2009). The three geometric indices are: area index (A_index), shape index (S_index), and connectivity index (C_index).

Table 1 displays the equations used to calculate these indices; note that each is a value between zero and one inclusive. The area index (A_index) measures how convex versus concave a field is. For example, a circle is convex where a C-shaped object is concave. When applied to an entire field, A_index describes how structured or dispersive the field is overall. Values of A_index that are closer to one indicate a more structured pattern, and those closer to zero indicate a more dispersive one.

Table 1.

Equations for the geometric indices. Each index falls between zero and one inclusive. N_P is the number of nonzero grid points in the event area and N_C is the number of isolated clusters of events (called connected components).

View Table

The shape index, S_index, is very similar to the area index, and typically provides redundant information. Nevertheless, as some of the fields will illustrate, it does inform about different properties. Values close to zero indicate more dispersive fields while values closer to one indicate a more compact area of nonzero grid points that is close to circular in shape.

The connectivity index, C_index, measures how connected a field is. Values near one indicate a highly connected field and those near zero indicate a less connected field (e.g., with numerous smaller features). C_index is identically one for a single feature, so it is perhaps most useful when applied to an entire field.

If no events occur in the field A_{convex hull} = 0, P = 0, N_P = 0, and N_C = 0 and thus the values of all three indices are not defined because of the divisions by zero. Finally, note that a single circular object should have S_index = 1, but small errors relating to the computations on the grids accumulate to where the estimated values will be less than unity.

b. Distance measures

Different distance measures might measure forecast quality in a variety of ways, but their common property is that they all provide results in terms of spatial distance. One interesting point to keep in mind is that while the test fields represent perfectly known situations, for example two identical circles where one is displaced, different methods give different information about the similarity between two fields. One distance measure, for example, might provide information about the average or total amount of transformation of one field to another where another might provide some kind of average summary of actual distances between events, which may differ from the precise transformation invoked. Moreover, some methods might be designed to provide a measure of spatial distance that would reflect a subjective estimate of spatial displacement of events in one field compared to events in the other field. It is hoped that these comparisons will help illuminate how each method informs about the similarity and dissimilarity between two fields for potential users.

The distance measures and geometric indices investigated in this paper are applied to the entire field as a single entity, rather than on individual features within a field (as is done, e.g., in MODE, Davis et al. 2006a,b, 2009). The measures considered in this work could, however, be applied to individual features within a field. In fact, some of the case comparisons proposed could prove challenging for a method, such as MODE, in terms of identifying merges and/or matches of features.

One attribute for a distance measure comparing a field with event sets A and B that can be important for some purposes is for it to be a true mathematical metric, m(A, B) ≥ 0. The criteria requires that three properties be met: (i) identity m(A, B) = 0 if and only if A = B, (ii) symmetry m(A, B) = m(B, A), and (iii) the triangle inequality m(A, C) ≤ m(A, B) + m(B, C). The first is an obviously good property to have. The second property, however, may or may not be ideal in a forecast verification setting (Gilleland 2017) as the asymmetry of a measure may inform about false alarms as opposed to misses. Finally, the third property is important because it establishes a coherence in the measure. Namely, it ensures that if forecast A is closer to observation C than B in terms of the metric, then m(A, C) < m(B, C).

A general description of each measure is given below. For completeness, the equations defining each measure is provided in Table 2.

1. Centroid distance: The centroid distance (CDST) is the distance between the centers of mass of two fields. It is often used in feature-based methods as a measure of spatial closeness between two features (cf. Davis et al. 2009). It is a quick and easy measure to calculate, and is readily interpretable. Some of the new fields presented here will also highlight some of its shortcomings. Recall that, here, the distance is calculated between the centroids from the two entire field domains rather than a single feature within the domain. Some of the shortcomings might be mitigated by considering only individual features.

2. Baddeley’s Δ: Baddeley’s Δ metric (Baddeley 1992) is easily described using Fig. 1, which shows two binary fields, A and B in the top row. The next row shows their distance maps, which give the shortest distance from every point in the domain to the nearest point in the event sets. Baddeley’s Δ is a type of average of panel g) in the figure, which shows the absolute differences between the two distance maps for A and B. The result is a type of average distance between the two binary event sets. It is a modification of the well-known Hausdorff distance metric. Both are sensitive to the size and shape of the domain of the verification set.

   • Denote the nonzero grid points in one field by A and those in the other by B. The shortest distance from every point $\mathbf{s}\in \mathcal{D}$ to the nearest nonzero grid point in A (and similarly for B) are calculated and denoted by d(s, A) and d(s, B), respectively. These resulting fields defined by d(s, A) and d(s, B) are called distance maps, and can be calculated in a computationally efficient manner (e.g., Borgefors 1986; Meijster et al. 2000; Fabbri et al. 2008).

   • The metric is an L_p norm of the magnitude difference between the distance maps for A and B where p is a user-selected parameter where p = ∞ recovers the Hausdorff distance. The cutoff transformation [i.e., w(x) = min{x, c} for a user-chosen constant c] has been found to remove some of the sensitivity to outliers and domain effects (e.g., Schwedler and Baldwin 2011). However, Gilleland (2011) found that results can be sensitive to the choice of constant c. Results reported here are without any transformation because it was found that applying the cutoff transformation did not change any of the pertinent findings related to these geometric comparisons.

   • For the pathological case where Z(s) = 0 everywhere, a special rule is invoked with the distance map defined to be ∞ everywhere, and in practical applications, this value is reduced to be N; the size of the domain. Different definitions for N are possible. If the grid is n × m, then the size might be taken to be n + m, n × m (used here), $\sqrt{n^2+m^2}$, etc.

3. Squared and centered Baddeley Δ: One way to alleviate the issue of domain placement is to reposition the sets so that they are centered on a new square domain; but where they remain in their same relative positions to each other. Call the resulting metric the squared and centered Baddeley Δ, and denote it by ð (eth). If this approach is performed for all subsequent comparisons of different sets A and B, then ð will be consistent across the comparisons where Δ would not. In practice the new domain should be at least as large as the largest verification set to be compared in order to contain both event sets, A and B, so in general the new domain will be larger than the original $\mathcal{D}$. ð has the same definition as Δ when Z(s) = 0 everywhere, except that |N| will usually be larger. Of course, ð is still sensitive to |N|.

4. Mean-error distance: Again using Fig. 1 to illustrate, the mean-error distance (MED; Baddeley 1992; Gilleland 2017) measures the average distance of one event set to the nearest points in another event set. MED (B, A) is the average value of the nonzero values shown in Fig. 1e of the figure, and MED (A, B) is the average of the nonzero values in Fig. 1f. Note that the domain is not shown in these panels to emphasize that the distances are only averaged over the events defined by the respective binary sets.

   • The MED measures the average shortest distance of nonzero grid points in one field to the nearest nonzero grid points in another. It is an asymmetric measure because MED(A, B) ≠ MED(B, A). This asymmetry has a useful interpretation for forecast verification because MED(forecast, observation) gives a spatial distance measure of misses and MED(observation, forecast) gives a spatial distance measure of false alarms. While any measure for d(s, ⋅) can be used, the computationally fast distance transform used to calculate Baddeley’s Δ can be exploited here as well so that the computations can be made in an operational setting. Unlike Δ, however, the MED is insensitive to the specific domain’s size or shape; meaning that its value doesn’t change if, for example, the domain size were increased. A few modifications of the MED, that are symmetric, and therefore true mathematical metrics include:

     $\begin{array}{c}\text{avg MED}\left(A,B\right)={\displaystyle \frac{1}{2}}\left[\text{MED}\left(A,B\right)+\text{MED}\left(B,A\right)\right],\\ \text{minMED}\left(A,B\right)=\min \left\{\text{MED}\left(A,B\right),\text{MED}\left(B,A\right)\right\}\\ \text{maxMED}\left(A,B\right)=\max \left\{\text{MED}\left(A,B\right),\text{MED}\left(B,A\right)\right\}.\end{array}$(1)

   • Again, because of the rule that the distance map is ∞, taken to be |N|, at every grid point when Z(s) = 0 everywhere, MED(A, B) is still defined when A is empty (zero everywhere), if B > 0 somewhere. However, MED(B, A) would be undefined.

5. Zhu’s measure: Zhu’s measure (Zhu et al. 2011) is designed to be a true mathematical metric in the case where two competing forecast fields are to be compared against the same observation field. However, in the simplified version that compares a single forecast field to an observed field, it is not a true metric, and is therefore referred to as Zhu’s measure here. It is defined by a weighted average between the root-mean square deviation between the two binary fields and MED (A, B). The reliance on MED in the second term demonstrates that Z(A, B) ≠ Z(B, A) in general, and is subsequently not symmetric unless λ₂ = 0. Here, λ₁ = λ₂ = 1/2 so that each term is weighted equally.

6. Fractions skill score displacement: The fractions skill score displacement (dFSS; Skok and Roberts 2018) uses the fractions skill score (FSS; Roberts and Lean 2008; Roberts 2008) to provide information about the spatial displacement between events in one field compared to the events in another field. The FSS gives a measure of forecast skill at different spatial scales and the dFSS value is obtained via determining the scale at which FSS exceeds 1/2 while separately treating the regions with and without overlap.

Table 2.

Equations for the distance measures. Here, ||⋅|| is any distance norm (e.g., great-circle or, here, Euclidean), $\mathcal{D}$ is the spatial domain with coordinates $\mathbf{s}=\left(x,y\right)\in \mathcal{D}$, and $A\subset \mathcal{D}$ and $B\subset \mathcal{D}$ are binary event sets for two different fields Z(s). I_A(s) and I_B(s) represent the binary fields containing the sets A and B, respectively. d(s, A) is the distance map for I_A(s), N is the size of the domain, and n_A is the number of points in the set A. w(⋅) is any concave function [i.e., w(t + u) ≤ w(t) + w(u)], and is usually taken to be w(x) = min{x, c} for some user-chosen constant c. See the text for details on p.

View Table

View Full Size

(a),(b) Example of two binary fields (A and B) to be compared. (c),(d) Distance maps for A and B, respectively. (e) Distance map of A multiplied by the binary field B. (f) Distance map of B multiplied by the binary field A. (g) Magnitude difference in distance maps for A and B. Domain boundaries are not displayed for (e) and (f) for instructional purposes. See text.

Citation: Monthly Weather Review 148, 4; 10.1175/MWR-D-19-0256.1

• Download Figure
• Download figure as PowerPoint slide

The dFSS was designed with the aim of providing a measure of spatial distance that would reflect a subjective estimate of spatial displacement of events in the two fields. In Skok and Roberts (2018) the behavior of dFSS was analyzed for various idealized and real setups and the results showed that the dFSS can indeed be used to determine spatial displacement in a meaningful way. The results also showed that the larger events have a larger influence on the resulting score value. Further, dFSS exhibits some dependence of the resulting value on the orientation of the displacement and the presence of the nearby domain border; however, the influence of these two factors on the score value tends to be small.

The dFSS requires that the frequency bias be small (the number of nonzero points in the two fields should be similar) and Skok and Roberts (2018) recommend that it should not be used in cases when the frequency bias is outside of the interval between 1/2 and 2; an appropriate value of bias can usually be achieved by using a percentile threshold.

a. Geometric indices

Figures 2 and 3 show the geometric indices introduced by AghaKouchak et al. (2011) for the pathological fields. These indices are not comparative by themselves as they are only applied to individual fields from the verification set, rather than evaluating how closely one field matches the other. Comparing their values across fields, however, tells about the similarity/difference in terms of the overall spatial pattern, or texture of the fields. None of the indices is defined when no events occur in the field because of division by zero.

By its definition, A_index should be 1.00 for any single point in $\mathcal{D}$. However, the software used to calculate the value for this index (R package SpatialVx; Gilleland 2018) returns a missing value when there is only a single nonzero value placed in either corner. Further, the computation returns 0.50 for the single event in the middle of the field because the function returns the value 2 for the area of the convex hull. In theory, however, for each single-point case, the area should be one. Results for S_index for these fields are very similar, but the software returns NA for single event points. C_index is not defined for P1, and is not very interesting for the rest of these fields as they are mostly single contiguous features so that C_index is identically one for all fields. For the four-point fields, its value is 0.50.

Figure 5 shows the geometric indices for each of the circular fields. The A_index is found to have a slight edge effect, as is evidenced by the small difference in its value for C3 compared to those of C1 and C2. This effect may simply be the result of computational idiosyncracies of the software used to calculate them; the difference is only on the order of one hundredth. The similar S_index does not suffer from the edge effect problem.

To illustrate how the area and shape indices inform about dispersiveness, compare C6 with C7. The two identical circles are slightly farther apart for C7 than for C6, yielding slightly lower values of these indices, and even lower for C8 where they are separated even more. The connectivity index, on the other hand, is the same for all three fields. The C_index is again less interesting for these fields as they are all composed of one or a couple continuous event areas. They do demonstrate how having only two sets of isolated clusters only lowers the value by a negligible amount.

The elliptical fields (Fig. 7) demonstrate the primary difference between A_index and S_index. The former is nearly one as the ellipse is fully concave, but it is far from a perfect circle so that S_index is much lower ranging from about 0.54 to about 0.65. E17 and E18 have a low A_index of about 0.16 despite having a tight concave area of activity, but nevertheless consist of small “storm” cells. The S_index is slightly higher but still very low at about 0.24, which is not surprising as the nonzero grid cells form an elliptical region. The reason why S_index is not identical for fields E1 to E4 is that E1 and E3 both have 1553 events where E2 and E4 have slightly more at 1573, which is a result of the way in which the fields are created.

For the small-scale scattered events (S1 to S3 in Fig. 9), the area index simply reflects the frequency of occurrence of the rain areas (5%) because it is essentially the area of the circular envelope divided by the total area of rain for each of the fields. The shape index is similarly defined to the area index, but is related to the square root of frequency because perimeters instead of areas are used to calculate the index value. The square root of 0.05 is approximately 0.22 which is similar to 0.2 given by the index. The small difference occurs because of the way P_min is estimated. One might consider a more precise calculation for the perimeter, but it is unclear what the added value of the S_index is over the A_index in this setting; if considering a single object within a field, then the S_index can provide useful additional information. The C_index for these fields describes the texture of the field as having numerous discontiguous rain areas with very low values; all of which are rounded to about 0.09.

The fields with holes (H1 and H2 in Fig. 9) are not very dispersive, which is captured by these indices. Because all one-valued grid cells are connected, C_index is one for both fields as all one-valued grid points are connected.

For the noisy fields (N1 to N4 in Fig. 9), the shape and area indices are heavily influenced by the noise; giving very low values suggestive of high dispersiveness. All the values for C4 alone without P5 added to the field give shape, area and connectivity indices at or near 1, indicating undispersive fields. After adding P5 to C4 (i.e., N3) or P3 (i.e., N4), both shape and area indices drop drastically to around 60%; showing a high sensitiviy of these indices to noise because a single added event can cause a substantial difference in the index value. The connectivity index perhaps gives more reasonable information under this situation because its relatively low values of 0.56 and 0.55 for the cases with multiple scattered events is suggestive of low, but not very low connectivity. On the other hand, the addition of the single point does not reduce its value much; going from 1.00 to 0.97 with the additional event. It should be noted that AghaKouchak et al. (2011) were aware of the high sensitivity of these indices to isolated outliers and recommended removing noise before applying the measure.

b. Distance measures

With so many cases, it is difficult to navigate the numerous possible results. In what follows, a summary of each measure’s propoerties for each set of comparison cases is given with an aim of only highlighting interesting properties. An overall summary is given at the end.

1. Pathological comparisons: Figure 4 shows the pathological comparisons with results for the distance measures. The centroid for the null fields is undefined. A single event (one grid point), however, is all that is necessary for it to be defined; and a single event is its own centroid. For example, the centroid for P3 is (1, 1), while for P4 it is (200, 200) and P5 it is (100, 100). Therefore, the CDST for P1P1, P1P3, P1P4 and P1P5 is undefined. As expected all the measures give a perfect value (zero distance) for the comparison P2P2, which consist of two identical full fields.

   • For the P2P5 comparison the point is displaced by $1/\sqrt{2}=0.71$ from the center of the domain, because the domain does not have an exact center, so is also the result given by the CDST. The P6P7 comparison gives a CDST of about 0.35. The P2P6 CDST is calculated to be identically zero because P6 has a single point in each of the four corners so that the centroids of both P2 and P6 are the same. Finally, the CDST for (r)P1C1 is the same as for (r)P1P5 at about 141.42. The CDST for (r)P2C1 is about 0.71.

   • As previously described, the distance map used to calculate Δ is defined to be c everywhere when no one-valued grid points are in the domain, and for practical purposes, when this value is ∞, it is reduced to the size of the domain; in this case, it is reduced to 400. When the entire domain is one-valued, it is simply zero. Therefore, the P1P1 (and P2P2) comparison is identically zero for Δ, a perfect score. (r)P1P2 yields the max value of c in each case, which for ð is just slightly larger because of the new domain on which it is constructed.

   • It is interesting that Δ gives the same answer, as would be desired, for the comparisons (r)P1P3 and P1P4, which both compare the null field against the field with only one nonzero grid point, but where the one-valued grid point is in two opposite corners from P3 to P4. However, Δ gives a different answer for P1P5, which is the same as the other two except that the single one-valued grid point is now in the center of the field instead of a corner. For Zhu’s measure, the MED component is undefined for P1P1 because the set over which the average is to be taken is empty. If λ₂ = 0, then Zhu’s measure is also zero. One might consider defining MED to be zero for this case, as is considered in the results here, because both sets A and B are empty.

   • Because the majority of pathological comparisons include either one or two fully empty fields or exhibit very large frequency bias, the dFSS value is not defined for most of these comparisons. The only exceptions are P2P2, where the dFSS gives a perfect result of 0 and P6P7 where the dFSS gives a result of 63 points, which is an underestimation compared to the true displacements of 100 points. This underestimation is probably related to the fact that P6P7 exhibits an extreme example of the border effect because all the isolated points are located on the opposite sides of the domain at the borders.

   • Generally, none of the measures provides satisfactory results for the limiting case of no event areas, and each is fairly sensitive to the addition of only a few events. A possible solution for a user needing to summarize over numerous comparisons (e.g., over many time points) would be to report such cases as not applicable for these measures, and complement every case with the frequency bias and the size of event areas in each field of the verification set. Determining a lower bound of event area size is a question best left to specific users to decide depending on the variable being studied and their particular needs.

2. Circle test-case comparisons: Figure 6 shows the results of the circle test-case comparisons for the various distance measures. Baddeley’s Δ metric is using p = 2 and the constant, c = ∞. Because this metric is a true mathematical one, it is symmetric. The sensitivity of the metric to the location of the pairs of objects in the field can be seen as the value changes depending on the location of the two objects. For example, C1C2 is identical to C2C3 in every way except their position within the domain, where the latter is near the boundary. Additionally, the comparison C2C4 is also identical to C1C2 and C2C3 except that they are positioned vertically instead of horizontally to one another, but the values for Δ changes, although only slightly, for each comparison at 29, 31, and 28, respectively. The Hausdorff distance does not suffer from this issue and results in a value of about 40 for all three comparisons. Interestingly, CDST also equals 40 meaning that the average translation error is equal to the maximum translation error for this set of test-case comparisons.

   • Δ penalizes C2C11 more than C1C2, C2C3 or C2C4, which are effectively subset cases of C2C11. That is, it does penalize for having over or under forecast the areal extent. The Hausdorff distance does not detect this behavior; having the same value for each of these comparisons.

   • The squared and centered version of Baddeley’s Δ, ð, is not shown in the figure but yields more consistent results as expected; its value being the same for C1C4 and C3C4. Moreover, it penalizes C2C11 more than it does the first three comparisons, although only by about half a grid point when c = ∞. It also yields a major difference between comparisons C6C7 and C6C8, which involve a similar error, but where C6C8 has a much larger displacement for the southern circle than C6C7. ð slighlty favors comparison C1C9 over C1C10, but by only about 1/2 and 1/3 of a grid point for c = 50 and c = ∞, respectively. C1C9 has some overlapping events while C1C10 has no areas of overlap. On the other hand, C1C10 has more correct negative area, which is the reason for the closeness in the two ð values. Both Δ and ð give sensible values for the comparisons where the circles are the same size and shape and have a slight enough translation error that they overlap (C2C5, C3C5, and C6C12).

   • The dFSS gives a reasonable estimate for the translational displacement for the comparisons with a single displaced circle (C1C2, C2C3, C2C4, C1C4, C3C4, C2C5, C3C5) albeit somewhat underestimating the true size of the displacement. Similarly to Δ the dFSS is also somewhat sensitive to the positioning of the circle pairs as its value also changes slightly depending on their relative positions within the domain (34, 32, and 34, for C1C2, C2C3, and C2C4, respectively), mainly changing depending on how close to the border the circles are located. It also gives close to the true average displacement for the C6C7/C6C8 comparisons, which are composed of two sets of circles where one is displaced and the other is not displaced. For example, the true displacement of the bottom set of circles in C6C7 is 40 points for an average displacement of 20, whereas the dFSS gives 17 points. A somewhat similar comparison is C13C14 where the smaller upper set is displaced by 25 points while the larger bottom set is displaced by 50 points. Because larger events have a larger influence on the dFSS, the resulting value of 43 is closer to 50 than to 25 points. dFSS also does a good job for comparison C6C12 where the two sets of circles are displaced by $20\sqrt{2}\approx 28$ points while the dFSS gives 25 points. For comparisons C2C11, C1C6, C1C9, and C1C10 the dFSS value is not defined because the frequency bias is too large.

   • For the comparisons with a single displaced circle the CDST gives a perfect estimate of translational displacement, and similarly for C6C7, C6C8, and C13C14. The limitations of the CDST become very apparent in comparisons C1C6, C1C9, C1C10, and C6C12 where the resulting CDST is 0 despite that the fields are very different from one another; but each pair has the same centroids as each other. Of course, if the intent is to compare the detailed structure of different fields, it would not be recommended to use a summary measure such as the CDST in the same way as the mean should not be used to measure variability, and these results illustrate the need to understand what sort of information each method provides.

   • Because Zhu’s measure is a linear combination of RMSE and MED, it is not surprising that the values are higher, but analogous as the MED. In each case, these values give a reasonable summary of how closely one field matches the other, where the asymmetry, again, indicates whether or not one event set is a subset of the other or not; Zhu’s method emphasizes the asymmetry less because of the symmetric RMSE component.

3. Elliptical comparisons: Elliptical comparisons with the distance measure results are shown in Fig. 8. It has already been seen that Baddeley’s Δ and dFSS are somewhat sensitive to the relative positions of event areas in the domain so it is not surprising that these values fluctuate a bit for comparisons E1E9, E2E10, E3E11, and E4E12, which all have the same two ellipses separated by the same translation error, but rotated in different directions. Similarly, as mentioned earlier, the dFSS value is not defined in the presence of large bias. Finally, the deficiencies with CDST have already been addressed as well. Otherwise, the various distance measures provide similar results for the elliptical scenarios.

   • Comparisons E1E3 and E1E4 both have zero CDSTs with a rotational displacement. Similar comparisons were analyzed using dFSS in (Skok and Roberts 2018). Which of these cases is “best” will, as usual, depend on the specific user. For E1E3 and E1E4 the dFSS-derived distances are about 17% and 13% of the ellipse’s major axis length, which is consistent with the fact that about a quarter of the ellipse is overlapping (24% for E1E3 and 33% for E1E4). Similar results are given by Δ, MED and Zhu’s measure; noting that MED gives lower values and Zhu’s measure gives higher values, but they yield similar information on their own scales.

   • Comparisons E3E11 and E7E11 represent almost the same situation, but where one ellipse is much smaller (E7E11) than in the other (E3E11). Δ is slightly lower (indicating a better agreement between the two fields) for the more similar objects in E3E11 than E7E11. Similarly for avg MED(A, B), which is 17 for E3E11 and 20 for E7E11. On the other hand, min MED(A, B) = 17 for E3E11 and only 16 for E7E11, suggesting that E7E11 would be preferred while max MED(A, B) = 17 for E3E11 and 24 for E7E11, so that it more emphatically prefers E3E11 over E7E11. Again, which of E3E11 and E7E11 is “better” will depend on a specific user’s needs. An asymmetric measure such as MED, applied in both directions, provides a more thorough summary in this situation.

   • The most general conclusion from the elliptical comparisons by looking at Fig. 8 for each method is that greater overlap is the most important property to reduce each measure’s value closer to perfect. Reducing the size of one feature also seems to reduce the values of the measures; implying that a forecast could be hedged to improve performance by underforecasting event areas.

4. Scattered, holes, and noisy comparisons: Fig. 10 shows the results for the scattered, holes and noisy comparisons. The comparisons S1S2 and S1S3 are designed to test how the methods handle small-scale events scattered inside an envelope. In S1S2 the envelopes overlap. Because the positioning of nonzero grid points inside the envelope is random the displacement can be interpreted in terms of average nearest neighbor distance from each nonzero grid point to the closest nonzero grid point in the other field, which is about 2.35 grid points. The Δ, MED and dFSS all give very similar results that reflect the average nearest neighbor distance well. Zhu’s measure gives a much larger value while, as could be expected, the CDST gives a smaller value because the centers of mass of the two envelopes are near each other. In S1S3 the envelopes are displaced by 100 grid points and is reasonably well represented by values of CDST, dFSS, and MED; while Δ and especially Zhu’s measure give smaller values.

   • The comparison H1H2 is designed to test how the methods behave when the majority of the domain is covered by events with holes present in the fields. The results of H1H2 can be compared to C1C2 because these comparisons use identical fields with inverted values. As already mentioned, the result for H1H2 and C1C2 would be the same for the RMSE metric because RMSE treats zero and nonzero areas in an equal manner. However, all the tested measures give a better (smaller distance) value for the H1H2 than for C1C2. With the exception of Zhu’s measure, they all provide a very small distance; ranging between 0.3 and 2.3 grid points for the H1H2, while the distances for C1C2 tend to be about 10 times larger. This result shows that the tested distance measures tend to focus more on analyzing the spatial matching of event areas (i.e., region of nonzero values, which have a very good overlap in H1H2) as opposed to analyzing the nonevent areas (which have a very good overlap in C1C2). This property is also reflected in the fact that all of the tested methods have problems dealing with empty fields.

   • The comparisons N1N2, C1N3, and C1N4 are designed to test how sensitive the methods are to the presence of noise in the fields. The results of N1N2 can be compared to the result of C1C4. Compared to C1C4 the MED and Δ values are reduced by about 50% for N1N2, which indicates a strong sensitivity of these methods to noise. At the same time, the value of Zhu’s measure is reduced by about 20% while CDST and dFSS do not exhibit much sensitivity to noise. The position of the added noise is also a factor as noise close to existing features has less impact than noise that is far away from existing events in the same field and closer to the events in the other field. This effect is shown in C1N3 where a single nonzero value is added into the middle of the event in the other field; thus considerably reducing the distances provided by MED and Δ. On the other hand, in C1N4 the single added point is placed too far from the event in the other field to cause any major impact on the results of any measure.

5. Overall comparison: As mentioned previously, so many cases can make it difficult to synthesize all of the information. It is suggested that these comparisons be investigated for any new measure that primarily, or in part, accounts for location errors. While many cases may not shed any light on a particular measure’s properties, it is highly likely that at least one case will uncover important information. Such a case can then be emphasized when reporting on the measure’s properties. Additionally, in order to avoid cherry-picking results, it is important to try to display the results for all of the cases.

Figure 11 shows scatterplots for each measure for all but the pathological comparisons. In general, except for centroid distance, the measures rank the comparisons similarly, suggesting that each of these measures typically provides similar information regarding forecast performance. However, a closer look at the figure suggests some variations and differences in the information they provide; for example, Hausdorff distance is an outlier in the C6C8 comparison; moreover, the ordering of distance measures varies by comparison. In terms of general sensitivities, the most obvious conclusion is that greater overlap of event areas is the most important factor in reducing each measure’s value toward the perfect score, which is zero for each of these measures.

View Full Size

Scatterplots of results for (top left) circular, (top right) elliptical, and (bottom left) random rain, holes, and noisy comparisons. For the elliptical comparisons, the dashed vertical lines separate the different error types with the first set’s being translation-only, followed by size-only, then translation and size, followed by rotation-only, and finishing with size, rotation, and translation.

Citation: Monthly Weather Review 148, 4; 10.1175/MWR-D-19-0256.1

• Download Figure
• Download figure as PowerPoint slide