• Ångström, A. K., 1922: On the effectivity of weather warnings. Nord. Stat. Tidskr., 1 , 394408.

  • Katz, R. W., and Murphy A. H. , 1997: Forecast value: Prototype decision-making models. Economic Value of Weather and Climate Forecasts, R. W. Katz and A. H. Murphy, Eds., Cambridge University Press, 183–217.

    • Search Google Scholar
    • Export Citation
  • Liljas, E., and Murphy A. H. , 1994: Anders Ångström and his early papers on probability forecasting and the use/value of weather forecasts. Bull. Amer. Meteor. Soc., 75 , 12271236.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1966: A note on the utility of probabilistic predictions and the probability score in the cost–loss ratio decision situation. J. Appl. Meteor., 5 , 534537.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., and Wilks D. S. , 1998: A case study of the use of statistical models in forecast verification: Precipitation probability forecasts. Wea. Forecasting, 13 , 795810.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mylne, K. R., and Legg T. P. , 2002: Early warnings of severe weather from the ECMWF ensemble prediction system. Preprints, 16th Conf. on Probability and Statistics in the Atmospheric Sciences, Orlando, FL, Amer. Meteor. Soc., J254.

    • Search Google Scholar
    • Export Citation
  • Richardson, D. S., 2000: Skill and relative economic value of the ECMWF ensemble prediction system,. Quart. J. Roy. Meteor. Soc., 126 , 649667.

  • Slovic, P., Fischhoff B. , Lichtenstein S. , Corrigan B. , and Combs B. , 1977: Preference for insuring against probable small losses: Insurance implications. J. Risk Insur., XLIV. 237257.

    • Search Google Scholar
    • Export Citation
  • Stewart, T. R., 1997: Forecast value: Descriptive decision studies. Economic Value of Weather and Climate Forecasts, R. W. Katz, and A. H. Murphy, Eds., Cambridge University Press, 147–181.

    • Search Google Scholar
    • Export Citation
  • View in gallery
    Fig. 1.

    The frequency with which probabilistic forecasts of an event with a particular probability are issued, as a function of the parameter α. Low values of α correspond to events for which high-probability forecasts are relatively common. High values of α correspond to events for which high-probability forecasts are rare.

  • View in gallery
    Fig. 2.

    The compliance rate, q, as a function of the probability threshold at which warnings are issued: four different values of the parameter λ are shown. The model of compliance, q, as a function of warning threshold, pw, is q = pλw. Thus, a value of λ = 0 corresponds to full compliance (q = 1) regardless of the warning threshold

  • View in gallery
    Fig. 3.

    (a) The optimum warning probability threshold as a function of event rarity parameterized by α and intolerance of false alarms parameterized by λ when the cost–loss ratio is 0.01. (b) The reduction (%) in the expected cost when the optimum threshold is used rather than the cost–loss ratio

  • View in gallery
    Fig. 4.

    As in Fig. 3, except for a cost–loss ratio of 0.1. Note that with this higher cost–loss ratio the maximum cost reduction is reduced to about 30%

  • View in gallery
    Fig. 5.

    As in Fig. 3, except for a cost–loss ratio of 0.5. Note that with this cost–loss ratio there is very little cost reduction and the optimum warning thresholds do not deviate from the cost–loss ratio as much as they did in Figs. 3 and 4

  • View in gallery
    Fig. 6.

    The ratio (%) of the value (average cost reduction) of the forecasts when there is imperfect compliance to the value when there is perfect compliance. A value of 100% indicates that incompliance has no impact on forecast value, while a value of 10% indicates that the actual value of the forecast is only 10% of the value that would be calculated using the conventional cost–loss method. The optimum warning threshold for the parameter values was chosen for the imperfect compliance, and a threshold of pw = C/L = 0.1 was used for the standard cost–loss comparison

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 869 287 14
PDF Downloads 591 233 12

The Boy Who Cried Wolf Revisited: The Impact of False Alarm Intolerance on Cost–Loss Scenarios

M. S. Roulston Pembroke College, Oxford University, Oxford, and Centre for the Analysis of Time Series, London School of Economics, London, United Kingdom

Search for other papers by M. S. Roulston in
Current site
Google Scholar
PubMed
Close
and
L. A. Smith Pembroke College, Oxford University, Oxford, and Centre for the Analysis of Time Series, London School of Economics, London, United Kingdom

Search for other papers by L. A. Smith in
Current site
Google Scholar
PubMed
Close
Full access

Abstract

Meteorologists often interpret the value of a probabilistic weather forecast using the binary cost–loss scenario. The socioeconomic benefit of such a forecast will depend on the compliance rate of users and, hence, the number of warnings that are not followed by the corresponding high-impact weather. A modified version of the canonical binary cost–loss problem in which the compliance rate of users is a function of the warning probability threshold, and hence of the “false alarm rate”, is presented. In this version of the problem, the value of the forecast can be enhanced by choosing a probability warning threshold that is higher than the cost–loss ratio. It is found that the advantage of modifying the probability warning threshold is greatest when the frequency of highly confident forecasts of an event is relatively high and when users are moderately intolerant of false alarms. Using this simple example it is illustrated that forecasters who issue nonprobabilistic, or “unequivocal,” forecasts are making implicit assumptions about the false alarm intolerance of users, as well as assumptions about their cost– loss ratios. Adopting a probabilistic approach to forecasting avoids these assumptions and separates the activity of forecasting from the activity of decision making.

Corresponding author address: Mark S. Roulston, Pembroke College, Oxford OX1 1DW, United Kingdom. Email: roulston@maths.ox.ac.uk

Abstract

Meteorologists often interpret the value of a probabilistic weather forecast using the binary cost–loss scenario. The socioeconomic benefit of such a forecast will depend on the compliance rate of users and, hence, the number of warnings that are not followed by the corresponding high-impact weather. A modified version of the canonical binary cost–loss problem in which the compliance rate of users is a function of the warning probability threshold, and hence of the “false alarm rate”, is presented. In this version of the problem, the value of the forecast can be enhanced by choosing a probability warning threshold that is higher than the cost–loss ratio. It is found that the advantage of modifying the probability warning threshold is greatest when the frequency of highly confident forecasts of an event is relatively high and when users are moderately intolerant of false alarms. Using this simple example it is illustrated that forecasters who issue nonprobabilistic, or “unequivocal,” forecasts are making implicit assumptions about the false alarm intolerance of users, as well as assumptions about their cost– loss ratios. Adopting a probabilistic approach to forecasting avoids these assumptions and separates the activity of forecasting from the activity of decision making.

Corresponding author address: Mark S. Roulston, Pembroke College, Oxford OX1 1DW, United Kingdom. Email: roulston@maths.ox.ac.uk

1. Introduction

Aesop's fable, the “The Boy who Cried Wolf,” is about a young shepherd boy guarding the village flock who cries that there is a wolf worrying the sheep. The villagers rush out to protect the sheep, but there is no wolf. This event is repeated two or three times before a wolf actually does show up on the hillside. The boy cries “Wolf!,” but to no avail: the villagers, no longer regarding the warning as credible, fail to act and the wolf decimates the flock. The traditional moral of this tale is that liars are not believed, even when telling the truth. But, are we being too harsh on the shepherd boy? Perhaps his mistake was to overestimate the rationality of the villagers. This traditional tale can be expressed in terms of a cost–loss problem, used in the analysis of the utility of weather forecasts (Ångström 1922; Murphy 1966; Liljas and Murphy 1994; Richardson 2000). The villagers' cost–loss matrix is
i1520-0434-19-2-391-eq1
where C is the price the villagers paid when they ran to aid the shepherd boy. This price is largely an opportunity cost, the value of the goods and services they could have been producing had they not been running about the hills. The loss associated with losing part of the flock to a lupine predator is represented by L. If the shepherd boy believed the probability that there was a wolf about is p, then the expected cost to the village is1
i1520-0434-19-2-391-e1
To minimize the expected cost, the boy should have cried wolf if p > C/L. If we assume that the flock of sheep was one of the village's most important assets, and that a wolf attack would result in the loss of several sheep, it seems reasonable to say that the villagers' cost– loss ratio, C/L, was quite modest. An estimate of C/L using contemporary prices gives
i1520-0434-19-2-391-e2
where three has been taken as a reasonable numerical value of “several.” The cost–loss analysis implies that the shepherd boy should have issued a warning even if he believed there was only a 10% probability of lupine activity. While this is a rational strategy, it would obviously have resulted in a high rate of “false alarms”— possibly as high as 90%, if the probabilities underlying the forecasts were reliable.2 The fact that the villagers were apparently unprepared to tolerate a false alarm rate of around 67% seems to suggest that they had not performed a cost–benefit analysis of responding to the cries of “Wolf!” Unfortunately, such behavior is not confined to the characters in stories written over two millenia ago. Psychological factors influence people's perception of risk, and this perception affects the way that they interpret meteorological forecasts. In this paper, we examine how the public's compliance with warnings might affect the value of the forecasts, and whether the forecast value can be increased by taking the user's response into account. We do this in the context of the familiar cost– loss scenario. The standard cost–loss scenario is a prescriptive decision model, which specifies how a user should respond. In contrast, descriptive decision models describe how users actually do respond (Stewart 1997). The modified cost–loss model we present is an idealized descriptive model, but it provides a quantitative illustration of the descriptive–prescriptive distinction. We hope that a better appreciation of the importance of this distinction will motivate further study in the area of descriptive modeling. The classical cost–loss problem illustrates that any forecast that is nonprobabilistic includes an implicit assumption about the cost–loss ratios of users. The extension presented in this paper demonstrates that such nonprobabilistic forecasts also include an implicit assumption concerning the tolerance of users to false alarms. Forecasters who issue strictly probabilistic forecasts do not make any such tacit assumptions. With probabilistic forecasts, the process of forecasting can be explictly separated from the process of decision making in a way that is not possible with unequivocal (nonprobabilistic) forecasts.

2. Cost–loss scenario with imperfect compliance

To model imperfect compliance with warnings in the cost–loss scenario we introduce a compliance rate, q. This is the probability that action will be taken if a warning is given. The expected cost, E[COST], if the forecast probability is p is thus
i1520-0434-19-2-391-e3
The first term on the lower line of the right-hand side (rhs) of Eq. (3) denotes the expected cost of protective action being taken, while the second term is the expected cost of a loss if no protective action is taken. If warnings are issued when the forecast probability exceeds some threshold, pw, then the expected cost averaged over time is given by
i1520-0434-19-2-391-e4
where ρ(p) is the frequency distribution of forecasted probabilities. The first term on the rhs of Eq. (4) represents days on which no warning is issued, and the second term is days on which a warning is issued. Equation (4) can be rewritten as
i1520-0434-19-2-391-e5
The problem becomes analytically tractable if we assume that the forecast probability, p, is drawn from a uniform distribution; that is, ρ(p) = 1 on the interval [0, 1]. In this case, Eq. (5) can be rewritten
i1520-0434-19-2-391-e6
where J is the expected cost per unit loss and c = C/L is the cost–loss ratio. If the compliance rate, q, is independent of the threshold at which warnings are issued, pw, then the minimum of J occurs when pw = c; this is the standard cost–loss scenario.
A more interesting situation arises if we assume that the compliance rate is a function of the false alarm rate. The definition of the false alarm rate we are using is
i1520-0434-19-2-391-e7
Assuming that the forecasts are reliable, and that ρ(p) = 1, Eq. (8) gives
i1520-0434-19-2-391-e9
Note that the false alarm rate is not, in general, equal to (1 − pw). The expected false alarm rate is a monotonically nonincreasing function of the warning threshold, pw, so we can write q = q(pw). As pw increases the false alarm rate decreases, and we can expect the compliance rate to rise. Thus, a reasonable form of q(pw) is a monotonically increasing function of pw. The simplest function of this form is q = pw. Assuming that relationship and differentiating J with respect to pw gives
i1520-0434-19-2-391-e10
Thus, the value of the warning threshold, pw, that minimizes the cost per unit loss is
i1520-0434-19-2-391-e11
If the cost of acting is small relative to the potential loss if no action is taken then c ≪ 1 and Eq. (11) gives
i1520-0434-19-2-391-e12
This threshold is close to the 60% threshold that the U.K. Met Office's National Severe Weather Warning Service (NSWWS) uses for its “early warnings” (Mylne and Legg 2002). Given the low cost–loss ratios often associated with severe weather, this threshold would appear too high with respect to the standard cost–loss analysis. Imperfect compliance on the part of the public provides a possible justification for the high value of the operational threshold.

3. Imperfect compliance when warnings are rare

While analytically tractable, the assumption of a uniform distribution of forecasted probabilities is rarely appropriate when considering extreme and rare events. Murphy and Wilks (1998) used a beta distribution to model the frequency of forecasted probabilities in a precipitation forecasting system. This type of distribution can represent the bimodality that an ideal forecasting system would have, with most forecasts being close to 0% or 100%. In this study an exponential distribution of forecasted probabilities is used. The frequency of forecast probabilities generated by a severe weather forecasting system based on the European Centre for Medium-Range Weather Forecasts (ECMWF) ensemble forecasts (Mylne and Legg 2002) generally declines with increasing probability—a pattern that can be represented by an exponential distribution described by a single parameter. The analysis presented could be applied to other forms of ρ(p), including estimates of ρ(p) made using actual forecasts. The exponential distribution is given by
i1520-0434-19-2-391-e13
where α parameterizes the rarity of higher forecast probabilities. The normalization of ρ(p) is such that 10 ρ(p) dp = 1. When the distribution of forecast probabilities is given by Eq. (13) then the frequency with which the forecast probability is equal to or exceeds r is
i1520-0434-19-2-391-e14
As expected, as α increases the frequency of probability forecasts exceeding r decreases. Figure 1 shows the form of ρ(p) for three different values of α. Values of α close to zero correspond to an almost uniform distribution on the interval [0, 1]. If the forecasted probabilities are reliable, then the climatological probability of the event is the expected value of p; that is,
i1520-0434-19-2-391-e15
Given that the warning threshold is pw, and the compliance rate is q, the time-averaged expected cost is given by substitution of Eq. (13) into Eq. (5). Integration gives
i1520-0434-19-2-391-e16
where β = eα and γ = 1 − eα. We seek the value of pw that minimizes J. If the compliance rate, q, is independent of pw then setting the derivative of dJ/dpw equal to zero gives the standard cost–loss scenario result, p*w = c. We can now quantify how user intolerance to false alarms impacts the forecast value.

In the remainder of this section we perform the same analysis as for the case of a uniform distribution of forecast probabilities, but, in addition to using an exponential distribution of forecast probabilities, we introduce a more general dependence of the compliance rate on the warning threshold. We do this to show how the value of unequivocal forecasts varies with the frequency of the event, the cost–loss ratio of the user, and the intolerance of the user to false alarms.

We parameterize the intolerance of the user to false alarms with the following model:
qpλwλ
The parameter λ is a quantification of the strength of the “cry wolf effect.” It will also be assumed that the probability forecasts are reliable. If λ = 0 then the compliance rate is 1 and the standard cost–loss scenario is recovered. If λ is small then the compliance rate remains quite high until the warning threshold, pw, is set very low (the case of frequent warnings). In the case of pw = 0, warnings occur all the time, and the compliance rate falls to zero. In this case, there is a major difference between ideal rational users of the traditional cost–loss scenario who will always take protective action if c = 0 and the intolerant users of the modified scenario who will never take action. This unwillingless to protect against improbable, but potentially catastrophic, events has been studied in the context of attitudes to insurance (Slovic et al. 1977). If pw is set to one, warnings are only issued when there is certainty, and total compliance is achieved. The extreme cases of pw = 0, 1 are the same for all values of λ > 0, but higher values of λ correspond to less tolerance of false alarms. The higher the value of λ the more rapidly the compliance rate falls as the warning threshold is reduced from near certainty. Figure 2 shows the compliance rate as a function of the warning threshold for four different values of λ.

In this more general case, finding the global minimum value of the function J(pw) is not analytically tractable. The minimum value of J, and the corresponding optimum value of pw that gives this minimum value, can be found numerically. For cost–loss ratios of c = 0.01, c = 0.1, and c = 0.5 the results are shown in Figs. 3, 4, and 5, respectively. Figure 3a shows the optimum value of pw as a function of α and λ. Figure 3b gives the percent reduction in the average expected cost when this optimum is used, rather than a value of pw = c = 0.01, which is what the conventional cost–loss analysis would suggest. The maximum cost reduction is over 50% and is obtained for α ≪ 1 and λ ≈ 0.2. This corresponds to an almost uniform frequency distribution of forecast probabilities and a relatively, but not perfectly, compliant user. The optimal warning threshold in this case is pw ≈ 0.30, which is considerably higher than the cost–loss ratio of 0.01. Inspection of Fig. 3a suggests that the value of 60% used by the NSWWS suggests a value of λ > 1, and since the effective value of α for extreme events is likely to be at the high end of the range shown, the implied value of λ may be more than 3. Such a value would correspond to a user that is quite intolerant of false alarms. Reference to Fig. 3b indicates that in this case the reduction in the average expected cost obtained by inflating the warning threshold is quite low, less than 5%. Figure 4 shows the same analysis as Fig. 3, except for the case of a cost– loss ratio of c = 0.1. Again, the 60% contour corresponds to relatively high values of λ, and again the cost reductions obtained for high values of α and λ are below 5%. In this region of the αλ plane, the high value of λ means that the user tends to only respond to almost certain forecasts, but the high value of α means that such forecasts are highly uncommon; hence, the overall impact of accounting for imperfect compliance is small. Figure 5b indicates that the cost reductions possible by accounting for imperfect compliance when the cost–loss ratio is 0.5 are not very large for any combination of α and λ. In Figs. 3 and 4 it can be seen that the biggest reductions in expected cost, for modest values of α, are obtained for values of 0 < λ < 1. In this range, the user deviates from the perfect compliance enough so that taking this deviation into account matters, yet they do not deviate so much that it is infeasible to compensate for their intolerance of false alarms.

The introduction of imperfect compliance increases the average cost incurred relative to the standard cost– loss scenario. The value of the forecast is the difference between the average cost incurred when the forecasts are used compared to the average cost incurred in the absence of forecasts. The ratio of the values of the forecasts in the imperfect compliance case, with an optimal warning threshold, and in the standard perfect compliance case is given by the following expression:
i1520-0434-19-2-391-e18
Figure 6 shows the value of the ratio in Eq. (18) for different values of λ and α. The cost–loss ratio used was c = 0.1. From Fig. 6 it can be seen that as the rarity of events increases, and the user's intolerance for false alarms increases, the actual potential value of the forecasts falls in relation to the value that the standard cost–loss analysis predicts. Replacing COST (zero compliance) in Eq. (18) with the average cost when decisions are based on climatological probabilities, COST (climatology) will reduce the value of the forecasts and reduce the ratio defined by Eq. (18). That is, if we assume that users, in the absence of a forecast, make better decisions than zero compliance, then the value of forecasts with imperfect compliance will be further reduced relative to the forecasts with perfect compliance.

4. Discussion and implications

We have shown that introducing a compliance rate that is a function of the false alarm rate in the cost–loss model can have a substantial impact on the optimal choice of warning threshold and the value of forecasts. The extent to which false alarm intolerance modifies the results of the cost–loss analysis depends upon the frequency of forecasted probabilities (α in our model), the cost–loss ratio (c), and the intolerance of the users to false alarms (λ in our model). The modification is most pronounced for low cost–loss ratios (c ≪ 0.1), relatively high frequency events (α < 1), and users who are moderately intolerant of false alarms (0 < λ < 1). For such situations the optimal warning threshold can be many times the cost–loss ratio, and the savings obtained by changing the warning threshold can exceed 25%.

Establishing the value of α, or a more appropriate form of ρ(p), for a particular type of forecast is a problem that lies comfortably inside the domain of meteorologists. Past forecasts can be used to estimate ρ(p). Determination of the cost–loss ratio, c, is more a problem of economics than meteorology. Nevertheless, the cost–loss type of decision model is now well established in the field of weather forecasting (Katz and Murphy 1997; Richardson 2000). The value of the parameter, λ, indeed the functional relationship q(pw), is poorly understood. At its simplest, q can be interpreted as the probability that a random individual will comply with forecast warning. The compliance rate can also be interpreted as the fraction of individuals who will comply with a warning. These two interpretations are equivalent if each individual's decision is independent of the decisions made by other individuals, but in practice this is unlikely to be the case. For example, an individual's decision on whether to evacuate their home may depend on what his or her neighbors are doing. This means that the form of q(pw) will be an emergent property of a system of interacting individual choices. In addition, q may depend on the false alarm rate during a finite period in the recent past. In such a situation the optimum warning threshold will be time dependent.

The simplified problem addressed in this paper illustrates the importance of including the actual user response in models of forecast value. This descriptive component cannot be neglected if estimates of the true value of forecasting systems to the economy and society are required. The results in this paper also indicate that consideration of users' response by forecasters can increase the realized value of their forecasts. Many forecasters will feel uncomfortable about modifying their forecasts in an attempt to compensate for imperfections in user responses. If forecasters make only probabilistic forecasts then they need not, and should not, report forecasts that deviate from their best estimate of the probabilities of future weather outcomes. Whenever forecasters issue nonprobabilistic forecasts, however, they are making implicit assumptions about both the cost– loss ratios of users and their tolerance of false alarms. Unless forecasters are prepared to make these assumptions explicit, and to justify them, then they should exclusively issue probabilistic forecasts and thus untangle themselves from the process of decision making.

Acknowledgments

The authors would like to thank C. Bishop, H. Brooks, and K. Mylne, discussions with whom inspired this paper, and an anonymous reviewer whose suggestions improved it. This work was supported by ONR DRI Grant N00014-99-1-0056.

REFERENCES

  • Ångström, A. K., 1922: On the effectivity of weather warnings. Nord. Stat. Tidskr., 1 , 394408.

  • Katz, R. W., and Murphy A. H. , 1997: Forecast value: Prototype decision-making models. Economic Value of Weather and Climate Forecasts, R. W. Katz and A. H. Murphy, Eds., Cambridge University Press, 183–217.

    • Search Google Scholar
    • Export Citation
  • Liljas, E., and Murphy A. H. , 1994: Anders Ångström and his early papers on probability forecasting and the use/value of weather forecasts. Bull. Amer. Meteor. Soc., 75 , 12271236.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., 1966: A note on the utility of probabilistic predictions and the probability score in the cost–loss ratio decision situation. J. Appl. Meteor., 5 , 534537.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Murphy, A. H., and Wilks D. S. , 1998: A case study of the use of statistical models in forecast verification: Precipitation probability forecasts. Wea. Forecasting, 13 , 795810.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • Mylne, K. R., and Legg T. P. , 2002: Early warnings of severe weather from the ECMWF ensemble prediction system. Preprints, 16th Conf. on Probability and Statistics in the Atmospheric Sciences, Orlando, FL, Amer. Meteor. Soc., J254.

    • Search Google Scholar
    • Export Citation
  • Richardson, D. S., 2000: Skill and relative economic value of the ECMWF ensemble prediction system,. Quart. J. Roy. Meteor. Soc., 126 , 649667.

  • Slovic, P., Fischhoff B. , Lichtenstein S. , Corrigan B. , and Combs B. , 1977: Preference for insuring against probable small losses: Insurance implications. J. Risk Insur., XLIV. 237257.

    • Search Google Scholar
    • Export Citation
  • Stewart, T. R., 1997: Forecast value: Descriptive decision studies. Economic Value of Weather and Climate Forecasts, R. W. Katz, and A. H. Murphy, Eds., Cambridge University Press, 147–181.

    • Search Google Scholar
    • Export Citation

Fig. 1.
Fig. 1.

The frequency with which probabilistic forecasts of an event with a particular probability are issued, as a function of the parameter α. Low values of α correspond to events for which high-probability forecasts are relatively common. High values of α correspond to events for which high-probability forecasts are rare.

Citation: Weather and Forecasting 19, 2; 10.1175/1520-0434(2004)019<0391:TBWCWR>2.0.CO;2

Fig. 2.
Fig. 2.

The compliance rate, q, as a function of the probability threshold at which warnings are issued: four different values of the parameter λ are shown. The model of compliance, q, as a function of warning threshold, pw, is q = pλw. Thus, a value of λ = 0 corresponds to full compliance (q = 1) regardless of the warning threshold

Citation: Weather and Forecasting 19, 2; 10.1175/1520-0434(2004)019<0391:TBWCWR>2.0.CO;2

Fig. 3.
Fig. 3.

(a) The optimum warning probability threshold as a function of event rarity parameterized by α and intolerance of false alarms parameterized by λ when the cost–loss ratio is 0.01. (b) The reduction (%) in the expected cost when the optimum threshold is used rather than the cost–loss ratio

Citation: Weather and Forecasting 19, 2; 10.1175/1520-0434(2004)019<0391:TBWCWR>2.0.CO;2

Fig. 4.
Fig. 4.

As in Fig. 3, except for a cost–loss ratio of 0.1. Note that with this higher cost–loss ratio the maximum cost reduction is reduced to about 30%

Citation: Weather and Forecasting 19, 2; 10.1175/1520-0434(2004)019<0391:TBWCWR>2.0.CO;2

Fig. 5.
Fig. 5.

As in Fig. 3, except for a cost–loss ratio of 0.5. Note that with this cost–loss ratio there is very little cost reduction and the optimum warning thresholds do not deviate from the cost–loss ratio as much as they did in Figs. 3 and 4

Citation: Weather and Forecasting 19, 2; 10.1175/1520-0434(2004)019<0391:TBWCWR>2.0.CO;2

Fig. 6.
Fig. 6.

The ratio (%) of the value (average cost reduction) of the forecasts when there is imperfect compliance to the value when there is perfect compliance. A value of 100% indicates that incompliance has no impact on forecast value, while a value of 10% indicates that the actual value of the forecast is only 10% of the value that would be calculated using the conventional cost–loss method. The optimum warning threshold for the parameter values was chosen for the imperfect compliance, and a threshold of pw = C/L = 0.1 was used for the standard cost–loss comparison

Citation: Weather and Forecasting 19, 2; 10.1175/1520-0434(2004)019<0391:TBWCWR>2.0.CO;2

1

Whether or not the probability p was a reliable estimate of the probability would not affect the shepherd boy's behavior, as long as the shepherd boy believed p to be a reliable estimate.

2

In this paper we use the phrase “false alarm rate” to mean the proportion of times the warning is issued but the event does not occur. The term “false alarm rate” is unfortunate in that is perpertuates the belief that one can evaluate a single so-called probability forecast: perfect probability forecasts would still generate “false” alarms or nonevent warnings. We prefer not to introduce new jargon here but instead suggest that compliance will improve when decision makers understand this point.

Save